2024-03-12T03:06:47.591 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-12T03:06:47.591 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --verbose --name teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869 --description orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} -- /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869/orig.config.yaml 2024-03-12T03:06:47.624 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T03:06:47.703 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869 branch: main description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591869' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c owner: scheduled_teuthology@teuthology priority: 200 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 6815 sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c targets: smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGVehr8HZ5N4vY73bOvh5Z9eSmyze1Jejhc1NhvLYo7mQCQkXzcb9WKKxhZunYOw4V4+ECTo1qMqMx0iO7gVMk= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLRR6gqvY4I1TKPA4iluTrw1r/Hv7coz9fW43/ByTE1ShCkAXmybEMEt9U8s5Em4LNPbsTG3jJTQsC+l2x0SpLc= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_20:08:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-12T03:06:47.704 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa; will attempt to use it 2024-03-12T03:06:47.704 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks 2024-03-12T03:06:47.705 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-12T03:06:47.706 INFO:teuthology.task.internal:Checking packages... 2024-03-12T03:06:47.731 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2b0101eb74d60610d544cb8a473e23a35ac1cc0b' 2024-03-12T03:06:47.732 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T03:06:47.732 INFO:teuthology.packaging:ref: None 2024-03-12T03:06:47.732 INFO:teuthology.packaging:tag: None 2024-03-12T03:06:47.732 INFO:teuthology.packaging:branch: main 2024-03-12T03:06:47.732 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:06:47.732 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-12T03:06:48.155 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1998.g1c349451 2024-03-12T03:06:48.156 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-12T03:06:48.165 INFO:teuthology.task.internal:no buildpackages task found 2024-03-12T03:06:48.165 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-12T03:06:48.193 INFO:teuthology.task.internal:Saving configuration 2024-03-12T03:06:48.206 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-12T03:06:48.238 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T03:06:48.265 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 03:01:30.638213', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGVehr8HZ5N4vY73bOvh5Z9eSmyze1Jejhc1NhvLYo7mQCQkXzcb9WKKxhZunYOw4V4+ECTo1qMqMx0iO7gVMk='} 2024-03-12T03:06:48.289 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 03:01:30.636107', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLRR6gqvY4I1TKPA4iluTrw1r/Hv7coz9fW43/ByTE1ShCkAXmybEMEt9U8s5Em4LNPbsTG3jJTQsC+l2x0SpLc='} 2024-03-12T03:06:48.289 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-12T03:06:48.294 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-12T03:06:48.294 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-12T03:06:48.295 INFO:teuthology.run_tasks:Running task console_log... 2024-03-12T03:06:48.382 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8eb30f9f70>, signals=[15]) 2024-03-12T03:06:48.382 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-12T03:06:48.388 INFO:teuthology.task.internal:Opening connections... 2024-03-12T03:06:48.388 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-03-12T03:06:48.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:06:48.481 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-03-12T03:06:48.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:06:48.571 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-12T03:06:48.577 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-03-12T03:06:48.595 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-03-12T03:06:48.595 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-03-12T03:06:48.656 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-03-12T03:06:48.656 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2024-03-12T03:06:48.656 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-03-12T03:06:48.656 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-03-12T03:06:48.656 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2024-03-12T03:06:48.656 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2024-03-12T03:06:48.656 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T03:06:48.656 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-03-12T03:06:48.656 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2024-03-12T03:06:48.656 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T03:06:48.656 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-03-12T03:06:48.656 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T03:06:48.657 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T03:06:48.657 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T03:06:48.657 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-03-12T03:06:48.679 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-03-12T03:06:48.697 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-03-12T03:06:48.697 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-03-12T03:06:48.754 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-03-12T03:06:48.754 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-03-12T03:06:48.754 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-03-12T03:06:48.754 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-03-12T03:06:48.754 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-03-12T03:06:48.754 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-03-12T03:06:48.754 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T03:06:48.755 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-03-12T03:06:48.755 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-03-12T03:06:48.755 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T03:06:48.755 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-03-12T03:06:48.755 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T03:06:48.755 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T03:06:48.755 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T03:06:48.755 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-03-12T03:06:48.779 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-12T03:06:48.787 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-12T03:06:48.808 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-12T03:06:48.808 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-03-12T03:06:48.811 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-03-12T03:06:48.827 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-12T03:06:48.841 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-12T03:06:48.842 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-03-12T03:06:48.871 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-03-12T03:06:48.970 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-12T03:06:49.022 INFO:teuthology.run_tasks:Running task kernel... 2024-03-12T03:06:49.038 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-12T03:06:49.038 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-12T03:06:49.039 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T03:06:49.039 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T03:06:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:06:49.055 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-03-12T03:06:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:06:49.056 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-12T03:06:49.111 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-361.el9.x86_64 2024-03-12T03:06:49.111 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-361.el9.x86_64 2024-03-12T03:06:49.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-03-12T03:06:49.114 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-361.el9.x86_64 2024-03-12T03:06:49.114 INFO:teuthology.task.kernel:Running kernel on smithi169: 5.14.0-361.el9.x86_64 2024-03-12T03:06:49.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2024-03-12T03:06:51.219 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-12T03:06:51.327 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-12T03:06:52.611 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-12T03:06:52.961 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-12T03:06:53.360 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-12T03:06:55.420 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-12T03:06:59.640 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.1 MB 00:00 2024-03-12T03:07:01.105 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 15 kB 00:00 2024-03-12T03:07:02.533 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 889 kB/s | 8.0 MB 00:09 2024-03-12T03:07:03.131 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-12T03:07:08.164 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-12T03:07:10.354 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T03:07:10.354 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T03:07:10.464 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:07:10.472 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-03-12T03:07:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:07:10.474 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 100 M 2024-03-12T03:07:10.474 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 123 M 2024-03-12T03:07:10.474 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:07:12.315 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): kernel-core-5.14.0-427.el9.x86_64.rpm 12 MB/s | 21 MB 00:01 2024-03-12T03:07:12.390 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): kernel-5.14.0-427.el9.x86_64.rpm 3.3 MB/s | 6.0 MB 00:01 2024-03-12T03:07:19.048 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 5.0 MB/s | 33 MB 00:06 2024-03-12T03:07:20.998 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 3.8 MB/s | 39 MB 00:10 2024-03-12T03:07:20.998 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:07:20.998 INFO:teuthology.orchestra.run.smithi173.stdout:Total 9.5 MB/s | 100 MB 00:10 2024-03-12T03:07:21.673 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 19 MB 00:18 2024-03-12T03:07:21.887 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:07:22.291 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:07:22.291 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:07:23.640 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:07:23.640 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:07:25.616 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-12T03:07:26.437 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:07:26.917 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 87 kB/s | 15 kB 00:00 2024-03-12T03:07:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:07:27.206 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:07:28.360 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-12T03:07:30.008 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:07:30.234 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:07:33.327 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-12T03:07:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:07:35.164 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:07:35.508 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T03:07:35.508 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T03:07:35.601 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:07:35.612 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:07:35.613 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-12T03:07:35.613 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:07:35.613 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:07:35.613 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T03:07:35.613 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-12T03:07:35.613 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T03:07:35.613 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T03:07:35.613 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T03:07:35.613 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:07:35.613 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:07:35.613 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:07:35.613 INFO:teuthology.orchestra.run.smithi169.stdout:Install 4 Packages 2024-03-12T03:07:35.614 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:07:35.614 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 100 M 2024-03-12T03:07:35.614 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 123 M 2024-03-12T03:07:35.614 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:07:39.913 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:07:44.851 INFO:teuthology.orchestra.run.smithi169.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 679 kB/s | 6.0 MB 00:09 2024-03-12T03:08:07.058 INFO:teuthology.orchestra.run.smithi169.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 683 kB/s | 21 MB 00:31 2024-03-12T03:08:09.060 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:08:09.060 INFO:teuthology.orchestra.run.smithi173.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T03:08:09.060 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:08:09.145 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:08:09.578 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:08:09.578 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:08:09.578 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:08:09.578 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:08:10.009 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:08:10.009 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:08:10.009 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:08:10.009 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T03:08:10.009 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T03:08:10.009 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T03:08:10.009 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T03:08:10.009 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:08:10.009 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:08:10.522 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:08:10.704 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:08:10.705 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:08:10.705 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T03:08:10.705 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T03:08:10.705 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T03:08:10.705 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-12T03:08:10.705 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T03:08:10.705 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:08:10.826 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:08:10.826 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:08:10.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-03-12T03:08:10.869 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-03-12T03:08:10.870 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-03-12T03:08:10.870 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.06 2024-03-12T03:08:10.870 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 68.el9 2024-03-12T03:08:10.870 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-03-12T03:08:10.870 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T03:08:10.870 INFO:teuthology.orchestra.run.smithi173.stdout:Group : Unspecified 2024-03-12T03:08:10.870 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 8264254 2024-03-12T03:08:10.870 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-03-12T03:08:10.870 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T03:08:10.870 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T03:08:10.870 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T03:08:10.870 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T03:08:10.871 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : builder@centos.org 2024-03-12T03:08:10.871 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-03-12T03:08:10.871 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T03:08:10.871 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-03-12T03:08:10.871 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-03-12T03:08:10.871 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:08:10.871 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T03:08:10.871 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T03:08:10.871 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T03:08:10.871 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-03-12T03:08:10.871 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:08:10.871 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T03:08:10.872 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T03:08:10.872 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 5.14.0-427.el9.x86_64 2024-03-12T03:08:10.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T03:08:11.516 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-03-12T03:08:12.877 INFO:teuthology.orchestra.run.smithi173.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T03:08:12.912 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-03-12T03:08:12.914 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-03-12T03:08:12.929 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.lq0Cm81lJN 2024-03-12T03:08:12.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lq0Cm81lJN 2024-03-12T03:08:13.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.lq0Cm81lJN 2024-03-12T03:08:13.141 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.lq0Cm81lJN is 6KB 2024-03-12T03:08:13.190 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.lq0Cm81lJN 2024-03-12T03:08:13.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T03:08:13.269 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T03:08:13.270 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T03:08:13.270 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T03:08:13.270 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T03:08:13.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T03:08:13.431 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-03-12T03:08:32.137 INFO:teuthology.orchestra.run.smithi169.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 715 kB/s | 39 MB 00:56 2024-03-12T03:08:33.587 INFO:teuthology.orchestra.run.smithi169.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 702 kB/s | 33 MB 00:48 2024-03-12T03:08:33.587 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:08:33.588 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.7 MB/s | 100 MB 00:57 2024-03-12T03:08:34.473 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:08:34.854 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:08:34.854 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:08:36.187 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:08:36.187 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:08:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:08:39.544 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:08:39.680 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:08:42.498 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:08:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:08:43.434 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-12T03:08:43.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-12T03:08:43.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:08:47.448 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:08:47.660 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:08:52.455 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:09:01.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-03-12T03:09:10.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-12T03:09:10.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:09:13.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-03-12T03:09:21.458 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:09:21.458 INFO:teuthology.orchestra.run.smithi169.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T03:09:21.458 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:09:21.542 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:09:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:09:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:09:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:09:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:09:22.359 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:09:22.359 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:09:22.359 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:09:22.359 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T03:09:22.359 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T03:09:22.359 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T03:09:22.359 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T03:09:22.359 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:09:22.359 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:09:22.903 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:09:23.051 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:09:23.052 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:09:23.052 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T03:09:23.052 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T03:09:23.052 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T03:09:23.052 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-12T03:09:23.052 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T03:09:23.052 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:09:23.174 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:09:23.174 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:09:23.174 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2024-03-12T03:09:23.215 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2024-03-12T03:09:23.215 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2024-03-12T03:09:23.215 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.06 2024-03-12T03:09:23.215 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 68.el9 2024-03-12T03:09:23.215 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2024-03-12T03:09:23.215 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T03:09:23.215 INFO:teuthology.orchestra.run.smithi169.stdout:Group : Unspecified 2024-03-12T03:09:23.215 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 8264254 2024-03-12T03:09:23.215 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2024-03-12T03:09:23.215 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T03:09:23.215 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : builder@centos.org 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:09:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T03:09:23.217 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T03:09:23.217 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 5.14.0-427.el9.x86_64 2024-03-12T03:09:23.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T03:09:23.842 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2024-03-12T03:09:25.243 INFO:teuthology.orchestra.run.smithi169.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T03:09:25.276 INFO:teuthology.orchestra.run.smithi169.stderr:done 2024-03-12T03:09:25.278 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-03-12T03:09:25.293 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.wQgS0zyXSC 2024-03-12T03:09:25.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wQgS0zyXSC 2024-03-12T03:09:25.363 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.wQgS0zyXSC 2024-03-12T03:09:25.501 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.wQgS0zyXSC is 6KB 2024-03-12T03:09:25.550 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.wQgS0zyXSC 2024-03-12T03:09:25.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T03:09:25.630 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T03:09:25.630 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T03:09:25.630 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T03:09:25.630 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T03:09:25.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T03:09:25.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2024-03-12T03:09:25.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-12T03:09:25.920 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:09:55.814 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-12T03:09:55.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-12T03:09:55.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:09:57.828 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-03-12T03:09:58.283 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-12T03:09:58.284 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-12T03:09:58.284 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-03-12T03:09:58.297 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-427.el9.x86_64 2024-03-12T03:09:58.298 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T03:09:58.298 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T03:09:58.298 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T03:09:59.298 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-12T03:09:59.299 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T03:09:59.401 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-03-12T03:09:59.435 DEBUG:teuthology.parallel:result is None 2024-03-12T03:10:14.302 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-12T03:10:23.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-12T03:10:23.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:10:26.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-12T03:10:38.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-12T03:10:38.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:10:56.802 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-12T03:11:11.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-12T03:11:11.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:11:14.395 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-12T03:11:14.960 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-12T03:11:14.960 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-12T03:11:14.960 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-12T03:11:14.974 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-427.el9.x86_64 2024-03-12T03:11:14.974 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T03:11:14.974 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T03:11:14.975 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T03:11:15.975 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-12T03:11:15.975 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T03:11:16.070 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2024-03-12T03:11:16.113 DEBUG:teuthology.parallel:result is None 2024-03-12T03:11:16.113 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-12T03:11:16.121 INFO:teuthology.task.internal:Creating test directory... 2024-03-12T03:11:16.121 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T03:11:16.123 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T03:11:16.142 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-12T03:11:16.170 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-12T03:11:16.201 INFO:teuthology.task.internal:Creating archive directory... 2024-03-12T03:11:16.201 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T03:11:16.203 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T03:11:16.251 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-12T03:11:16.259 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-12T03:11:16.259 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T03:11:16.268 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T03:11:16.295 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:11:16.308 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:11:16.331 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:11:16.346 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:11:16.348 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-12T03:11:16.356 INFO:teuthology.task.internal:Configuring sudo... 2024-03-12T03:11:16.356 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T03:11:16.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T03:11:16.388 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-12T03:11:16.397 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-12T03:11:16.398 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T03:11:16.427 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T03:11:16.444 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:11:16.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:11:16.574 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:11:16.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:11:16.696 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:11:16.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T03:11:16.764 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:11:16.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:11:16.858 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:11:16.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:11:16.987 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:11:16.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T03:11:17.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-03-12T03:11:17.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-03-12T03:11:17.102 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T03:11:17.161 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T03:11:17.567 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-12T03:11:17.575 INFO:teuthology.task.internal:Starting timer... 2024-03-12T03:11:17.575 INFO:teuthology.run_tasks:Running task pcp... 2024-03-12T03:11:17.612 INFO:teuthology.run_tasks:Running task selinux... 2024-03-12T03:11:17.644 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-12T03:11:17.645 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-03-12T03:11:17.697 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-03-12T03:11:17.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-03-12T03:11:17.759 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-03-12T03:11:17.761 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-12T03:11:17.761 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-03-12T03:11:17.776 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-03-12T03:11:17.776 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-03-12T03:11:17.800 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-03-12T03:11:17.800 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi169.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-03-12T03:11:17.800 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T03:11:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:11:17.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T03:11:17.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:11:17.875 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-12T03:11:17.875 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-03-12T03:11:17.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-03-12T03:11:17.969 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-12T03:11:17.979 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-12T03:11:17.995 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-12T03:11:17.996 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi169.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-12T03:19:41.765 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi169.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-03-12T03:19:41.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-12T03:19:41.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:19:41.857 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-12T03:19:41.930 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-12T03:19:41.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-12T03:19:41.931 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:19:42.018 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-03-12T03:19:42.052 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-12T03:19:42.053 INFO:teuthology.run_tasks:Running task clock... 2024-03-12T03:19:42.064 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-12T03:19:42.064 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T03:19:42.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:19:42.067 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T03:19:42.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:19:42.096 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T03:19:42.112 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T03:19:42.122 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T03:19:42.139 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T03:19:42.142 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-03-12T03:19:42.155 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-03-12T03:19:42.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-03-12T03:19:42.172 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T03:19:42.182 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-03-12T03:19:42.188 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T03:19:42.198 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T03:19:42.215 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T03:19:42.240 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:19:42.244 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:19:42.244 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-12T03:19:42.244 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:19:42.244 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:19:42.244 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:19:42.245 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:19:42.266 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:19:42.270 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:19:42.270 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-12T03:19:42.270 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:19:42.270 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:19:42.270 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:19:42.270 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:19:42.271 INFO:teuthology.run_tasks:Running task install... 2024-03-12T03:19:42.280 DEBUG:teuthology.task.install:project ceph 2024-03-12T03:19:42.280 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}} 2024-03-12T03:19:42.280 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'} 2024-03-12T03:19:42.280 INFO:teuthology.task.install:Using flavor: default 2024-03-12T03:19:42.286 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-12T03:19:42.286 INFO:teuthology.task.install:extra packages: [] 2024-03-12T03:19:42.287 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'tag': None, 'wait_for_package': False} 2024-03-12T03:19:42.287 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:19:42.289 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'tag': None, 'wait_for_package': False} 2024-03-12T03:19:42.289 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:19:42.429 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/ 2024-03-12T03:19:42.430 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1979.g2b0101eb 2024-03-12T03:19:42.512 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/ 2024-03-12T03:19:42.512 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1979.g2b0101eb 2024-03-12T03:19:42.546 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-12T03:19:42.547 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:19:42.547 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-12T03:19:42.575 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-12T03:19:42.575 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-12T03:19:42.623 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-12T03:19:42.624 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:19:42.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-12T03:19:42.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-12T03:19:42.654 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-12T03:19:42.655 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-12T03:19:42.724 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-12T03:19:42.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-12T03:19:42.790 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-03-12T03:19:42.792 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-03-12T03:19:42.807 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-12T03:19:42.874 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-03-12T03:19:42.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-03-12T03:19:43.033 INFO:teuthology.orchestra.run.smithi169.stdout:74 files removed 2024-03-12T03:19:43.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw 2024-03-12T03:19:43.114 INFO:teuthology.orchestra.run.smithi173.stdout:74 files removed 2024-03-12T03:19:43.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2024-03-12T03:19:43.791 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 199 kB/s | 82 kB 00:00 2024-03-12T03:19:43.980 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 157 kB/s | 82 kB 00:00 2024-03-12T03:19:44.234 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-12T03:19:44.305 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 36 kB/s | 10 kB 00:00 2024-03-12T03:19:44.648 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-03-12T03:19:44.660 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-12T03:19:45.153 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-12T03:19:45.309 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-12T03:19:46.713 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-03-12T03:19:46.734 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-12T03:19:47.633 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-12T03:19:47.980 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 8.5 MB/s | 8.0 MB 00:00 2024-03-12T03:19:50.070 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-12T03:19:54.102 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-12T03:19:55.920 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 15 kB 00:00 2024-03-12T03:19:57.416 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-12T03:20:02.338 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-12T03:20:04.941 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:20:04.962 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:20:04.962 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-12T03:20:04.962 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:20:04.962 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:20:04.962 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 9.8 M 2024-03-12T03:20:04.962 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-03-12T03:20:04.962 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.4 M 2024-03-12T03:20:04.962 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-12T03:20:04.962 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-12T03:20:04.963 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options 2024-03-12T03:20:04.963 INFO:teuthology.orchestra.run.smithi173.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-12T03:20:04.963 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.3 M 2024-03-12T03:20:04.963 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 20 M 2024-03-12T03:20:04.963 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 25 k 2024-03-12T03:20:04.963 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-12T03:20:04.963 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-12T03:20:04.963 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 745 k 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 501 k 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.2 M 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 45 k 2024-03-12T03:20:04.964 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 131 k 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 162 k 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 319 k 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 298 k 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 99 k 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:20:04.965 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:20:04.966 INFO:teuthology.orchestra.run.smithi173.stdout:Install 30 Packages 2024-03-12T03:20:04.966 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-03-12T03:20:04.966 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:04.966 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 58 M 2024-03-12T03:20:04.966 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:20:06.981 INFO:teuthology.orchestra.run.smithi173.stdout:(1/32): ceph-radosgw-19.0.0-1979.g2b0101eb.el9. 5.5 MB/s | 9.8 MB 00:01 2024-03-12T03:20:07.031 INFO:teuthology.orchestra.run.smithi173.stdout:(2/32): ceph-selinux-19.0.0-1979.g2b0101eb.el9. 506 kB/s | 25 kB 00:00 2024-03-12T03:20:07.107 INFO:teuthology.orchestra.run.smithi173.stdout:(3/32): libcephfs2-19.0.0-1979.g2b0101eb.el9.x8 9.7 MB/s | 745 kB 00:00 2024-03-12T03:20:07.174 INFO:teuthology.orchestra.run.smithi173.stdout:(4/32): libradosstriper1-19.0.0-1979.g2b0101eb. 7.4 MB/s | 501 kB 00:00 2024-03-12T03:20:07.491 INFO:teuthology.orchestra.run.smithi173.stdout:(5/32): librgw2-19.0.0-1979.g2b0101eb.el9.x86_6 16 MB/s | 5.2 MB 00:00 2024-03-12T03:20:07.532 INFO:teuthology.orchestra.run.smithi173.stdout:(6/32): python3-ceph-argparse-19.0.0-1979.g2b01 1.1 MB/s | 45 kB 00:00 2024-03-12T03:20:07.644 INFO:teuthology.orchestra.run.smithi173.stdout:(7/32): python3-ceph-common-19.0.0-1979.g2b0101 1.2 MB/s | 131 kB 00:00 2024-03-12T03:20:07.691 INFO:teuthology.orchestra.run.smithi173.stdout:(8/32): python3-cephfs-19.0.0-1979.g2b0101eb.el 3.3 MB/s | 162 kB 00:00 2024-03-12T03:20:07.750 INFO:teuthology.orchestra.run.smithi173.stdout:(9/32): python3-rados-19.0.0-1979.g2b0101eb.el9 5.3 MB/s | 319 kB 00:00 2024-03-12T03:20:07.809 INFO:teuthology.orchestra.run.smithi173.stdout:(10/32): python3-rbd-19.0.0-1979.g2b0101eb.el9. 5.0 MB/s | 298 kB 00:00 2024-03-12T03:20:07.859 INFO:teuthology.orchestra.run.smithi173.stdout:(11/32): python3-rgw-19.0.0-1979.g2b0101eb.el9. 1.9 MB/s | 99 kB 00:00 2024-03-12T03:20:07.926 INFO:teuthology.orchestra.run.smithi173.stdout:(12/32): ceph-base-19.0.0-1979.g2b0101eb.el9.x8 1.9 MB/s | 5.3 MB 00:02 2024-03-12T03:20:08.068 INFO:teuthology.orchestra.run.smithi173.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 507 kB/s | 106 kB 00:00 2024-03-12T03:20:08.093 INFO:teuthology.orchestra.run.smithi173.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 953 kB/s | 159 kB 00:00 2024-03-12T03:20:08.110 INFO:teuthology.orchestra.run.smithi173.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-12T03:20:08.186 INFO:teuthology.orchestra.run.smithi173.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-12T03:20:08.211 INFO:teuthology.orchestra.run.smithi173.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.7 MB/s | 282 kB 00:00 2024-03-12T03:20:08.228 INFO:teuthology.orchestra.run.smithi173.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 990 kB/s | 41 kB 00:00 2024-03-12T03:20:08.320 INFO:teuthology.orchestra.run.smithi173.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 381 kB/s | 35 kB 00:00 2024-03-12T03:20:08.395 INFO:teuthology.orchestra.run.smithi173.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.2 MB/s | 216 kB 00:00 2024-03-12T03:20:08.421 INFO:teuthology.orchestra.run.smithi173.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 443 kB/s | 44 kB 00:00 2024-03-12T03:20:08.929 INFO:teuthology.orchestra.run.smithi173.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 579 kB/s | 309 kB 00:00 2024-03-12T03:20:09.196 INFO:teuthology.orchestra.run.smithi173.stdout:(23/32): ceph-common-19.0.0-1979.g2b0101eb.el9. 5.0 MB/s | 20 MB 00:04 2024-03-12T03:20:09.221 INFO:teuthology.orchestra.run.smithi173.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 89 kB/s | 26 kB 00:00 2024-03-12T03:20:09.330 INFO:teuthology.orchestra.run.smithi173.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 621 kB/s | 67 kB 00:00 2024-03-12T03:20:09.422 INFO:teuthology.orchestra.run.smithi173.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2024-03-12T03:20:09.530 INFO:teuthology.orchestra.run.smithi173.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-12T03:20:09.556 INFO:teuthology.orchestra.run.smithi173.stdout:(28/32): liboath-2.6.7-2.el9.x86_64.rpm 135 kB/s | 49 kB 00:00 2024-03-12T03:20:09.598 INFO:teuthology.orchestra.run.smithi173.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-03-12T03:20:09.848 INFO:teuthology.orchestra.run.smithi173.stdout:(30/32): librbd1-19.0.0-1979.g2b0101eb.el9.x86_ 13 MB/s | 3.1 MB 00:00 2024-03-12T03:20:10.215 INFO:teuthology.orchestra.run.smithi173.stdout:(31/32): librados2-19.0.0-1979.g2b0101eb.el9.x8 5.1 MB/s | 3.4 MB 00:00 2024-03-12T03:20:10.632 INFO:teuthology.orchestra.run.smithi173.stdout:(32/32): thrift-0.15.0-2.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-03-12T03:20:10.635 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:20:10.636 INFO:teuthology.orchestra.run.smithi173.stdout:Total 10 MB/s | 58 MB 00:05 2024-03-12T03:20:10.671 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:20:10.709 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:20:10.709 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:20:11.185 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:20:11.186 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:20:12.673 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:20:12.798 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-12T03:20:12.828 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-12T03:20:12.854 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-12T03:20:12.885 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-12T03:20:12.913 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-12T03:20:13.107 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-12T03:20:13.166 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-12T03:20:13.493 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-12T03:20:13.556 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-12T03:20:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-12T03:20:13.727 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 8/34 2024-03-12T03:20:13.749 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-12T03:20:13.812 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-12T03:20:13.841 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-12T03:20:13.891 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 11/34 2024-03-12T03:20:13.961 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-12T03:20:13.985 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-12T03:20:14.031 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-12T03:20:14.058 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-12T03:20:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-12T03:20:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-12T03:20:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-12T03:20:14.283 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-12T03:20:14.312 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-12T03:20:14.345 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-12T03:20:14.660 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-12T03:20:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-12T03:20:14.757 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-12T03:20:14.789 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 23/34 2024-03-12T03:20:14.828 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-12T03:20:15.285 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-12T03:20:15.311 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-12T03:20:15.382 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-12T03:20:15.452 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 27/34 2024-03-12T03:20:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 28/34 2024-03-12T03:20:17.677 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-12T03:20:17.719 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-12T03:20:18.309 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-12T03:20:18.373 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-12T03:20:18.466 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-12T03:20:18.466 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-12T03:20:18.466 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-12T03:20:18.466 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:18.499 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-12T03:20:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-12T03:20:28.103 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-03-12T03:20:28.103 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-03-12T03:20:28.103 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-03-12T03:20:28.103 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-03-12T03:20:28.103 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-03-12T03:20:28.103 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-03-12T03:20:28.103 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-03-12T03:20:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:28.984 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-12T03:20:29.020 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-12T03:20:29.021 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:20:29.021 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-12T03:20:29.021 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T03:20:29.021 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T03:20:29.021 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:29.023 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-12T03:20:29.113 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-12T03:20:29.113 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-12T03:20:30.287 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-12T03:20:30.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/34 2024-03-12T03:20:30.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/34 2024-03-12T03:20:30.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/34 2024-03-12T03:20:30.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/34 2024-03-12T03:20:30.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/34 2024-03-12T03:20:30.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 6/34 2024-03-12T03:20:30.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-12T03:20:30.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 8/34 2024-03-12T03:20:30.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 9/34 2024-03-12T03:20:30.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 10/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 11/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 13/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-12T03:20:30.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-12T03:20:30.289 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-12T03:20:30.289 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-12T03:20:30.289 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-12T03:20:30.289 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-12T03:20:30.290 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-12T03:20:30.290 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-12T03:20:30.290 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-12T03:20:30.290 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/34 2024-03-12T03:20:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-12T03:20:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:31.438 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-03-12T03:20:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.439 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:20:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-12T03:20:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-12T03:20:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-12T03:20:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-12T03:20:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-12T03:20:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-12T03:20:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-12T03:20:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-12T03:20:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-12T03:20:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-3.el9.x86_64 2024-03-12T03:20:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-12T03:20:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:31.445 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:20:31.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2024-03-12T03:20:32.361 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:30 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:20:32.510 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:20:32.511 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:20:32.511 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-12T03:20:32.511 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:20:32.511 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:20:32.511 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 48 M 2024-03-12T03:20:32.511 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-12T03:20:32.511 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-12T03:20:32.511 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 163 k 2024-03-12T03:20:32.512 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-12T03:20:32.512 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-12T03:20:32.512 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-12T03:20:32.512 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:32.512 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:20:32.512 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:20:32.512 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-03-12T03:20:32.512 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:32.513 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 48 M 2024-03-12T03:20:32.513 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 205 M 2024-03-12T03:20:32.513 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:20:32.743 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-12T03:20:32.794 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): libcephsqlite-19.0.0-1979.g2b0101eb.el9. 683 kB/s | 163 kB 00:00 2024-03-12T03:20:32.834 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 300 kB 00:00 2024-03-12T03:20:32.851 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-12T03:20:33.052 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 864 kB/s | 187 kB 00:00 2024-03-12T03:20:35.185 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): ceph-test-19.0.0-1979.g2b0101eb.el9.x86_ 18 MB/s | 48 MB 00:02 2024-03-12T03:20:35.189 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:20:35.189 INFO:teuthology.orchestra.run.smithi173.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-12T03:20:35.203 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:20:35.225 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:20:35.225 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:20:35.618 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:20:35.618 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:20:36.195 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:20:36.268 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-12T03:20:36.310 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-12T03:20:36.352 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-12T03:20:36.401 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-12T03:20:36.430 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-12T03:20:40.936 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-12T03:20:41.013 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-12T03:20:43.407 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-12T03:20:43.408 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/6 2024-03-12T03:20:43.408 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/6 2024-03-12T03:20:43.408 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-12T03:20:43.408 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-12T03:20:43.408 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-12T03:20:43.795 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-12T03:20:43.795 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:43.795 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:20:43.795 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:43.795 INFO:teuthology.orchestra.run.smithi173.stdout: jq-1.6-16.el9.x86_64 2024-03-12T03:20:43.795 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:43.795 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-12T03:20:43.795 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-12T03:20:43.796 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-12T03:20:43.796 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:43.796 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:20:44.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2024-03-12T03:20:44.656 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:42 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:20:44.823 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:20:44.825 INFO:teuthology.orchestra.run.smithi173.stdout:========================================================================================= 2024-03-12T03:20:44.825 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-12T03:20:44.825 INFO:teuthology.orchestra.run.smithi173.stdout:========================================================================================= 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 6.6 k 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 2.1 M 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 1.5 M 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 248 k 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 4.7 M 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 17 M 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-12T03:20:44.826 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-12T03:20:44.827 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.run.smithi173.stdout:========================================================================================= 2024-03-12T03:20:44.829 INFO:teuthology.orchestra.run.smithi173.stdout:Install 36 Packages 2024-03-12T03:20:44.829 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:44.829 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 30 M 2024-03-12T03:20:44.829 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 107 M 2024-03-12T03:20:44.829 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:20:45.596 INFO:teuthology.orchestra.run.smithi173.stdout:(1/36): ceph-19.0.0-1979.g2b0101eb.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-03-12T03:20:45.738 INFO:teuthology.orchestra.run.smithi173.stdout:(2/36): ceph-mgr-19.0.0-1979.g2b0101eb.el9.x86_ 4.2 MB/s | 1.5 MB 00:00 2024-03-12T03:20:46.155 INFO:teuthology.orchestra.run.smithi173.stdout:(3/36): ceph-mds-19.0.0-1979.g2b0101eb.el9.x86_ 2.8 MB/s | 2.1 MB 00:00 2024-03-12T03:20:46.246 INFO:teuthology.orchestra.run.smithi173.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1979.g2b01 2.6 MB/s | 248 kB 00:00 2024-03-12T03:20:46.397 INFO:teuthology.orchestra.run.smithi173.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-12T03:20:46.447 INFO:teuthology.orchestra.run.smithi173.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-12T03:20:46.622 INFO:teuthology.orchestra.run.smithi173.stdout:(7/36): ceph-osd-19.0.0-1979.g2b0101eb.el9.x86_ 19 MB/s | 17 MB 00:00 2024-03-12T03:20:46.639 INFO:teuthology.orchestra.run.smithi173.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 164 kB/s | 32 kB 00:00 2024-03-12T03:20:46.656 INFO:teuthology.orchestra.run.smithi173.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-12T03:20:46.673 INFO:teuthology.orchestra.run.smithi173.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-12T03:20:46.707 INFO:teuthology.orchestra.run.smithi173.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-12T03:20:46.774 INFO:teuthology.orchestra.run.smithi173.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-12T03:20:46.799 INFO:teuthology.orchestra.run.smithi173.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-12T03:20:46.824 INFO:teuthology.orchestra.run.smithi173.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-12T03:20:46.858 INFO:teuthology.orchestra.run.smithi173.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-12T03:20:47.117 INFO:teuthology.orchestra.run.smithi173.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-03-12T03:20:47.359 INFO:teuthology.orchestra.run.smithi173.stdout:(17/36): ceph-mon-19.0.0-1979.g2b0101eb.el9.x86 2.7 MB/s | 4.7 MB 00:01 2024-03-12T03:20:47.384 INFO:teuthology.orchestra.run.smithi173.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 223 kB/s | 125 kB 00:00 2024-03-12T03:20:47.543 INFO:teuthology.orchestra.run.smithi173.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 355 kB/s | 151 kB 00:00 2024-03-12T03:20:47.735 INFO:teuthology.orchestra.run.smithi173.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 116 kB/s | 43 kB 00:00 2024-03-12T03:20:47.860 INFO:teuthology.orchestra.run.smithi173.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 361 kB/s | 172 kB 00:00 2024-03-12T03:20:47.910 INFO:teuthology.orchestra.run.smithi173.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 61 kB/s | 11 kB 00:00 2024-03-12T03:20:48.060 INFO:teuthology.orchestra.run.smithi173.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 693 kB/s | 358 kB 00:00 2024-03-12T03:20:48.127 INFO:teuthology.orchestra.run.smithi173.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 66 kB/s | 18 kB 00:00 2024-03-12T03:20:48.169 INFO:teuthology.orchestra.run.smithi173.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 89 kB/s | 23 kB 00:00 2024-03-12T03:20:48.336 INFO:teuthology.orchestra.run.smithi173.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 70 kB/s | 19 kB 00:00 2024-03-12T03:20:48.395 INFO:teuthology.orchestra.run.smithi173.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 73 kB/s | 20 kB 00:00 2024-03-12T03:20:48.511 INFO:teuthology.orchestra.run.smithi173.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 135 kB/s | 46 kB 00:00 2024-03-12T03:20:48.679 INFO:teuthology.orchestra.run.smithi173.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 230 kB/s | 79 kB 00:00 2024-03-12T03:20:48.779 INFO:teuthology.orchestra.run.smithi173.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 61 kB/s | 16 kB 00:00 2024-03-12T03:20:48.920 INFO:teuthology.orchestra.run.smithi173.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 518 kB/s | 272 kB 00:00 2024-03-12T03:20:49.054 INFO:teuthology.orchestra.run.smithi173.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 240 kB/s | 90 kB 00:00 2024-03-12T03:20:49.080 INFO:teuthology.orchestra.run.smithi173.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 119 kB/s | 36 kB 00:00 2024-03-12T03:20:49.346 INFO:teuthology.orchestra.run.smithi173.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 75 kB/s | 20 kB 00:00 2024-03-12T03:20:49.421 INFO:teuthology.orchestra.run.smithi173.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 459 kB/s | 230 kB 00:00 2024-03-12T03:20:49.663 INFO:teuthology.orchestra.run.smithi173.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 701 kB/s | 427 kB 00:00 2024-03-12T03:20:49.668 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:20:49.668 INFO:teuthology.orchestra.run.smithi173.stdout:Total 6.1 MB/s | 30 MB 00:04 2024-03-12T03:20:49.693 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:20:49.736 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:20:49.736 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:20:50.143 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:20:50.143 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:20:51.456 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:20:51.499 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-12T03:20:51.538 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-12T03:20:51.569 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-12T03:20:51.641 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-12T03:20:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-12T03:20:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-12T03:20:51.801 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-12T03:20:51.870 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-12T03:20:51.911 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-12T03:20:51.983 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-12T03:20:52.036 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-12T03:20:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-12T03:20:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-12T03:20:52.170 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-12T03:20:52.360 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-12T03:20:52.404 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-12T03:20:52.452 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-12T03:20:52.477 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-12T03:20:52.507 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-12T03:20:52.583 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-12T03:20:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-12T03:20:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-12T03:20:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-12T03:20:52.743 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-12T03:20:52.777 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-12T03:20:52.867 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-12T03:20:52.916 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-12T03:20:52.989 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-12T03:20:53.126 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-12T03:20:53.249 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-12T03:20:53.416 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 31/36 2024-03-12T03:20:53.447 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-12T03:20:53.472 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-12T03:20:53.472 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:20:53.472 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-12T03:20:53.473 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T03:20:53.473 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T03:20:53.473 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:54.906 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-12T03:20:54.932 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-12T03:20:54.932 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:20:54.932 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-12T03:20:54.932 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T03:20:54.932 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T03:20:54.932 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-12T03:20:55.396 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-12T03:20:55.396 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:20:55.397 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-12T03:20:55.397 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T03:20:55.397 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T03:20:55.397 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:55.607 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-12T03:20:55.630 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-12T03:20:55.630 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:20:55.631 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-12T03:20:55.631 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T03:20:55.631 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T03:20:55.631 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:55.699 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-12T03:20:57.524 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-12T03:20:57.525 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/36 2024-03-12T03:20:57.525 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/36 2024-03-12T03:20:57.525 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/36 2024-03-12T03:20:57.525 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/36 2024-03-12T03:20:57.525 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/36 2024-03-12T03:20:57.525 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 6/36 2024-03-12T03:20:57.525 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-12T03:20:57.525 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-12T03:20:57.525 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-12T03:20:57.525 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-12T03:20:57.525 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-12T03:20:57.525 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-12T03:20:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-12T03:20:57.527 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-12T03:20:57.527 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-12T03:20:57.527 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-12T03:20:57.527 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-12T03:20:57.528 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-12T03:20:57.528 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-12T03:20:57.528 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-12T03:20:57.528 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-12T03:20:57.528 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-12T03:20:57.528 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-12T03:20:58.318 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-12T03:20:58.318 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:58.318 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:20:58.318 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:58.318 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:58.318 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-12T03:20:58.319 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-12T03:20:58.320 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-12T03:20:58.321 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-12T03:20:58.321 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-12T03:20:58.321 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-12T03:20:58.321 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-12T03:20:58.321 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-12T03:20:58.321 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-12T03:20:58.321 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:20:58.321 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:20:58.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2024-03-12T03:20:59.237 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:57 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:20:59.335 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:20:59.387 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:20:59.388 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-12T03:20:59.388 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:20:59.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2024-03-12T03:21:00.033 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:58 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:21:00.181 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:21:00.182 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:21:00.182 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-12T03:21:00.182 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:21:00.182 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:21:00.182 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 748 k 2024-03-12T03:21:00.183 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:00.183 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:21:00.183 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:21:00.183 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-12T03:21:00.183 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:00.183 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 748 k 2024-03-12T03:21:00.183 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 755 k 2024-03-12T03:21:00.183 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:21:00.623 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-19.0.0-1979.g2b0101eb.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-12T03:21:00.624 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:21:00.624 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-12T03:21:00.624 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:21:00.629 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:21:00.629 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:21:00.635 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:21:00.635 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:21:00.710 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:21:01.122 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:21:01.211 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:21:01.602 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:21:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:21:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:01.977 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:21:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:21:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:01.977 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:21:02.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2024-03-12T03:21:02.774 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:00 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:21:02.930 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:21:02.931 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:21:02.931 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-12T03:21:02.931 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:21:02.931 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:21:02.931 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 145 k 2024-03-12T03:21:02.931 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:02.931 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:21:02.931 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:21:02.931 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-12T03:21:02.931 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:02.931 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 145 k 2024-03-12T03:21:02.932 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 436 k 2024-03-12T03:21:02.932 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:21:03.278 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-19.0.0-1979.g2b0101 419 kB/s | 145 kB 00:00 2024-03-12T03:21:03.279 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:21:03.279 INFO:teuthology.orchestra.run.smithi173.stdout:Total 417 kB/s | 145 kB 00:00 2024-03-12T03:21:03.279 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:21:03.287 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:21:03.287 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:21:03.338 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:21:03.339 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:21:03.473 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:21:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-12T03:21:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-12T03:21:03.571 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:21:03.571 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-12T03:21:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:04.573 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-12T03:21:04.573 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:04.573 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:21:04.573 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:21:04.573 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:04.573 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:21:04.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2024-03-12T03:21:05.385 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:03 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:21:05.481 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:21:05.533 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:21:05.534 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-12T03:21:05.534 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:21:05.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-dashboard 2024-03-12T03:21:06.165 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:04 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:21:06.314 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:21:06.315 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-03-12T03:21:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-12T03:21:06.315 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-03-12T03:21:06.315 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:21:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 1.8 M 2024-03-12T03:21:06.315 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-12T03:21:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 34 k 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 16 k 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-03-12T03:21:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:06.317 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 6.9 M 2024-03-12T03:21:06.317 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2024-03-12T03:21:06.317 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:21:06.790 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1979.g2b 182 kB/s | 34 kB 00:00 2024-03-12T03:21:06.883 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1979.g2b0 59 kB/s | 16 kB 00:00 2024-03-12T03:21:07.000 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-12T03:21:07.025 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-12T03:21:07.092 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1979.g2b0101e 3.7 MB/s | 1.8 MB 00:00 2024-03-12T03:21:07.209 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.0 MB/s | 863 kB 00:00 2024-03-12T03:21:07.234 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-03-12T03:21:08.318 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 133 kB/s | 144 kB 00:01 2024-03-12T03:21:08.676 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 86 kB/s | 31 kB 00:00 2024-03-12T03:21:10.101 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 132 kB/s | 188 kB 00:01 2024-03-12T03:21:10.743 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 147 kB/s | 548 kB 00:03 2024-03-12T03:21:21.617 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 145 kB/s | 2.0 MB 00:14 2024-03-12T03:21:21.620 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:21:21.620 INFO:teuthology.orchestra.run.smithi173.stdout:Total 460 kB/s | 6.9 MB 00:15 2024-03-12T03:21:21.639 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:21:21.658 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:21:21.658 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:21:21.827 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:21:21.827 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:21:22.356 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:21:22.433 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-12T03:21:22.477 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-12T03:21:22.523 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-12T03:21:22.554 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-12T03:21:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-12T03:21:22.779 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-12T03:21:22.881 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-12T03:21:22.926 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-12T03:21:22.955 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-12T03:21:22.978 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 10/12 2024-03-12T03:21:23.214 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 11/12 2024-03-12T03:21:23.244 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-12T03:21:24.057 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-12T03:21:24.057 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 1/12 2024-03-12T03:21:24.057 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 2/12 2024-03-12T03:21:24.057 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 3/12 2024-03-12T03:21:24.058 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-12T03:21:24.058 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-12T03:21:24.058 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-12T03:21:24.058 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-12T03:21:24.058 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-12T03:21:24.058 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-12T03:21:24.058 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-12T03:21:24.058 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-12T03:21:24.425 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-12T03:21:24.425 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:24.425 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:21:24.425 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-12T03:21:24.425 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:21:24.425 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:21:24.426 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:21:24.426 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-12T03:21:24.426 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-12T03:21:24.426 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-12T03:21:24.426 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-12T03:21:24.426 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-12T03:21:24.426 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-12T03:21:24.426 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-12T03:21:24.426 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-12T03:21:24.426 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:24.426 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:21:24.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-12T03:21:25.250 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:23 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:21:25.400 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:21:25.401 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-12T03:21:25.401 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-12T03:21:25.401 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-12T03:21:25.401 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:21:25.401 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 7.4 M 2024-03-12T03:21:25.402 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-12T03:21:25.402 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-12T03:21:25.402 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-12T03:21:25.402 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-12T03:21:25.402 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-12T03:21:25.402 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-12T03:21:25.402 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-12T03:21:25.402 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-12T03:21:25.402 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-12T03:21:25.402 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-12T03:21:25.403 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-12T03:21:25.403 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-12T03:21:25.403 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:25.403 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:21:25.403 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-12T03:21:25.403 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-03-12T03:21:25.403 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:25.404 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 37 M 2024-03-12T03:21:25.404 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 209 M 2024-03-12T03:21:25.404 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:21:25.899 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 207 kB/s | 32 kB 00:00 2024-03-12T03:21:25.933 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 520 kB/s | 17 kB 00:00 2024-03-12T03:21:25.966 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-12T03:21:26.025 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-12T03:21:26.200 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-12T03:21:26.275 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-12T03:21:26.300 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-12T03:21:26.517 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 65 MB/s | 16 MB 00:00 2024-03-12T03:21:26.659 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-19 8.1 MB/s | 7.4 MB 00:00 2024-03-12T03:21:26.835 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-12T03:21:27.101 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 335 kB/s | 195 kB 00:00 2024-03-12T03:21:27.135 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 965 kB/s | 803 kB 00:00 2024-03-12T03:21:27.137 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:21:27.137 INFO:teuthology.orchestra.run.smithi173.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-12T03:21:27.177 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:21:27.201 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:21:27.201 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:21:27.570 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:21:27.570 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:21:28.269 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:21:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-12T03:21:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-12T03:21:28.463 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-12T03:21:28.501 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-12T03:21:28.679 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-12T03:21:28.716 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-12T03:21:28.792 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-12T03:21:29.435 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-12T03:21:29.578 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-12T03:21:30.807 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-12T03:21:31.845 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-12T03:21:31.874 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-12T03:21:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-12T03:21:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 1/12 2024-03-12T03:21:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-12T03:21:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-12T03:21:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-12T03:21:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-12T03:21:34.347 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-12T03:21:34.347 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-12T03:21:34.347 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-12T03:21:34.347 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-12T03:21:34.347 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-12T03:21:34.347 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-12T03:21:34.759 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-12T03:21:34.759 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:34.759 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:21:34.759 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:21:34.759 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-12T03:21:34.759 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-12T03:21:34.759 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-12T03:21:34.760 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-12T03:21:34.760 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-12T03:21:34.760 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-12T03:21:34.760 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-12T03:21:34.760 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-12T03:21:34.760 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-12T03:21:34.760 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-12T03:21:34.760 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-12T03:21:34.760 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:34.760 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:21:35.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-rook 2024-03-12T03:21:35.703 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:33 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:21:35.855 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:21:35.856 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-12T03:21:35.856 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-12T03:21:35.856 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-12T03:21:35.857 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:21:35.857 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 49 k 2024-03-12T03:21:35.857 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-12T03:21:35.857 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-12T03:21:35.857 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-12T03:21:35.857 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-12T03:21:35.857 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-12T03:21:35.857 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-12T03:21:35.857 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-12T03:21:35.857 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-12T03:21:35.857 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-12T03:21:35.858 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-12T03:21:35.858 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-12T03:21:35.858 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-12T03:21:35.858 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-12T03:21:35.858 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:35.858 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:21:35.858 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-12T03:21:35.858 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2024-03-12T03:21:35.858 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:35.859 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 2.0 M 2024-03-12T03:21:35.859 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2024-03-12T03:21:35.859 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:21:36.174 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 130 kB/s | 18 kB 00:00 2024-03-12T03:21:36.199 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 149 kB/s | 24 kB 00:00 2024-03-12T03:21:36.258 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-12T03:21:36.283 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-12T03:21:36.308 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-12T03:21:36.334 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-03-12T03:21:36.359 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): ceph-mgr-rook-19.0.0-1979.g2b0101eb.el9 152 kB/s | 49 kB 00:00 2024-03-12T03:21:36.518 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): python3-google-auth-2.28.0-1.el9.noarch 1.3 MB/s | 203 kB 00:00 2024-03-12T03:21:36.685 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 43 kB/s | 15 kB 00:00 2024-03-12T03:21:36.860 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): python3-cachetools-4.2.4-1.el9.noarch. 58 kB/s | 32 kB 00:00 2024-03-12T03:21:36.968 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 2.3 MB/s | 1.0 MB 00:00 2024-03-12T03:21:37.435 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 79 kB/s | 59 kB 00:00 2024-03-12T03:21:37.860 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 90 kB/s | 90 kB 00:01 2024-03-12T03:21:37.863 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:21:37.863 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.0 MB/s | 2.0 MB 00:02 2024-03-12T03:21:37.863 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:21:37.878 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:21:37.878 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:21:37.986 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:21:37.987 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:21:38.503 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:21:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-12T03:21:38.772 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-12T03:21:38.833 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-12T03:21:38.862 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-12T03:21:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-12T03:21:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-12T03:21:39.106 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-12T03:21:39.156 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-12T03:21:39.875 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-12T03:21:39.919 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-12T03:21:39.961 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-12T03:21:40.009 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-12T03:21:40.031 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-12T03:21:40.870 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-12T03:21:40.870 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 1/13 2024-03-12T03:21:40.870 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-12T03:21:40.870 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-12T03:21:40.870 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-12T03:21:40.870 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-12T03:21:40.870 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-12T03:21:40.870 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-12T03:21:40.870 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-12T03:21:40.870 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-12T03:21:40.871 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-12T03:21:40.871 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-12T03:21:40.871 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-12T03:21:41.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-12T03:21:41.240 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:41.240 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:21:41.240 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:21:41.240 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-12T03:21:41.240 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-12T03:21:41.240 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-12T03:21:41.240 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-12T03:21:41.240 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-12T03:21:41.240 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-12T03:21:41.240 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-12T03:21:41.240 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-12T03:21:41.241 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-12T03:21:41.241 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-12T03:21:41.241 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-12T03:21:41.241 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-12T03:21:41.241 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:41.241 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:21:41.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-cephadm 2024-03-12T03:21:42.084 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:40 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:21:42.239 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:21:42.240 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-12T03:21:42.240 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-12T03:21:42.240 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-12T03:21:42.240 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:21:42.240 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 139 k 2024-03-12T03:21:42.240 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-12T03:21:42.240 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-12T03:21:42.241 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-12T03:21:42.241 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-12T03:21:42.241 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-12T03:21:42.241 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-12T03:21:42.241 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:42.241 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:21:42.241 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-12T03:21:42.241 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-03-12T03:21:42.241 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:21:42.242 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 6.8 M 2024-03-12T03:21:42.242 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 33 M 2024-03-12T03:21:42.242 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:22:02.137 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 145 kB/s | 19 MB 02:13 2024-03-12T03:22:06.195 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-12T03:22:07.796 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 15 kB 00:00 2024-03-12T03:22:13.397 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 3.8 MB/s | 21 MB 00:05 2024-03-12T03:22:18.388 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-12T03:22:20.933 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:22:20.944 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:22:20.944 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-12T03:22:20.944 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:22:20.944 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:22:20.944 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 9.8 M 2024-03-12T03:22:20.944 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.4 M 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.3 M 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 20 M 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 25 k 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 745 k 2024-03-12T03:22:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-12T03:22:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-12T03:22:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-12T03:22:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 501 k 2024-03-12T03:22:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-12T03:22:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.2 M 2024-03-12T03:22:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-12T03:22:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-12T03:22:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-12T03:22:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse 2024-03-12T03:22:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 45 k 2024-03-12T03:22:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 131 k 2024-03-12T03:22:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 162 k 2024-03-12T03:22:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-12T03:22:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 319 k 2024-03-12T03:22:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 298 k 2024-03-12T03:22:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 99 k 2024-03-12T03:22:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-12T03:22:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-12T03:22:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-12T03:22:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-12T03:22:20.948 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-03-12T03:22:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-12T03:22:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:22:20.949 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:22:20.949 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:22:20.949 INFO:teuthology.orchestra.run.smithi169.stdout:Install 30 Packages 2024-03-12T03:22:20.949 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 2 Packages 2024-03-12T03:22:20.949 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:22:20.950 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 58 M 2024-03-12T03:22:20.950 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:22:22.431 INFO:teuthology.orchestra.run.smithi169.stdout:(1/32): ceph-common-19.0.0-1979.g2b0101eb.el9.x 14 MB/s | 20 MB 00:01 2024-03-12T03:22:22.473 INFO:teuthology.orchestra.run.smithi169.stdout:(2/32): ceph-selinux-19.0.0-1979.g2b0101eb.el9. 606 kB/s | 25 kB 00:00 2024-03-12T03:22:22.582 INFO:teuthology.orchestra.run.smithi169.stdout:(3/32): libcephfs2-19.0.0-1979.g2b0101eb.el9.x8 6.7 MB/s | 745 kB 00:00 2024-03-12T03:22:22.640 INFO:teuthology.orchestra.run.smithi169.stdout:(4/32): libradosstriper1-19.0.0-1979.g2b0101eb. 8.4 MB/s | 501 kB 00:00 2024-03-12T03:22:22.957 INFO:teuthology.orchestra.run.smithi169.stdout:(5/32): librgw2-19.0.0-1979.g2b0101eb.el9.x86_6 16 MB/s | 5.2 MB 00:00 2024-03-12T03:22:22.999 INFO:teuthology.orchestra.run.smithi169.stdout:(6/32): python3-ceph-argparse-19.0.0-1979.g2b01 1.1 MB/s | 45 kB 00:00 2024-03-12T03:22:23.066 INFO:teuthology.orchestra.run.smithi169.stdout:(7/32): ceph-base-19.0.0-1979.g2b0101eb.el9.x86 2.7 MB/s | 5.3 MB 00:02 2024-03-12T03:22:23.091 INFO:teuthology.orchestra.run.smithi169.stdout:(8/32): python3-ceph-common-19.0.0-1979.g2b0101 1.4 MB/s | 131 kB 00:00 2024-03-12T03:22:23.150 INFO:teuthology.orchestra.run.smithi169.stdout:(9/32): python3-cephfs-19.0.0-1979.g2b0101eb.el 1.9 MB/s | 162 kB 00:00 2024-03-12T03:22:23.175 INFO:teuthology.orchestra.run.smithi169.stdout:(10/32): python3-rados-19.0.0-1979.g2b0101eb.el 3.7 MB/s | 319 kB 00:00 2024-03-12T03:22:23.217 INFO:teuthology.orchestra.run.smithi169.stdout:(11/32): python3-rbd-19.0.0-1979.g2b0101eb.el9. 4.4 MB/s | 298 kB 00:00 2024-03-12T03:22:23.242 INFO:teuthology.orchestra.run.smithi169.stdout:(12/32): python3-rgw-19.0.0-1979.g2b0101eb.el9. 1.5 MB/s | 99 kB 00:00 2024-03-12T03:22:23.393 INFO:teuthology.orchestra.run.smithi169.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 604 kB/s | 106 kB 00:00 2024-03-12T03:22:23.418 INFO:teuthology.orchestra.run.smithi169.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-12T03:22:23.444 INFO:teuthology.orchestra.run.smithi169.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-12T03:22:23.510 INFO:teuthology.orchestra.run.smithi169.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-12T03:22:23.536 INFO:teuthology.orchestra.run.smithi169.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-12T03:22:23.561 INFO:teuthology.orchestra.run.smithi169.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 823 kB/s | 41 kB 00:00 2024-03-12T03:22:23.678 INFO:teuthology.orchestra.run.smithi169.stdout:(19/32): ceph-radosgw-19.0.0-1979.g2b0101eb.el9 3.7 MB/s | 9.8 MB 00:02 2024-03-12T03:22:23.745 INFO:teuthology.orchestra.run.smithi169.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 190 kB/s | 35 kB 00:00 2024-03-12T03:22:23.770 INFO:teuthology.orchestra.run.smithi169.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 485 kB/s | 44 kB 00:00 2024-03-12T03:22:23.804 INFO:teuthology.orchestra.run.smithi169.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 808 kB/s | 216 kB 00:00 2024-03-12T03:22:24.071 INFO:teuthology.orchestra.run.smithi169.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-03-12T03:22:24.238 INFO:teuthology.orchestra.run.smithi169.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 627 kB/s | 309 kB 00:00 2024-03-12T03:22:24.263 INFO:teuthology.orchestra.run.smithi169.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-03-12T03:22:24.338 INFO:teuthology.orchestra.run.smithi169.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-12T03:22:24.472 INFO:teuthology.orchestra.run.smithi169.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-12T03:22:25.214 INFO:teuthology.orchestra.run.smithi169.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 4.4 MB 00:01 2024-03-12T03:22:25.372 INFO:teuthology.orchestra.run.smithi169.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 1.8 MB/s | 1.6 MB 00:00 2024-03-12T03:22:25.564 INFO:teuthology.orchestra.run.smithi169.stdout:(30/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 644 kB/s | 838 kB 00:01 2024-03-12T03:22:25.648 INFO:teuthology.orchestra.run.smithi169.stdout:(31/32): librbd1-19.0.0-1979.g2b0101eb.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-12T03:22:27.381 INFO:teuthology.orchestra.run.smithi169.stdout:(32/32): librados2-19.0.0-1979.g2b0101eb.el9.x8 1.6 MB/s | 3.4 MB 00:02 2024-03-12T03:22:27.384 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:22:27.385 INFO:teuthology.orchestra.run.smithi169.stdout:Total 9.0 MB/s | 58 MB 00:06 2024-03-12T03:22:27.418 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:22:27.457 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:22:27.457 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:22:27.931 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:22:27.931 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:22:29.393 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:22:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-12T03:22:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-12T03:22:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-12T03:22:29.601 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-12T03:22:29.617 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-12T03:22:29.818 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-12T03:22:29.873 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-12T03:22:30.194 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-12T03:22:30.246 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-12T03:22:30.321 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-12T03:22:30.409 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 8/34 2024-03-12T03:22:30.440 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-12T03:22:30.502 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-12T03:22:30.531 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-12T03:22:30.570 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 11/34 2024-03-12T03:22:30.635 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-12T03:22:30.659 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-12T03:22:30.734 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-12T03:22:30.891 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-12T03:22:31.000 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-12T03:22:31.034 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-12T03:22:31.075 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-12T03:22:31.107 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-12T03:22:31.136 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-12T03:22:31.169 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-12T03:22:31.484 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-12T03:22:31.507 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-12T03:22:31.588 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-12T03:22:31.620 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 23/34 2024-03-12T03:22:31.655 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-12T03:22:32.118 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-12T03:22:32.143 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-12T03:22:32.197 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-12T03:22:32.266 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 27/34 2024-03-12T03:22:32.296 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 28/34 2024-03-12T03:22:34.386 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-12T03:22:34.426 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-12T03:22:34.992 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-12T03:22:35.029 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-12T03:22:35.122 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-12T03:22:35.122 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-12T03:22:35.122 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-12T03:22:35.122 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:22:35.156 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-12T03:22:44.409 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-12T03:22:44.409 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-03-12T03:22:44.409 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-03-12T03:22:44.409 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-03-12T03:22:44.409 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-03-12T03:22:44.409 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-03-12T03:22:44.410 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /root 2024-03-12T03:22:44.410 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-03-12T03:22:44.410 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:22:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-12T03:22:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-12T03:22:45.359 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:22:45.359 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-12T03:22:45.359 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T03:22:45.359 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T03:22:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:22:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-12T03:22:45.443 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-12T03:22:45.444 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-12T03:22:46.584 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-12T03:22:46.584 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/34 2024-03-12T03:22:46.584 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/34 2024-03-12T03:22:46.584 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/34 2024-03-12T03:22:46.584 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/34 2024-03-12T03:22:46.584 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/34 2024-03-12T03:22:46.584 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 6/34 2024-03-12T03:22:46.584 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-12T03:22:46.584 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 8/34 2024-03-12T03:22:46.584 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 9/34 2024-03-12T03:22:46.584 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 10/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 11/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 13/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-12T03:22:46.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-12T03:22:46.586 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-12T03:22:46.586 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-12T03:22:46.586 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-12T03:22:46.587 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-12T03:22:46.587 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-12T03:22:46.587 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-12T03:22:46.587 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-12T03:22:46.587 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-12T03:22:46.587 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/34 2024-03-12T03:22:47.845 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-12T03:22:47.845 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:22:47.845 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-03-12T03:22:47.845 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-12T03:22:47.846 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-12T03:22:47.847 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-12T03:22:47.848 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-12T03:22:47.848 INFO:teuthology.orchestra.run.smithi169.stdout: re2-1:20211101-3.el9.x86_64 2024-03-12T03:22:47.848 INFO:teuthology.orchestra.run.smithi169.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-12T03:22:47.848 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:22:47.848 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:22:48.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test 2024-03-12T03:22:48.666 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:30 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:22:48.824 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:22:48.825 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:22:48.825 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-12T03:22:48.825 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:22:48.825 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:22:48.825 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 48 M 2024-03-12T03:22:48.825 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-12T03:22:48.825 INFO:teuthology.orchestra.run.smithi169.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-12T03:22:48.825 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 163 k 2024-03-12T03:22:48.826 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-12T03:22:48.826 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-12T03:22:48.826 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-12T03:22:48.826 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:22:48.826 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:22:48.826 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:22:48.826 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-03-12T03:22:48.826 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:22:48.827 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 48 M 2024-03-12T03:22:48.827 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 205 M 2024-03-12T03:22:48.827 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:22:49.056 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-12T03:22:49.115 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-12T03:22:49.148 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-12T03:22:49.240 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-12T03:22:49.599 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): libcephsqlite-19.0.0-1979.g2b0101eb.el9. 232 kB/s | 163 kB 00:00 2024-03-12T03:22:52.057 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): ceph-test-19.0.0-1979.g2b0101eb.el9.x86_ 15 MB/s | 48 MB 00:03 2024-03-12T03:22:52.060 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:22:52.060 INFO:teuthology.orchestra.run.smithi169.stdout:Total 15 MB/s | 48 MB 00:03 2024-03-12T03:22:52.075 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:22:52.096 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:22:52.097 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:22:52.485 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:22:52.485 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:22:53.073 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:22:53.149 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-12T03:22:53.189 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-12T03:22:53.221 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-12T03:22:53.274 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-12T03:22:53.302 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-12T03:22:57.810 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-12T03:22:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-12T03:23:00.190 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-12T03:23:00.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/6 2024-03-12T03:23:00.191 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/6 2024-03-12T03:23:00.191 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-12T03:23:00.191 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-12T03:23:00.191 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-12T03:23:00.577 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-12T03:23:00.577 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:23:00.577 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:23:00.577 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:23:00.577 INFO:teuthology.orchestra.run.smithi169.stdout: jq-1.6-16.el9.x86_64 2024-03-12T03:23:00.577 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:23:00.577 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-12T03:23:00.577 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-12T03:23:00.577 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-12T03:23:00.577 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:23:00.578 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:23:00.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph 2024-03-12T03:23:01.468 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:43 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:23:01.631 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:23:01.634 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-03-12T03:23:01.634 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-12T03:23:01.634 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-03-12T03:23:01.634 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:23:01.634 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 6.6 k 2024-03-12T03:23:01.634 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-12T03:23:01.634 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 2.1 M 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 1.5 M 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 248 k 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 4.7 M 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 17 M 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-12T03:23:01.635 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-12T03:23:01.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-12T03:23:01.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-12T03:23:01.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-12T03:23:01.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-12T03:23:01.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-12T03:23:01.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-12T03:23:01.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-12T03:23:01.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-12T03:23:01.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-12T03:23:01.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-12T03:23:01.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-03-12T03:23:01.637 INFO:teuthology.orchestra.run.smithi169.stdout:Install 36 Packages 2024-03-12T03:23:01.638 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:23:01.638 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 30 M 2024-03-12T03:23:01.638 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 107 M 2024-03-12T03:23:01.638 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:23:54.485 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-12T03:23:54.552 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1979.g2b0101eb.e 595 kB/s | 139 kB 00:00 2024-03-12T03:23:54.660 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-12T03:23:54.927 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 155 kB/s | 58 kB 00:00 2024-03-12T03:23:55.010 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 146 kB/s | 51 kB 00:00 2024-03-12T03:23:55.069 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 939 kB/s | 548 kB 00:00 2024-03-12T03:23:55.071 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:23:55.071 INFO:teuthology.orchestra.run.smithi173.stdout:Total 52 kB/s | 6.8 MB 02:12 2024-03-12T03:23:55.084 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:23:55.096 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:23:55.096 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:23:55.192 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:23:55.192 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:23:55.503 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:23:55.667 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-12T03:23:55.720 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-12T03:23:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-12T03:23:56.322 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-12T03:23:56.384 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-12T03:23:56.407 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-12T03:23:57.199 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-12T03:23:57.199 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/6 2024-03-12T03:23:57.199 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-12T03:23:57.199 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-12T03:23:57.199 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-12T03:23:57.199 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-12T03:23:57.535 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-12T03:23:57.535 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:23:57.535 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:23:57.535 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:23:57.535 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-12T03:23:57.535 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-12T03:23:57.535 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-12T03:23:57.535 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-12T03:23:57.535 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-12T03:23:57.535 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:23:57.535 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:23:57.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2024-03-12T03:23:58.367 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:03:56 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:23:58.517 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:23:58.518 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:23:58.518 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-12T03:23:58.518 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:23:58.518 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:23:58.518 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 888 k 2024-03-12T03:23:58.518 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-12T03:23:58.518 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-12T03:23:58.518 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:23:58.518 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:23:58.518 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:23:58.519 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-03-12T03:23:58.519 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:23:58.519 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 966 k 2024-03-12T03:23:58.519 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 2.8 M 2024-03-12T03:23:58.519 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:23:58.663 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 549 kB/s | 79 kB 00:00 2024-03-12T03:23:58.829 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-fuse-19.0.0-1979.g2b0101eb.el9.x86_ 2.8 MB/s | 888 kB 00:00 2024-03-12T03:23:58.831 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:23:58.831 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.0 MB/s | 966 kB 00:00 2024-03-12T03:23:58.831 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:23:58.851 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:23:58.852 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:23:58.929 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:23:58.929 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:23:59.108 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:23:59.277 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-12T03:23:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-12T03:24:00.085 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-12T03:24:00.085 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/2 2024-03-12T03:24:00.408 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-12T03:24:00.408 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:00.409 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:24:00.409 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-12T03:24:00.409 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:00.409 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:00.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-volume 2024-03-12T03:24:01.276 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:03:59 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:01.426 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:01.427 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-12T03:24:01.427 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:01.427 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:24:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 264 k 2024-03-12T03:24:01.427 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-12T03:24:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-12T03:24:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:01.427 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:24:01.428 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:01.428 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-03-12T03:24:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:01.428 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 332 k 2024-03-12T03:24:01.428 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 1.5 M 2024-03-12T03:24:01.428 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:24:01.590 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 427 kB/s | 69 kB 00:00 2024-03-12T03:24:01.844 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-volume-19.0.0-1979.g2b0101eb.el9.no 635 kB/s | 264 kB 00:00 2024-03-12T03:24:01.845 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:24:01.846 INFO:teuthology.orchestra.run.smithi173.stdout:Total 795 kB/s | 332 kB 00:00 2024-03-12T03:24:01.846 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:24:01.854 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:24:01.854 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:24:01.899 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:24:01.899 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:24:02.052 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:24:02.241 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-12T03:24:02.280 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-12T03:24:02.296 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-12T03:24:02.296 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:24:02.296 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-12T03:24:02.296 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:03.038 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 1/2 2024-03-12T03:24:03.389 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-12T03:24:03.390 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:03.390 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:24:03.390 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:24:03.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-12T03:24:03.390 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:03.390 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:03.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2024-03-12T03:24:04.293 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:02 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:04.446 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:04.447 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:04.447 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-12T03:24:04.447 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:04.447 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:24:04.447 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 127 k 2024-03-12T03:24:04.447 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:04.447 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:24:04.447 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:04.447 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-12T03:24:04.447 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:04.447 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 127 k 2024-03-12T03:24:04.447 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 456 k 2024-03-12T03:24:04.448 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:24:04.783 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-19.0.0-1979.g2b0101eb.el9.x86_64 377 kB/s | 127 kB 00:00 2024-03-12T03:24:04.784 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:24:04.784 INFO:teuthology.orchestra.run.smithi173.stdout:Total 375 kB/s | 127 kB 00:00 2024-03-12T03:24:04.784 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:24:04.790 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:24:04.790 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:24:04.845 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:24:04.845 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:24:04.997 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:24:05.115 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:05.608 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:05.940 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:05.940 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:05.940 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:24:05.940 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:24:05.940 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:05.941 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:06.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2024-03-12T03:24:06.810 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:04 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:06.910 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:24:06.962 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:06.963 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-12T03:24:06.963 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:07.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2024-03-12T03:24:07.592 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:05 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:07.741 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:07.742 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-12T03:24:07.742 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:07.742 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:24:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 31 k 2024-03-12T03:24:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:07.742 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:24:07.743 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:07.743 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-12T03:24:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:07.743 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 31 k 2024-03-12T03:24:07.743 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 140 k 2024-03-12T03:24:07.743 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:24:07.903 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-19.0.0-1979.g2b0101eb.el9.x86_6 195 kB/s | 31 kB 00:00 2024-03-12T03:24:07.904 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:24:07.904 INFO:teuthology.orchestra.run.smithi173.stdout:Total 193 kB/s | 31 kB 00:00 2024-03-12T03:24:07.904 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:24:07.907 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:24:07.907 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:24:07.933 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:24:07.934 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:24:08.080 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:24:08.176 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:08.855 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:08.855 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:08.855 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:24:08.855 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:24:08.856 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:08.856 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:09.009 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2024-03-12T03:24:09.568 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:07 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:09.665 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:24:09.719 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:09.720 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-12T03:24:09.720 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:09.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2024-03-12T03:24:10.317 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:08 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:10.414 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:24:10.467 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:10.468 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-12T03:24:10.468 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:10.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2024-03-12T03:24:11.087 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:09 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:11.183 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:24:11.234 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:11.235 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-12T03:24:11.235 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:11.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2024-03-12T03:24:11.847 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:09 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:11.943 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:24:11.994 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:11.995 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-12T03:24:11.995 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:12.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2024-03-12T03:24:12.600 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:10 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:12.698 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:24:12.749 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:12.750 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-12T03:24:12.750 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:12.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2024-03-12T03:24:13.364 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:11 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:13.460 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:24:13.511 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:13.512 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-12T03:24:13.512 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:13.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2024-03-12T03:24:14.121 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:12 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:14.272 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:14.273 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:14.273 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-03-12T03:24:14.273 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:14.273 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:24:14.273 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 85 k 2024-03-12T03:24:14.273 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:14.273 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:24:14.273 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:14.273 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-12T03:24:14.273 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:14.274 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 85 k 2024-03-12T03:24:14.274 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 231 k 2024-03-12T03:24:14.274 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:24:14.490 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 394 kB/s | 85 kB 00:00 2024-03-12T03:24:14.490 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:24:14.491 INFO:teuthology.orchestra.run.smithi173.stdout:Total 392 kB/s | 85 kB 00:00 2024-03-12T03:24:14.491 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:24:14.497 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:24:14.497 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:24:14.553 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:24:14.553 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:24:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:24:14.787 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:15.184 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:15.555 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:24:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:24:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:15.555 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:15.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2024-03-12T03:24:16.354 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:14 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:16.504 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:16.505 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:16.505 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-12T03:24:16.505 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:16.505 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:24:16.505 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-12T03:24:16.505 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:16.505 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:24:16.505 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:16.505 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-12T03:24:16.505 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:16.506 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.1 M 2024-03-12T03:24:16.506 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 13 M 2024-03-12T03:24:16.506 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:24:16.892 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 8.0 MB/s | 3.1 MB 00:00 2024-03-12T03:24:16.893 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:24:16.893 INFO:teuthology.orchestra.run.smithi173.stdout:Total 8.0 MB/s | 3.1 MB 00:00 2024-03-12T03:24:16.893 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:24:16.902 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:24:16.903 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:24:16.975 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:24:16.975 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:24:17.388 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:24:17.452 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:17.475 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:17.475 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:24:17.475 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-12T03:24:17.475 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T03:24:17.475 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T03:24:17.475 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:18.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:18.671 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:18.672 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:24:18.672 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:24:18.672 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:18.672 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:18.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2024-03-12T03:24:19.473 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:17 ago on Tue 12 Mar 2024 03:20:02 AM UTC. 2024-03-12T03:24:19.623 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-12T03:24:19.623 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:19.624 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-03-12T03:24:19.624 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:19.624 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-12T03:24:19.624 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 170 k 2024-03-12T03:24:19.624 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:19.624 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-12T03:24:19.624 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-12T03:24:19.624 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-12T03:24:19.624 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:19.625 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 170 k 2024-03-12T03:24:19.625 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 490 k 2024-03-12T03:24:19.625 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-12T03:24:19.837 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 798 kB/s | 170 kB 00:00 2024-03-12T03:24:19.838 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:24:19.838 INFO:teuthology.orchestra.run.smithi173.stdout:Total 794 kB/s | 170 kB 00:00 2024-03-12T03:24:19.838 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-12T03:24:19.846 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-12T03:24:19.846 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-12T03:24:19.901 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-12T03:24:19.901 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-12T03:24:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-12T03:24:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:20.611 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:24:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:20.920 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-12T03:24:20.921 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:24:20.921 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:24:20.921 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-12T03:24:21.153 DEBUG:teuthology.parallel:result is None 2024-03-12T03:25:14.883 INFO:teuthology.orchestra.run.smithi169.stdout:(1/36): ceph-19.0.0-1979.g2b0101eb.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-12T03:25:15.532 INFO:teuthology.orchestra.run.smithi169.stdout:(2/36): ceph-mds-19.0.0-1979.g2b0101eb.el9.x86_ 2.0 MB/s | 2.1 MB 00:01 2024-03-12T03:25:15.599 INFO:teuthology.orchestra.run.smithi169.stdout:(3/36): ceph-mgr-19.0.0-1979.g2b0101eb.el9.x86_ 1.3 MB/s | 1.5 MB 00:01 2024-03-12T03:25:15.707 INFO:teuthology.orchestra.run.smithi169.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1979.g2b01 2.2 MB/s | 248 kB 00:00 2024-03-12T03:25:15.858 INFO:teuthology.orchestra.run.smithi169.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-12T03:25:15.900 INFO:teuthology.orchestra.run.smithi169.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-12T03:25:15.942 INFO:teuthology.orchestra.run.smithi169.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 759 kB/s | 32 kB 00:00 2024-03-12T03:25:15.975 INFO:teuthology.orchestra.run.smithi169.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-12T03:25:16.009 INFO:teuthology.orchestra.run.smithi169.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-12T03:25:16.059 INFO:teuthology.orchestra.run.smithi169.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-12T03:25:16.134 INFO:teuthology.orchestra.run.smithi169.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-12T03:25:16.177 INFO:teuthology.orchestra.run.smithi169.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-12T03:25:16.218 INFO:teuthology.orchestra.run.smithi169.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-12T03:25:16.277 INFO:teuthology.orchestra.run.smithi169.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-12T03:25:16.527 INFO:teuthology.orchestra.run.smithi169.stdout:(15/36): ceph-mon-19.0.0-1979.g2b0101eb.el9.x86 2.8 MB/s | 4.7 MB 00:01 2024-03-12T03:25:16.552 INFO:teuthology.orchestra.run.smithi169.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 454 kB/s | 125 kB 00:00 2024-03-12T03:25:16.769 INFO:teuthology.orchestra.run.smithi169.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 698 kB/s | 151 kB 00:00 2024-03-12T03:25:17.161 INFO:teuthology.orchestra.run.smithi169.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 111 kB/s | 43 kB 00:00 2024-03-12T03:25:17.245 INFO:teuthology.orchestra.run.smithi169.stdout:(19/36): lua-devel-5.4.4-4.el9.x86_64.rpm 31 kB/s | 22 kB 00:00 2024-03-12T03:25:17.554 INFO:teuthology.orchestra.run.smithi169.stdout:(20/36): ceph-osd-19.0.0-1979.g2b0101eb.el9.x86 8.3 MB/s | 17 MB 00:02 2024-03-12T03:25:17.587 INFO:teuthology.orchestra.run.smithi169.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 404 kB/s | 172 kB 00:00 2024-03-12T03:25:17.712 INFO:teuthology.orchestra.run.smithi169.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 768 kB/s | 358 kB 00:00 2024-03-12T03:25:17.729 INFO:teuthology.orchestra.run.smithi169.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 61 kB/s | 11 kB 00:00 2024-03-12T03:25:17.830 INFO:teuthology.orchestra.run.smithi169.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 82 kB/s | 18 kB 00:00 2024-03-12T03:25:17.905 INFO:teuthology.orchestra.run.smithi169.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 121 kB/s | 23 kB 00:00 2024-03-12T03:25:17.930 INFO:teuthology.orchestra.run.smithi169.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 97 kB/s | 19 kB 00:00 2024-03-12T03:25:18.022 INFO:teuthology.orchestra.run.smithi169.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 103 kB/s | 20 kB 00:00 2024-03-12T03:25:18.081 INFO:teuthology.orchestra.run.smithi169.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 264 kB/s | 46 kB 00:00 2024-03-12T03:25:18.139 INFO:teuthology.orchestra.run.smithi169.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 427 kB/s | 79 kB 00:00 2024-03-12T03:25:18.198 INFO:teuthology.orchestra.run.smithi169.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 142 kB/s | 16 kB 00:00 2024-03-12T03:25:18.223 INFO:teuthology.orchestra.run.smithi169.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-12T03:25:18.307 INFO:teuthology.orchestra.run.smithi169.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 540 kB/s | 90 kB 00:00 2024-03-12T03:25:18.366 INFO:teuthology.orchestra.run.smithi169.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 215 kB/s | 36 kB 00:00 2024-03-12T03:25:18.449 INFO:teuthology.orchestra.run.smithi169.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-03-12T03:25:18.474 INFO:teuthology.orchestra.run.smithi169.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 186 kB/s | 20 kB 00:00 2024-03-12T03:25:18.549 INFO:teuthology.orchestra.run.smithi169.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2024-03-12T03:25:18.553 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:25:18.553 INFO:teuthology.orchestra.run.smithi169.stdout:Total 222 kB/s | 30 MB 02:16 2024-03-12T03:25:18.572 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:25:18.611 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:25:18.611 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:25:19.018 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:25:19.018 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:25:20.449 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:25:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-12T03:25:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-12T03:25:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-12T03:25:20.650 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-12T03:25:20.694 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-12T03:25:20.727 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-12T03:25:20.820 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-12T03:25:20.898 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-12T03:25:20.938 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-12T03:25:21.010 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-12T03:25:21.064 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-12T03:25:21.103 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-12T03:25:21.160 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-12T03:25:21.216 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-12T03:25:21.402 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-12T03:25:21.448 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-12T03:25:21.491 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-12T03:25:21.522 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-12T03:25:21.552 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-12T03:25:21.628 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-12T03:25:21.662 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-12T03:25:21.704 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-12T03:25:21.744 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-12T03:25:21.779 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-12T03:25:21.814 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-12T03:25:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-12T03:25:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-12T03:25:22.017 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-12T03:25:22.163 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-12T03:25:22.289 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-12T03:25:22.453 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 31/36 2024-03-12T03:25:22.483 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-12T03:25:22.507 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-12T03:25:22.507 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:25:22.508 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-12T03:25:22.508 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T03:25:22.508 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T03:25:22.508 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:23.951 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-12T03:25:23.974 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-12T03:25:23.975 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:25:23.975 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-12T03:25:23.975 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T03:25:23.975 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T03:25:23.975 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-12T03:25:24.458 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-12T03:25:24.459 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:25:24.459 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-12T03:25:24.459 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T03:25:24.459 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T03:25:24.459 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:24.677 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-12T03:25:24.699 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-12T03:25:24.699 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:25:24.700 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-12T03:25:24.700 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T03:25:24.700 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T03:25:24.700 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:24.759 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-12T03:25:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-12T03:25:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/36 2024-03-12T03:25:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/36 2024-03-12T03:25:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/36 2024-03-12T03:25:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/36 2024-03-12T03:25:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/36 2024-03-12T03:25:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 6/36 2024-03-12T03:25:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-12T03:25:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-12T03:25:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-12T03:25:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-12T03:25:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-12T03:25:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-12T03:25:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-12T03:25:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-12T03:25:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-12T03:25:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-12T03:25:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-12T03:25:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-12T03:25:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-12T03:25:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-12T03:25:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-12T03:25:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-12T03:25:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-12T03:25:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-12T03:25:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:27.497 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:25:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:25:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:25:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:25:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:25:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: lua-5.4.4-4.el9.x86_64 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-12T03:25:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-12T03:25:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-12T03:25:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-12T03:25:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-12T03:25:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-12T03:25:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-12T03:25:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-12T03:25:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply-3.11-14.el9.noarch 2024-03-12T03:25:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-12T03:25:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-12T03:25:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-12T03:25:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-12T03:25:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-12T03:25:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-12T03:25:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-12T03:25:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-12T03:25:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-12T03:25:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-12T03:25:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-12T03:25:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:27.500 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:25:27.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base 2024-03-12T03:25:28.448 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:03:10 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:25:28.545 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:25:28.596 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:25:28.597 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-12T03:25:28.597 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:25:28.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install cephadm 2024-03-12T03:25:29.232 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:03:11 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:25:29.380 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:25:29.381 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:25:29.381 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-12T03:25:29.381 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:25:29.381 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:25:29.381 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 748 k 2024-03-12T03:25:29.381 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:29.381 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:25:29.381 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:25:29.381 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-12T03:25:29.382 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:29.382 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 748 k 2024-03-12T03:25:29.382 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 755 k 2024-03-12T03:25:29.382 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:25:29.652 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm-19.0.0-1979.g2b0101eb.el9.noarch.rpm 2.7 MB/s | 748 kB 00:00 2024-03-12T03:25:29.653 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:25:29.653 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.7 MB/s | 748 kB 00:00 2024-03-12T03:25:29.653 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:25:29.658 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:25:29.658 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:25:29.663 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:25:29.664 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:25:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:25:30.210 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:25:30.298 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:25:30.714 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:25:30.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-12T03:25:30.998 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:30.998 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:25:30.998 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:25:30.998 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:30.998 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:25:31.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache 2024-03-12T03:25:31.769 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:03:13 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:25:31.925 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:25:31.926 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:25:31.926 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-12T03:25:31.926 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:25:31.926 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:25:31.926 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 145 k 2024-03-12T03:25:31.926 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:31.927 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:25:31.927 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:25:31.927 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-12T03:25:31.927 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:31.927 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 145 k 2024-03-12T03:25:31.927 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 436 k 2024-03-12T03:25:31.927 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:25:32.400 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-19.0.0-1979.g2b0101 307 kB/s | 145 kB 00:00 2024-03-12T03:25:32.401 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:25:32.401 INFO:teuthology.orchestra.run.smithi169.stdout:Total 306 kB/s | 145 kB 00:00 2024-03-12T03:25:32.401 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:25:32.408 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:25:32.409 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:25:32.462 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:25:32.463 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:25:32.599 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:25:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-12T03:25:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-12T03:25:32.693 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:25:32.693 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-12T03:25:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:33.681 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-12T03:25:33.681 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:33.681 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:25:33.681 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:25:33.681 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:33.681 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:25:33.932 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr 2024-03-12T03:25:34.510 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:03:16 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:25:34.612 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:25:34.664 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:25:34.665 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-12T03:25:34.665 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:25:34.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-dashboard 2024-03-12T03:25:35.309 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:03:17 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:25:35.458 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:25:35.459 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-03-12T03:25:35.459 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-12T03:25:35.459 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-03-12T03:25:35.459 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:25:35.459 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 1.8 M 2024-03-12T03:25:35.459 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 34 k 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 16 k 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:35.460 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:25:35.461 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-03-12T03:25:35.461 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-03-12T03:25:35.461 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:25:35.461 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 6.9 M 2024-03-12T03:25:35.461 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 26 M 2024-03-12T03:25:35.461 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:25:35.904 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1979.g2b 132 kB/s | 34 kB 00:00 2024-03-12T03:25:35.929 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1979.g2b0 58 kB/s | 16 kB 00:00 2024-03-12T03:25:36.104 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-03-12T03:25:36.138 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-12T03:25:36.313 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.1 MB/s | 863 kB 00:00 2024-03-12T03:25:36.355 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1979.g2b0101e 2.5 MB/s | 1.8 MB 00:00 2024-03-12T03:25:36.622 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 63 kB/s | 19 kB 00:00 2024-03-12T03:25:39.114 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 58 kB/s | 144 kB 00:02 2024-03-12T03:25:39.381 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 115 kB/s | 31 kB 00:00 2024-03-12T03:25:41.081 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 111 kB/s | 188 kB 00:01 2024-03-12T03:25:42.539 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 86 kB/s | 548 kB 00:06 2024-03-12T03:26:01.954 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 82 kB/s | 2.0 MB 00:25 2024-03-12T03:26:01.957 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:26:01.957 INFO:teuthology.orchestra.run.smithi169.stdout:Total 266 kB/s | 6.9 MB 00:26 2024-03-12T03:26:01.976 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:26:01.995 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:26:01.995 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:26:02.183 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:26:02.184 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:26:02.707 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:26:02.789 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-12T03:26:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-12T03:26:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-12T03:26:02.900 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-12T03:26:03.014 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-12T03:26:03.147 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-12T03:26:03.235 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-12T03:26:03.279 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-12T03:26:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-12T03:26:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 10/12 2024-03-12T03:26:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 11/12 2024-03-12T03:26:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-12T03:26:04.390 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-12T03:26:04.390 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 1/12 2024-03-12T03:26:04.390 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 2/12 2024-03-12T03:26:04.390 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 3/12 2024-03-12T03:26:04.390 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-12T03:26:04.390 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-12T03:26:04.390 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-12T03:26:04.390 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-12T03:26:04.390 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-12T03:26:04.390 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-12T03:26:04.391 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-12T03:26:04.391 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-12T03:26:04.760 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-12T03:26:04.760 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:04.760 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:26:04.760 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-12T03:26:04.760 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:26:04.760 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:26:04.760 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:26:04.760 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-12T03:26:04.761 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-12T03:26:04.761 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-12T03:26:04.761 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-12T03:26:04.761 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-12T03:26:04.761 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-12T03:26:04.761 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-12T03:26:04.761 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-12T03:26:04.761 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:04.761 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:04.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-12T03:26:05.559 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:03:47 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:05.712 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:05.713 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-12T03:26:05.713 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-12T03:26:05.713 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-12T03:26:05.713 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 7.4 M 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-12T03:26:05.714 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:05.715 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:26:05.715 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-12T03:26:05.715 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-03-12T03:26:05.715 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:05.715 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 37 M 2024-03-12T03:26:05.715 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 209 M 2024-03-12T03:26:05.715 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:26:06.043 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 214 kB/s | 32 kB 00:00 2024-03-12T03:26:06.085 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 426 kB/s | 17 kB 00:00 2024-03-12T03:26:06.135 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-12T03:26:06.161 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 469 kB/s | 34 kB 00:00 2024-03-12T03:26:06.319 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-12T03:26:06.386 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-12T03:26:06.420 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-12T03:26:06.645 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 61 MB/s | 16 MB 00:00 2024-03-12T03:26:06.846 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-19 7.8 MB/s | 7.4 MB 00:00 2024-03-12T03:26:07.046 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-12T03:26:07.146 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 390 kB/s | 195 kB 00:00 2024-03-12T03:26:07.388 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 832 kB/s | 803 kB 00:00 2024-03-12T03:26:07.390 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:26:07.390 INFO:teuthology.orchestra.run.smithi169.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-12T03:26:07.431 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:26:07.457 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:26:07.457 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:26:07.823 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:26:07.823 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:26:08.561 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:26:08.627 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-12T03:26:08.659 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-12T03:26:08.754 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-12T03:26:08.796 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-12T03:26:08.976 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-12T03:26:09.011 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-12T03:26:09.087 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-12T03:26:09.729 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-12T03:26:09.840 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-12T03:26:11.084 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-12T03:26:12.104 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-12T03:26:12.127 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-12T03:26:14.686 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-12T03:26:14.686 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 1/12 2024-03-12T03:26:14.686 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-12T03:26:14.686 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-12T03:26:14.686 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-12T03:26:14.686 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-12T03:26:14.687 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-12T03:26:14.687 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-12T03:26:14.687 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-12T03:26:14.687 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-12T03:26:14.687 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-12T03:26:14.687 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-12T03:26:15.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-12T03:26:15.189 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:15.189 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:26:15.189 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:26:15.190 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-12T03:26:15.190 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-12T03:26:15.190 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-12T03:26:15.190 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-12T03:26:15.190 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-12T03:26:15.190 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-12T03:26:15.190 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-12T03:26:15.190 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-12T03:26:15.190 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-12T03:26:15.190 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-12T03:26:15.190 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-12T03:26:15.190 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:15.191 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:15.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-rook 2024-03-12T03:26:16.110 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:03:58 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:16.262 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:16.263 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-12T03:26:16.263 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-12T03:26:16.263 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-12T03:26:16.263 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:26:16.263 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 49 k 2024-03-12T03:26:16.263 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-12T03:26:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:16.265 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:26:16.265 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-12T03:26:16.265 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2024-03-12T03:26:16.265 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:16.265 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 2.0 M 2024-03-12T03:26:16.265 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 26 M 2024-03-12T03:26:16.266 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:26:30.202 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 178 kB/s | 24 kB 00:00 2024-03-12T03:26:30.218 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2024-03-12T03:26:30.293 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-12T03:26:30.319 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-03-12T03:26:30.344 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-12T03:26:30.369 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-12T03:26:30.494 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): ceph-mgr-rook-19.0.0-1979.g2b0101eb.el9 115 kB/s | 49 kB 00:00 2024-03-12T03:26:30.653 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-03-12T03:26:30.678 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 96 kB/s | 32 kB 00:00 2024-03-12T03:26:30.862 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-03-12T03:26:31.004 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 398 kB/s | 203 kB 00:00 2024-03-12T03:26:31.046 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 489 kB/s | 90 kB 00:00 2024-03-12T03:26:31.254 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-03-12T03:26:31.256 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:26:31.256 INFO:teuthology.orchestra.run.smithi169.stdout:Total 140 kB/s | 2.0 MB 00:14 2024-03-12T03:26:31.257 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:26:31.271 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:26:31.272 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:26:31.380 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:26:31.380 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:26:31.897 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:26:32.068 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-12T03:26:32.122 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-12T03:26:32.177 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-12T03:26:32.214 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-12T03:26:32.264 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-12T03:26:32.367 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-12T03:26:32.476 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-12T03:26:32.532 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-12T03:26:33.257 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-12T03:26:33.304 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-12T03:26:33.338 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-12T03:26:33.387 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-12T03:26:33.408 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-12T03:26:34.156 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-12T03:26:34.156 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 1/13 2024-03-12T03:26:34.156 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-12T03:26:34.156 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-12T03:26:34.156 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-12T03:26:34.156 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-12T03:26:34.156 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-12T03:26:34.156 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-12T03:26:34.156 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-12T03:26:34.157 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-12T03:26:34.157 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-12T03:26:34.157 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-12T03:26:34.157 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-12T03:26:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-12T03:26:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:34.533 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:26:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:26:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-12T03:26:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-12T03:26:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-12T03:26:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-12T03:26:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-12T03:26:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-12T03:26:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-12T03:26:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-12T03:26:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-12T03:26:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-12T03:26:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-12T03:26:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-12T03:26:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:34.534 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:34.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-cephadm 2024-03-12T03:26:35.402 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:17 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:35.561 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:35.562 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-12T03:26:35.562 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-12T03:26:35.562 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-12T03:26:35.562 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:26:35.562 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 139 k 2024-03-12T03:26:35.562 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-12T03:26:35.562 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-12T03:26:35.562 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-12T03:26:35.562 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-12T03:26:35.562 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-12T03:26:35.562 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-12T03:26:35.563 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:35.563 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:26:35.563 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-12T03:26:35.563 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-03-12T03:26:35.563 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:35.564 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 6.8 M 2024-03-12T03:26:35.564 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 33 M 2024-03-12T03:26:35.564 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:26:36.356 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-12T03:26:36.581 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1979.g2b0101eb.e 276 kB/s | 139 kB 00:00 2024-03-12T03:26:36.931 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-03-12T03:26:36.957 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 949 kB/s | 548 kB 00:00 2024-03-12T03:26:37.057 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 407 kB/s | 51 kB 00:00 2024-03-12T03:26:37.182 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 225 kB/s | 248 kB 00:01 2024-03-12T03:26:37.183 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:26:37.184 INFO:teuthology.orchestra.run.smithi169.stdout:Total 4.2 MB/s | 6.8 MB 00:01 2024-03-12T03:26:37.197 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:26:37.207 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:26:37.207 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:26:37.299 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:26:37.299 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:26:37.591 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:26:37.763 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-12T03:26:37.817 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-12T03:26:38.338 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-12T03:26:38.422 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-12T03:26:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-12T03:26:38.506 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-12T03:26:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-12T03:26:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/6 2024-03-12T03:26:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-12T03:26:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-12T03:26:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-12T03:26:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-12T03:26:39.713 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-12T03:26:39.713 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:39.713 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:26:39.713 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:26:39.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-12T03:26:39.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-12T03:26:39.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-12T03:26:39.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-12T03:26:39.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-12T03:26:39.714 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:39.714 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:39.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse 2024-03-12T03:26:40.535 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:22 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:40.685 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:40.686 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:40.686 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-12T03:26:40.686 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:40.686 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:26:40.686 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 888 k 2024-03-12T03:26:40.686 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-12T03:26:40.686 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-12T03:26:40.686 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:40.686 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:26:40.687 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:40.687 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-03-12T03:26:40.687 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:40.687 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 966 k 2024-03-12T03:26:40.687 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.8 M 2024-03-12T03:26:40.687 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:26:40.827 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 560 kB/s | 79 kB 00:00 2024-03-12T03:26:40.994 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-fuse-19.0.0-1979.g2b0101eb.el9.x86_ 2.8 MB/s | 888 kB 00:00 2024-03-12T03:26:40.995 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:26:40.996 INFO:teuthology.orchestra.run.smithi169.stdout:Total 3.0 MB/s | 966 kB 00:00 2024-03-12T03:26:40.996 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:26:41.017 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:26:41.017 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:26:41.096 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:26:41.096 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:26:41.283 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:26:41.429 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-12T03:26:41.502 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-12T03:26:42.202 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-12T03:26:42.202 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/2 2024-03-12T03:26:42.529 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-12T03:26:42.530 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:42.530 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:26:42.530 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-12T03:26:42.530 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:42.530 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:42.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-volume 2024-03-12T03:26:43.429 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:25 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:43.581 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:43.581 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:43.581 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-12T03:26:43.581 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:43.581 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:26:43.581 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 264 k 2024-03-12T03:26:43.581 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-12T03:26:43.582 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-12T03:26:43.582 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:43.582 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:26:43.582 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:43.582 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-03-12T03:26:43.582 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:43.583 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 332 k 2024-03-12T03:26:43.583 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.5 M 2024-03-12T03:26:43.583 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:26:43.732 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 462 kB/s | 69 kB 00:00 2024-03-12T03:26:43.940 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-volume-19.0.0-1979.g2b0101eb.el9.no 738 kB/s | 264 kB 00:00 2024-03-12T03:26:43.941 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:26:43.942 INFO:teuthology.orchestra.run.smithi169.stdout:Total 924 kB/s | 332 kB 00:00 2024-03-12T03:26:43.942 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:26:43.950 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:26:43.950 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:26:43.988 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:26:43.988 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:26:44.135 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:26:44.324 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-12T03:26:44.360 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-12T03:26:44.375 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-12T03:26:44.375 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:26:44.375 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-12T03:26:44.375 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:45.150 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 1/2 2024-03-12T03:26:45.508 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-12T03:26:45.508 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:45.508 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:26:45.508 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-12T03:26:45.509 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-12T03:26:45.509 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:45.509 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:45.799 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel 2024-03-12T03:26:46.398 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:28 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:46.549 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:46.550 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:46.550 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-12T03:26:46.551 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:46.551 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:26:46.551 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 127 k 2024-03-12T03:26:46.551 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:46.551 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:26:46.551 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:46.551 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-12T03:26:46.551 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:46.551 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 127 k 2024-03-12T03:26:46.551 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 456 k 2024-03-12T03:26:46.551 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:26:46.863 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-19.0.0-1979.g2b0101eb.el9.x86_64 406 kB/s | 127 kB 00:00 2024-03-12T03:26:46.864 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:26:46.864 INFO:teuthology.orchestra.run.smithi169.stdout:Total 404 kB/s | 127 kB 00:00 2024-03-12T03:26:46.864 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:26:46.870 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:26:46.870 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:26:46.926 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:26:46.926 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:26:47.069 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:26:47.187 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:26:47.659 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:26:47.995 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:26:47.995 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:47.995 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:26:47.995 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:26:47.995 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:47.995 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:48.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2 2024-03-12T03:26:48.845 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:30 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:48.943 INFO:teuthology.orchestra.run.smithi169.stdout:Package libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:26:48.998 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:48.999 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-12T03:26:49.000 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:49.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs-devel 2024-03-12T03:26:49.659 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:31 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:49.809 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:49.810 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:49.810 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-12T03:26:49.810 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:49.810 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:26:49.810 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 31 k 2024-03-12T03:26:49.810 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:49.810 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:26:49.811 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:49.811 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-12T03:26:49.811 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:49.811 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 31 k 2024-03-12T03:26:49.811 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 140 k 2024-03-12T03:26:49.811 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:26:50.050 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs-devel-19.0.0-1979.g2b0101eb.el9.x86_6 130 kB/s | 31 kB 00:00 2024-03-12T03:26:50.051 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:26:50.051 INFO:teuthology.orchestra.run.smithi169.stdout:Total 130 kB/s | 31 kB 00:00 2024-03-12T03:26:50.051 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:26:50.054 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:26:50.054 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:26:50.085 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:26:50.085 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:26:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:26:50.314 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:26:50.725 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:26:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:26:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:51.005 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:26:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:26:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:51.006 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:51.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2 2024-03-12T03:26:51.758 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:33 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:51.855 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:26:51.906 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:51.907 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-12T03:26:51.908 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:51.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1 2024-03-12T03:26:52.520 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:34 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:52.617 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:26:52.668 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:52.669 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-12T03:26:52.669 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:52.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados 2024-03-12T03:26:53.290 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:35 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:53.387 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:26:53.437 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:53.438 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-12T03:26:53.439 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:53.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw 2024-03-12T03:26:54.064 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:36 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:54.160 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:26:54.211 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:54.212 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-12T03:26:54.212 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:54.271 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs 2024-03-12T03:26:54.835 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:36 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:54.932 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:26:54.984 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:54.985 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-12T03:26:54.985 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:55.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd 2024-03-12T03:26:55.606 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:37 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:55.702 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-12T03:26:55.753 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:55.754 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-12T03:26:55.754 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:55.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse 2024-03-12T03:26:56.377 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:38 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:56.525 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:56.525 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:56.526 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-12T03:26:56.526 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:56.526 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:26:56.526 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 85 k 2024-03-12T03:26:56.526 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:56.526 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:26:56.526 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:56.526 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-12T03:26:56.526 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:56.526 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 85 k 2024-03-12T03:26:56.526 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 231 k 2024-03-12T03:26:56.526 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:26:56.911 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 221 kB/s | 85 kB 00:00 2024-03-12T03:26:56.912 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:26:56.912 INFO:teuthology.orchestra.run.smithi169.stdout:Total 221 kB/s | 85 kB 00:00 2024-03-12T03:26:56.912 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:26:56.919 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:26:56.919 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:26:56.974 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:26:56.974 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:26:57.108 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:26:57.226 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:26:57.648 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:26:57.960 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:26:57.960 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:57.960 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:26:57.960 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:26:57.960 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:57.960 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:26:58.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror 2024-03-12T03:26:58.781 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:40 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:26:58.937 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:26:58.938 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:58.938 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-12T03:26:58.938 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:58.938 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:26:58.938 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-12T03:26:58.938 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:58.938 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:26:58.938 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:26:58.938 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-12T03:26:58.938 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:26:58.938 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 3.1 M 2024-03-12T03:26:58.939 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 13 M 2024-03-12T03:26:58.939 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:27:01.622 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 1.2 MB/s | 3.1 MB 00:02 2024-03-12T03:27:01.623 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:27:01.623 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.2 MB/s | 3.1 MB 00:02 2024-03-12T03:27:01.624 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:27:01.634 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:27:01.634 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:27:01.711 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:27:01.711 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:27:02.138 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:27:02.206 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:27:02.231 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:27:02.231 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:27:02.231 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-12T03:27:02.231 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T03:27:02.231 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T03:27:02.231 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:27:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:27:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:27:03.399 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:27:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:27:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:27:03.399 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:27:03.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd 2024-03-12T03:27:04.210 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:46 ago on Tue 12 Mar 2024 03:22:18 AM UTC. 2024-03-12T03:27:04.362 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-12T03:27:04.363 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:27:04.365 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-12T03:27:04.365 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:27:04.365 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-12T03:27:04.365 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 170 k 2024-03-12T03:27:04.365 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:27:04.365 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-12T03:27:04.365 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-12T03:27:04.365 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-12T03:27:04.366 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:27:04.366 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 170 k 2024-03-12T03:27:04.366 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 490 k 2024-03-12T03:27:04.366 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-12T03:27:04.950 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 291 kB/s | 170 kB 00:00 2024-03-12T03:27:04.951 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:27:04.951 INFO:teuthology.orchestra.run.smithi169.stdout:Total 290 kB/s | 170 kB 00:00 2024-03-12T03:27:04.951 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-12T03:27:04.959 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-12T03:27:04.959 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-12T03:27:05.016 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-12T03:27:05.017 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-12T03:27:05.175 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-12T03:27:05.282 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:27:05.747 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:27:06.078 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-12T03:27:06.078 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:27:06.078 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-12T03:27:06.078 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-12T03:27:06.078 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:27:06.078 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-12T03:27:06.328 DEBUG:teuthology.parallel:result is None 2024-03-12T03:27:06.329 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:27:06.474 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-12T03:27:06.495 INFO:teuthology.orchestra.run.smithi169.stdout:19.0.0-1979.g2b0101eb.el9 2024-03-12T03:27:06.495 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979.g2b0101eb.el9 2024-03-12T03:27:06.496 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979.g2b0101eb is installed. 2024-03-12T03:27:06.497 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:27:06.716 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-12T03:27:06.743 INFO:teuthology.orchestra.run.smithi173.stdout:19.0.0-1979.g2b0101eb.el9 2024-03-12T03:27:06.743 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979.g2b0101eb.el9 2024-03-12T03:27:06.743 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979.g2b0101eb is installed. 2024-03-12T03:27:06.745 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-12T03:27:06.745 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:27:06.745 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-12T03:27:06.774 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:27:06.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-12T03:27:06.816 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-12T03:27:06.816 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:27:06.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-03-12T03:27:06.843 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-12T03:27:06.909 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:27:06.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-03-12T03:27:06.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-12T03:27:07.008 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-12T03:27:07.008 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:27:07.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-12T03:27:07.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-12T03:27:07.101 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:27:07.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-12T03:27:07.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-12T03:27:07.197 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-12T03:27:07.197 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:27:07.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-03-12T03:27:07.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-12T03:27:07.290 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:27:07.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-03-12T03:27:07.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-12T03:27:07.385 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-12T03:27:07.490 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'cephadm_mode': 'root'} 2024-03-12T03:27:07.490 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:27:07.491 INFO:tasks.cephadm:Cluster fsid is 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:27:07.491 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-12T03:27:07.491 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.169', 'mon.c': '[v2:172.21.15.169:3301,v1:172.21.15.169:6790]', 'mon.b': '172.21.15.173'} 2024-03-12T03:27:07.491 INFO:tasks.cephadm:First mon is mon.a on smithi169 2024-03-12T03:27:07.491 INFO:tasks.cephadm:First mgr is y 2024-03-12T03:27:07.491 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-12T03:27:07.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname $(hostname -s) 2024-03-12T03:27:07.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-03-12T03:27:07.547 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-12T03:27:07.547 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:27:07.685 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54565/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1979-g2b0101eb', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1979.g2b0101eb'}, 'url': 'https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-11 00:47:00.272739', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-12T03:27:07.902 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref main, sha1 2b0101eb74d60610d544cb8a473e23a35ac1cc0b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:27:07.903 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm 2024-03-12T03:27:07.903 DEBUG:teuthology.orchestra.run.smithi169:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T03:27:09.738 INFO:teuthology.orchestra.run.smithi169.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 12 03:27 /home/ubuntu/cephtest/cephadm 2024-03-12T03:27:09.738 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T03:27:11.087 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 12 03:27 /home/ubuntu/cephtest/cephadm 2024-03-12T03:27:11.088 DEBUG:teuthology.orchestra.run.smithi169:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T03:27:11.107 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T03:27:11.138 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b on all hosts... 2024-03-12T03:27:11.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-12T03:27:11.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-12T03:27:11.940 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:27:12.018 INFO:teuthology.orchestra.run.smithi169.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:27:41.808 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:27:41.808 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-12T03:27:41.808 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-12T03:27:41.808 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-03-12T03:27:41.808 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-12T03:27:41.809 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:27:41.809 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:27:44.478 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:27:44.478 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-12T03:27:44.479 INFO:teuthology.orchestra.run.smithi169.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-12T03:27:44.479 INFO:teuthology.orchestra.run.smithi169.stdout: "repo_digests": [ 2024-03-12T03:27:44.479 INFO:teuthology.orchestra.run.smithi169.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-12T03:27:44.479 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:27:44.479 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:27:44.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph 2024-03-12T03:27:44.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-03-12T03:27:44.572 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /etc/ceph 2024-03-12T03:27:44.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-03-12T03:27:44.642 INFO:tasks.cephadm:Writing seed config... 2024-03-12T03:27:44.643 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-12T03:27:44.643 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-12T03:27:44.643 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-12T03:27:44.643 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-12T03:27:44.644 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-12T03:27:44.645 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-12T03:27:44.645 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-12T03:27:44.645 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-12T03:27:44.645 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-12T03:27:44.645 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-12T03:27:44.645 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-12T03:27:44.645 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-12T03:27:44.646 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:27:44.646 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-12T03:27:44.662 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 677d624a-e020-11ee-95c8-87774f69a715 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-12T03:27:44.663 DEBUG:teuthology.orchestra.run.smithi169:mon.a> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.a.service 2024-03-12T03:27:44.705 DEBUG:teuthology.orchestra.run.smithi169:mgr.y> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@mgr.y.service 2024-03-12T03:27:44.751 INFO:tasks.cephadm:Bootstrapping... 2024-03-12T03:27:44.751 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b -v bootstrap --fsid 677d624a-e020-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.169 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:27:44.943 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:27:44.943 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b', '-v', 'bootstrap', '--fsid', '677d624a-e020-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.169', '--skip-admin-label'] 2024-03-12T03:27:44.968 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T03:27:44.969 INFO:teuthology.orchestra.run.smithi169.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-12T03:27:44.969 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying podman|docker is present... 2024-03-12T03:27:44.992 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T03:27:44.992 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying lvm2 is present... 2024-03-12T03:27:44.992 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying time synchronization is in place... 2024-03-12T03:27:45.000 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T03:27:45.000 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T03:27:45.006 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T03:27:45.006 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout inactive 2024-03-12T03:27:45.012 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout enabled 2024-03-12T03:27:45.018 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout active 2024-03-12T03:27:45.018 INFO:teuthology.orchestra.run.smithi169.stdout:Unit chronyd.service is enabled and running 2024-03-12T03:27:45.018 INFO:teuthology.orchestra.run.smithi169.stdout:Repeating the final host check... 2024-03-12T03:27:45.041 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T03:27:45.042 INFO:teuthology.orchestra.run.smithi169.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-12T03:27:45.042 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl is present 2024-03-12T03:27:45.042 INFO:teuthology.orchestra.run.smithi169.stdout:lvcreate is present 2024-03-12T03:27:45.048 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T03:27:45.048 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T03:27:45.054 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T03:27:45.054 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout inactive 2024-03-12T03:27:45.060 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout enabled 2024-03-12T03:27:45.066 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout active 2024-03-12T03:27:45.066 INFO:teuthology.orchestra.run.smithi169.stdout:Unit chronyd.service is enabled and running 2024-03-12T03:27:45.066 INFO:teuthology.orchestra.run.smithi169.stdout:Host looks OK 2024-03-12T03:27:45.066 INFO:teuthology.orchestra.run.smithi169.stdout:Cluster fsid: 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:27:45.066 INFO:teuthology.orchestra.run.smithi169.stdout:Acquiring lock 139694855147424 on /run/cephadm/677d624a-e020-11ee-95c8-87774f69a715.lock 2024-03-12T03:27:45.066 INFO:teuthology.orchestra.run.smithi169.stdout:Lock 139694855147424 acquired on /run/cephadm/677d624a-e020-11ee-95c8-87774f69a715.lock 2024-03-12T03:27:45.067 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying IP 172.21.15.169 port 3300 ... 2024-03-12T03:27:45.067 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying IP 172.21.15.169 port 6789 ... 2024-03-12T03:27:45.067 INFO:teuthology.orchestra.run.smithi169.stdout:Base mon IP(s) is [172.21.15.169:3300, 172.21.15.169:6789], mon addrv is [v2:172.21.15.169:3300,v1:172.21.15.169:6789] 2024-03-12T03:27:45.070 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.169 metric 100 2024-03-12T03:27:45.070 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.169 metric 100 2024-03-12T03:27:45.073 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-12T03:27:45.073 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-12T03:27:45.073 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1738sec hoplimit 64 pref medium 2024-03-12T03:27:45.075 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-12T03:27:45.075 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-12T03:27:45.075 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T03:27:45.075 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-12T03:27:45.075 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7f4d/64 scope link 2024-03-12T03:27:45.075 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T03:27:45.076 INFO:teuthology.orchestra.run.smithi169.stdout:Mon IP `172.21.15.169` is in CIDR network `172.21.0.0/20` 2024-03-12T03:27:45.076 INFO:teuthology.orchestra.run.smithi169.stdout:Mon IP `172.21.15.169` is in CIDR network `172.21.0.0/20` 2024-03-12T03:27:45.076 INFO:teuthology.orchestra.run.smithi169.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-12T03:27:45.076 INFO:teuthology.orchestra.run.smithi169.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-12T03:27:45.078 INFO:teuthology.orchestra.run.smithi169.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:27:45.566 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stdout 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-12T03:27:45.566 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:27:45.566 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stderr Getting image source signatures 2024-03-12T03:27:45.566 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stderr Copying blob sha256:4392e0adce526b0883d65bbedc13478c05c0e708702f1c7015b93fe77c162cd5 2024-03-12T03:27:45.566 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-12T03:27:45.566 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stderr Copying config sha256:4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-12T03:27:45.566 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-12T03:27:46.530 INFO:teuthology.orchestra.run.smithi169.stdout:ceph: stdout ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-12T03:27:46.530 INFO:teuthology.orchestra.run.smithi169.stdout:Ceph version: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-12T03:27:46.530 INFO:teuthology.orchestra.run.smithi169.stdout:Extracting ceph user uid/gid from container image... 2024-03-12T03:27:47.418 INFO:teuthology.orchestra.run.smithi169.stdout:stat: stdout 167 167 2024-03-12T03:27:47.419 INFO:teuthology.orchestra.run.smithi169.stdout:Creating initial keys... 2024-03-12T03:27:48.292 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph-authtool: stdout AQCzy+9lccoEKxAAUUfgpS6hbt83+8k/goUXLQ== 2024-03-12T03:27:49.106 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph-authtool: stdout AQC0y+9lyqlNJRAAEhrDGBTLYGK+fK5aBE5FyQ== 2024-03-12T03:27:50.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph-authtool: stdout AQC1y+9lcp8sGxAAokg1kOkl8KRe30ZKhgvhbg== 2024-03-12T03:27:50.014 INFO:teuthology.orchestra.run.smithi169.stdout:Creating initial monmap... 2024-03-12T03:27:50.953 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T03:27:50.953 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-12T03:27:50.953 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:27:50.953 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T03:27:50.954 INFO:teuthology.orchestra.run.smithi169.stdout:monmaptool for a [v2:172.21.15.169:3300,v1:172.21.15.169:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T03:27:50.954 INFO:teuthology.orchestra.run.smithi169.stdout:setting min_mon_release = quincy 2024-03-12T03:27:50.954 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/monmaptool: set fsid to 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:27:50.954 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T03:27:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:27:50.954 INFO:teuthology.orchestra.run.smithi169.stdout:Creating mon... 2024-03-12T03:27:51.909 INFO:teuthology.orchestra.run.smithi169.stdout:create mon.a on 2024-03-12T03:27:52.120 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-12T03:27:52.258 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-12T03:27:52.424 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-677d624a-e020-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-677d624a-e020-11ee-95c8-87774f69a715.target. 2024-03-12T03:27:52.425 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-677d624a-e020-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-677d624a-e020-11ee-95c8-87774f69a715.target. 2024-03-12T03:27:52.652 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.a 2024-03-12T03:27:52.652 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Failed to reset failed state of unit ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-12T03:27:52.814 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-677d624a-e020-11ee-95c8-87774f69a715.target.wants/ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-677d624a-e020-11ee-95c8-87774f69a715@.service. 2024-03-12T03:27:53.352 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:27:53.353 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout disabled 2024-03-12T03:27:53.360 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:27:53.360 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout inactive 2024-03-12T03:27:53.360 INFO:teuthology.orchestra.run.smithi169.stdout:firewalld.service is not enabled 2024-03-12T03:27:53.361 INFO:teuthology.orchestra.run.smithi169.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T03:27:53.361 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for mon to start... 2024-03-12T03:27:53.361 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for mon... 2024-03-12T03:27:54.397 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout cluster: 2024-03-12T03:27:54.397 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout id: 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:27:54.397 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-12T03:27:54.397 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-12T03:27:54.397 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout services: 2024-03-12T03:27:54.397 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.531503s) 2024-03-12T03:27:54.397 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-12T03:27:54.397 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-12T03:27:54.397 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-12T03:27:54.397 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout data: 2024-03-12T03:27:54.398 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-12T03:27:54.398 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-12T03:27:54.398 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-12T03:27:54.398 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout pgs: 2024-03-12T03:27:54.398 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-12T03:27:54.398 INFO:teuthology.orchestra.run.smithi169.stdout:mon is available 2024-03-12T03:27:54.398 INFO:teuthology.orchestra.run.smithi169.stdout:Assimilating anything we can from ceph.conf... 2024-03-12T03:27:55.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:55 smithi169 ceph-mon[28964]: from='client.? 172.21.15.169:0/2179347680' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-12T03:27:55.427 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-12T03:27:55.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T03:27:55.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout fsid = 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:27:55.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.169:3300,v1:172.21.15.169:6789] 2024-03-12T03:27:55.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T03:27:55.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T03:27:55.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T03:27:55.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T03:27:55.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-12T03:27:55.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T03:27:55.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T03:27:55.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-12T03:27:55.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T03:27:55.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T03:27:55.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T03:27:55.429 INFO:teuthology.orchestra.run.smithi169.stdout:Generating new minimal ceph.conf... 2024-03-12T03:27:56.425 INFO:teuthology.orchestra.run.smithi169.stdout:Restarting the monitor... 2024-03-12T03:27:56.442 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:56 smithi169 ceph-mon[28964]: from='client.? 172.21.15.169:0/3223989656' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:27:56.693 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:56 smithi169 systemd[1]: Stopping Ceph mon.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:27:57.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:56 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a[28959]: 2024-03-12T03:27:56.690+0000 7f4a521cf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:27:57.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:56 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a[28959]: 2024-03-12T03:27:56.691+0000 7f4a521cf700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-12T03:27:57.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:56 smithi169 podman[29218]: 2024-03-12 03:27:56.885694523 +0000 UTC m=+0.303564798 container died 81c86191ed290684aef3fe93fbc6bd490db30290bd7719842807c91ff1f0f5ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:27:57.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:57 smithi169 podman[29218]: 2024-03-12 03:27:57.241105153 +0000 UTC m=+0.658975418 container cleanup 81c86191ed290684aef3fe93fbc6bd490db30290bd7719842807c91ff1f0f5ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:27:57.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:57 smithi169 bash[29218]: ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a 2024-03-12T03:27:57.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:57 smithi169 podman[29231]: 2024-03-12 03:27:57.508701271 +0000 UTC m=+0.622656748 container remove 81c86191ed290684aef3fe93fbc6bd490db30290bd7719842807c91ff1f0f5ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-12T03:27:57.854 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:57 smithi169 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-12T03:27:57.854 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:57 smithi169 systemd[1]: Stopped Ceph mon.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:27:57.854 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:57 smithi169 systemd[1]: Starting Ceph mon.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:27:58.101 INFO:teuthology.orchestra.run.smithi169.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-12T03:27:58.134 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:57 smithi169 podman[29325]: 2024-03-12 03:27:57.851425593 +0000 UTC m=+0.097516149 container create 9b290f8cbe46fb22732f566e597407cf2b564a7696de432e93df77444f7d35e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-12T03:27:58.134 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:57 smithi169 podman[29325]: 2024-03-12 03:27:57.77220988 +0000 UTC m=+0.018300399 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 podman[29325]: 2024-03-12 03:27:58.028626841 +0000 UTC m=+0.274717364 container init 9b290f8cbe46fb22732f566e597407cf2b564a7696de432e93df77444f7d35e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 podman[29325]: 2024-03-12 03:27:58.031419502 +0000 UTC m=+0.277510016 container start 9b290f8cbe46fb22732f566e597407cf2b564a7696de432e93df77444f7d35e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, ceph=True) 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 2 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: pidfile_write: ignore empty --pid-file 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: load: jerasure load: lrc 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: RocksDB version: 7.9.2 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Git sha 0 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: DB SUMMARY 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: DB Session ID: CFNU2WDN317A1HFO0VZR 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:27:58.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:27:58.136 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-12T03:27:58.136 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-12T03:27:58.136 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 91727 ; 2024-03-12T03:27:58.136 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:27:58.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:27:58.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:27:58.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T03:27:58.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:27:58.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T03:27:58.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.env: 0x5624c6ca4b20 2024-03-12T03:27:58.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T03:27:58.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.info_log: 0x5624c77698e0 2024-03-12T03:27:58.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:27:58.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.statistics: (nil) 2024-03-12T03:27:58.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.use_fsync: 0 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.db_log_dir: 2024-03-12T03:27:58.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.wal_dir: 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.write_buffer_manager: 0x5624c773d400 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:27:58.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.unordered_write: 0 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.row_cache: None 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.wal_filter: None 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.wal_compression: 0 2024-03-12T03:27:58.140 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:27:58.141 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_open_files: -1 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:27:58.142 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Compression algorithms supported: 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: kZSTD supported: 0 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T03:27:58.143 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.merge_operator: 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_filter: None 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5624c77699e0) 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout: index_type: 0 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout: data_block_index_type: 0 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout: index_shortening: 1 2024-03-12T03:27:58.144 INFO:journalctl@ceph.mon.a.smithi169.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: checksum: 4 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: no_block_cache: 0 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_cache: 0x5624c776f090 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_cache_options: 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: capacity : 536870912 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: num_shard_bits : 4 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: strict_capacity_limit : 0 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_cache_compressed: (nil) 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: persistent_cache: (nil) 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_size: 4096 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_size_deviation: 10 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_restart_interval: 16 2024-03-12T03:27:58.145 INFO:journalctl@ceph.mon.a.smithi169.stdout: index_block_restart_interval: 1 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: metadata_block_size: 4096 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: partition_filters: 0 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: use_delta_encoding: 1 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: filter_policy: bloomfilter 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: whole_key_filtering: 1 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: verify_compression: 0 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: format_version: 5 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: enable_index_compression: 1 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_align: 0 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: max_auto_readahead_size: 262144 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: prepopulate_block_cache: 0 2024-03-12T03:27:58.146 INFO:journalctl@ceph.mon.a.smithi169.stdout: initial_auto_readahead_size: 8192 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compression: NoCompression 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.num_levels: 7 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:27:58.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:27:58.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:27:58.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:27:58.150 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:27:58.151 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:27:58.151 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:27:58.151 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:27:58.151 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:27:58.151 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:27:58.151 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:27:58.152 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:27:58.152 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:27:58.152 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:27:58.152 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:27:58.152 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:27:58.152 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:27:58.152 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.ttl: 2592000 2024-03-12T03:27:58.152 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:27:58.152 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T03:27:58.152 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T03:27:58.152 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: edd1069a-56be-42bb-b7b1-4841f9a39952 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214078061767, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-12T03:27:58.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214078062721, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 87600, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 282, "table_properties": {"data_size": 85688, "index_size": 258, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11802, "raw_average_key_size": 48, "raw_value_size": 79223, "raw_average_value_size": 328, "num_data_blocks": 11, "num_entries": 241, "num_filter_entries": 241, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710214078, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "edd1069a-56be-42bb-b7b1-4841f9a39952", "db_session_id": "CFNU2WDN317A1HFO0VZR", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214078062805, "job": 1, "event": "recovery_finished"} 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5624c7862000 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: DB pointer 0x5624c784c000 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout: ** DB Stats ** 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:27:58.154 INFO:journalctl@ceph.mon.a.smithi169.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: ** Compaction Stats [default] ** 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: L0 2/0 87.38 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: Sum 2/0 87.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: ** Compaction Stats [default] ** 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 134.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:27:58.155 INFO:journalctl@ceph.mon.a.smithi169.stdout: 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: Cumulative compaction: 0.00 GB write, 28.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: Interval compaction: 0.00 GB write, 28.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: Block cache BinnedLRUCache@0x5624c776f090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-12T03:27:58.156 INFO:journalctl@ceph.mon.a.smithi169.stdout: 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: starting mon.a rank 0 at public addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] at bind addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: mon.a@-1(???) e1 preinit fsid 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: mon.a@-1(???).mds e1 new map 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: mon.a@-1(???).mds e1 print_map 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout: e1 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout: legacy client fscid: -1 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout: 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout: No filesystems configured 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:27:58.157 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:27:58.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-12T03:27:58.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 bash[29325]: 9b290f8cbe46fb22732f566e597407cf2b564a7696de432e93df77444f7d35e3 2024-03-12T03:27:58.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 systemd[1]: Started Ceph mon.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:27:58.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-12T03:27:58.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: monmap e1: 1 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:27:58.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: fsmap 2024-03-12T03:27:58.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: osdmap e1: 0 total, 0 up, 0 in 2024-03-12T03:27:58.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:58 smithi169 ceph-mon[29344]: mgrmap e1: no daemons active 2024-03-12T03:27:59.182 INFO:teuthology.orchestra.run.smithi169.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-12T03:27:59.204 INFO:teuthology.orchestra.run.smithi169.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:27:59.204 INFO:teuthology.orchestra.run.smithi169.stdout:Creating mgr... 2024-03-12T03:27:59.204 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-12T03:27:59.205 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-12T03:27:59.469 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-677d624a-e020-11ee-95c8-87774f69a715@mgr.y 2024-03-12T03:27:59.469 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Failed to reset failed state of unit ceph-677d624a-e020-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-677d624a-e020-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-12T03:27:59.615 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-677d624a-e020-11ee-95c8-87774f69a715.target.wants/ceph-677d624a-e020-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-677d624a-e020-11ee-95c8-87774f69a715@.service. 2024-03-12T03:27:59.832 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:27:59 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3786839091' entity='client.admin' 2024-03-12T03:28:00.109 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:00 smithi169 podman[29554]: 2024-03-12 03:28:00.03473078 +0000 UTC m=+0.302293892 container start eadae3e4956938f1e170b658b706e3429746291351b73442fd10e772f2323e26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-12T03:28:00.109 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:00 smithi169 bash[29554]: eadae3e4956938f1e170b658b706e3429746291351b73442fd10e772f2323e26 2024-03-12T03:28:00.122 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:28:00.122 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout disabled 2024-03-12T03:28:00.130 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:28:00.130 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout inactive 2024-03-12T03:28:00.130 INFO:teuthology.orchestra.run.smithi169.stdout:firewalld.service is not enabled 2024-03-12T03:28:00.130 INFO:teuthology.orchestra.run.smithi169.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T03:28:00.138 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:28:00.138 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout disabled 2024-03-12T03:28:00.146 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:28:00.146 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout inactive 2024-03-12T03:28:00.146 INFO:teuthology.orchestra.run.smithi169.stdout:firewalld.service is not enabled 2024-03-12T03:28:00.146 INFO:teuthology.orchestra.run.smithi169.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-12T03:28:00.146 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for mgr to start... 2024-03-12T03:28:00.146 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for mgr... 2024-03-12T03:28:00.368 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:00 smithi169 systemd[1]: Started Ceph mgr.y for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:28:00.650 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:00 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:00.590+0000 7fc90da15200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:28:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:00 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1933930021' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:28:00.925 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:00 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:00.690+0000 7fc90da15200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:28:01.158 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-12T03:28:01.158 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:28:01.158 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsid": "677d624a-e020-11ee-95c8-87774f69a715", 2024-03-12T03:28:01.158 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:28:01.158 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:28:01.158 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T03:28:01.159 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:28:01.160 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:01.160 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:28:01.160 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:28:01.160 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:28:01.160 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:28:01.160 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:28:01.160 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:28:01.160 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:28:01.160 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:28:01.160 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:28:01.161 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:28:01.162 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:27:53.320487+0000", 2024-03-12T03:28:01.163 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:28:01.163 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:01.163 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:28:01.163 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:28:01.163 INFO:teuthology.orchestra.run.smithi169.stdout:mgr not available, waiting (1/15)... 2024-03-12T03:28:01.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:01.345+0000 7fc90da15200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:28:01.757 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:01.437+0000 7fc90da15200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:28:01.757 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:01.578+0000 7fc90da15200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:28:01.757 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:01.672+0000 7fc90da15200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:28:01.757 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:01.754+0000 7fc90da15200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:28:02.305 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:02 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:02.016+0000 7fc90da15200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:28:02.305 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:02 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:02.124+0000 7fc90da15200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:28:02.305 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:02 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:02.209+0000 7fc90da15200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:28:02.305 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:02 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:02.302+0000 7fc90da15200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:28:02.673 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:02 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:02.479+0000 7fc90da15200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:28:03.601 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:03 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:03.462+0000 7fc90da15200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:28:03.601 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:03 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:03.598+0000 7fc90da15200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:28:03.877 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:03 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/694346727' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:28:04.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:03 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:03.944+0000 7fc90da15200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:28:04.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:04 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:04.034+0000 7fc90da15200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:28:04.489 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:04 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:04.226+0000 7fc90da15200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:28:04.489 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:04 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:04.410+0000 7fc90da15200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:28:04.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:04 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:04.700+0000 7fc90da15200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:28:05.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:04 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:04.969+0000 7fc90da15200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:28:05.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:05 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:05.204+0000 7fc90da15200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:28:06.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:06.015+0000 7fc90da15200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:28:06.909 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:06.594+0000 7fc90da15200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:28:06.909 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:06.691+0000 7fc90da15200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:28:06.909 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:06.907+0000 7fc90da15200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:28:07.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: Activating manager daemon y 2024-03-12T03:28:07.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: mgrmap e2: y(active, starting, since 0.00310972s) 2024-03-12T03:28:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: from='mgr.14100 172.21.15.169:0/3897252843' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:28:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: from='mgr.14100 172.21.15.169:0/3897252843' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:28:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: from='mgr.14100 172.21.15.169:0/3897252843' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:28:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: from='mgr.14100 172.21.15.169:0/3897252843' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:28:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: from='mgr.14100 172.21.15.169:0/3897252843' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:28:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: Manager daemon y is now available 2024-03-12T03:28:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: from='mgr.14100 172.21.15.169:0/3897252843' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:28:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: from='mgr.14100 172.21.15.169:0/3897252843' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:28:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: from='mgr.14100 172.21.15.169:0/3897252843' entity='mgr.y' 2024-03-12T03:28:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: from='mgr.14100 172.21.15.169:0/3897252843' entity='mgr.y' 2024-03-12T03:28:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:06 smithi169 ceph-mon[29344]: from='mgr.14100 172.21.15.169:0/3897252843' entity='mgr.y' 2024-03-12T03:28:09.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:08 smithi169 ceph-mon[29344]: mgrmap e3: y(active, since 1.00736s) 2024-03-12T03:28:09.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-12T03:28:09.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:28:09.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsid": "677d624a-e020-11ee-95c8-87774f69a715", 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:28:09.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:09.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:09.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:27:53.320487+0000", 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:28:09.658 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:28:09.658 INFO:teuthology.orchestra.run.smithi169.stdout:mgr not available, waiting (2/15)... 2024-03-12T03:28:11.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:10 smithi169 ceph-mon[29344]: mgrmap e4: y(active, since 3s) 2024-03-12T03:28:12.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:12 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2141366972' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:28:13.011 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-12T03:28:13.011 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:28:13.011 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsid": "677d624a-e020-11ee-95c8-87774f69a715", 2024-03-12T03:28:13.011 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:28:13.011 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:28:13.011 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:28:13.011 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:28:13.011 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:13.011 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:28:13.011 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:28:13.011 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:28:13.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:28:13.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:28:13.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:28:13.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:28:13.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:13.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:28:13.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:28:13.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:28:13.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:28:13.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:28:13.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:28:13.018 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:28:13.018 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:28:13.018 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:28:13.018 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:28:13.018 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:13.018 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:28:13.018 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:28:13.018 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:27:53.320487+0000", 2024-03-12T03:28:13.018 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:28:13.018 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:28:13.019 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:28:13.019 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:28:13.019 INFO:teuthology.orchestra.run.smithi169.stdout:mgr is available 2024-03-12T03:28:13.944 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:13 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/189040727' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T03:28:14.207 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout fsid = 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T03:28:14.208 INFO:teuthology.orchestra.run.smithi169.stdout:Enabling cephadm module... 2024-03-12T03:28:15.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:14 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3577854065' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-12T03:28:15.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:14 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ignoring --setuser ceph since I am not root 2024-03-12T03:28:15.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:14 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ignoring --setgroup ceph since I am not root 2024-03-12T03:28:15.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:15 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:15.519+0000 7f9469ab5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:28:15.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:15 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:15.615+0000 7f9469ab5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:28:16.258 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:15 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3577854065' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-12T03:28:16.259 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:15 smithi169 ceph-mon[29344]: mgrmap e5: y(active, since 8s) 2024-03-12T03:28:16.588 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:16.256+0000 7f9469ab5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:28:16.588 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:16.347+0000 7f9469ab5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:28:16.588 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:16.490+0000 7f9469ab5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:28:16.588 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:16.586+0000 7f9469ab5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:28:16.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:16.668+0000 7f9469ab5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:28:17.225 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:16.933+0000 7f9469ab5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:28:17.225 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:17 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:17.042+0000 7f9469ab5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:28:17.226 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:17 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:17.128+0000 7f9469ab5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:28:17.226 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:17 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:17.223+0000 7f9469ab5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:28:17.673 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:17 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:17.402+0000 7f9469ab5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:28:18.673 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:18 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:18.379+0000 7f9469ab5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:28:18.673 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:18 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:18.513+0000 7f9469ab5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:28:19.152 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:18 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:18.859+0000 7f9469ab5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:28:19.153 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:18 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:18.953+0000 7f9469ab5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:28:19.153 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:19 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:19.150+0000 7f9469ab5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:28:19.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:19 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:19.338+0000 7f9469ab5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:28:19.821 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:19 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:19.633+0000 7f9469ab5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:28:20.131 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:19 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/202915596' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T03:28:20.131 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:19 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:19.907+0000 7f9469ab5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:28:20.399 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:20 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:20.146+0000 7f9469ab5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:28:20.402 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:28:20.402 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-12T03:28:20.402 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T03:28:20.402 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-12T03:28:20.402 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T03:28:20.402 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:28:20.402 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for the mgr to restart... 2024-03-12T03:28:20.402 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for mgr epoch 5... 2024-03-12T03:28:21.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:20 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:20.971+0000 7f9469ab5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:28:21.857 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:21.541+0000 7f9469ab5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:28:21.857 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:21.639+0000 7f9469ab5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:28:21.857 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:21.854+0000 7f9469ab5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:28:22.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: Active manager daemon y restarted 2024-03-12T03:28:22.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: Activating manager daemon y 2024-03-12T03:28:22.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: osdmap e2: 0 total, 0 up, 0 in 2024-03-12T03:28:22.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: mgrmap e6: y(active, starting, since 0.00344337s) 2024-03-12T03:28:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:28:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:28:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:28:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:28:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:28:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: Manager daemon y is now available 2024-03-12T03:28:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:28:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:21 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:28:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:22 smithi169 ceph-mon[29344]: Found migration_current of "None". Setting to last migration. 2024-03-12T03:28:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:22 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:28:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:22 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:28:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:22 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:28:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:22 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:22 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:23.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:22 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:28:23.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:22 smithi169 ceph-mon[29344]: mgrmap e7: y(active, since 1.00557s) 2024-03-12T03:28:23.492 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:28:23.492 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-12T03:28:23.492 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T03:28:23.492 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:28:23.492 INFO:teuthology.orchestra.run.smithi169.stdout:mgr epoch 5 is available 2024-03-12T03:28:23.492 INFO:teuthology.orchestra.run.smithi169.stdout:Setting orchestrator backend to cephadm... 2024-03-12T03:28:24.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:23 smithi169 ceph-mon[29344]: [12/Mar/2024:03:28:22] ENGINE Bus STARTING 2024-03-12T03:28:24.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:23 smithi169 ceph-mon[29344]: [12/Mar/2024:03:28:22] ENGINE Serving on https://172.21.15.169:7150 2024-03-12T03:28:24.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:23 smithi169 ceph-mon[29344]: [12/Mar/2024:03:28:22] ENGINE Serving on http://172.21.15.169:8765 2024-03-12T03:28:24.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:23 smithi169 ceph-mon[29344]: [12/Mar/2024:03:28:22] ENGINE Bus STARTED 2024-03-12T03:28:24.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:23 smithi169 ceph-mon[29344]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T03:28:24.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:23 smithi169 ceph-mon[29344]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T03:28:25.031 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:24 smithi169 ceph-mon[29344]: mgrmap e8: y(active, since 2s) 2024-03-12T03:28:25.032 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:24 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:25.032 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:24 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:28:25.889 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-12T03:28:25.889 INFO:teuthology.orchestra.run.smithi169.stdout:Generating ssh key... 2024-03-12T03:28:26.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:25 smithi169 ceph-mon[29344]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:28:26.884 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: Generating public/private rsa key pair. 2024-03-12T03:28:26.884 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: Your identification has been saved in /tmp/tmpqcxlpkbn/key. 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: Your public key has been saved in /tmp/tmpqcxlpkbn/key.pub. 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: The key fingerprint is: 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: SHA256:O45K4fmhnkfcBW9ZgvK4/SQTcKRfs2l8JEj+VX/qiJY ceph-677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: The key's randomart image is: 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: +---[RSA 3072]----+ 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: | .o. . | 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: | o++.. .. . | 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: | .*oo++o o| 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: | ..o+=B ..| 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: | .. +S+* . . | 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: | . o+ =o.+ o | 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: | +.. o=E . . | 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: | . +.+ o. | 2024-03-12T03:28:26.885 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: | .=oo . | 2024-03-12T03:28:26.886 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: +----[SHA256]-----+ 2024-03-12T03:28:27.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-mon[29344]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:28:27.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:27.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:26 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:28.213 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:27 smithi169 ceph-mon[29344]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:28:28.213 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:27 smithi169 ceph-mon[29344]: Generating ssh key... 2024-03-12T03:28:28.302 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdWyGw+C22cid4EfNVbJMcKPmhwPmngtxaVxnUXFJuWLgusZYWDySJ3096VeTA8S7Gs599KMkwymfFWVqdxiqAgq6z82SIjTL3NfOV0sPU2YTfDxONLwFYdVFk4RoU9zrHxeGfnOQrLYxxaaApnTbuEqNbdK6Wckn3XsmiQJncEUMIEnLWHM33nvvHSqv5+NBkYWF+xvvz88FQZDyDz6/68YId2jfcQCpd4JbOHFOUSS6SE2hN0S4t/bpnY2oiLP6Rbh24mYeozdfnYTWyZJd3/3ComGCQueZWhZGKCMcS/OAxLlQw0xyalAY6CXOz1vSfdf9jYMx2B7pa9QCjdZrmJ1paFYJfzAmRgdfnkqOlBe8z7k7FXM5ETDKSmDWcUpUxhtMKUEgn7mRzQFLZYIcQFNtTrnmN/kn5CrdPRmSeCaxPS9/oauQjJvMTiqVjAf2lVVVYyFjXWZ9ABkSWwAkYOK3UopznLdm3ouzRnoSA3jPXbSbHCR8AsYdBXLhPCPE= ceph-677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:28:28.302 INFO:teuthology.orchestra.run.smithi169.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-12T03:28:28.302 INFO:teuthology.orchestra.run.smithi169.stdout:Adding key to root@localhost authorized_keys... 2024-03-12T03:28:28.303 INFO:teuthology.orchestra.run.smithi169.stdout:Adding host smithi169... 2024-03-12T03:28:29.272 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:28 smithi169 ceph-mon[29344]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:28:30.078 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:29 smithi169 ceph-mon[29344]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "addr": "172.21.15.169", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:28:31.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:30 smithi169 ceph-mon[29344]: Deploying cephadm binary to smithi169 2024-03-12T03:28:33.903 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout Added host 'smithi169' with addr '172.21.15.169' 2024-03-12T03:28:33.903 INFO:teuthology.orchestra.run.smithi169.stdout:Deploying unmanaged mon service... 2024-03-12T03:28:34.541 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:34 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:34.542 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:34 smithi169 ceph-mon[29344]: Added host smithi169 2024-03-12T03:28:34.542 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:34 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:28:35.328 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-12T03:28:35.328 INFO:teuthology.orchestra.run.smithi169.stdout:Deploying unmanaged mgr service... 2024-03-12T03:28:35.540 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:35 smithi169 ceph-mon[29344]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:28:35.540 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:35 smithi169 ceph-mon[29344]: Saving service mon spec with placement count:5 2024-03-12T03:28:35.541 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:35 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:37.082 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-12T03:28:37.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:37 smithi169 ceph-mon[29344]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:28:37.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:37 smithi169 ceph-mon[29344]: Saving service mgr spec with placement count:2 2024-03-12T03:28:37.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:37 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:37.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:37 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:37.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:37 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:38.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:38 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3998570507' entity='client.admin' 2024-03-12T03:28:38.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:38 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:40.150 INFO:teuthology.orchestra.run.smithi169.stdout:Enabling the dashboard module... 2024-03-12T03:28:40.780 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:40 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/157760873' entity='client.admin' 2024-03-12T03:28:40.780 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:40 smithi169 ceph-mon[29344]: from='mgr.14118 172.21.15.169:0/1975474273' entity='mgr.y' 2024-03-12T03:28:41.876 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:41 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1010787607' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-12T03:28:41.877 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ignoring --setuser ceph since I am not root 2024-03-12T03:28:41.877 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ignoring --setgroup ceph since I am not root 2024-03-12T03:28:42.169 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:42 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:42.123+0000 7f0014009200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:28:42.422 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:42 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:42.221+0000 7f0014009200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:28:42.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:42 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1010787607' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-12T03:28:42.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:42 smithi169 ceph-mon[29344]: mgrmap e9: y(active, since 19s) 2024-03-12T03:28:43.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:42 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:42.877+0000 7f0014009200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:28:43.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:42 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:42.975+0000 7f0014009200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:28:43.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:43 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:43.117+0000 7f0014009200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:28:43.486 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:43 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:43.213+0000 7f0014009200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:28:43.487 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:43 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:43.296+0000 7f0014009200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:28:43.491 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:28:43.491 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-12T03:28:43.491 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T03:28:43.491 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-12T03:28:43.491 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T03:28:43.491 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:28:43.491 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for the mgr to restart... 2024-03-12T03:28:43.491 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for mgr epoch 9... 2024-03-12T03:28:43.757 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:43 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1682536247' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T03:28:43.757 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:43 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:43.573+0000 7f0014009200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:28:43.757 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:43 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:43.683+0000 7f0014009200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:28:44.050 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:43 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:43.772+0000 7f0014009200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:28:44.051 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:43 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:43.866+0000 7f0014009200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:28:44.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:44 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:44.048+0000 7f0014009200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:28:45.045 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:45.015+0000 7f0014009200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:28:45.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:45.154+0000 7f0014009200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:28:45.815 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:45.517+0000 7f0014009200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:28:45.815 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:45.611+0000 7f0014009200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:28:45.815 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:45.812+0000 7f0014009200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:28:46.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:46 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:46.003+0000 7f0014009200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:28:46.555 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:46 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:46.291+0000 7f0014009200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:28:46.556 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:46 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:46.553+0000 7f0014009200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:28:46.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:46 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:46.791+0000 7f0014009200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:28:47.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:47 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:47.608+0000 7f0014009200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:28:48.507 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:48.183+0000 7f0014009200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:28:48.507 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:48.283+0000 7f0014009200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:28:48.507 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:28:48.505+0000 7f0014009200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:28:48.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-mon[29344]: Active manager daemon y restarted 2024-03-12T03:28:48.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-mon[29344]: Activating manager daemon y 2024-03-12T03:28:48.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-mon[29344]: osdmap e3: 0 total, 0 up, 0 in 2024-03-12T03:28:48.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-mon[29344]: mgrmap e10: y(active, starting, since 0.00356875s) 2024-03-12T03:28:48.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:28:48.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:28:48.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:28:48.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:28:48.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:28:48.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-mon[29344]: Manager daemon y is now available 2024-03-12T03:28:48.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:28:48.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:48 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:28:49.735 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:49 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:28:49.735 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:49 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:28:49.735 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:49 smithi169 ceph-mon[29344]: mgrmap e11: y(active, since 1.00671s) 2024-03-12T03:28:50.441 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:28:50.441 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-12T03:28:50.441 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T03:28:50.442 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:28:50.442 INFO:teuthology.orchestra.run.smithi169.stdout:mgr epoch 9 is available 2024-03-12T03:28:50.442 INFO:teuthology.orchestra.run.smithi169.stdout:Generating a dashboard self-signed certificate... 2024-03-12T03:28:50.907 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:50 smithi169 ceph-mon[29344]: [12/Mar/2024:03:28:49] ENGINE Bus STARTING 2024-03-12T03:28:50.908 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:50 smithi169 ceph-mon[29344]: [12/Mar/2024:03:28:49] ENGINE Serving on https://172.21.15.169:7150 2024-03-12T03:28:50.908 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:50 smithi169 ceph-mon[29344]: [12/Mar/2024:03:28:49] ENGINE Serving on http://172.21.15.169:8765 2024-03-12T03:28:50.908 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:50 smithi169 ceph-mon[29344]: [12/Mar/2024:03:28:49] ENGINE Bus STARTED 2024-03-12T03:28:50.908 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:50 smithi169 ceph-mon[29344]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T03:28:50.908 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:50 smithi169 ceph-mon[29344]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T03:28:50.908 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:50 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:28:51.947 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-12T03:28:51.947 INFO:teuthology.orchestra.run.smithi169.stdout:Creating initial admin user... 2024-03-12T03:28:52.322 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:52 smithi169 ceph-mon[29344]: mgrmap e12: y(active, since 2s) 2024-03-12T03:28:52.322 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:52 smithi169 ceph-mon[29344]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:28:52.322 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:52 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:28:52.323 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:52 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:28:52.323 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:52 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:28:53.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:53 smithi169 ceph-mon[29344]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:28:53.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:53 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:28:54.221 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$cfDoWOuf3LmJES393MdW3./.7iEi6pH/jPaMxCpjp655MRd8a98k2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710214132, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-12T03:28:54.221 INFO:teuthology.orchestra.run.smithi169.stdout:Fetching dashboard port number... 2024-03-12T03:28:55.367 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 8443 2024-03-12T03:28:55.377 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:28:55.377 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout disabled 2024-03-12T03:28:55.383 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:28:55.384 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout inactive 2024-03-12T03:28:55.384 INFO:teuthology.orchestra.run.smithi169.stdout:firewalld.service is not enabled 2024-03-12T03:28:55.384 INFO:teuthology.orchestra.run.smithi169.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-12T03:28:55.385 INFO:teuthology.orchestra.run.smithi169.stdout:Ceph Dashboard is now available at: 2024-03-12T03:28:55.385 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:28:55.385 INFO:teuthology.orchestra.run.smithi169.stdout: URL: https://smithi169.front.sepia.ceph.com:8443/ 2024-03-12T03:28:55.385 INFO:teuthology.orchestra.run.smithi169.stdout: User: admin 2024-03-12T03:28:55.385 INFO:teuthology.orchestra.run.smithi169.stdout: Password: i9gr8pl0g3 2024-03-12T03:28:55.385 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:28:55.386 INFO:teuthology.orchestra.run.smithi169.stdout:Saving cluster configuration to /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config directory 2024-03-12T03:28:55.386 INFO:teuthology.orchestra.run.smithi169.stdout:Enabling autotune for osd_memory_target 2024-03-12T03:28:55.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:55 smithi169 ceph-mon[29344]: mgrmap e13: y(active, since 6s) 2024-03-12T03:28:55.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:55 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1359449903' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-12T03:28:57.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-12T03:28:57.838 INFO:teuthology.orchestra.run.smithi169.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-12T03:28:57.838 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout:Or, if you are only running a single cluster on this host: 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: ceph telemetry on 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout:For more information see: 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:28:57.839 INFO:teuthology.orchestra.run.smithi169.stdout:Bootstrap complete. 2024-03-12T03:28:57.881 INFO:tasks.cephadm:Fetching config... 2024-03-12T03:28:57.881 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:28:57.881 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-12T03:28:57.897 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-12T03:28:57.897 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:28:57.898 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-12T03:28:57.953 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-12T03:28:57.953 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:28:57.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-12T03:28:58.020 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-12T03:28:58.020 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:28:58.020 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-12T03:28:58.074 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-12T03:28:58.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdWyGw+C22cid4EfNVbJMcKPmhwPmngtxaVxnUXFJuWLgusZYWDySJ3096VeTA8S7Gs599KMkwymfFWVqdxiqAgq6z82SIjTL3NfOV0sPU2YTfDxONLwFYdVFk4RoU9zrHxeGfnOQrLYxxaaApnTbuEqNbdK6Wckn3XsmiQJncEUMIEnLWHM33nvvHSqv5+NBkYWF+xvvz88FQZDyDz6/68YId2jfcQCpd4JbOHFOUSS6SE2hN0S4t/bpnY2oiLP6Rbh24mYeozdfnYTWyZJd3/3ComGCQueZWhZGKCMcS/OAxLlQw0xyalAY6CXOz1vSfdf9jYMx2B7pa9QCjdZrmJ1paFYJfzAmRgdfnkqOlBe8z7k7FXM5ETDKSmDWcUpUxhtMKUEgn7mRzQFLZYIcQFNtTrnmN/kn5CrdPRmSeCaxPS9/oauQjJvMTiqVjAf2lVVVYyFjXWZ9ABkSWwAkYOK3UopznLdm3ouzRnoSA3jPXbSbHCR8AsYdBXLhPCPE= ceph-677d624a-e020-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T03:28:58.153 INFO:teuthology.orchestra.run.smithi169.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdWyGw+C22cid4EfNVbJMcKPmhwPmngtxaVxnUXFJuWLgusZYWDySJ3096VeTA8S7Gs599KMkwymfFWVqdxiqAgq6z82SIjTL3NfOV0sPU2YTfDxONLwFYdVFk4RoU9zrHxeGfnOQrLYxxaaApnTbuEqNbdK6Wckn3XsmiQJncEUMIEnLWHM33nvvHSqv5+NBkYWF+xvvz88FQZDyDz6/68YId2jfcQCpd4JbOHFOUSS6SE2hN0S4t/bpnY2oiLP6Rbh24mYeozdfnYTWyZJd3/3ComGCQueZWhZGKCMcS/OAxLlQw0xyalAY6CXOz1vSfdf9jYMx2B7pa9QCjdZrmJ1paFYJfzAmRgdfnkqOlBe8z7k7FXM5ETDKSmDWcUpUxhtMKUEgn7mRzQFLZYIcQFNtTrnmN/kn5CrdPRmSeCaxPS9/oauQjJvMTiqVjAf2lVVVYyFjXWZ9ABkSWwAkYOK3UopznLdm3ouzRnoSA3jPXbSbHCR8AsYdBXLhPCPE= ceph-677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:28:58.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdWyGw+C22cid4EfNVbJMcKPmhwPmngtxaVxnUXFJuWLgusZYWDySJ3096VeTA8S7Gs599KMkwymfFWVqdxiqAgq6z82SIjTL3NfOV0sPU2YTfDxONLwFYdVFk4RoU9zrHxeGfnOQrLYxxaaApnTbuEqNbdK6Wckn3XsmiQJncEUMIEnLWHM33nvvHSqv5+NBkYWF+xvvz88FQZDyDz6/68YId2jfcQCpd4JbOHFOUSS6SE2hN0S4t/bpnY2oiLP6Rbh24mYeozdfnYTWyZJd3/3ComGCQueZWhZGKCMcS/OAxLlQw0xyalAY6CXOz1vSfdf9jYMx2B7pa9QCjdZrmJ1paFYJfzAmRgdfnkqOlBe8z7k7FXM5ETDKSmDWcUpUxhtMKUEgn7mRzQFLZYIcQFNtTrnmN/kn5CrdPRmSeCaxPS9/oauQjJvMTiqVjAf2lVVVYyFjXWZ9ABkSWwAkYOK3UopznLdm3ouzRnoSA3jPXbSbHCR8AsYdBXLhPCPE= ceph-677d624a-e020-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T03:28:58.214 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdWyGw+C22cid4EfNVbJMcKPmhwPmngtxaVxnUXFJuWLgusZYWDySJ3096VeTA8S7Gs599KMkwymfFWVqdxiqAgq6z82SIjTL3NfOV0sPU2YTfDxONLwFYdVFk4RoU9zrHxeGfnOQrLYxxaaApnTbuEqNbdK6Wckn3XsmiQJncEUMIEnLWHM33nvvHSqv5+NBkYWF+xvvz88FQZDyDz6/68YId2jfcQCpd4JbOHFOUSS6SE2hN0S4t/bpnY2oiLP6Rbh24mYeozdfnYTWyZJd3/3ComGCQueZWhZGKCMcS/OAxLlQw0xyalAY6CXOz1vSfdf9jYMx2B7pa9QCjdZrmJ1paFYJfzAmRgdfnkqOlBe8z7k7FXM5ETDKSmDWcUpUxhtMKUEgn7mRzQFLZYIcQFNtTrnmN/kn5CrdPRmSeCaxPS9/oauQjJvMTiqVjAf2lVVVYyFjXWZ9ABkSWwAkYOK3UopznLdm3ouzRnoSA3jPXbSbHCR8AsYdBXLhPCPE= ceph-677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:28:58.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-12T03:28:58.323 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:28:58 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4164992484' entity='client.admin' 2024-03-12T03:28:58.469 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:29:00.931 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:00 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:00.932 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:00 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:00.932 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:00 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:29:00.932 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:00 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:00.932 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:00 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:00.932 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:00 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:01.819 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-12T03:29:01.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-12T03:29:02.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:01 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3972085253' entity='client.admin' 2024-03-12T03:29:02.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:01 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:02.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:01 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:02.522 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:29:05.119 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-03-12T03:29:05.119 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:29:05.119 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-03-12T03:29:05.137 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:29:05.137 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:29:05.194 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-03-12T03:29:05.194 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch host add smithi173 2024-03-12T03:29:05.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:05 smithi169 ceph-mon[29344]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:29:05.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:05.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:05.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:05 smithi169 ceph-mon[29344]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-12T03:29:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:05 smithi169 ceph-mon[29344]: Updating smithi169:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:29:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:05.431 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:29:06.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:06 smithi169 ceph-mon[29344]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:29:06.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:06 smithi169 ceph-mon[29344]: Updating smithi169:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:29:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:07 smithi169 ceph-mon[29344]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:29:09.847 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:09 smithi169 ceph-mon[29344]: Deploying cephadm binary to smithi173 2024-03-12T03:29:09.848 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:09 smithi169 ceph-mon[29344]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:11.294 INFO:teuthology.orchestra.run.smithi169.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-03-12T03:29:11.627 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:11 smithi169 ceph-mon[29344]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:11.627 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:11.627 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:11.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-12T03:29:12.166 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:29:12.581 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:12 smithi169 ceph-mon[29344]: Added host smithi173 2024-03-12T03:29:12.581 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:12 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:12.581 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:12 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:13.621 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:13 smithi169 ceph-mon[29344]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:13.857 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:29:13.857 INFO:teuthology.orchestra.run.smithi169.stdout:[{"addr": "172.21.15.169", "hostname": "smithi169", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-03-12T03:29:14.359 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-12T03:29:14.359 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-12T03:29:14.607 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:29:15.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:14 smithi169 ceph-mon[29344]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:29:15.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:14 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:15.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:14 smithi169 ceph-mon[29344]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:16.230 INFO:teuthology.orchestra.run.smithi169.stderr:adjusted tunables profile to default 2024-03-12T03:29:16.520 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:16 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1180593117' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T03:29:16.851 INFO:tasks.cephadm:Adding mon.a on smithi169 2024-03-12T03:29:16.851 INFO:tasks.cephadm:Adding mon.c on smithi169 2024-03-12T03:29:16.851 INFO:tasks.cephadm:Adding mon.b on smithi173 2024-03-12T03:29:16.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi169:172.21.15.169=a;smithi169:[v2:172.21.15.169:3301,v1:172.21.15.169:6790]=c;smithi173:172.21.15.173=b' 2024-03-12T03:29:17.090 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:29:17.139 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:29:17.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:17 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1180593117' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T03:29:17.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:17 smithi169 ceph-mon[29344]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:29:17.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:17 smithi169 ceph-mon[29344]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:17.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:17.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:17.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:17.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:17.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:29:17.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:17.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:18 smithi169 ceph-mon[29344]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-12T03:29:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:18 smithi169 ceph-mon[29344]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:29:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:18 smithi169 ceph-mon[29344]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:29:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:18 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:18 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:18 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:18.882 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-03-12T03:29:19.450 DEBUG:teuthology.orchestra.run.smithi169:mon.c> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.c.service 2024-03-12T03:29:19.453 DEBUG:teuthology.orchestra.run.smithi173:mon.b> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.b.service 2024-03-12T03:29:19.455 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:29:19.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:29:19.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:19 smithi169 ceph-mon[29344]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:29:19.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:19 smithi169 ceph-mon[29344]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:19.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:19 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:19 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:19 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:19 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:19 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:19 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:19 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:20.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:20 smithi169 ceph-mon[29344]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi169:172.21.15.169=a;smithi169:[v2:172.21.15.169:3301,v1:172.21.15.169:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:29:20.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:20 smithi169 ceph-mon[29344]: Saving service mon spec with placement smithi169:172.21.15.169=a;smithi169:[v2:172.21.15.169:3301,v1:172.21.15.169:6790]=c;smithi173:172.21.15.173=b;count:3 2024-03-12T03:29:20.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:20 smithi169 ceph-mon[29344]: Deploying daemon mon.b on smithi173 2024-03-12T03:29:21.576 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:29:21.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:21 smithi169 ceph-mon[29344]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:23.492 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:29:23.493 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"677d624a-e020-11ee-95c8-87774f69a715","modified":"2024-03-12T03:27:50.360850Z","created":"2024-03-12T03:27:50.360850Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:29:23.495 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-03-12T03:29:25.251 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:29:25.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:29:25.497 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:29:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:29:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: mon.a calling monitor election 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: mon.b calling monitor election 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: monmap e2: 2 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: fsmap 2024-03-12T03:29:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:29:28.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: mgrmap e13: y(active, since 40s) 2024-03-12T03:29:28.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: overall HEALTH_OK 2024-03-12T03:29:28.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:28.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:28.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:28.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:28.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:28.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:28 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:29 smithi169 ceph-mon[29344]: Deploying daemon mon.c on smithi169 2024-03-12T03:29:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:29 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:29 smithi173 ceph-mon[33544]: Deploying daemon mon.c on smithi169 2024-03-12T03:29:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:29 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:30.345 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:29:30.345 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"677d624a-e020-11ee-95c8-87774f69a715","modified":"2024-03-12T03:29:23.557464Z","created":"2024-03-12T03:27:50.360850Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T03:29:30.347 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-03-12T03:29:30.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:30 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/1643630715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:29:30.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:30 smithi173 ceph-mon[33544]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:30 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/1643630715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:29:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:30 smithi169 ceph-mon[29344]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:31.921 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:29:31.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:29:31.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:31 smithi169 systemd[1]: Starting Ceph mon.c for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:29:32.165 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:29:32.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:31 smithi169 podman[39930]: 2024-03-12 03:29:31.83733104 +0000 UTC m=+0.017917498 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:29:32.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:31 smithi169 podman[39930]: 2024-03-12 03:29:31.942502765 +0000 UTC m=+0.123089209 container create ec8786add2919ed9d632de50254f54dba8fee6125f5dc2cd2f1fad362fb6ade3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-12T03:29:32.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 podman[39930]: 2024-03-12 03:29:32.186457974 +0000 UTC m=+0.367044418 container init ec8786add2919ed9d632de50254f54dba8fee6125f5dc2cd2f1fad362fb6ade3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-c, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-12T03:29:32.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 podman[39930]: 2024-03-12 03:29:32.189845433 +0000 UTC m=+0.370431880 container start ec8786add2919ed9d632de50254f54dba8fee6125f5dc2cd2f1fad362fb6ade3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-12T03:29:32.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:29:32.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 2 2024-03-12T03:29:32.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: pidfile_write: ignore empty --pid-file 2024-03-12T03:29:32.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: load: jerasure load: lrc 2024-03-12T03:29:32.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: RocksDB version: 7.9.2 2024-03-12T03:29:32.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Git sha 0 2024-03-12T03:29:32.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: DB SUMMARY 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: DB Session ID: H25YU20W3QWPVE8TEXNM 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:29:32.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.env: 0x55bd18b25b20 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.info_log: 0x55bd1b3718e0 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.statistics: (nil) 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.use_fsync: 0 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:29:32.484 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.db_log_dir: 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.wal_dir: 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.write_buffer_manager: 0x55bd1b345400 2024-03-12T03:29:32.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.unordered_write: 0 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.row_cache: None 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.wal_filter: None 2024-03-12T03:29:32.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:29:32.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:29:32.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:29:32.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:29:32.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.wal_compression: 0 2024-03-12T03:29:32.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:29:32.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:29:32.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:29:32.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:29:32.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_open_files: -1 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Compression algorithms supported: 2024-03-12T03:29:32.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: kZSTD supported: 0 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.merge_operator: 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_filter: None 2024-03-12T03:29:32.491 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bd1b3719e0) 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout: index_type: 0 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout: data_block_index_type: 0 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout: index_shortening: 1 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout: checksum: 4 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout: no_block_cache: 0 2024-03-12T03:29:32.492 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_cache: 0x55bd1b377090 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_cache_options: 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: capacity : 536870912 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: num_shard_bits : 4 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: strict_capacity_limit : 0 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_cache_compressed: (nil) 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: persistent_cache: (nil) 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_size: 4096 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_size_deviation: 10 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_restart_interval: 16 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: index_block_restart_interval: 1 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: metadata_block_size: 4096 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: partition_filters: 0 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: use_delta_encoding: 1 2024-03-12T03:29:32.493 INFO:journalctl@ceph.mon.c.smithi169.stdout: filter_policy: bloomfilter 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout: whole_key_filtering: 1 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout: verify_compression: 0 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout: format_version: 5 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout: enable_index_compression: 1 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout: block_align: 0 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout: max_auto_readahead_size: 262144 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout: prepopulate_block_cache: 0 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout: initial_auto_readahead_size: 8192 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compression: NoCompression 2024-03-12T03:29:32.494 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.num_levels: 7 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:29:32.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:29:32.496 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:29:32.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:29:32.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.ttl: 2592000 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:29:32.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f4bb11a6-ffe6-4d26-be71-5f629319571d 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214172219606, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214172220023, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710214172, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f4bb11a6-ffe6-4d26-be71-5f629319571d", "db_session_id": "H25YU20W3QWPVE8TEXNM", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-12T03:29:32.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214172220122, "job": 1, "event": "recovery_finished"} 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bd1b46a000 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: DB pointer 0x55bd1b454000 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout: ** DB Stats ** 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:29:32.501 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: ** Compaction Stats [default] ** 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: ** Compaction Stats [default] ** 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T03:29:32.502 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: Interval compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: Block cache BinnedLRUCache@0x55bd1b377090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-12T03:29:32.503 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: using public_addrv [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: starting mon.c rank -1 at public addrs [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] at bind addrs [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(???) e0 preinit fsid 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).mds e1 new map 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout: e1 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout: legacy client fscid: -1 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout: No filesystems configured 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-12T03:29:32.504 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1180593117' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1180593117' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:32.505 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:29:32.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi169:172.21.15.169=a;smithi169:[v2:172.21.15.169:3301,v1:172.21.15.169:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: Saving service mon spec with placement smithi169:172.21.15.169=a;smithi169:[v2:172.21.15.169:3301,v1:172.21.15.169:6790]=c;smithi173:172.21.15.173=b;count:3 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: Deploying daemon mon.b on smithi173 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.a calling monitor election 2024-03-12T03:29:32.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.b calling monitor election 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: monmap e2: 2 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: fsmap 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mgrmap e13: y(active, since 40s) 2024-03-12T03:29:32.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: overall HEALTH_OK 2024-03-12T03:29:32.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:32.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:32.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:32.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: Deploying daemon mon.c on smithi169 2024-03-12T03:29:32.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:32.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/1643630715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:29:32.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:32.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 ceph-mon[39949]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-12T03:29:32.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 bash[39930]: ec8786add2919ed9d632de50254f54dba8fee6125f5dc2cd2f1fad362fb6ade3 2024-03-12T03:29:32.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:32 smithi169 systemd[1]: Started Ceph mon.c for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:29:37.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:29:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: mon.a calling monitor election 2024-03-12T03:29:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: mon.b calling monitor election 2024-03-12T03:29:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: mon.c calling monitor election 2024-03-12T03:29:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: monmap e3: 3 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: fsmap 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: mgrmap e13: y(active, since 48s) 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: overall HEALTH_OK 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:29:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: mon.a calling monitor election 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: mon.b calling monitor election 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: mon.c calling monitor election 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-12T03:29:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: monmap e3: 3 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:29:37.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: fsmap 2024-03-12T03:29:37.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:29:37.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: mgrmap e13: y(active, since 48s) 2024-03-12T03:29:37.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: overall HEALTH_OK 2024-03-12T03:29:37.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:37.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:37.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:37.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:29:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: mon.a calling monitor election 2024-03-12T03:29:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: mon.b calling monitor election 2024-03-12T03:29:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:29:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: mon.c calling monitor election 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: monmap e3: 3 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: fsmap 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: mgrmap e13: y(active, since 48s) 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: overall HEALTH_OK 2024-03-12T03:29:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:37.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:37.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:37.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-12T03:29:38.249 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":3,"fsid":"677d624a-e020-11ee-95c8-87774f69a715","modified":"2024-03-12T03:29:32.248028Z","created":"2024-03-12T03:27:50.360850Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3301","nonce":0},{"type":"v1","addr":"172.21.15.169:6790","nonce":0}]},"addr":"172.21.15.169:6790/0","public_addr":"172.21.15.169:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-12T03:29:38.251 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 3 2024-03-12T03:29:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:38 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:38 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/3795176258' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:29:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:38 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:38.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:38 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:38.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:38 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/3795176258' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:29:38.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:38 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:38.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:38 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:38.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:38 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/3795176258' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:29:38.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:38 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:29:39.197 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-12T03:29:39.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-12T03:29:39.449 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:29:39.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:39.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:29:39.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:39.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:39.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:29:39.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:39.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:39.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:29:39.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:40.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:40 smithi169 ceph-mon[29344]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-12T03:29:40.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:40 smithi169 ceph-mon[29344]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-12T03:29:40.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:40 smithi169 ceph-mon[29344]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:29:40.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:40 smithi169 ceph-mon[29344]: Updating smithi169:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:29:40.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:40 smithi169 ceph-mon[29344]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T03:29:40.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:40 smithi169 ceph-mon[29344]: Reconfiguring daemon mon.a on smithi169 2024-03-12T03:29:40.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:40 smithi169 ceph-mon[39949]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-12T03:29:40.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:40 smithi169 ceph-mon[39949]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-12T03:29:40.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:40 smithi169 ceph-mon[39949]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:29:40.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:40 smithi169 ceph-mon[39949]: Updating smithi169:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:29:40.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:40 smithi169 ceph-mon[39949]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T03:29:40.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:40 smithi169 ceph-mon[39949]: Reconfiguring daemon mon.a on smithi169 2024-03-12T03:29:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:40 smithi173 ceph-mon[33544]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-12T03:29:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:40 smithi173 ceph-mon[33544]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-12T03:29:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:40 smithi173 ceph-mon[33544]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:29:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:40 smithi173 ceph-mon[33544]: Updating smithi169:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:29:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:40 smithi173 ceph-mon[33544]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T03:29:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:40 smithi173 ceph-mon[33544]: Reconfiguring daemon mon.a on smithi169 2024-03-12T03:29:41.827 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:41 smithi169 ceph-mon[29344]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:41.827 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:41 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:41.827 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:41 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:41.828 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:41 smithi169 ceph-mon[39949]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:41.828 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:41 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:41.828 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:41 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:41 smithi173 ceph-mon[33544]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:41 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:41 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:42.242 INFO:teuthology.orchestra.run.smithi169.stdout:# minimal ceph.conf for 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:42.242 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-03-12T03:29:42.242 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:42.242 INFO:teuthology.orchestra.run.smithi169.stdout: mon_host = [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] [v2:172.21.15.169:3301/0,v1:172.21.15.169:6790/0] 2024-03-12T03:29:42.840 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-12T03:29:42.840 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:29:42.840 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T03:29:42.865 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:42 smithi169 ceph-mon[39949]: Reconfiguring mon.c (monmap changed)... 2024-03-12T03:29:42.865 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:42 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:42.865 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:42 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:29:42.865 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:42 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:42.865 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:42 smithi169 ceph-mon[39949]: Reconfiguring daemon mon.c on smithi169 2024-03-12T03:29:42.865 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:42 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1535855794' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:42.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:42 smithi169 ceph-mon[29344]: Reconfiguring mon.c (monmap changed)... 2024-03-12T03:29:42.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:42 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:42.866 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:42 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:29:42.866 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:42 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:42.866 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:42 smithi169 ceph-mon[29344]: Reconfiguring daemon mon.c on smithi169 2024-03-12T03:29:42.866 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:42 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1535855794' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:42.869 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:29:42.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:29:42.940 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:29:42.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T03:29:42.970 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:42 smithi173 ceph-mon[33544]: Reconfiguring mon.c (monmap changed)... 2024-03-12T03:29:42.970 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:42 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:42.970 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:42 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:29:42.970 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:42 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:42.970 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:42 smithi173 ceph-mon[33544]: Reconfiguring daemon mon.c on smithi169 2024-03-12T03:29:42.970 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:42 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1535855794' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:42.975 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:29:42.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:29:43.047 INFO:tasks.cephadm:Adding mgr.y on smithi169 2024-03-12T03:29:43.047 INFO:tasks.cephadm:Adding mgr.x on smithi173 2024-03-12T03:29:43.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch apply mgr '2;smithi169=y;smithi173=x' 2024-03-12T03:29:43.324 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:29:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:43 smithi173 ceph-mon[33544]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:43 smithi169 ceph-mon[29344]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:43.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:43 smithi169 ceph-mon[39949]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:44.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:44 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:44.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:44 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:44.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:44 smithi173 ceph-mon[33544]: Reconfiguring mon.b (monmap changed)... 2024-03-12T03:29:44.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:44 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:44.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:44 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:29:44.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:44 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:44.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:44 smithi173 ceph-mon[33544]: Reconfiguring daemon mon.b on smithi173 2024-03-12T03:29:44.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:44 smithi173 ceph-mon[33544]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:44.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:44.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:44.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[39949]: Reconfiguring mon.b (monmap changed)... 2024-03-12T03:29:44.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:44.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:29:44.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:44.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[39949]: Reconfiguring daemon mon.b on smithi173 2024-03-12T03:29:44.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[39949]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:44.938 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:44.938 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:44.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[29344]: Reconfiguring mon.b (monmap changed)... 2024-03-12T03:29:44.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:29:44.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:29:44.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:44.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[29344]: Reconfiguring daemon mon.b on smithi173 2024-03-12T03:29:44.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:44 smithi169 ceph-mon[29344]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:45.139 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mgr update... 2024-03-12T03:29:46.025 DEBUG:teuthology.orchestra.run.smithi173:mgr.x> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@mgr.x.service 2024-03-12T03:29:46.028 INFO:tasks.cephadm:Deploying OSDs... 2024-03-12T03:29:46.028 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:29:46.028 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:29:46.045 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T03:29:46.045 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2024-03-12T03:29:46.100 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T03:29:46.100 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:29:46.100 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-12T03:29:46.100 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:29:46.100 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:29:46.100 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-12 03:29:31.658688452 +0000 2024-03-12T03:29:46.100 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-12 03:19:20.222324899 +0000 2024-03-12T03:29:46.100 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-12 03:19:20.222324899 +0000 2024-03-12T03:29:46.100 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-12 03:19:20.222324899 +0000 2024-03-12T03:29:46.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T03:29:46.167 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-12T03:29:46.168 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-12T03:29:46.168 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000289727 s, 1.8 MB/s 2024-03-12T03:29:46.169 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T03:29:46.225 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2024-03-12T03:29:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T03:29:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:29:46.280 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-12T03:29:46.280 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:29:46.280 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:29:46.280 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-12 03:29:31.659688431 +0000 2024-03-12T03:29:46.280 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-12 03:19:20.578317762 +0000 2024-03-12T03:29:46.280 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-12 03:19:20.578317762 +0000 2024-03-12T03:29:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-12 03:19:20.578317762 +0000 2024-03-12T03:29:46.280 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T03:29:46.346 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-12T03:29:46.346 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-12T03:29:46.346 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000198209 s, 2.6 MB/s 2024-03-12T03:29:46.347 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T03:29:46.403 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2024-03-12T03:29:46.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:46 smithi173 ceph-mon[33544]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi169=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:29:46.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:46 smithi173 ceph-mon[33544]: Saving service mgr spec with placement smithi169=y;smithi173=x;count:2 2024-03-12T03:29:46.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:46 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:46.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:46 smithi169 ceph-mon[29344]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi169=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:29:46.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:46 smithi169 ceph-mon[29344]: Saving service mgr spec with placement smithi169=y;smithi173=x;count:2 2024-03-12T03:29:46.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:46 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:46.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:46 smithi169 ceph-mon[39949]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi169=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:29:46.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:46 smithi169 ceph-mon[39949]: Saving service mgr spec with placement smithi169=y;smithi173=x;count:2 2024-03-12T03:29:46.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:46 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:46.437 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T03:29:46.437 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:29:46.437 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-12T03:29:46.438 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:29:46.438 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:29:46.438 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-12 03:29:31.659688431 +0000 2024-03-12T03:29:46.438 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-12 03:19:20.912311065 +0000 2024-03-12T03:29:46.438 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-12 03:19:20.912311065 +0000 2024-03-12T03:29:46.438 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-12 03:19:20.912311065 +0000 2024-03-12T03:29:46.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T03:29:46.504 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-12T03:29:46.504 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-12T03:29:46.504 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000257829 s, 2.0 MB/s 2024-03-12T03:29:46.505 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T03:29:46.562 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2024-03-12T03:29:46.616 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T03:29:46.616 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:29:46.616 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-12T03:29:46.617 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:29:46.617 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:29:46.617 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-12 03:29:31.659688431 +0000 2024-03-12T03:29:46.617 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-12 03:19:21.250304289 +0000 2024-03-12T03:29:46.617 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-12 03:19:21.250304289 +0000 2024-03-12T03:29:46.617 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-12 03:19:21.250304289 +0000 2024-03-12T03:29:46.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T03:29:46.682 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-12T03:29:46.683 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-12T03:29:46.683 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000176954 s, 2.9 MB/s 2024-03-12T03:29:46.684 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T03:29:46.739 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:29:46.740 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:29:46.755 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T03:29:46.755 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-03-12T03:29:46.810 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T03:29:46.810 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:29:46.810 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-12T03:29:46.810 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:29:46.810 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:29:46.810 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-12 03:29:22.562898270 +0000 2024-03-12T03:29:46.810 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-12 03:17:15.413247162 +0000 2024-03-12T03:29:46.810 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-12 03:17:15.413247162 +0000 2024-03-12T03:29:46.810 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-12 03:17:15.413247162 +0000 2024-03-12T03:29:46.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T03:29:46.876 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-12T03:29:46.876 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-12T03:29:46.876 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000187663 s, 2.7 MB/s 2024-03-12T03:29:46.877 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T03:29:46.934 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-03-12T03:29:46.990 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T03:29:46.990 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:29:46.990 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-12T03:29:46.990 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:29:46.990 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:29:46.990 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-12 03:29:22.562898270 +0000 2024-03-12T03:29:46.991 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-12 03:17:15.763239655 +0000 2024-03-12T03:29:46.991 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-12 03:17:15.763239655 +0000 2024-03-12T03:29:46.991 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-12 03:17:15.763239655 +0000 2024-03-12T03:29:46.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T03:29:47.058 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-12T03:29:47.059 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-12T03:29:47.059 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000179966 s, 2.8 MB/s 2024-03-12T03:29:47.059 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T03:29:47.115 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-03-12T03:29:47.170 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T03:29:47.170 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:29:47.170 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-12T03:29:47.170 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:29:47.170 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:29:47.170 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-12 03:29:22.563898248 +0000 2024-03-12T03:29:47.171 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-12 03:17:16.111232191 +0000 2024-03-12T03:29:47.171 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-12 03:17:16.111232191 +0000 2024-03-12T03:29:47.171 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-12 03:17:16.111232191 +0000 2024-03-12T03:29:47.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T03:29:47.235 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-12T03:29:47.235 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-12T03:29:47.235 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000194806 s, 2.6 MB/s 2024-03-12T03:29:47.236 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T03:29:47.292 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-03-12T03:29:47.346 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T03:29:47.347 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:29:47.347 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-12T03:29:47.347 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:29:47.347 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:29:47.347 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-12 03:29:22.563898248 +0000 2024-03-12T03:29:47.347 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-12 03:17:16.462224663 +0000 2024-03-12T03:29:47.347 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-12 03:17:16.462224663 +0000 2024-03-12T03:29:47.347 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-12 03:17:16.462224663 +0000 2024-03-12T03:29:47.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T03:29:47.413 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-12T03:29:47.413 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-12T03:29:47.413 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000208642 s, 2.5 MB/s 2024-03-12T03:29:47.415 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T03:29:47.470 INFO:tasks.cephadm:Deploying osd.0 on smithi169 with /dev/vg_nvme/lv_4... 2024-03-12T03:29:47.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T03:29:47.717 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:29:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:29:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:29:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:29:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:47 smithi173 ceph-mon[33544]: Deploying daemon mgr.x on smithi173 2024-03-12T03:29:47.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:47 smithi173 ceph-mon[33544]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:47.839 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:47.839 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:47.839 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[29344]: Deploying daemon mgr.x on smithi173 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[29344]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:47.840 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:47.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:47.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:47.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:29:47.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:29:47.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:29:47.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:47.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[39949]: Deploying daemon mgr.x on smithi173 2024-03-12T03:29:47.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:47 smithi169 ceph-mon[39949]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:48.707 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:48 smithi173 systemd[1]: Starting Ceph mgr.x for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:29:48.968 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:48 smithi173 podman[35394]: 2024-03-12 03:29:48.608323172 +0000 UTC m=+0.017549631 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:29:48.968 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:48 smithi173 podman[35394]: 2024-03-12 03:29:48.716475771 +0000 UTC m=+0.125702213 container create d08b03e30b6b02f2089ebe167b5ad5236f6d70907a866a41c457e5d26643b0f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=) 2024-03-12T03:29:48.968 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:48 smithi173 podman[35394]: 2024-03-12 03:29:48.899642394 +0000 UTC m=+0.308868842 container init d08b03e30b6b02f2089ebe167b5ad5236f6d70907a866a41c457e5d26643b0f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:29:48.968 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:48 smithi173 podman[35394]: 2024-03-12 03:29:48.902275374 +0000 UTC m=+0.311501819 container start d08b03e30b6b02f2089ebe167b5ad5236f6d70907a866a41c457e5d26643b0f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-12T03:29:48.968 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:48 smithi173 ceph-mgr[35413]: -- 172.21.15.173:0/637495020 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d00aed05a0 con 0x55d00aea5000 2024-03-12T03:29:48.968 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:48 smithi173 bash[35394]: d08b03e30b6b02f2089ebe167b5ad5236f6d70907a866a41c457e5d26643b0f4 2024-03-12T03:29:49.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:49 smithi169 ceph-mon[29344]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:49.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:49 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:49.148 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:49 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:49.148 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:49 smithi169 ceph-mon[39949]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:49.148 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:49 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:49.148 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:49 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:49.236 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:48 smithi173 systemd[1]: Started Ceph mgr.x for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:29:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:49 smithi173 ceph-mon[33544]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:49 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:49 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:49.605 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T03:29:49.606 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T03:29:49.606 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-03-12T03:29:49.606 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-03-12T03:29:49.606 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134031 s, 782 MB/s 2024-03-12T03:29:49.606 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-03-12T03:29:49.736 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:49.457+0000 7f4a8d06f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:29:49.736 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:49.551+0000 7f4a8d06f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:29:50.231 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:50 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:50.232 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:50 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:50.232 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:50 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:50 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:50 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:50 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:50 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:50 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:50 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:50.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_4 2024-03-12T03:29:50.486 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:50 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:50.229+0000 7f4a8d06f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:29:50.486 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:50 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:50.320+0000 7f4a8d06f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:29:50.486 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:50 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:50.469+0000 7f4a8d06f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:29:50.711 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:29:50.911 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:50 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:50.565+0000 7f4a8d06f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:29:50.912 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:50 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:50.650+0000 7f4a8d06f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:29:50.912 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:50 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:50.909+0000 7f4a8d06f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:29:51.197 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:51.017+0000 7f4a8d06f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:29:51.197 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:51.101+0000 7f4a8d06f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:29:51.197 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:51.194+0000 7f4a8d06f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:29:51.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:51.369+0000 7f4a8d06f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:29:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-mon[33544]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:51.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-mon[33544]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T03:29:51.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:29:51.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:29:51.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:51.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:51 smithi173 ceph-mon[33544]: Reconfiguring daemon mgr.y on smithi169 2024-03-12T03:29:51.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:51.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:51.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:51.757 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:51.757 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:51.757 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[29344]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:51.757 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[29344]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T03:29:51.757 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:29:51.757 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:29:51.757 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:51.757 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[29344]: Reconfiguring daemon mgr.y on smithi169 2024-03-12T03:29:51.757 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:51.757 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:51.757 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:51.757 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:51.758 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:51.758 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[39949]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:51.758 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[39949]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T03:29:51.758 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:29:51.758 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:29:51.758 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:51.758 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:51 smithi169 ceph-mon[39949]: Reconfiguring daemon mgr.y on smithi169 2024-03-12T03:29:52.736 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:52.330+0000 7f4a8d06f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:29:52.736 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:52.458+0000 7f4a8d06f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:29:53.079 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:52.796+0000 7f4a8d06f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:29:53.079 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:52.884+0000 7f4a8d06f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:29:53.079 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:53.076+0000 7f4a8d06f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:29:53.486 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:53.261+0000 7f4a8d06f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:29:53.801 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:53.537+0000 7f4a8d06f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:29:53.801 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:53 smithi173 ceph-mon[33544]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:53.801 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:53 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:29:53.801 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:53 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:29:53.801 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:53 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:53.802 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:53 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:53.802 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:53 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:53.802 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:53 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:53.841 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[29344]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:53.842 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:29:53.842 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:29:53.842 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:53.842 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:53.842 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:53.842 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:53.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[39949]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:53.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:29:53.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:29:53.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:53.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:53.843 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:53.843 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:53 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:29:54.236 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:53.798+0000 7f4a8d06f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:29:54.236 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:54.032+0000 7f4a8d06f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:29:54.656 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:54 smithi169 ceph-mon[29344]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:29:54.656 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:54 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:54.657 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:54 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:54.657 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:54 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:54.657 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:54 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:54.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:54 smithi169 ceph-mon[39949]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:29:54.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:54 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:54.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:54 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:54.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:54 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:54.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:54 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:54.986 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:54.818+0000 7f4a8d06f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:29:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:54 smithi173 ceph-mon[33544]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:29:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:54 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:54 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:54 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:29:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:54 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:29:55.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:55 smithi169 ceph-mon[29344]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:55.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:55 smithi169 ceph-mon[39949]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:55.694 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:55 smithi173 ceph-mon[33544]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:55.694 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:55 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:55.383+0000 7f4a8d06f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:29:55.694 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:55 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:55.479+0000 7f4a8d06f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:29:55.694 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:29:55 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:29:55.691+0000 7f4a8d06f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:29:56.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[39949]: Standby manager daemon x started 2024-03-12T03:29:56.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[39949]: from='mgr.? 172.21.15.173:0/3689405235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:29:56.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[39949]: from='mgr.? 172.21.15.173:0/3689405235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[39949]: from='mgr.? 172.21.15.173:0/3689405235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[39949]: from='mgr.? 172.21.15.173:0/3689405235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/703445011' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c44466b0-3112-44dc-b5f3-d6494b958761"}]: dispatch 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c44466b0-3112-44dc-b5f3-d6494b958761"}]: dispatch 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c44466b0-3112-44dc-b5f3-d6494b958761"}]': finished 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[39949]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3321872804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[29344]: Standby manager daemon x started 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[29344]: from='mgr.? 172.21.15.173:0/3689405235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[29344]: from='mgr.? 172.21.15.173:0/3689405235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[29344]: from='mgr.? 172.21.15.173:0/3689405235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:29:56.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[29344]: from='mgr.? 172.21.15.173:0/3689405235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:29:56.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/703445011' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c44466b0-3112-44dc-b5f3-d6494b958761"}]: dispatch 2024-03-12T03:29:56.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c44466b0-3112-44dc-b5f3-d6494b958761"}]: dispatch 2024-03-12T03:29:56.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c44466b0-3112-44dc-b5f3-d6494b958761"}]': finished 2024-03-12T03:29:56.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[29344]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T03:29:56.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:29:56.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:56 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3321872804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:29:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:56 smithi173 ceph-mon[33544]: Standby manager daemon x started 2024-03-12T03:29:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:56 smithi173 ceph-mon[33544]: from='mgr.? 172.21.15.173:0/3689405235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:29:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:56 smithi173 ceph-mon[33544]: from='mgr.? 172.21.15.173:0/3689405235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:29:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:56 smithi173 ceph-mon[33544]: from='mgr.? 172.21.15.173:0/3689405235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:29:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:56 smithi173 ceph-mon[33544]: from='mgr.? 172.21.15.173:0/3689405235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:29:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:56 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/703445011' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c44466b0-3112-44dc-b5f3-d6494b958761"}]: dispatch 2024-03-12T03:29:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:56 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c44466b0-3112-44dc-b5f3-d6494b958761"}]: dispatch 2024-03-12T03:29:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:56 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c44466b0-3112-44dc-b5f3-d6494b958761"}]': finished 2024-03-12T03:29:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:56 smithi173 ceph-mon[33544]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T03:29:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:56 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:29:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:56 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3321872804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:29:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:57 smithi169 ceph-mon[29344]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:57 smithi169 ceph-mon[29344]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-12T03:29:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:57 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:29:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:57 smithi169 ceph-mon[39949]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:57 smithi169 ceph-mon[39949]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-12T03:29:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:57 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:29:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:57 smithi173 ceph-mon[33544]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:57 smithi173 ceph-mon[33544]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-12T03:29:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:57 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:29:59.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:29:59 smithi169 ceph-mon[29344]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:59.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:29:59 smithi169 ceph-mon[39949]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:29:59.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:29:59 smithi173 ceph-mon[33544]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:00.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:00 smithi169 ceph-mon[39949]: overall HEALTH_OK 2024-03-12T03:30:00.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:00 smithi169 ceph-mon[29344]: overall HEALTH_OK 2024-03-12T03:30:00.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:00 smithi173 ceph-mon[33544]: overall HEALTH_OK 2024-03-12T03:30:01.861 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:01 smithi169 ceph-mon[29344]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:01.861 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:01 smithi169 ceph-mon[39949]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:01 smithi173 ceph-mon[33544]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:02 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:30:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:02 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:02.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:02 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:30:02.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:02 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:02 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:30:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:02 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:03 smithi169 ceph-mon[29344]: Deploying daemon osd.0 on smithi169 2024-03-12T03:30:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:03 smithi169 ceph-mon[29344]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:03 smithi169 ceph-mon[39949]: Deploying daemon osd.0 on smithi169 2024-03-12T03:30:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:03 smithi169 ceph-mon[39949]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:03 smithi173 ceph-mon[33544]: Deploying daemon osd.0 on smithi169 2024-03-12T03:30:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:03 smithi173 ceph-mon[33544]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:05.840 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:05 smithi169 ceph-mon[29344]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:05.840 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:05 smithi169 ceph-mon[39949]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:05 smithi173 ceph-mon[33544]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:07 smithi169 ceph-mon[29344]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:07 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:07 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:07 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:07 smithi169 ceph-mon[39949]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:07 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:07 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:07 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:07 smithi173 ceph-mon[33544]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:07 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:07 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:07 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:09.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:09 smithi169 ceph-mon[29344]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:09.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:09 smithi169 ceph-mon[39949]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:09 smithi173 ceph-mon[33544]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:10.881 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 0 on host 'smithi169' 2024-03-12T03:30:11.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:11.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[39949]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:30:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[39949]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[29344]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:30:11.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[29344]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:11.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:11.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:11 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:11 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:11 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:11.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:11 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:11.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:11 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:11 smithi173 ceph-mon[33544]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:30:11.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:11 smithi173 ceph-mon[33544]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:11.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:11 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:11 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:11.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:11 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:11.505 DEBUG:teuthology.orchestra.run.smithi169:osd.0> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.0.service 2024-03-12T03:30:11.507 INFO:tasks.cephadm:Deploying osd.1 on smithi169 with /dev/vg_nvme/lv_3... 2024-03-12T03:30:11.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T03:30:12.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:12 smithi169 ceph-mon[39949]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:30:12.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:12 smithi169 ceph-mon[39949]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:30:12.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:12 smithi169 ceph-mon[39949]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:30:12.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:12 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:12.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:12 smithi169 ceph-mon[29344]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:30:12.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:12 smithi169 ceph-mon[29344]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:30:12.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:12 smithi169 ceph-mon[29344]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:30:12.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:12 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:12.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:12 smithi173 ceph-mon[33544]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:30:12.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:12 smithi173 ceph-mon[33544]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:30:12.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:12 smithi173 ceph-mon[33544]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:30:12.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:12 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:12.833 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:30:12 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0[43969]: 2024-03-12T03:30:12.529+0000 7f48d1940700 -1 osd.0 0 waiting for initial osdmap 2024-03-12T03:30:12.834 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:30:12 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0[43969]: 2024-03-12T03:30:12.532+0000 7f48c8f2a700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:30:13.032 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:30:13.371 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[29344]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-12T03:30:13.372 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[29344]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:30:13.372 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:13.372 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:13.372 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[29344]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' 2024-03-12T03:30:13.372 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[29344]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:13.372 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:13.372 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[39949]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-12T03:30:13.372 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[39949]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:30:13.372 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:13.372 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:13.372 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[39949]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' 2024-03-12T03:30:13.373 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[39949]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:13.373 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:13 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:13 smithi173 ceph-mon[33544]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-12T03:30:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:13 smithi173 ceph-mon[33544]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:30:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:13 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:13 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:13 smithi173 ceph-mon[33544]: from='osd.0 v2:172.21.15.169:6801/676779916' entity='osd.0' 2024-03-12T03:30:13.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:13 smithi173 ceph-mon[33544]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:13.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:13 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:14.329 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:14 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:30:14.329 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:14 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:30:14.329 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:14 smithi169 ceph-mon[29344]: osd.0 v2:172.21.15.169:6801/676779916 boot 2024-03-12T03:30:14.329 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:14 smithi169 ceph-mon[29344]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T03:30:14.330 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:14 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:14.330 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:14 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:30:14.330 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:14 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:30:14.330 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:14 smithi169 ceph-mon[39949]: osd.0 v2:172.21.15.169:6801/676779916 boot 2024-03-12T03:30:14.330 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:14 smithi169 ceph-mon[39949]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T03:30:14.330 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:14 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:14.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:14 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:30:14.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:14 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:30:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:14 smithi173 ceph-mon[33544]: osd.0 v2:172.21.15.169:6801/676779916 boot 2024-03-12T03:30:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:14 smithi173 ceph-mon[33544]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T03:30:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:14 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:30:15.628 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[39949]: Detected new or changed devices on smithi169 2024-03-12T03:30:15.628 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:15.628 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:15.628 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:30:15.629 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[39949]: Adjusting osd_memory_target on smithi169 to 16064M 2024-03-12T03:30:15.629 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:15.629 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:15.629 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:15.629 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:15.629 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[39949]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:15.629 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[29344]: Detected new or changed devices on smithi169 2024-03-12T03:30:15.629 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:15.629 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:15.629 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:30:15.630 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[29344]: Adjusting osd_memory_target on smithi169 to 16064M 2024-03-12T03:30:15.630 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:15.630 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:15.630 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:15.630 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:15.630 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:15 smithi169 ceph-mon[29344]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:15.630 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T03:30:15.630 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T03:30:15.630 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-03-12T03:30:15.630 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-03-12T03:30:15.630 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112465 s, 932 MB/s 2024-03-12T03:30:15.631 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-03-12T03:30:15.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:15 smithi173 ceph-mon[33544]: Detected new or changed devices on smithi169 2024-03-12T03:30:15.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:15 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:15.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:15 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:15.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:15 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:30:15.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:15 smithi173 ceph-mon[33544]: Adjusting osd_memory_target on smithi169 to 16064M 2024-03-12T03:30:15.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:15 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:15.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:15 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:15.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:15 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:15.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:15 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:15.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:15 smithi173 ceph-mon[33544]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:16.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_3 2024-03-12T03:30:16.450 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:30:16.572 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:16 smithi169 ceph-mon[29344]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:30:16.572 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:16 smithi169 ceph-mon[39949]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:30:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:16 smithi173 ceph-mon[33544]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:30:17.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:17 smithi169 ceph-mon[29344]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:17.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:17 smithi169 ceph-mon[39949]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:17 smithi173 ceph-mon[33544]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:18.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:18 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:30:18.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:18 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:30:18.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:18 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:18 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:30:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:18 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:30:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:18 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:18 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:30:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:18 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:30:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:18 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:19.479 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:19 smithi169 ceph-mon[29344]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:19.479 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:19 smithi169 ceph-mon[29344]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:19.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:19 smithi169 ceph-mon[39949]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:19.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:19 smithi169 ceph-mon[39949]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:19 smithi173 ceph-mon[33544]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:19 smithi173 ceph-mon[33544]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:20 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1861116656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "184c5c97-a992-4be9-bcd3-c08cb58a0e56"}]: dispatch 2024-03-12T03:30:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:20 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "184c5c97-a992-4be9-bcd3-c08cb58a0e56"}]: dispatch 2024-03-12T03:30:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:20 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "184c5c97-a992-4be9-bcd3-c08cb58a0e56"}]': finished 2024-03-12T03:30:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:20 smithi169 ceph-mon[29344]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T03:30:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:20 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1861116656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "184c5c97-a992-4be9-bcd3-c08cb58a0e56"}]: dispatch 2024-03-12T03:30:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:20 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "184c5c97-a992-4be9-bcd3-c08cb58a0e56"}]: dispatch 2024-03-12T03:30:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:20 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "184c5c97-a992-4be9-bcd3-c08cb58a0e56"}]': finished 2024-03-12T03:30:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:20 smithi169 ceph-mon[39949]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T03:30:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:20 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1861116656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "184c5c97-a992-4be9-bcd3-c08cb58a0e56"}]: dispatch 2024-03-12T03:30:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:20 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "184c5c97-a992-4be9-bcd3-c08cb58a0e56"}]: dispatch 2024-03-12T03:30:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:20 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "184c5c97-a992-4be9-bcd3-c08cb58a0e56"}]': finished 2024-03-12T03:30:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:20 smithi173 ceph-mon[33544]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T03:30:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:21.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:21 smithi169 ceph-mon[29344]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:21.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:21 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3357576571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:30:21.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:21 smithi169 ceph-mon[39949]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:21.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:21 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3357576571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:30:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:21 smithi173 ceph-mon[33544]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:21 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3357576571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:30:23.820 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:23 smithi169 ceph-mon[29344]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:23.821 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:23 smithi169 ceph-mon[39949]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:23 smithi173 ceph-mon[33544]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:25.804 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:25 smithi169 ceph-mon[29344]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:25.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:25 smithi169 ceph-mon[39949]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:25 smithi173 ceph-mon[33544]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:27.846 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:27 smithi169 ceph-mon[29344]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:27.846 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:27 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:30:27.846 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:27 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:27.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:27 smithi169 ceph-mon[39949]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:27.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:27 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:30:27.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:27 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:27 smithi173 ceph-mon[33544]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:27 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:30:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:27 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:28.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:28 smithi169 ceph-mon[29344]: Deploying daemon osd.1 on smithi169 2024-03-12T03:30:28.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:28 smithi169 ceph-mon[39949]: Deploying daemon osd.1 on smithi169 2024-03-12T03:30:28.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:28 smithi173 ceph-mon[33544]: Deploying daemon osd.1 on smithi169 2024-03-12T03:30:29.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:29 smithi173 ceph-mon[33544]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:29 smithi169 ceph-mon[29344]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:29.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:29 smithi169 ceph-mon[39949]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:31.919 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:31 smithi169 ceph-mon[29344]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:31.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:31 smithi169 ceph-mon[39949]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:31 smithi173 ceph-mon[33544]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:33.473 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:33 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:33.474 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:33 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:33.474 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:33 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:33.474 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:33 smithi169 ceph-mon[29344]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:33.474 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:33 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:33.474 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:33 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:33.474 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:33 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:33.474 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:33 smithi169 ceph-mon[39949]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:33.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:33 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:33.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:33 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:33.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:33 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:33.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:33 smithi173 ceph-mon[33544]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:35.789 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 1 on host 'smithi169' 2024-03-12T03:30:35.789 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[39949]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:35.789 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:35.789 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:35.789 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:35.789 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:35.789 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:35.790 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[39949]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:30:35.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[29344]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:35.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:35.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:35.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:35.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:35.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:35.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:35 smithi169 ceph-mon[29344]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:30:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:35 smithi173 ceph-mon[33544]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:35 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:35 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:35 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:35 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:35.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:35 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:35.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:35 smithi173 ceph-mon[33544]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:30:36.431 DEBUG:teuthology.orchestra.run.smithi169:osd.1> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.1.service 2024-03-12T03:30:36.434 INFO:tasks.cephadm:Deploying osd.2 on smithi169 with /dev/vg_nvme/lv_2... 2024-03-12T03:30:36.434 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T03:30:37.170 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:37.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:37.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:37.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[39949]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:30:37.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[39949]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:30:37.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[39949]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:30:37.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:37.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[39949]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:37.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:37.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:37.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:37.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[29344]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:30:37.172 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[29344]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:30:37.172 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[29344]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:30:37.172 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:37.172 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:36 smithi169 ceph-mon[29344]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:36 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:36 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:36 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:36 smithi173 ceph-mon[33544]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:30:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:36 smithi173 ceph-mon[33544]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:30:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:36 smithi173 ceph-mon[33544]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:30:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:36 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:36 smithi173 ceph-mon[33544]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:37.426 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:30:37 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1[49981]: 2024-03-12T03:30:37.354+0000 7fb2a73af700 -1 osd.1 0 waiting for initial osdmap 2024-03-12T03:30:37.426 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:30:37 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1[49981]: 2024-03-12T03:30:37.357+0000 7fb29f19a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:30:37.986 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:30:38.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:37 smithi169 ceph-mon[39949]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-12T03:30:38.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:37 smithi169 ceph-mon[39949]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:30:38.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:38.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:38.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:37 smithi169 ceph-mon[39949]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' 2024-03-12T03:30:38.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:37 smithi169 ceph-mon[29344]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-12T03:30:38.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:37 smithi169 ceph-mon[29344]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:30:38.101 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:38.101 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:38.101 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:37 smithi169 ceph-mon[29344]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' 2024-03-12T03:30:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:37 smithi173 ceph-mon[33544]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-12T03:30:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:37 smithi173 ceph-mon[33544]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:30:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:38.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:38.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:37 smithi173 ceph-mon[33544]: from='osd.1 v2:172.21.15.169:6805/130042595' entity='osd.1' 2024-03-12T03:30:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:38 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:30:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:38 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:30:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:38 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:38 smithi173 ceph-mon[33544]: osd.1 v2:172.21.15.169:6805/130042595 boot 2024-03-12T03:30:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:38 smithi173 ceph-mon[33544]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T03:30:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:38 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:38 smithi173 ceph-mon[33544]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:39.276 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:30:39.276 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:30:39.276 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:39.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[39949]: osd.1 v2:172.21.15.169:6805/130042595 boot 2024-03-12T03:30:39.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[39949]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T03:30:39.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:39.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[39949]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:39.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:30:39.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:30:39.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:39.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[29344]: osd.1 v2:172.21.15.169:6805/130042595 boot 2024-03-12T03:30:39.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[29344]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T03:30:39.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:30:39.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:38 smithi169 ceph-mon[29344]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:30:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[29344]: Detected new or changed devices on smithi169 2024-03-12T03:30:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:30:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:30:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[29344]: Adjusting osd_memory_target on smithi169 to 8032M 2024-03-12T03:30:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[39949]: Detected new or changed devices on smithi169 2024-03-12T03:30:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:30:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:30:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[39949]: Adjusting osd_memory_target on smithi169 to 8032M 2024-03-12T03:30:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:40 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:40.273 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T03:30:40.273 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T03:30:40.274 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-03-12T03:30:40.274 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-03-12T03:30:40.274 INFO:teuthology.orchestra.run.smithi169.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0129288 s, 811 MB/s 2024-03-12T03:30:40.274 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 2024-03-12T03:30:40.274 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-03-12T03:30:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:40 smithi173 ceph-mon[33544]: Detected new or changed devices on smithi169 2024-03-12T03:30:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:40 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:40 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:40.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:40 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:30:40.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:40 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:30:40.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:40 smithi173 ceph-mon[33544]: Adjusting osd_memory_target on smithi169 to 8032M 2024-03-12T03:30:40.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:40 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:40.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:40 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:40.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:40 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:40.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:40 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:40.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_2 2024-03-12T03:30:41.177 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:30:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:41 smithi173 ceph-mon[33544]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:30:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:41 smithi173 ceph-mon[33544]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:41.579 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:41 smithi169 ceph-mon[29344]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:30:41.579 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:41 smithi169 ceph-mon[29344]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:41.579 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:41 smithi169 ceph-mon[39949]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:30:41.579 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:41 smithi169 ceph-mon[39949]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:43.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:43 smithi169 ceph-mon[29344]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:43.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:30:43.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:30:43.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:43.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:43 smithi169 ceph-mon[39949]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:43.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:30:43.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:30:43.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:43 smithi173 ceph-mon[33544]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:30:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:30:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:44.896 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:44 smithi169 ceph-mon[29344]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:44.896 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:44 smithi169 ceph-mon[39949]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:44.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:44 smithi173 ceph-mon[33544]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:45 smithi169 ceph-mon[29344]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:45 smithi169 ceph-mon[39949]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:45.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:45 smithi173 ceph-mon[33544]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:46.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1876329565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "beb15e06-1ca4-46a7-b640-efd1991367e8"}]: dispatch 2024-03-12T03:30:46.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1876329565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "beb15e06-1ca4-46a7-b640-efd1991367e8"}]': finished 2024-03-12T03:30:46.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:46 smithi169 ceph-mon[29344]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T03:30:46.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:46 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:30:46.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2855262151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:30:46.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1876329565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "beb15e06-1ca4-46a7-b640-efd1991367e8"}]: dispatch 2024-03-12T03:30:46.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1876329565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "beb15e06-1ca4-46a7-b640-efd1991367e8"}]': finished 2024-03-12T03:30:46.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:46 smithi169 ceph-mon[39949]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T03:30:46.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:46 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:30:46.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2855262151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:30:46.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1876329565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "beb15e06-1ca4-46a7-b640-efd1991367e8"}]: dispatch 2024-03-12T03:30:46.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1876329565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "beb15e06-1ca4-46a7-b640-efd1991367e8"}]': finished 2024-03-12T03:30:46.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:46 smithi173 ceph-mon[33544]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T03:30:46.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:46 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:30:46.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2855262151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:30:47.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:47 smithi169 ceph-mon[39949]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:47 smithi169 ceph-mon[29344]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:47.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:47 smithi173 ceph-mon[33544]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:49.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:49 smithi169 ceph-mon[29344]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:49.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:49 smithi169 ceph-mon[39949]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:49.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:49 smithi173 ceph-mon[33544]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:51.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:51 smithi169 ceph-mon[29344]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:51.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:51 smithi169 ceph-mon[39949]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:51 smithi173 ceph-mon[33544]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:52.917 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:52 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:30:52.918 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:52 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:52.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:52 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:30:52.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:52 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:52.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:52 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:30:52.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:52 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:53.858 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:53 smithi169 ceph-mon[29344]: Deploying daemon osd.2 on smithi169 2024-03-12T03:30:53.858 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:53 smithi169 ceph-mon[29344]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:53.858 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:53 smithi169 ceph-mon[39949]: Deploying daemon osd.2 on smithi169 2024-03-12T03:30:53.858 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:53 smithi169 ceph-mon[39949]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:53 smithi173 ceph-mon[33544]: Deploying daemon osd.2 on smithi169 2024-03-12T03:30:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:53 smithi173 ceph-mon[33544]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:55.804 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:55 smithi169 ceph-mon[29344]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:55.804 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:55 smithi169 ceph-mon[39949]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:55 smithi173 ceph-mon[33544]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:57.907 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:57 smithi169 ceph-mon[29344]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:57.907 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:57 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:57.907 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:57 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:57.907 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:57 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:57.907 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:57 smithi169 ceph-mon[39949]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:57.907 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:57 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:57.907 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:57 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:57.907 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:57 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:57 smithi173 ceph-mon[33544]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:57 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:57 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:30:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:57 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:59.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:30:59 smithi169 ceph-mon[29344]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:59.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:30:59 smithi169 ceph-mon[39949]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:30:59.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:30:59 smithi173 ceph-mon[33544]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:31:01.143 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 2 on host 'smithi169' 2024-03-12T03:31:01.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:01 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:01.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:01 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:01.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:01 smithi169 ceph-mon[39949]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:31:01.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:01 smithi169 ceph-mon[39949]: from='osd.2 v2:172.21.15.169:6809/2751816702' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:31:01.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:01 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:01.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:01 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:01 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:01 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:01 smithi169 ceph-mon[29344]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:31:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:01 smithi169 ceph-mon[29344]: from='osd.2 v2:172.21.15.169:6809/2751816702' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:31:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:01 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:01 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:01 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:01 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:01 smithi173 ceph-mon[33544]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:31:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:01 smithi173 ceph-mon[33544]: from='osd.2 v2:172.21.15.169:6809/2751816702' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:31:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:01 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:01 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:01.957 DEBUG:teuthology.orchestra.run.smithi169:osd.2> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.2.service 2024-03-12T03:31:01.959 INFO:tasks.cephadm:Deploying osd.3 on smithi169 with /dev/vg_nvme/lv_1... 2024-03-12T03:31:01.959 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-12T03:31:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:02 smithi169 ceph-mon[29344]: from='osd.2 v2:172.21.15.169:6809/2751816702' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:31:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:02 smithi169 ceph-mon[29344]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:31:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:02 smithi169 ceph-mon[29344]: from='osd.2 v2:172.21.15.169:6809/2751816702' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:31:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:02 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:02 smithi169 ceph-mon[39949]: from='osd.2 v2:172.21.15.169:6809/2751816702' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:31:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:02 smithi169 ceph-mon[39949]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:31:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:02 smithi169 ceph-mon[39949]: from='osd.2 v2:172.21.15.169:6809/2751816702' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:31:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:02 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:02.674 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:31:02 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2[56055]: 2024-03-12T03:31:02.339+0000 7ff1a1072700 -1 osd.2 0 waiting for initial osdmap 2024-03-12T03:31:02.674 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:31:02 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2[56055]: 2024-03-12T03:31:02.342+0000 7ff19ce65700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:31:02.679 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:31:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:02 smithi173 ceph-mon[33544]: from='osd.2 v2:172.21.15.169:6809/2751816702' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:31:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:02 smithi173 ceph-mon[33544]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:31:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:02 smithi173 ceph-mon[33544]: from='osd.2 v2:172.21.15.169:6809/2751816702' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:31:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:02 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:03.645 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:03 smithi169 ceph-mon[29344]: from='osd.2 v2:172.21.15.169:6809/2751816702' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-12T03:31:03.645 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:03 smithi169 ceph-mon[29344]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:31:03.646 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:03 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:03.646 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:03 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:03.646 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:03 smithi169 ceph-mon[29344]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:31:03.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:03 smithi169 ceph-mon[39949]: from='osd.2 v2:172.21.15.169:6809/2751816702' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-12T03:31:03.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:03 smithi169 ceph-mon[39949]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:31:03.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:03 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:03.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:03 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:03.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:03 smithi169 ceph-mon[39949]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:31:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:03 smithi173 ceph-mon[33544]: from='osd.2 v2:172.21.15.169:6809/2751816702' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-12T03:31:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:03 smithi173 ceph-mon[33544]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:31:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:03 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:03 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:03 smithi173 ceph-mon[33544]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:31:04.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:04 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:31:04.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:04 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:31:04.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:04 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:04.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:04 smithi169 ceph-mon[29344]: osd.2 v2:172.21.15.169:6809/2751816702 boot 2024-03-12T03:31:04.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:04 smithi169 ceph-mon[29344]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T03:31:04.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:04 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:04.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:04 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:31:04.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:04 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:31:04.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:04 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:04.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:04 smithi169 ceph-mon[39949]: osd.2 v2:172.21.15.169:6809/2751816702 boot 2024-03-12T03:31:04.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:04 smithi169 ceph-mon[39949]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T03:31:04.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:04 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:04 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:31:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:04 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:31:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:04 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:04 smithi173 ceph-mon[33544]: osd.2 v2:172.21.15.169:6809/2751816702 boot 2024-03-12T03:31:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:04 smithi173 ceph-mon[33544]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T03:31:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:04 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:31:05.236 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-12T03:31:05.236 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-12T03:31:05.237 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-03-12T03:31:05.237 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-03-12T03:31:05.237 INFO:teuthology.orchestra.run.smithi169.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125361 s, 836 MB/s 2024-03-12T03:31:05.237 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-03-12T03:31:05.543 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:05 smithi169 ceph-mon[29344]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:31:05.543 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:05 smithi169 ceph-mon[29344]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:05.543 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:31:05.543 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:05 smithi169 ceph-mon[39949]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:31:05.543 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:05 smithi169 ceph-mon[39949]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:05.543 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:05 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:31:05.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:05 smithi173 ceph-mon[33544]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:31:05.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:05 smithi173 ceph-mon[33544]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:05.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:05 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:31:06.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_1 2024-03-12T03:31:06.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T03:31:06.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:31:06.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:31:06.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T03:31:06.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:31:06.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:31:06.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:06.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:06.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:06.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:06.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:06.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:06.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:06.426 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:06.426 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.455 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:31:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T03:31:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:31:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:31:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:07.173 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59894]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:31:07.173 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59894]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:31:07.173 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59894]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:31:07.173 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59894]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:31:07.487 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:07 smithi169 ceph-mon[29344]: Detected new or changed devices on smithi169 2024-03-12T03:31:07.487 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:07 smithi169 ceph-mon[29344]: Adjusting osd_memory_target on smithi169 to 5354M 2024-03-12T03:31:07.487 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:07 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:31:07.487 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:07 smithi169 ceph-mon[29344]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:31:07.488 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:07 smithi169 ceph-mon[29344]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:07.488 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:07 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:07.488 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:07 smithi169 ceph-mon[39949]: Detected new or changed devices on smithi169 2024-03-12T03:31:07.488 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:07 smithi169 ceph-mon[39949]: Adjusting osd_memory_target on smithi169 to 5354M 2024-03-12T03:31:07.488 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:07 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:31:07.488 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:07 smithi169 ceph-mon[39949]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:31:07.488 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:07 smithi169 ceph-mon[39949]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:07.488 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:07 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:07.489 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59901]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:31:07.489 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59901]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:31:07.489 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59901]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:31:07.489 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59901]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:31:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:07 smithi173 ceph-mon[33544]: Detected new or changed devices on smithi169 2024-03-12T03:31:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:07 smithi173 ceph-mon[33544]: Adjusting osd_memory_target on smithi169 to 5354M 2024-03-12T03:31:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:07 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:31:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:07 smithi173 ceph-mon[33544]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:31:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:07 smithi173 ceph-mon[33544]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:07.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:07 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:07.782 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59906]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:31:07.782 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59906]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:31:07.783 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59906]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:31:07.783 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59906]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:31:08.052 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59932]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:31:08.053 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59932]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:31:08.053 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59932]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:31:08.053 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:31:07 smithi169 sudo[59932]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:31:08.319 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[59957]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:31:08.319 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[59957]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:31:08.319 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[59957]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:31:08.319 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[59957]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:31:08.575 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[60027]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:31:08.575 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[60027]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:31:08.575 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[60027]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:31:08.575 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[60027]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:31:08.831 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[60030]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:31:08.832 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[60030]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:31:08.832 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[60030]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:31:08.832 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[60030]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:31:09.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:08 smithi173 ceph-mon[33544]: osdmap e22: 3 total, 3 up, 3 in 2024-03-12T03:31:09.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:08 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:09.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:08 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:09.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:08 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:09.106 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:08 smithi173 ceph-mon[33544]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-12T03:31:09.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:08 smithi173 ceph-mon[33544]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:31:09.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 sudo[36126]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:31:09.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 sudo[36126]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:31:09.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 sudo[36126]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:31:09.107 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 sudo[36126]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:31:09.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:08 smithi169 ceph-mon[39949]: osdmap e22: 3 total, 3 up, 3 in 2024-03-12T03:31:09.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:08 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:09.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:08 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:09.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:08 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:09.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:08 smithi169 ceph-mon[39949]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-12T03:31:09.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:08 smithi169 ceph-mon[39949]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:31:09.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[60087]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:31:09.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[60087]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:31:09.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[60087]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:31:09.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 sudo[60087]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:31:09.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 ceph-mon[29344]: osdmap e22: 3 total, 3 up, 3 in 2024-03-12T03:31:09.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:09.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:09.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:09.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 ceph-mon[29344]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-12T03:31:09.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:08 smithi169 ceph-mon[29344]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:31:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 sudo[36129]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:31:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 sudo[36129]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:31:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 sudo[36129]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:31:09.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 sudo[36129]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:31:09.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 sudo[60110]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:31:09.875 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 sudo[60110]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:31:09.875 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 sudo[60110]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:31:09.875 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 sudo[60110]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:31:10.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:31:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:31:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:31:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:10.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:10.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 sudo[60134]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:31:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 sudo[60134]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:31:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 sudo[60134]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:31:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 sudo[60134]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:31:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:31:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:10.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:31:10.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:10.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:10.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:10.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:31:10.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:10.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:10.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:10.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:10.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:10.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:11.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:10 smithi169 ceph-mon[39949]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:31:11.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:10 smithi169 ceph-mon[39949]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:31:11.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:10 smithi169 ceph-mon[39949]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:10 smithi169 ceph-mon[29344]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:31:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:10 smithi169 ceph-mon[29344]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:31:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:10 smithi169 ceph-mon[29344]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:11.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:10 smithi173 ceph-mon[33544]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:31:11.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:10 smithi173 ceph-mon[33544]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:31:11.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:10 smithi173 ceph-mon[33544]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:12.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:11 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3434760935' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a00b4c50-e064-4c9e-85dc-b0a75ea5bf37"}]: dispatch 2024-03-12T03:31:12.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:11 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a00b4c50-e064-4c9e-85dc-b0a75ea5bf37"}]: dispatch 2024-03-12T03:31:12.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:11 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a00b4c50-e064-4c9e-85dc-b0a75ea5bf37"}]': finished 2024-03-12T03:31:12.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:11 smithi169 ceph-mon[29344]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T03:31:12.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:12.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:11 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1375119903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:31:12.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:11 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3434760935' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a00b4c50-e064-4c9e-85dc-b0a75ea5bf37"}]: dispatch 2024-03-12T03:31:12.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:11 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a00b4c50-e064-4c9e-85dc-b0a75ea5bf37"}]: dispatch 2024-03-12T03:31:12.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:11 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a00b4c50-e064-4c9e-85dc-b0a75ea5bf37"}]': finished 2024-03-12T03:31:12.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:11 smithi169 ceph-mon[39949]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T03:31:12.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:11 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:12.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:11 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1375119903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:31:12.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:11 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3434760935' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a00b4c50-e064-4c9e-85dc-b0a75ea5bf37"}]: dispatch 2024-03-12T03:31:12.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:11 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a00b4c50-e064-4c9e-85dc-b0a75ea5bf37"}]: dispatch 2024-03-12T03:31:12.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:11 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a00b4c50-e064-4c9e-85dc-b0a75ea5bf37"}]': finished 2024-03-12T03:31:12.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:11 smithi173 ceph-mon[33544]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T03:31:12.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:11 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:12.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:11 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1375119903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:31:13.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:12 smithi169 ceph-mon[39949]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:12 smithi169 ceph-mon[29344]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:13.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:12 smithi173 ceph-mon[33544]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:15.874 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:15 smithi169 ceph-mon[29344]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-12T03:31:15.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:15 smithi169 ceph-mon[39949]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-12T03:31:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:15 smithi173 ceph-mon[33544]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-12T03:31:17.873 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:17 smithi169 ceph-mon[29344]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-12T03:31:17.873 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T03:31:17.874 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:17.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:17 smithi169 ceph-mon[39949]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-12T03:31:17.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:17 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T03:31:17.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:17 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:17 smithi173 ceph-mon[33544]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-12T03:31:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:17 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T03:31:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:17 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:18.766 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:18 smithi169 ceph-mon[29344]: Deploying daemon osd.3 on smithi169 2024-03-12T03:31:18.766 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:18 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:31:18.766 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:18 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:31:18.766 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:18 smithi169 ceph-mon[39949]: Deploying daemon osd.3 on smithi169 2024-03-12T03:31:18.767 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:18 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:31:18.767 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:18 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:31:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:18 smithi173 ceph-mon[33544]: Deploying daemon osd.3 on smithi169 2024-03-12T03:31:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:18 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:31:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:18 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:31:19.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:19 smithi169 ceph-mon[39949]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-12T03:31:19.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:19 smithi169 ceph-mon[39949]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T03:31:19.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:19 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:19 smithi169 ceph-mon[29344]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-12T03:31:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:19 smithi169 ceph-mon[29344]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T03:31:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:19 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:19 smithi173 ceph-mon[33544]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-12T03:31:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:19 smithi173 ceph-mon[33544]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T03:31:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:19 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:21.787 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:21 smithi169 ceph-mon[29344]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 82 KiB/s wr, 5 op/s 2024-03-12T03:31:21.788 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:21 smithi169 ceph-mon[39949]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 82 KiB/s wr, 5 op/s 2024-03-12T03:31:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:21 smithi173 ceph-mon[33544]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 82 KiB/s wr, 5 op/s 2024-03-12T03:31:23.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:23 smithi169 ceph-mon[39949]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-12T03:31:23.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:23 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:23.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:23 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:23.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:23 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:23 smithi169 ceph-mon[29344]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-12T03:31:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:23 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:23 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:23 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:23 smithi173 ceph-mon[33544]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-12T03:31:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:23 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:23 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:23 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:25.793 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:25 smithi169 ceph-mon[29344]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:25.793 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:25 smithi169 ceph-mon[39949]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:25.869 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 3 on host 'smithi169' 2024-03-12T03:31:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:25 smithi173 ceph-mon[33544]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:26.467 DEBUG:teuthology.orchestra.run.smithi169:osd.3> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3.service 2024-03-12T03:31:26.469 INFO:tasks.cephadm:Deploying osd.4 on smithi173 with /dev/vg_nvme/lv_4... 2024-03-12T03:31:26.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T03:31:26.713 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:31:26.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:26 smithi173 ceph-mon[33544]: from='osd.3 v2:172.21.15.169:6813/195572266' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:31:26.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:26 smithi173 ceph-mon[33544]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:31:26.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:26 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:26.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:26 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:26.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:26 smithi169 ceph-mon[29344]: from='osd.3 v2:172.21.15.169:6813/195572266' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:31:26.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:26 smithi169 ceph-mon[29344]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:31:26.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:26 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:26.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:26 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:26.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:26 smithi169 ceph-mon[39949]: from='osd.3 v2:172.21.15.169:6813/195572266' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:31:26.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:26 smithi169 ceph-mon[39949]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:31:26.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:26 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:26.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:26 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:27.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:27 smithi173 ceph-mon[33544]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:27.889 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:27 smithi173 ceph-mon[33544]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:31:27.889 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:27 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:27.889 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:27 smithi173 ceph-mon[33544]: osdmap e25: 4 total, 3 up, 4 in 2024-03-12T03:31:27.889 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:27 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:27.889 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:27 smithi173 ceph-mon[33544]: from='osd.3 v2:172.21.15.169:6813/195572266' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:31:27.889 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:27 smithi173 ceph-mon[33544]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:31:27.889 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:27 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[29344]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[29344]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:31:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[29344]: osdmap e25: 4 total, 3 up, 4 in 2024-03-12T03:31:27.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:27.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[29344]: from='osd.3 v2:172.21.15.169:6813/195572266' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:31:27.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[29344]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:31:27.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[39949]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[39949]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:31:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[39949]: osdmap e25: 4 total, 3 up, 4 in 2024-03-12T03:31:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[39949]: from='osd.3 v2:172.21.15.169:6813/195572266' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:31:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[39949]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:31:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:27.925 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[61852]: 2024-03-12T03:31:27.638+0000 7f07f302a700 -1 osd.3 0 waiting for initial osdmap 2024-03-12T03:31:27.925 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:31:27 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[61852]: 2024-03-12T03:31:27.641+0000 7f07eee1d700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:31:28.555 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T03:31:28.555 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T03:31:28.555 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-12T03:31:28.555 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-12T03:31:28.555 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128721 s, 815 MB/s 2024-03-12T03:31:28.555 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-12T03:31:28.861 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:28 smithi173 ceph-mon[33544]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-12T03:31:28.861 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:28 smithi173 ceph-mon[33544]: osdmap e26: 4 total, 3 up, 4 in 2024-03-12T03:31:28.861 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:28 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:28.861 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:28 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:28 smithi169 ceph-mon[29344]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-12T03:31:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:28 smithi169 ceph-mon[29344]: osdmap e26: 4 total, 3 up, 4 in 2024-03-12T03:31:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:28.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:28 smithi169 ceph-mon[39949]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-12T03:31:28.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:28 smithi169 ceph-mon[39949]: osdmap e26: 4 total, 3 up, 4 in 2024-03-12T03:31:28.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:28 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:28.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:28 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:29.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-03-12T03:31:29.431 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:31:29.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:29 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:31:29.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:29 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:31:29.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:29 smithi173 ceph-mon[33544]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:29.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:29 smithi173 ceph-mon[33544]: osd.3 v2:172.21.15.169:6813/195572266 boot 2024-03-12T03:31:29.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:29 smithi173 ceph-mon[33544]: osdmap e27: 4 total, 4 up, 4 in 2024-03-12T03:31:29.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:29 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:29.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:29 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:31:29.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:29 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:31:29.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:29 smithi169 ceph-mon[39949]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:29.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:29 smithi169 ceph-mon[39949]: osd.3 v2:172.21.15.169:6813/195572266 boot 2024-03-12T03:31:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:29 smithi169 ceph-mon[39949]: osdmap e27: 4 total, 4 up, 4 in 2024-03-12T03:31:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:29 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:29 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:31:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:29 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:31:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:29 smithi169 ceph-mon[29344]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:31:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:29 smithi169 ceph-mon[29344]: osd.3 v2:172.21.15.169:6813/195572266 boot 2024-03-12T03:31:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:29 smithi169 ceph-mon[29344]: osdmap e27: 4 total, 4 up, 4 in 2024-03-12T03:31:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:29 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:31:30.661 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: osdmap e28: 4 total, 4 up, 4 in 2024-03-12T03:31:30.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: Detected new or changed devices on smithi169 2024-03-12T03:31:30.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:30.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:30.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:30.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:30.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: Adjusting osd_memory_target on smithi169 to 4016M 2024-03-12T03:31:30.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:30.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:30.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.663 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:30.663 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:30.663 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:30.663 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.663 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:30 smithi173 ceph-mon[33544]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:30.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: osdmap e28: 4 total, 4 up, 4 in 2024-03-12T03:31:30.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: Detected new or changed devices on smithi169 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: Adjusting osd_memory_target on smithi169 to 4016M 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:30.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[39949]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:30.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: osdmap e28: 4 total, 4 up, 4 in 2024-03-12T03:31:30.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: Detected new or changed devices on smithi169 2024-03-12T03:31:30.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:30.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:30.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:30.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:30.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: Adjusting osd_memory_target on smithi169 to 4016M 2024-03-12T03:31:30.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:30.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:30.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:30.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:30.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:30.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:30.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:30 smithi169 ceph-mon[29344]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:31.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:31 smithi169 ceph-mon[29344]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:31.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:31 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:31.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:31 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:31.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:31 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:31.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:31 smithi169 ceph-mon[39949]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:31.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:31 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:31.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:31 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:31.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:31 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:31.950 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:31 smithi173 ceph-mon[33544]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:31.950 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:31 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:31.951 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:31 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:31.951 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:31 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:32.921 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:32 smithi173 ceph-mon[33544]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:32.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:32 smithi169 ceph-mon[29344]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:32.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:32 smithi169 ceph-mon[39949]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:34.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:33 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/2628791468' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a6121ef-6309-41b7-a964-6772c010b3a0"}]: dispatch 2024-03-12T03:31:34.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:33 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a6121ef-6309-41b7-a964-6772c010b3a0"}]: dispatch 2024-03-12T03:31:34.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:33 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a6121ef-6309-41b7-a964-6772c010b3a0"}]': finished 2024-03-12T03:31:34.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:33 smithi169 ceph-mon[29344]: osdmap e29: 5 total, 4 up, 5 in 2024-03-12T03:31:34.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:33 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:33 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/2628791468' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a6121ef-6309-41b7-a964-6772c010b3a0"}]: dispatch 2024-03-12T03:31:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:33 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a6121ef-6309-41b7-a964-6772c010b3a0"}]: dispatch 2024-03-12T03:31:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:33 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a6121ef-6309-41b7-a964-6772c010b3a0"}]': finished 2024-03-12T03:31:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:33 smithi169 ceph-mon[39949]: osdmap e29: 5 total, 4 up, 5 in 2024-03-12T03:31:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:33 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:33 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/2628791468' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a6121ef-6309-41b7-a964-6772c010b3a0"}]: dispatch 2024-03-12T03:31:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:33 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a6121ef-6309-41b7-a964-6772c010b3a0"}]: dispatch 2024-03-12T03:31:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:33 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a6121ef-6309-41b7-a964-6772c010b3a0"}]': finished 2024-03-12T03:31:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:33 smithi173 ceph-mon[33544]: osdmap e29: 5 total, 4 up, 5 in 2024-03-12T03:31:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:33 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:35.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:34 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/2963584885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:31:35.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:34 smithi169 ceph-mon[29344]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:35.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:34 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/2963584885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:31:35.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:34 smithi169 ceph-mon[39949]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:35.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:34 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/2963584885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:31:35.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:34 smithi173 ceph-mon[33544]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:37.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:37 smithi173 ceph-mon[33544]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:37.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:37 smithi169 ceph-mon[29344]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:37.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:37 smithi169 ceph-mon[39949]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:39 smithi173 ceph-mon[33544]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:39.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:39 smithi169 ceph-mon[29344]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:39.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:39 smithi169 ceph-mon[39949]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:40.718 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:40 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T03:31:40.740 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:40 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:40.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:40 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T03:31:40.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:40 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:40.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:40 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T03:31:40.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:40 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:41.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:41 smithi173 ceph-mon[33544]: Deploying daemon osd.4 on smithi173 2024-03-12T03:31:41.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:41 smithi173 ceph-mon[33544]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:41.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:41 smithi169 ceph-mon[29344]: Deploying daemon osd.4 on smithi173 2024-03-12T03:31:41.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:41 smithi169 ceph-mon[29344]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:41.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:41 smithi169 ceph-mon[39949]: Deploying daemon osd.4 on smithi173 2024-03-12T03:31:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:41 smithi169 ceph-mon[39949]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:43.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:43 smithi173 ceph-mon[33544]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:43 smithi169 ceph-mon[29344]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:43.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:43 smithi169 ceph-mon[39949]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:45 smithi173 ceph-mon[33544]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:45 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:45 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:45 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:45 smithi169 ceph-mon[29344]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:45 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:45 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:45 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:45 smithi169 ceph-mon[39949]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:45 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:45 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:45 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:47.876 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:47 smithi173 ceph-mon[33544]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:47 smithi169 ceph-mon[29344]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:47.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:47 smithi169 ceph-mon[39949]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:49.080 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:48 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:49.080 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:48 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:49.080 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:48 smithi173 ceph-mon[33544]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:49.080 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:48 smithi173 ceph-mon[33544]: from='osd.4 v2:172.21.15.173:6800/1397212305' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:31:49.081 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:48 smithi173 ceph-mon[33544]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:31:49.081 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:48 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:49.081 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:48 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:49.081 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:48 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:49.081 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 4 on host 'smithi173' 2024-03-12T03:31:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[29344]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[29344]: from='osd.4 v2:172.21.15.173:6800/1397212305' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:31:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[29344]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:31:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[39949]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[39949]: from='osd.4 v2:172.21.15.173:6800/1397212305' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:31:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[39949]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:31:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:48 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:49.693 DEBUG:teuthology.orchestra.run.smithi173:osd.4> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.4.service 2024-03-12T03:31:49.695 INFO:tasks.cephadm:Deploying osd.5 on smithi173 with /dev/vg_nvme/lv_3... 2024-03-12T03:31:49.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T03:31:50.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:50 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:50.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:50 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:50 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:50 smithi173 ceph-mon[33544]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T03:31:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:50 smithi173 ceph-mon[33544]: from='osd.4 v2:172.21.15.173:6800/1397212305' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:31:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:50 smithi173 ceph-mon[33544]: osdmap e30: 5 total, 4 up, 5 in 2024-03-12T03:31:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:50 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:50 smithi173 ceph-mon[33544]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:31:50.237 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:31:50 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:31:50.117+0000 7fe0d8177700 -1 osd.4 0 waiting for initial osdmap 2024-03-12T03:31:50.237 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:31:50 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:31:50.121+0000 7fe0d3f6a700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:31:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:50.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:50.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:50.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[29344]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T03:31:50.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[29344]: from='osd.4 v2:172.21.15.173:6800/1397212305' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:31:50.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[29344]: osdmap e30: 5 total, 4 up, 5 in 2024-03-12T03:31:50.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:50.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[29344]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:31:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[39949]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T03:31:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[39949]: from='osd.4 v2:172.21.15.173:6800/1397212305' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:31:50.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[39949]: osdmap e30: 5 total, 4 up, 5 in 2024-03-12T03:31:50.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:50.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:50 smithi169 ceph-mon[39949]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:31:50.790 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:31:51.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:51 smithi169 ceph-mon[29344]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-12T03:31:51.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:51 smithi169 ceph-mon[29344]: osdmap e31: 5 total, 4 up, 5 in 2024-03-12T03:31:51.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:51 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:51.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:51 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:51.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:51 smithi169 ceph-mon[29344]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:51.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:51 smithi169 ceph-mon[39949]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-12T03:31:51.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:51 smithi169 ceph-mon[39949]: osdmap e31: 5 total, 4 up, 5 in 2024-03-12T03:31:51.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:51 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:51.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:51 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:51.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:51 smithi169 ceph-mon[39949]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:51.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:51 smithi173 ceph-mon[33544]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-12T03:31:51.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:51 smithi173 ceph-mon[33544]: osdmap e31: 5 total, 4 up, 5 in 2024-03-12T03:31:51.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:51 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:51.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:51 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:51.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:51 smithi173 ceph-mon[33544]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:31:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:52 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:31:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:52 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:31:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:52 smithi173 ceph-mon[33544]: osd.4 v2:172.21.15.173:6800/1397212305 boot 2024-03-12T03:31:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:52 smithi173 ceph-mon[33544]: osdmap e32: 5 total, 5 up, 5 in 2024-03-12T03:31:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:52 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:52.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:52 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:31:52.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:52 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:31:52.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:52 smithi169 ceph-mon[29344]: osd.4 v2:172.21.15.173:6800/1397212305 boot 2024-03-12T03:31:52.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:52 smithi169 ceph-mon[29344]: osdmap e32: 5 total, 5 up, 5 in 2024-03-12T03:31:52.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:52 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:52.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:52 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:31:52.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:52 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:31:52.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:52 smithi169 ceph-mon[39949]: osd.4 v2:172.21.15.173:6800/1397212305 boot 2024-03-12T03:31:52.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:52 smithi169 ceph-mon[39949]: osdmap e32: 5 total, 5 up, 5 in 2024-03-12T03:31:52.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:52 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:31:53.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:53 smithi169 ceph-mon[29344]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T03:31:53.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:53 smithi169 ceph-mon[29344]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:31:53.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:53 smithi169 ceph-mon[29344]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T03:31:53.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:53 smithi169 ceph-mon[39949]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T03:31:53.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:53 smithi169 ceph-mon[39949]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:31:53.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:53 smithi169 ceph-mon[39949]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T03:31:53.471 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:53 smithi173 ceph-mon[33544]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T03:31:53.471 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:53 smithi173 ceph-mon[33544]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:31:53.471 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:53 smithi173 ceph-mon[33544]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T03:31:53.472 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T03:31:53.472 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T03:31:53.472 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-12T03:31:53.472 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-12T03:31:53.472 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111803 s, 938 MB/s 2024-03-12T03:31:53.472 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 2024-03-12T03:31:53.472 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-12T03:31:54.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-03-12T03:31:54.383 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:31:54.789 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:54 smithi173 ceph-mon[33544]: Detected new or changed devices on smithi173 2024-03-12T03:31:54.789 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:54 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:54.789 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:54 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:54.790 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:54 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:54.790 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:54 smithi173 ceph-mon[33544]: Adjusting osd_memory_target on smithi173 to 17088M 2024-03-12T03:31:54.790 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:54 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:54.790 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:54 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:54.790 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:54 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:54.790 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:54 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:54.790 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:54 smithi173 ceph-mon[33544]: osdmap e35: 5 total, 5 up, 5 in 2024-03-12T03:31:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[29344]: Detected new or changed devices on smithi173 2024-03-12T03:31:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[29344]: Adjusting osd_memory_target on smithi173 to 17088M 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[29344]: osdmap e35: 5 total, 5 up, 5 in 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[39949]: Detected new or changed devices on smithi173 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[39949]: Adjusting osd_memory_target on smithi173 to 17088M 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:31:54.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:54 smithi169 ceph-mon[39949]: osdmap e35: 5 total, 5 up, 5 in 2024-03-12T03:31:55.704 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:55 smithi173 ceph-mon[33544]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T03:31:55.704 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:55 smithi173 ceph-mon[33544]: osdmap e36: 5 total, 5 up, 5 in 2024-03-12T03:31:55.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:55 smithi169 ceph-mon[29344]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T03:31:55.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:55 smithi169 ceph-mon[29344]: osdmap e36: 5 total, 5 up, 5 in 2024-03-12T03:31:55.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:55 smithi169 ceph-mon[39949]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T03:31:55.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:55 smithi169 ceph-mon[39949]: osdmap e36: 5 total, 5 up, 5 in 2024-03-12T03:31:56.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:56 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:56.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:56 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:56.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:56 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:56 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:56 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:56 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:56.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:56 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:56.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:56 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:56.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:56 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:57.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:57 smithi173 ceph-mon[33544]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:57.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:57 smithi173 ceph-mon[33544]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-12T03:31:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:57 smithi169 ceph-mon[29344]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:57 smithi169 ceph-mon[29344]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-12T03:31:57.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:57 smithi169 ceph-mon[39949]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:57.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:57 smithi169 ceph-mon[39949]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-12T03:31:59.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:59 smithi169 ceph-mon[29344]: pgmap v121: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:31:59.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:59 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/1342500423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c2f6dd9-9784-4a07-8710-55ac3de82964"}]: dispatch 2024-03-12T03:31:59.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:59 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c2f6dd9-9784-4a07-8710-55ac3de82964"}]: dispatch 2024-03-12T03:31:59.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:59 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c2f6dd9-9784-4a07-8710-55ac3de82964"}]': finished 2024-03-12T03:31:59.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:59 smithi169 ceph-mon[29344]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T03:31:59.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:31:59 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:31:59.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:59 smithi169 ceph-mon[39949]: pgmap v121: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:31:59.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:59 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/1342500423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c2f6dd9-9784-4a07-8710-55ac3de82964"}]: dispatch 2024-03-12T03:31:59.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:59 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c2f6dd9-9784-4a07-8710-55ac3de82964"}]: dispatch 2024-03-12T03:31:59.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:59 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c2f6dd9-9784-4a07-8710-55ac3de82964"}]': finished 2024-03-12T03:31:59.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:59 smithi169 ceph-mon[39949]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T03:31:59.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:31:59 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:31:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:59 smithi173 ceph-mon[33544]: pgmap v121: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:31:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:59 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/1342500423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c2f6dd9-9784-4a07-8710-55ac3de82964"}]: dispatch 2024-03-12T03:31:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:59 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c2f6dd9-9784-4a07-8710-55ac3de82964"}]: dispatch 2024-03-12T03:31:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:59 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c2f6dd9-9784-4a07-8710-55ac3de82964"}]': finished 2024-03-12T03:31:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:59 smithi173 ceph-mon[33544]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T03:31:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:31:59 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:00.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:00 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/534453157' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:00.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:00 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/534453157' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:00.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:00 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/534453157' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:01.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:01 smithi169 ceph-mon[29344]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:01.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:01 smithi169 ceph-mon[39949]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:01 smithi173 ceph-mon[33544]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:03.877 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:03 smithi173 ceph-mon[33544]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:03.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:03 smithi169 ceph-mon[29344]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:03.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:03 smithi169 ceph-mon[39949]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:05.872 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:05 smithi173 ceph-mon[33544]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:05.872 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:05 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T03:32:05.873 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:05 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:05 smithi169 ceph-mon[29344]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T03:32:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:05 smithi169 ceph-mon[39949]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:05 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T03:32:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:05 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:06 smithi173 ceph-mon[33544]: Deploying daemon osd.5 on smithi173 2024-03-12T03:32:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:06 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:32:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:06 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:32:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:06 smithi169 ceph-mon[29344]: Deploying daemon osd.5 on smithi173 2024-03-12T03:32:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:06 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:32:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:06 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:32:06.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:06 smithi169 ceph-mon[39949]: Deploying daemon osd.5 on smithi173 2024-03-12T03:32:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:06 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:32:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:06 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:32:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:07 smithi173 ceph-mon[33544]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:07 smithi173 ceph-mon[33544]: osdmap e38: 6 total, 5 up, 6 in 2024-03-12T03:32:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:07 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:07 smithi169 ceph-mon[29344]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:07 smithi169 ceph-mon[29344]: osdmap e38: 6 total, 5 up, 6 in 2024-03-12T03:32:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:07 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:07 smithi169 ceph-mon[39949]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:07 smithi169 ceph-mon[39949]: osdmap e38: 6 total, 5 up, 6 in 2024-03-12T03:32:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:07 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:09.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:09 smithi169 ceph-mon[29344]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:09.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:09 smithi169 ceph-mon[39949]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:09 smithi173 ceph-mon[33544]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:11.735 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:11 smithi173 ceph-mon[33544]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:11.735 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:11 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:11.735 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:11 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:11.735 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:11 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:11.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:11 smithi169 ceph-mon[29344]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:11.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:11.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:11.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:11 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:11 smithi169 ceph-mon[39949]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:11 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:11 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:11 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:13.664 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:13 smithi173 ceph-mon[33544]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:13.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:13 smithi169 ceph-mon[29344]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:13.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:13 smithi169 ceph-mon[39949]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:14.012 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 5 on host 'smithi173' 2024-03-12T03:32:14.602 DEBUG:teuthology.orchestra.run.smithi173:osd.5> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.5.service 2024-03-12T03:32:14.604 INFO:tasks.cephadm:Deploying osd.6 on smithi173 with /dev/vg_nvme/lv_2... 2024-03-12T03:32:14.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T03:32:14.888 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:32:14.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:14 smithi169 ceph-mon[39949]: from='osd.5 v2:172.21.15.173:6804/3525888417' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:32:14.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:14 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:14.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:14 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:14 smithi169 ceph-mon[29344]: from='osd.5 v2:172.21.15.173:6804/3525888417' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:32:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:14 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:14 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:14 smithi173 ceph-mon[33544]: from='osd.5 v2:172.21.15.173:6804/3525888417' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:32:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:14 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:14 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:15.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[39949]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:15.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[39949]: from='osd.5 v2:172.21.15.173:6804/3525888417' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T03:32:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[39949]: osdmap e39: 6 total, 5 up, 6 in 2024-03-12T03:32:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[39949]: from='osd.5 v2:172.21.15.173:6804/3525888417' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:32:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[29344]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[29344]: from='osd.5 v2:172.21.15.173:6804/3525888417' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T03:32:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[29344]: osdmap e39: 6 total, 5 up, 6 in 2024-03-12T03:32:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[29344]: from='osd.5 v2:172.21.15.173:6804/3525888417' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:32:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:15 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:15 smithi173 ceph-mon[33544]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:15 smithi173 ceph-mon[33544]: from='osd.5 v2:172.21.15.173:6804/3525888417' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T03:32:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:15 smithi173 ceph-mon[33544]: osdmap e39: 6 total, 5 up, 6 in 2024-03-12T03:32:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:15 smithi173 ceph-mon[33544]: from='osd.5 v2:172.21.15.173:6804/3525888417' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:32:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:15 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:15 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:15 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:15.987 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:32:15 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:32:15.630+0000 7fe261692700 -1 osd.5 0 waiting for initial osdmap 2024-03-12T03:32:15.987 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:32:15 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:32:15.636+0000 7fe25d485700 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:32:16.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:16 smithi169 ceph-mon[29344]: from='osd.5 v2:172.21.15.173:6804/3525888417' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-12T03:32:16.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:16 smithi169 ceph-mon[29344]: osdmap e40: 6 total, 5 up, 6 in 2024-03-12T03:32:16.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:16 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:16.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:16 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:16.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:16 smithi169 ceph-mon[39949]: from='osd.5 v2:172.21.15.173:6804/3525888417' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-12T03:32:16.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:16 smithi169 ceph-mon[39949]: osdmap e40: 6 total, 5 up, 6 in 2024-03-12T03:32:16.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:16 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:16.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:16 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:16.952 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:16 smithi173 ceph-mon[33544]: from='osd.5 v2:172.21.15.173:6804/3525888417' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-12T03:32:16.953 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:16 smithi173 ceph-mon[33544]: osdmap e40: 6 total, 5 up, 6 in 2024-03-12T03:32:16.953 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:16 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:16.953 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:16 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:17.504 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T03:32:17.504 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T03:32:17.504 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-12T03:32:17.504 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-12T03:32:17.504 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127203 s, 824 MB/s 2024-03-12T03:32:17.504 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-12T03:32:17.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:17 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:32:17.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:17 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:32:17.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:17 smithi169 ceph-mon[29344]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:17.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:17 smithi169 ceph-mon[29344]: osd.5 v2:172.21.15.173:6804/3525888417 boot 2024-03-12T03:32:17.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:17 smithi169 ceph-mon[29344]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T03:32:17.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:17 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:32:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:17 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:32:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:17 smithi169 ceph-mon[39949]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:17 smithi169 ceph-mon[39949]: osd.5 v2:172.21.15.173:6804/3525888417 boot 2024-03-12T03:32:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:17 smithi169 ceph-mon[39949]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T03:32:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:17 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:17 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:32:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:17 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:32:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:17 smithi173 ceph-mon[33544]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:32:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:17 smithi173 ceph-mon[33544]: osd.5 v2:172.21.15.173:6804/3525888417 boot 2024-03-12T03:32:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:17 smithi173 ceph-mon[33544]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T03:32:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:17 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:32:18.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-03-12T03:32:18.786 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:32:18.900 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:18 smithi173 ceph-mon[33544]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T03:32:18.900 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:18 smithi173 ceph-mon[33544]: osdmap e43: 6 total, 6 up, 6 in 2024-03-12T03:32:18.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:18 smithi169 ceph-mon[29344]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T03:32:18.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:18 smithi169 ceph-mon[29344]: osdmap e43: 6 total, 6 up, 6 in 2024-03-12T03:32:18.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:18 smithi169 ceph-mon[39949]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T03:32:18.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:18 smithi169 ceph-mon[39949]: osdmap e43: 6 total, 6 up, 6 in 2024-03-12T03:32:19.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:19 smithi169 ceph-mon[29344]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:19.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:19 smithi169 ceph-mon[39949]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:19.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:19 smithi173 ceph-mon[33544]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:20.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:32:20.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:32:20.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: osdmap e44: 6 total, 6 up, 6 in 2024-03-12T03:32:20.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:20.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:20.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:20.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:20.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:20.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:20.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:20.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:32:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:32:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: osdmap e44: 6 total, 6 up, 6 in 2024-03-12T03:32:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:32:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: osdmap e44: 6 total, 6 up, 6 in 2024-03-12T03:32:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:20.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:21.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:21 smithi173 ceph-mon[33544]: Detected new or changed devices on smithi173 2024-03-12T03:32:21.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:21 smithi173 ceph-mon[33544]: Adjusting osd_memory_target on smithi173 to 8544M 2024-03-12T03:32:21.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:21 smithi173 ceph-mon[33544]: pgmap v140: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:21.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:21 smithi173 ceph-mon[33544]: osdmap e45: 6 total, 6 up, 6 in 2024-03-12T03:32:21.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:21 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:21.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:21 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:21.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:21 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:21.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[29344]: Detected new or changed devices on smithi173 2024-03-12T03:32:21.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[29344]: Adjusting osd_memory_target on smithi173 to 8544M 2024-03-12T03:32:21.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[29344]: pgmap v140: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:21.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[29344]: osdmap e45: 6 total, 6 up, 6 in 2024-03-12T03:32:21.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:21.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:21.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:21.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[39949]: Detected new or changed devices on smithi173 2024-03-12T03:32:21.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[39949]: Adjusting osd_memory_target on smithi173 to 8544M 2024-03-12T03:32:21.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[39949]: pgmap v140: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:21.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[39949]: osdmap e45: 6 total, 6 up, 6 in 2024-03-12T03:32:21.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:21.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:21.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:21 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:22.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:22 smithi169 ceph-mon[29344]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:22.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:22 smithi169 ceph-mon[39949]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:22 smithi173 ceph-mon[33544]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:23 smithi169 ceph-mon[29344]: 1.0 scrub starts 2024-03-12T03:32:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:23 smithi169 ceph-mon[29344]: 1.0 scrub ok 2024-03-12T03:32:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:23 smithi169 ceph-mon[29344]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:23 smithi169 ceph-mon[39949]: 1.0 scrub starts 2024-03-12T03:32:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:23 smithi169 ceph-mon[39949]: 1.0 scrub ok 2024-03-12T03:32:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:23 smithi169 ceph-mon[39949]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:23 smithi173 ceph-mon[33544]: 1.0 scrub starts 2024-03-12T03:32:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:23 smithi173 ceph-mon[33544]: 1.0 scrub ok 2024-03-12T03:32:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:23 smithi173 ceph-mon[33544]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/1540141260' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7759df7-1978-4d70-94a2-721540804f59"}]: dispatch 2024-03-12T03:32:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7759df7-1978-4d70-94a2-721540804f59"}]: dispatch 2024-03-12T03:32:24.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7759df7-1978-4d70-94a2-721540804f59"}]': finished 2024-03-12T03:32:24.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[29344]: osdmap e46: 7 total, 6 up, 7 in 2024-03-12T03:32:24.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:24.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/2185881291' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:24.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[29344]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/1540141260' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7759df7-1978-4d70-94a2-721540804f59"}]: dispatch 2024-03-12T03:32:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7759df7-1978-4d70-94a2-721540804f59"}]: dispatch 2024-03-12T03:32:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7759df7-1978-4d70-94a2-721540804f59"}]': finished 2024-03-12T03:32:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[39949]: osdmap e46: 7 total, 6 up, 7 in 2024-03-12T03:32:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/2185881291' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:24 smithi169 ceph-mon[39949]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:24 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/1540141260' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7759df7-1978-4d70-94a2-721540804f59"}]: dispatch 2024-03-12T03:32:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:24 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7759df7-1978-4d70-94a2-721540804f59"}]: dispatch 2024-03-12T03:32:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:24 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7759df7-1978-4d70-94a2-721540804f59"}]': finished 2024-03-12T03:32:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:24 smithi173 ceph-mon[33544]: osdmap e46: 7 total, 6 up, 7 in 2024-03-12T03:32:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:24 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:24.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:24 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/2185881291' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:24.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:24 smithi173 ceph-mon[33544]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:27 smithi169 ceph-mon[29344]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:27.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:27 smithi169 ceph-mon[39949]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:27.969 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:27 smithi173 ceph-mon[33544]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:29 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:32:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:29 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:32:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:29 smithi169 ceph-mon[29344]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:29 smithi169 ceph-mon[29344]: osdmap e47: 7 total, 6 up, 7 in 2024-03-12T03:32:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:29 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:29 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:32:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:29 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:32:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:29 smithi169 ceph-mon[39949]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:29 smithi169 ceph-mon[39949]: osdmap e47: 7 total, 6 up, 7 in 2024-03-12T03:32:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:29 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:29 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:32:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:29 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:32:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:29 smithi173 ceph-mon[33544]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:29 smithi173 ceph-mon[33544]: osdmap e47: 7 total, 6 up, 7 in 2024-03-12T03:32:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:29 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:30.664 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T03:32:30.664 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:30 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T03:32:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:30 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T03:32:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:30 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:31.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:31 smithi169 ceph-mon[29344]: Deploying daemon osd.6 on smithi173 2024-03-12T03:32:32.021 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:31 smithi169 ceph-mon[29344]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:32.022 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:31 smithi169 ceph-mon[39949]: Deploying daemon osd.6 on smithi173 2024-03-12T03:32:32.022 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:31 smithi169 ceph-mon[39949]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:32.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:31 smithi173 ceph-mon[33544]: Deploying daemon osd.6 on smithi173 2024-03-12T03:32:32.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:31 smithi173 ceph-mon[33544]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:33.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:33 smithi173 ceph-mon[33544]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:33.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:33 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:32:33.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:33 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:32:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:33 smithi173 ceph-mon[33544]: osdmap e48: 7 total, 6 up, 7 in 2024-03-12T03:32:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:33 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:33.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:33 smithi169 ceph-mon[29344]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:33.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:33 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:32:33.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:33 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:32:33.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:33 smithi169 ceph-mon[29344]: osdmap e48: 7 total, 6 up, 7 in 2024-03-12T03:32:33.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:33 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:33.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:33 smithi169 ceph-mon[39949]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:33.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:33 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:32:33.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:33 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:32:33.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:33 smithi169 ceph-mon[39949]: osdmap e48: 7 total, 6 up, 7 in 2024-03-12T03:32:33.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:33 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:35.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:35 smithi169 ceph-mon[29344]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:35.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:35 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:35 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:35 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:35.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:35 smithi169 ceph-mon[39949]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:35.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:35 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:35.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:35 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:35.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:35 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:35 smithi173 ceph-mon[33544]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:35 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:35 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:35 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:37 smithi173 ceph-mon[33544]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:37.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:37 smithi169 ceph-mon[29344]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:37.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:37 smithi169 ceph-mon[39949]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:39.221 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 6 on host 'smithi173' 2024-03-12T03:32:39.858 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.858 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.858 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:39.858 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:39.858 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.858 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:39 smithi173 ceph-mon[33544]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:39.858 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:39 smithi173 ceph-mon[33544]: from='osd.6 v2:172.21.15.173:6808/2147192629' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:32:39.858 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:39 smithi173 ceph-mon[33544]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:32:39.858 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.858 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.859 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:39.889 DEBUG:teuthology.orchestra.run.smithi173:osd.6> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.6.service 2024-03-12T03:32:39.892 INFO:tasks.cephadm:Deploying osd.7 on smithi173 with /dev/vg_nvme/lv_1... 2024-03-12T03:32:39.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-12T03:32:39.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:39.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:39.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[29344]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:39.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[29344]: from='osd.6 v2:172.21.15.173:6808/2147192629' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:32:39.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[29344]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:32:39.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:39.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:39.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:39.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[39949]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:39.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[39949]: from='osd.6 v2:172.21.15.173:6808/2147192629' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:32:39.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[39949]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:32:39.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:39.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:40.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:40 smithi169 ceph-mon[29344]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T03:32:40.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:40 smithi169 ceph-mon[29344]: osdmap e49: 7 total, 6 up, 7 in 2024-03-12T03:32:40.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:40 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:40.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:40 smithi169 ceph-mon[29344]: from='osd.6 v2:172.21.15.173:6808/2147192629' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:32:40.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:40 smithi169 ceph-mon[29344]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:32:40.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:40 smithi169 ceph-mon[39949]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T03:32:40.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:40 smithi169 ceph-mon[39949]: osdmap e49: 7 total, 6 up, 7 in 2024-03-12T03:32:40.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:40 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:40.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:40 smithi169 ceph-mon[39949]: from='osd.6 v2:172.21.15.173:6808/2147192629' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:32:40.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:40 smithi169 ceph-mon[39949]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:32:40.981 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:40 smithi173 ceph-mon[33544]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T03:32:40.981 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:40 smithi173 ceph-mon[33544]: osdmap e49: 7 total, 6 up, 7 in 2024-03-12T03:32:40.981 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:40 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:40.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:40 smithi173 ceph-mon[33544]: from='osd.6 v2:172.21.15.173:6808/2147192629' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:32:40.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:40 smithi173 ceph-mon[33544]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:32:40.982 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:32:40 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:32:40.524+0000 7fa43197d700 -1 osd.6 0 waiting for initial osdmap 2024-03-12T03:32:40.982 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:32:40 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:32:40.530+0000 7fa42d770700 -1 osd.6 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:32:41.443 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:32:41.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:41 smithi173 ceph-mon[33544]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-12T03:32:41.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:41 smithi173 ceph-mon[33544]: osdmap e50: 7 total, 6 up, 7 in 2024-03-12T03:32:41.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:41 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:41.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:41 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:41.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:41 smithi173 ceph-mon[33544]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:41.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:41 smithi169 ceph-mon[29344]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-12T03:32:41.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:41 smithi169 ceph-mon[29344]: osdmap e50: 7 total, 6 up, 7 in 2024-03-12T03:32:41.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:41 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:41.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:41 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:41.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:41 smithi169 ceph-mon[29344]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:41 smithi169 ceph-mon[39949]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-12T03:32:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:41 smithi169 ceph-mon[39949]: osdmap e50: 7 total, 6 up, 7 in 2024-03-12T03:32:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:41 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:41 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:41 smithi169 ceph-mon[39949]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:32:42.859 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:42 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:32:42.859 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:42 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:32:42.859 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:42 smithi173 ceph-mon[33544]: osd.6 v2:172.21.15.173:6808/2147192629 boot 2024-03-12T03:32:42.859 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:42 smithi173 ceph-mon[33544]: osdmap e51: 7 total, 7 up, 7 in 2024-03-12T03:32:42.859 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:42 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:42.860 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:42 smithi173 ceph-mon[33544]: osdmap e52: 7 total, 7 up, 7 in 2024-03-12T03:32:42.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:42 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:32:42.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:42 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:32:42.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:42 smithi169 ceph-mon[29344]: osd.6 v2:172.21.15.173:6808/2147192629 boot 2024-03-12T03:32:42.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:42 smithi169 ceph-mon[29344]: osdmap e51: 7 total, 7 up, 7 in 2024-03-12T03:32:42.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:42 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:42.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:42 smithi169 ceph-mon[29344]: osdmap e52: 7 total, 7 up, 7 in 2024-03-12T03:32:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:42 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:32:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:42 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:32:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:42 smithi169 ceph-mon[39949]: osd.6 v2:172.21.15.173:6808/2147192629 boot 2024-03-12T03:32:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:42 smithi169 ceph-mon[39949]: osdmap e51: 7 total, 7 up, 7 in 2024-03-12T03:32:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:42 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:32:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:42 smithi169 ceph-mon[39949]: osdmap e52: 7 total, 7 up, 7 in 2024-03-12T03:32:43.656 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:43 smithi173 ceph-mon[33544]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:43.657 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:43.657 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:43.657 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:43.657 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:43.657 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:43.657 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:43.657 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:43.657 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:43.657 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:43.657 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:43 smithi173 ceph-mon[33544]: osdmap e53: 7 total, 7 up, 7 in 2024-03-12T03:32:43.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[29344]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:43.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:43.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:43.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[29344]: osdmap e53: 7 total, 7 up, 7 in 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[39949]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:43.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:43.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:43.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:32:43.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:43 smithi169 ceph-mon[39949]: osdmap e53: 7 total, 7 up, 7 in 2024-03-12T03:32:44.206 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-12T03:32:44.206 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-12T03:32:44.206 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-12T03:32:44.207 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-12T03:32:44.207 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123454 s, 849 MB/s 2024-03-12T03:32:44.207 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-12T03:32:44.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:44 smithi173 ceph-mon[33544]: Detected new or changed devices on smithi173 2024-03-12T03:32:44.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:44 smithi173 ceph-mon[33544]: Adjusting osd_memory_target on smithi173 to 5696M 2024-03-12T03:32:44.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-03-12T03:32:44.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:44 smithi169 ceph-mon[29344]: Detected new or changed devices on smithi173 2024-03-12T03:32:44.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:44 smithi169 ceph-mon[29344]: Adjusting osd_memory_target on smithi173 to 5696M 2024-03-12T03:32:44.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:44 smithi169 ceph-mon[39949]: Detected new or changed devices on smithi173 2024-03-12T03:32:44.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:44 smithi169 ceph-mon[39949]: Adjusting osd_memory_target on smithi173 to 5696M 2024-03-12T03:32:45.094 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:32:45.883 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:45 smithi173 ceph-mon[33544]: osdmap e54: 7 total, 7 up, 7 in 2024-03-12T03:32:45.883 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:45 smithi173 ceph-mon[33544]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T03:32:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:45 smithi169 ceph-mon[29344]: osdmap e54: 7 total, 7 up, 7 in 2024-03-12T03:32:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:45 smithi169 ceph-mon[29344]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T03:32:45.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:45 smithi169 ceph-mon[39949]: osdmap e54: 7 total, 7 up, 7 in 2024-03-12T03:32:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:45 smithi169 ceph-mon[39949]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T03:32:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:46 smithi173 ceph-mon[33544]: osdmap e55: 7 total, 7 up, 7 in 2024-03-12T03:32:46.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:46 smithi169 ceph-mon[29344]: osdmap e55: 7 total, 7 up, 7 in 2024-03-12T03:32:46.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:46 smithi169 ceph-mon[39949]: osdmap e55: 7 total, 7 up, 7 in 2024-03-12T03:32:47.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:47 smithi173 ceph-mon[33544]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-12T03:32:47.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:47.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:47.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:47 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:47 smithi169 ceph-mon[29344]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-12T03:32:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:47 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:47 smithi169 ceph-mon[39949]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-12T03:32:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:47 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:48.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:48 smithi173 ceph-mon[33544]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:48.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:48 smithi169 ceph-mon[29344]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:48.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:48 smithi169 ceph-mon[39949]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:49.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:49 smithi169 ceph-mon[29344]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:32:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:49 smithi169 ceph-mon[39949]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:32:49.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:49 smithi173 ceph-mon[33544]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:32:50.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:50 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/3305075091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8843039e-2b17-4230-9cca-5788a2294d8c"}]: dispatch 2024-03-12T03:32:50.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:50 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8843039e-2b17-4230-9cca-5788a2294d8c"}]: dispatch 2024-03-12T03:32:50.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:50 smithi169 ceph-mon[29344]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8843039e-2b17-4230-9cca-5788a2294d8c"}]': finished 2024-03-12T03:32:50.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:50 smithi169 ceph-mon[29344]: osdmap e56: 8 total, 7 up, 8 in 2024-03-12T03:32:50.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:50 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:32:50.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:50 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/1135370311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:50 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/3305075091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8843039e-2b17-4230-9cca-5788a2294d8c"}]: dispatch 2024-03-12T03:32:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:50 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8843039e-2b17-4230-9cca-5788a2294d8c"}]: dispatch 2024-03-12T03:32:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:50 smithi169 ceph-mon[39949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8843039e-2b17-4230-9cca-5788a2294d8c"}]': finished 2024-03-12T03:32:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:50 smithi169 ceph-mon[39949]: osdmap e56: 8 total, 7 up, 8 in 2024-03-12T03:32:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:50 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:32:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:50 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/1135370311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:50 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/3305075091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8843039e-2b17-4230-9cca-5788a2294d8c"}]: dispatch 2024-03-12T03:32:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:50 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8843039e-2b17-4230-9cca-5788a2294d8c"}]: dispatch 2024-03-12T03:32:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:50 smithi173 ceph-mon[33544]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8843039e-2b17-4230-9cca-5788a2294d8c"}]': finished 2024-03-12T03:32:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:50 smithi173 ceph-mon[33544]: osdmap e56: 8 total, 7 up, 8 in 2024-03-12T03:32:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:50 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:32:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:50 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/1135370311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:51.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:51 smithi169 ceph-mon[29344]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:51.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:51 smithi169 ceph-mon[39949]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:51 smithi173 ceph-mon[33544]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:53.914 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:53 smithi173 ceph-mon[33544]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:53.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:53 smithi169 ceph-mon[29344]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:53.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:53 smithi169 ceph-mon[39949]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:54 smithi173 ceph-mon[33544]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:55.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:54 smithi169 ceph-mon[39949]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:55.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:54 smithi169 ceph-mon[29344]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:56 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T03:32:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:56 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:56.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:56 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T03:32:56.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:56 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:56.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:56 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T03:32:56.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:56 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:57 smithi173 ceph-mon[33544]: Deploying daemon osd.7 on smithi173 2024-03-12T03:32:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:57 smithi173 ceph-mon[33544]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:57.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:57 smithi169 ceph-mon[29344]: Deploying daemon osd.7 on smithi173 2024-03-12T03:32:57.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:57 smithi169 ceph-mon[29344]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:57.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:57 smithi169 ceph-mon[39949]: Deploying daemon osd.7 on smithi173 2024-03-12T03:32:57.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:57 smithi169 ceph-mon[39949]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:59.861 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:32:59 smithi173 ceph-mon[33544]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:59.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:32:59 smithi169 ceph-mon[29344]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:32:59.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:32:59 smithi169 ceph-mon[39949]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:01 smithi173 ceph-mon[33544]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:01 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:01 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:01 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:01 smithi169 ceph-mon[29344]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:01 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:01 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:01 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:01 smithi169 ceph-mon[39949]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:01 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:01 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:01 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:03.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:03 smithi169 ceph-mon[29344]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:03.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:03 smithi169 ceph-mon[39949]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:03.949 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:03 smithi173 ceph-mon[33544]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:05.173 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 7 on host 'smithi173' 2024-03-12T03:33:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:05 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:05 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:05 smithi173 ceph-mon[33544]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:05 smithi173 ceph-mon[33544]: from='osd.7 v2:172.21.15.173:6812/4014976876' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:33:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:05 smithi173 ceph-mon[33544]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:33:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:05 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:05 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:05.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:05.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[29344]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[29344]: from='osd.7 v2:172.21.15.173:6812/4014976876' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:33:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[29344]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:33:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[39949]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[39949]: from='osd.7 v2:172.21.15.173:6812/4014976876' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:33:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[39949]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:33:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:05 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:05.912 DEBUG:teuthology.orchestra.run.smithi173:osd.7> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.7.service 2024-03-12T03:33:05.914 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-12T03:33:05.914 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-12T03:33:06.168 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:33:06.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:06 smithi173 ceph-mon[33544]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T03:33:06.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:06 smithi173 ceph-mon[33544]: osdmap e57: 8 total, 7 up, 8 in 2024-03-12T03:33:06.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:06 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:33:06.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:06 smithi173 ceph-mon[33544]: from='osd.7 v2:172.21.15.173:6812/4014976876' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:33:06.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:06 smithi173 ceph-mon[33544]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:33:06.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:06 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:33:06.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:06 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:33:06.487 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:33:06 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:33:06.287+0000 7f09b6456700 -1 osd.7 0 waiting for initial osdmap 2024-03-12T03:33:06.487 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:33:06 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:33:06.294+0000 7f09b2249700 -1 osd.7 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:33:06.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[39949]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T03:33:06.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[39949]: osdmap e57: 8 total, 7 up, 8 in 2024-03-12T03:33:06.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:33:06.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[39949]: from='osd.7 v2:172.21.15.173:6812/4014976876' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:33:06.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[39949]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:33:06.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:33:06.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:33:06.625 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[29344]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T03:33:06.625 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[29344]: osdmap e57: 8 total, 7 up, 8 in 2024-03-12T03:33:06.625 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:33:06.626 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[29344]: from='osd.7 v2:172.21.15.173:6812/4014976876' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:33:06.626 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[29344]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-12T03:33:06.626 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:33:06.626 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:06 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:33:07.345 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:07 smithi173 ceph-mon[33544]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-12T03:33:07.345 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:07 smithi173 ceph-mon[33544]: osdmap e58: 8 total, 7 up, 8 in 2024-03-12T03:33:07.345 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:07 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:33:07.345 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:07 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:33:07.345 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:07 smithi173 ceph-mon[33544]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:07.345 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:07 smithi173 ceph-mon[33544]: osd.7 v2:172.21.15.173:6812/4014976876 boot 2024-03-12T03:33:07.345 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:07 smithi173 ceph-mon[33544]: osdmap e59: 8 total, 8 up, 8 in 2024-03-12T03:33:07.345 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:07 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:33:07.577 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[29344]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-12T03:33:07.578 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[29344]: osdmap e58: 8 total, 7 up, 8 in 2024-03-12T03:33:07.578 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:33:07.578 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:33:07.578 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[29344]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:07.578 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[29344]: osd.7 v2:172.21.15.173:6812/4014976876 boot 2024-03-12T03:33:07.578 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[29344]: osdmap e59: 8 total, 8 up, 8 in 2024-03-12T03:33:07.578 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:33:07.578 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[39949]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-12T03:33:07.578 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[39949]: osdmap e58: 8 total, 7 up, 8 in 2024-03-12T03:33:07.578 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:33:07.579 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:33:07.579 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[39949]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:33:07.579 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[39949]: osd.7 v2:172.21.15.173:6812/4014976876 boot 2024-03-12T03:33:07.579 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[39949]: osdmap e59: 8 total, 8 up, 8 in 2024-03-12T03:33:07.579 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:07 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:33:08.014 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:33:08.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:08 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:33:08.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:08 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:33:08.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:08 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1007793914' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:33:08.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:08 smithi173 ceph-mon[33544]: osdmap e60: 8 total, 8 up, 8 in 2024-03-12T03:33:08.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:08 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:33:08.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:08 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:33:08.588 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:08 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1007793914' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:33:08.588 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:08 smithi169 ceph-mon[29344]: osdmap e60: 8 total, 8 up, 8 in 2024-03-12T03:33:08.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:08 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:33:08.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:08 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:33:08.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:08 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1007793914' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:33:08.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:08 smithi169 ceph-mon[39949]: osdmap e60: 8 total, 8 up, 8 in 2024-03-12T03:33:08.614 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":59,"num_osds":8,"num_up_osds":8,"osd_up_since":1710214387,"num_in_osds":8,"osd_in_since":1710214369,"num_remapped_pgs":1} 2024-03-12T03:33:08.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:33:08.865 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:33:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:33:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:09.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:09.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:09.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[29344]: osdmap e61: 8 total, 8 up, 8 in 2024-03-12T03:33:09.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:33:09.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:09.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:09.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:09.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:09.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:09.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:09.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:09.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:09.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:09.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:09 smithi169 ceph-mon[39949]: osdmap e61: 8 total, 8 up, 8 in 2024-03-12T03:33:09.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:09.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:09 smithi173 ceph-mon[33544]: osdmap e61: 8 total, 8 up, 8 in 2024-03-12T03:33:10.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:10 smithi169 ceph-mon[29344]: Detected new or changed devices on smithi173 2024-03-12T03:33:10.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:10 smithi169 ceph-mon[29344]: Adjusting osd_memory_target on smithi173 to 4272M 2024-03-12T03:33:10.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:10 smithi169 ceph-mon[39949]: Detected new or changed devices on smithi173 2024-03-12T03:33:10.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:10 smithi169 ceph-mon[39949]: Adjusting osd_memory_target on smithi173 to 4272M 2024-03-12T03:33:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:33:10.674 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":62,"fsid":"677d624a-e020-11ee-95c8-87774f69a715","created":"2024-03-12T03:27:53.319986+0000","modified":"2024-03-12T03:33:10.297906+0000","last_up_change":"2024-03-12T03:33:07.282669+0000","last_in_change":"2024-03-12T03:32:49.642694+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:31:04.863493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c44466b0-3112-44dc-b5f3-d6494b958761","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6801","nonce":676779916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":676779916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":676779916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6803","nonce":676779916}]},"public_addr":"172.21.15.169:6801/676779916","cluster_addr":"172.21.15.169:6802/676779916","heartbeat_back_addr":"172.21.15.169:6804/676779916","heartbeat_front_addr":"172.21.15.169:6803/676779916","state":["exists","up"]},{"osd":1,"uuid":"184c5c97-a992-4be9-bcd3-c08cb58a0e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":130042595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":130042595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":130042595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6807","nonce":130042595}]},"public_addr":"172.21.15.169:6805/130042595","cluster_addr":"172.21.15.169:6806/130042595","heartbeat_back_addr":"172.21.15.169:6808/130042595","heartbeat_front_addr":"172.21.15.169:6807/130042595","state":["exists","up"]},{"osd":2,"uuid":"beb15e06-1ca4-46a7-b640-efd1991367e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":2751816702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":2751816702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":2751816702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6811","nonce":2751816702}]},"public_addr":"172.21.15.169:6809/2751816702","cluster_addr":"172.21.15.169:6810/2751816702","heartbeat_back_addr":"172.21.15.169:6812/2751816702","heartbeat_front_addr":"172.21.15.169:6811/2751816702","state":["exists","up"]},{"osd":3,"uuid":"a00b4c50-e064-4c9e-85dc-b0a75ea5bf37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6813","nonce":195572266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":195572266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":195572266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6815","nonce":195572266}]},"public_addr":"172.21.15.169:6813/195572266","cluster_addr":"172.21.15.169:6814/195572266","heartbeat_back_addr":"172.21.15.169:6816/195572266","heartbeat_front_addr":"172.21.15.169:6815/195572266","state":["exists","up"]},{"osd":4,"uuid":"3a6121ef-6309-41b7-a964-6772c010b3a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1397212305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":1397212305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":1397212305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1397212305}]},"public_addr":"172.21.15.173:6800/1397212305","cluster_addr":"172.21.15.173:6801/1397212305","heartbeat_back_addr":"172.21.15.173:6803/1397212305","heartbeat_front_addr":"172.21.15.173:6802/1397212305","state":["exists","up"]},{"osd":5,"uuid":"3c2f6dd9-9784-4a07-8710-55ac3de82964","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3525888417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":3525888417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":3525888417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3525888417}]},"public_addr":"172.21.15.173:6804/3525888417","cluster_addr":"172.21.15.173:6805/3525888417","heartbeat_back_addr":"172.21.15.173:6807/3525888417","heartbeat_front_addr":"172.21.15.173:6806/3525888417","state":["exists","up"]},{"osd":6,"uuid":"d7759df7-1978-4d70-94a2-721540804f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2147192629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":2147192629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":2147192629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2147192629}]},"public_addr":"172.21.15.173:6808/2147192629","cluster_addr":"172.21.15.173:6809/2147192629","heartbeat_back_addr":"172.21.15.173:6811/2147192629","heartbeat_front_addr":"172.21.15.173:6810/2147192629","state":["exists","up"]},{"osd":7,"uuid":"8843039e-2b17-4230-9cca-5788a2294d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":4014976876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":4014976876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6815","nonce":4014976876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":4014976876}]},"public_addr":"172.21.15.173:6812/4014976876","cluster_addr":"172.21.15.173:6813/4014976876","heartbeat_back_addr":"172.21.15.173:6815/4014976876","heartbeat_front_addr":"172.21.15.173:6814/4014976876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:32.567380+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:06.106617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:18.881209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:28.027359+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:05.474553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:14.887705+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:39.773606+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:05.582338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/3918553041":"2024-03-13T03:28:48.518732+0000","172.21.15.169:0/1020514981":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/284191239":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/1212459137":"2024-03-13T03:28:48.518732+0000","172.21.15.169:6800/204892396":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/2751007072":"2024-03-13T03:28:48.518732+0000","172.21.15.169:0/2349376510":"2024-03-13T03:28:21.868923+0000","172.21.15.169:6800/108891420":"2024-03-13T03:28:48.518732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:33:10.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:10 smithi173 ceph-mon[33544]: Detected new or changed devices on smithi173 2024-03-12T03:33:10.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:10 smithi173 ceph-mon[33544]: Adjusting osd_memory_target on smithi173 to 4272M 2024-03-12T03:33:11.226 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T03:31:04.863493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T03:33:11.227 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-12T03:33:11.472 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:33:11.593 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:11 smithi169 ceph-mon[29344]: osdmap e62: 8 total, 8 up, 8 in 2024-03-12T03:33:11.593 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:11 smithi169 ceph-mon[29344]: pgmap v183: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-03-12T03:33:11.594 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:11 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1117659089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:33:11.594 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:11 smithi169 ceph-mon[39949]: osdmap e62: 8 total, 8 up, 8 in 2024-03-12T03:33:11.594 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:11 smithi169 ceph-mon[39949]: pgmap v183: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-03-12T03:33:11.594 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:11 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1117659089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:33:11.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:11 smithi173 ceph-mon[33544]: osdmap e62: 8 total, 8 up, 8 in 2024-03-12T03:33:11.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:11 smithi173 ceph-mon[33544]: pgmap v183: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-03-12T03:33:11.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:11 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1117659089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:33:12.547 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:12 smithi169 ceph-mon[29344]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:33:12.547 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:12 smithi169 ceph-mon[29344]: osdmap e63: 8 total, 8 up, 8 in 2024-03-12T03:33:12.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:12 smithi169 ceph-mon[39949]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:33:12.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:12 smithi169 ceph-mon[39949]: osdmap e63: 8 total, 8 up, 8 in 2024-03-12T03:33:12.580 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:12 smithi173 ceph-mon[33544]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:33:12.580 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:12 smithi173 ceph-mon[33544]: osdmap e63: 8 total, 8 up, 8 in 2024-03-12T03:33:13.301 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-12T03:33:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:13 smithi169 ceph-mon[29344]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-12T03:33:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:13 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3619568240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:33:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:13 smithi169 ceph-mon[39949]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-12T03:33:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:13 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3619568240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:33:13.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:13 smithi173 ceph-mon[33544]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-12T03:33:13.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:13 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3619568240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:33:13.923 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi169 2024-03-12T03:33:13.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi169=foo.a' 2024-03-12T03:33:14.174 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:33:14.577 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:14 smithi173 ceph-mon[33544]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-12T03:33:14.577 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:14 smithi173 ceph-mon[33544]: Cluster is now healthy 2024-03-12T03:33:14.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:14 smithi169 ceph-mon[29344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-12T03:33:14.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:14 smithi169 ceph-mon[29344]: Cluster is now healthy 2024-03-12T03:33:14.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:14 smithi169 ceph-mon[39949]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-12T03:33:14.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:14 smithi169 ceph-mon[39949]: Cluster is now healthy 2024-03-12T03:33:15.599 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:15 smithi173 ceph-mon[33544]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:33:15.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:15 smithi169 ceph-mon[29344]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:33:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:15 smithi169 ceph-mon[39949]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T03:33:16.065 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled rgw.foo.a update... 2024-03-12T03:33:16.671 DEBUG:teuthology.orchestra.run.smithi169:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-12T03:33:16.673 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi173 2024-03-12T03:33:16.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-12T03:33:16.911 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:33:17.371 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:17 smithi173 ceph-mon[33544]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi169=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:17.372 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:17 smithi173 ceph-mon[33544]: Saving service rgw.foo.a spec with placement smithi169=foo.a;count:1 2024-03-12T03:33:17.372 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:17 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:17.372 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:17 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:17.372 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:17 smithi173 ceph-mon[33544]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-12T03:33:17.372 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:17 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:17.372 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:17 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:17.372 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:17 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:17.372 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:17 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:33:17.372 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:17 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:33:17.372 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:17 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:17.372 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:17 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:17.409 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[29344]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi169=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:17.409 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[29344]: Saving service rgw.foo.a spec with placement smithi169=foo.a;count:1 2024-03-12T03:33:17.409 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:17.409 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:17.409 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[29344]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-12T03:33:17.409 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:17.409 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:17.409 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:17.409 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:33:17.409 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:33:17.409 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:17.409 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[39949]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi169=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[39949]: Saving service rgw.foo.a spec with placement smithi169=foo.a;count:1 2024-03-12T03:33:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[39949]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-12T03:33:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:33:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:33:17.410 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:17.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:17 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:18.301 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:18 smithi173 ceph-mon[33544]: Deploying daemon rgw.foo.a on smithi169 2024-03-12T03:33:18.302 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:18 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:33:18.302 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:18 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:33:18.389 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:18 smithi169 ceph-mon[29344]: Deploying daemon rgw.foo.a on smithi169 2024-03-12T03:33:18.389 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:18 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:33:18.389 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:18 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:33:18.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:18 smithi169 ceph-mon[39949]: Deploying daemon rgw.foo.a on smithi169 2024-03-12T03:33:18.390 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:18 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:33:18.390 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:18 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:33:18.673 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:33:18 smithi169 systemd[1]: Starting Ceph rgw.foo.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:33:19.089 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:33:18 smithi169 podman[66663]: 2024-03-12 03:33:18.734868767 +0000 UTC m=+0.017704298 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:33:19.089 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:33:18 smithi169 podman[66663]: 2024-03-12 03:33:18.854639311 +0000 UTC m=+0.137474834 container create 0025bc06a0703e548bcf9283256b08a984ce108f4eeaec372e30cba18cf5951d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-12T03:33:19.089 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:33:19 smithi169 podman[66663]: 2024-03-12 03:33:19.057393989 +0000 UTC m=+0.340229523 container init 0025bc06a0703e548bcf9283256b08a984ce108f4eeaec372e30cba18cf5951d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-12T03:33:19.089 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:33:19 smithi169 podman[66663]: 2024-03-12 03:33:19.060217284 +0000 UTC m=+0.343052808 container start 0025bc06a0703e548bcf9283256b08a984ce108f4eeaec372e30cba18cf5951d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307) 2024-03-12T03:33:19.097 INFO:teuthology.orchestra.run.smithi173.stderr:pool 'datapool' created 2024-03-12T03:33:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:19 smithi169 ceph-mon[29344]: osdmap e64: 8 total, 8 up, 8 in 2024-03-12T03:33:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:19 smithi169 ceph-mon[29344]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:33:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:19 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/103877472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-12T03:33:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:19 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-12T03:33:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:19 smithi169 ceph-mon[39949]: osdmap e64: 8 total, 8 up, 8 in 2024-03-12T03:33:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:19 smithi169 ceph-mon[39949]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:33:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:19 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/103877472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-12T03:33:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:19 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-12T03:33:19.424 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:33:19 smithi169 bash[66663]: 0025bc06a0703e548bcf9283256b08a984ce108f4eeaec372e30cba18cf5951d 2024-03-12T03:33:19.424 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:33:19 smithi169 systemd[1]: Started Ceph rgw.foo.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:33:19.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:19 smithi173 ceph-mon[33544]: osdmap e64: 8 total, 8 up, 8 in 2024-03-12T03:33:19.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:19 smithi173 ceph-mon[33544]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:33:19.465 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:19 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/103877472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-12T03:33:19.465 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:19 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-12T03:33:19.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- rbd pool init datapool 2024-03-12T03:33:20.181 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:33:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:20 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-12T03:33:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:20 smithi173 ceph-mon[33544]: osdmap e65: 8 total, 8 up, 8 in 2024-03-12T03:33:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:20 smithi173 ceph-mon[33544]: Saving service rgw.foo.a spec with placement smithi169=foo.a;count:1 2024-03-12T03:33:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:20 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:20.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-12T03:33:20.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[29344]: osdmap e65: 8 total, 8 up, 8 in 2024-03-12T03:33:20.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[29344]: Saving service rgw.foo.a spec with placement smithi169=foo.a;count:1 2024-03-12T03:33:20.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:20.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-12T03:33:20.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[39949]: osdmap e65: 8 total, 8 up, 8 in 2024-03-12T03:33:20.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[39949]: Saving service rgw.foo.a spec with placement smithi169=foo.a;count:1 2024-03-12T03:33:20.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:20.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:20 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:21.405 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:21 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:33:21.405 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:21 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:33:21.405 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:21 smithi173 ceph-mon[33544]: osdmap e66: 8 total, 8 up, 8 in 2024-03-12T03:33:21.405 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:21 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1557333783' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:33:21.405 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:21 smithi173 ceph-mon[33544]: pgmap v192: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:33:21.405 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:21 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:21.405 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:21 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:21.405 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:21 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:21.405 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:21 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:21.406 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:21 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1557333783' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T03:33:21.406 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:21 smithi173 ceph-mon[33544]: osdmap e67: 8 total, 8 up, 8 in 2024-03-12T03:33:21.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:33:21.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:33:21.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[39949]: osdmap e66: 8 total, 8 up, 8 in 2024-03-12T03:33:21.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1557333783' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:33:21.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[39949]: pgmap v192: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:33:21.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1557333783' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[39949]: osdmap e67: 8 total, 8 up, 8 in 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[29344]: osdmap e66: 8 total, 8 up, 8 in 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1557333783' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[29344]: pgmap v192: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:21.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:21.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1557333783' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T03:33:21.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:21 smithi169 ceph-mon[29344]: osdmap e67: 8 total, 8 up, 8 in 2024-03-12T03:33:22.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:22 smithi169 ceph-mon[29344]: Checking dashboard <-> RGW credentials 2024-03-12T03:33:22.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:22 smithi169 ceph-mon[29344]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:33:22.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:22 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/2049986302' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-12T03:33:22.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:22 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-12T03:33:22.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:22 smithi169 ceph-mon[39949]: Checking dashboard <-> RGW credentials 2024-03-12T03:33:22.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:22 smithi169 ceph-mon[39949]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:33:22.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:22 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/2049986302' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-12T03:33:22.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:22 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-12T03:33:22.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:22 smithi173 ceph-mon[33544]: Checking dashboard <-> RGW credentials 2024-03-12T03:33:22.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:22 smithi173 ceph-mon[33544]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:33:22.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:22 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/2049986302' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-12T03:33:22.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:22 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-12T03:33:23.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-12T03:33:23.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[29344]: osdmap e68: 8 total, 8 up, 8 in 2024-03-12T03:33:23.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1393676617' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:33:23.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[29344]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[29344]: pgmap v195: 68 pgs: 1 creating+peering, 20 active+clean, 47 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[29344]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[29344]: osdmap e69: 8 total, 8 up, 8 in 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[39949]: osdmap e68: 8 total, 8 up, 8 in 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1393676617' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[39949]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[39949]: pgmap v195: 68 pgs: 1 creating+peering, 20 active+clean, 47 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:33:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[39949]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:33:23.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:23 smithi169 ceph-mon[39949]: osdmap e69: 8 total, 8 up, 8 in 2024-03-12T03:33:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:23 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-12T03:33:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:23 smithi173 ceph-mon[33544]: osdmap e68: 8 total, 8 up, 8 in 2024-03-12T03:33:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:23 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1393676617' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:33:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:23 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:33:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:23 smithi173 ceph-mon[33544]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:33:23.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:23 smithi173 ceph-mon[33544]: pgmap v195: 68 pgs: 1 creating+peering, 20 active+clean, 47 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-12T03:33:23.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:23 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:33:23.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:23 smithi173 ceph-mon[33544]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:33:23.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:23 smithi173 ceph-mon[33544]: osdmap e69: 8 total, 8 up, 8 in 2024-03-12T03:33:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:24 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:24 smithi173 ceph-mon[33544]: osdmap e70: 8 total, 8 up, 8 in 2024-03-12T03:33:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:24 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1393676617' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:33:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:24 smithi173 ceph-mon[33544]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:33:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:24 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:33:24.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.173 --placement '1;smithi173=iscsi.a' 2024-03-12T03:33:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:24 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:24 smithi169 ceph-mon[29344]: osdmap e70: 8 total, 8 up, 8 in 2024-03-12T03:33:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:24 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1393676617' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:33:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:24 smithi169 ceph-mon[29344]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:33:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:24 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:33:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:24 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:24 smithi169 ceph-mon[39949]: osdmap e70: 8 total, 8 up, 8 in 2024-03-12T03:33:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:24 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1393676617' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:33:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:24 smithi169 ceph-mon[39949]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:33:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:24 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:33:25.048 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:33:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:25 smithi173 ceph-mon[33544]: pgmap v198: 100 pgs: 5 creating+peering, 41 active+clean, 54 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T03:33:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:25 smithi173 ceph-mon[33544]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:33:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:25 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:33:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:25 smithi173 ceph-mon[33544]: osdmap e71: 8 total, 8 up, 8 in 2024-03-12T03:33:26.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:25 smithi169 ceph-mon[29344]: pgmap v198: 100 pgs: 5 creating+peering, 41 active+clean, 54 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T03:33:26.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:25 smithi169 ceph-mon[29344]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:33:26.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:25 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:33:26.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:25 smithi169 ceph-mon[29344]: osdmap e71: 8 total, 8 up, 8 in 2024-03-12T03:33:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:25 smithi169 ceph-mon[39949]: pgmap v198: 100 pgs: 5 creating+peering, 41 active+clean, 54 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T03:33:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:25 smithi169 ceph-mon[39949]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:33:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:25 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:33:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:25 smithi169 ceph-mon[39949]: osdmap e71: 8 total, 8 up, 8 in 2024-03-12T03:33:26.869 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled iscsi.datapool update... 2024-03-12T03:33:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:26 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:33:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:26 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:33:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:26 smithi173 ceph-mon[33544]: osdmap e72: 8 total, 8 up, 8 in 2024-03-12T03:33:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:26 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1393676617' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:33:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:26 smithi173 ceph-mon[33544]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:33:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:26 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:33:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:26 smithi173 ceph-mon[33544]: pgmap v201: 132 pgs: 9 creating+peering, 60 active+clean, 63 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-12T03:33:27.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:33:27.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:33:27.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[29344]: osdmap e72: 8 total, 8 up, 8 in 2024-03-12T03:33:27.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1393676617' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:33:27.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[29344]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:33:27.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:33:27.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[29344]: pgmap v201: 132 pgs: 9 creating+peering, 60 active+clean, 63 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-12T03:33:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:33:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:33:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[39949]: osdmap e72: 8 total, 8 up, 8 in 2024-03-12T03:33:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1393676617' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:33:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[39949]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:33:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:33:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:26 smithi169 ceph-mon[39949]: pgmap v201: 132 pgs: 9 creating+peering, 60 active+clean, 63 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-12T03:33:27.441 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-12T03:33:27.442 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:33:27.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-12T03:33:27.478 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:33:27.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-12T03:33:27.507 DEBUG:teuthology.orchestra.run.smithi173:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@iscsi.iscsi.a.service 2024-03-12T03:33:27.550 INFO:tasks.cephadm:Adding prometheus.a on smithi173 2024-03-12T03:33:27.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi173=a' 2024-03-12T03:33:27.823 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:33:28.144 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:27 smithi173 ceph-mon[33544]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:28.144 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:27 smithi173 ceph-mon[33544]: Saving service iscsi.datapool spec with placement smithi173=iscsi.a;count:1 2024-03-12T03:33:28.144 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:27 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:28.144 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:27 smithi173 ceph-mon[33544]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:33:28.144 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:27 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:33:28.145 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:27 smithi173 ceph-mon[33544]: osdmap e73: 8 total, 8 up, 8 in 2024-03-12T03:33:28.145 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:27 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1393676617' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:33:28.145 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:27 smithi173 ceph-mon[33544]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:33:28.145 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:27 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:33:28.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[29344]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:28.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[29344]: Saving service iscsi.datapool spec with placement smithi173=iscsi.a;count:1 2024-03-12T03:33:28.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:28.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[29344]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:33:28.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:33:28.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[29344]: osdmap e73: 8 total, 8 up, 8 in 2024-03-12T03:33:28.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1393676617' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:33:28.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[29344]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:33:28.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:33:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[39949]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[39949]: Saving service iscsi.datapool spec with placement smithi173=iscsi.a;count:1 2024-03-12T03:33:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[39949]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:33:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:33:28.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[39949]: osdmap e73: 8 total, 8 up, 8 in 2024-03-12T03:33:28.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1393676617' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:33:28.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[39949]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:33:28.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:27 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:33:29.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:28 smithi173 ceph-mon[33544]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:33:29.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:28 smithi173 ceph-mon[33544]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:33:29.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:28 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:33:29.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:28 smithi173 ceph-mon[33544]: osdmap e74: 8 total, 8 up, 8 in 2024-03-12T03:33:29.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:28 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:29.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:28 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:29.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:28 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:29.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:28 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:33:29.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:28 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-12T03:33:29.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:28 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:29.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:28 smithi173 ceph-mon[33544]: Deploying daemon iscsi.iscsi.a on smithi173 2024-03-12T03:33:29.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:28 smithi173 ceph-mon[33544]: pgmap v204: 132 pgs: 3 creating+activating, 18 creating+peering, 87 active+clean, 24 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-12T03:33:29.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[29344]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:33:29.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[29344]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:33:29.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:33:29.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[29344]: osdmap e74: 8 total, 8 up, 8 in 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[29344]: Deploying daemon iscsi.iscsi.a on smithi173 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[29344]: pgmap v204: 132 pgs: 3 creating+activating, 18 creating+peering, 87 active+clean, 24 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[39949]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[39949]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3041863637' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[39949]: osdmap e74: 8 total, 8 up, 8 in 2024-03-12T03:33:29.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:29.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:29.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:29.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:33:29.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-12T03:33:29.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:29.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[39949]: Deploying daemon iscsi.iscsi.a on smithi173 2024-03-12T03:33:29.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:28 smithi169 ceph-mon[39949]: pgmap v204: 132 pgs: 3 creating+activating, 18 creating+peering, 87 active+clean, 24 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-12T03:33:29.525 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled prometheus update... 2024-03-12T03:33:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[29344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:33:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[29344]: Cluster is now healthy 2024-03-12T03:33:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[29344]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[29344]: Saving service prometheus spec with placement smithi173=a;count:1 2024-03-12T03:33:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:30.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[29344]: 1.0 scrub starts 2024-03-12T03:33:30.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[29344]: 1.0 scrub ok 2024-03-12T03:33:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[39949]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:33:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[39949]: Cluster is now healthy 2024-03-12T03:33:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[39949]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[39949]: Saving service prometheus spec with placement smithi173=a;count:1 2024-03-12T03:33:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[39949]: 1.0 scrub starts 2024-03-12T03:33:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:29 smithi169 ceph-mon[39949]: 1.0 scrub ok 2024-03-12T03:33:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:29 smithi173 ceph-mon[33544]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:33:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:29 smithi173 ceph-mon[33544]: Cluster is now healthy 2024-03-12T03:33:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:29 smithi173 ceph-mon[33544]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:29 smithi173 ceph-mon[33544]: Saving service prometheus spec with placement smithi173=a;count:1 2024-03-12T03:33:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:29 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:29 smithi173 ceph-mon[33544]: 1.0 scrub starts 2024-03-12T03:33:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:29 smithi173 ceph-mon[33544]: 1.0 scrub ok 2024-03-12T03:33:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:30 smithi173 ceph-mon[33544]: pgmap v205: 132 pgs: 3 creating+activating, 9 creating+peering, 120 active+clean; 580 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 4.4 KiB/s wr, 173 op/s 2024-03-12T03:33:31.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:30 smithi169 ceph-mon[29344]: pgmap v205: 132 pgs: 3 creating+activating, 9 creating+peering, 120 active+clean; 580 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 4.4 KiB/s wr, 173 op/s 2024-03-12T03:33:31.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:30 smithi169 ceph-mon[39949]: pgmap v205: 132 pgs: 3 creating+activating, 9 creating+peering, 120 active+clean; 580 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 4.4 KiB/s wr, 173 op/s 2024-03-12T03:33:31.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:31 smithi173 systemd[1]: Starting Ceph iscsi.iscsi.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:33:31.403 DEBUG:teuthology.orchestra.run.smithi173:prometheus.a> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-12T03:33:31.405 INFO:tasks.cephadm:Adding node-exporter.a on smithi169 2024-03-12T03:33:31.405 INFO:tasks.cephadm:Adding node-exporter.b on smithi173 2024-03-12T03:33:31.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '2;smithi169=a;smithi173=b' 2024-03-12T03:33:31.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:31 smithi173 podman[60440]: 2024-03-12 03:33:31.481225175 +0000 UTC m=+0.019391800 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:33:31.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:31 smithi173 podman[60440]: 2024-03-12 03:33:31.605563696 +0000 UTC m=+0.143730324 container create ac42ed100c151baa7964ed750753707225444a9d6585c443187cc6fab399a024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-12T03:33:32.004 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:33:32.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:31 smithi173 podman[60440]: 2024-03-12 03:33:31.850020332 +0000 UTC m=+0.388186970 container init ac42ed100c151baa7964ed750753707225444a9d6585c443187cc6fab399a024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-12T03:33:32.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:31 smithi173 podman[60440]: 2024-03-12 03:33:31.853071861 +0000 UTC m=+0.391238488 container start ac42ed100c151baa7964ed750753707225444a9d6585c443187cc6fab399a024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-12T03:33:32.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:31 smithi173 bash[60440]: ac42ed100c151baa7964ed750753707225444a9d6585c443187cc6fab399a024 2024-03-12T03:33:32.135 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:31 smithi173 systemd[1]: Started Ceph iscsi.iscsi.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:33:32.384 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug Started the configuration object watcher 2024-03-12T03:33:32.384 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug Checking for config object changes every 1s 2024-03-12T03:33:32.385 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug Processing osd blocklist entries for this node 2024-03-12T03:33:32.698 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug Reading the configuration object to update local LIO configuration 2024-03-12T03:33:32.699 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug Configuration does not have an entry for this host(smithi173.front.sepia.ceph.com) - nothing to define to LIO 2024-03-12T03:33:32.699 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-12T03:33:33.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:33 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:33.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:33 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:33.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:33 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:33.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:33 smithi173 ceph-mon[33544]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-12T03:33:33.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:33 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:33.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:33 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:33.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:33 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/3232659032' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:33:33.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:33 smithi173 ceph-mon[33544]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 6.1 KiB/s wr, 252 op/s 2024-03-12T03:33:33.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:33.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:33.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:33.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[39949]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-12T03:33:33.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:33.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:33.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/3232659032' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:33:33.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[39949]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 6.1 KiB/s wr, 252 op/s 2024-03-12T03:33:33.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:33.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:33.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:33.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[29344]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-12T03:33:33.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:33.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:33.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/3232659032' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:33:33.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:33 smithi169 ceph-mon[29344]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 6.1 KiB/s wr, 252 op/s 2024-03-12T03:33:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:34 smithi173 ceph-mon[33544]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-12T03:33:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:34 smithi173 ceph-mon[33544]: osdmap e75: 8 total, 8 up, 8 in 2024-03-12T03:33:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:34 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:34.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:34 smithi169 ceph-mon[29344]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-12T03:33:34.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:34 smithi169 ceph-mon[29344]: osdmap e75: 8 total, 8 up, 8 in 2024-03-12T03:33:34.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:34 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:34.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:34 smithi169 ceph-mon[39949]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-12T03:33:34.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:34 smithi169 ceph-mon[39949]: osdmap e75: 8 total, 8 up, 8 in 2024-03-12T03:33:34.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:34 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:34.733 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled node-exporter update... 2024-03-12T03:33:35.348 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:35 smithi173 ceph-mon[33544]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 4.9 KiB/s wr, 217 op/s 2024-03-12T03:33:35.348 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:35 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:35.377 DEBUG:teuthology.orchestra.run.smithi169:node-exporter.a> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-12T03:33:35.380 DEBUG:teuthology.orchestra.run.smithi173:node-exporter.b> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-12T03:33:35.382 INFO:tasks.cephadm:Adding alertmanager.a on smithi169 2024-03-12T03:33:35.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi169=a' 2024-03-12T03:33:35.405 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:35 smithi169 ceph-mon[29344]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 4.9 KiB/s wr, 217 op/s 2024-03-12T03:33:35.405 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:35 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:35.405 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:35 smithi169 ceph-mon[39949]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 4.9 KiB/s wr, 217 op/s 2024-03-12T03:33:35.406 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:35 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:36.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:36 smithi169 ceph-mon[29344]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi169=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:36.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:36 smithi169 ceph-mon[29344]: Saving service node-exporter spec with placement smithi169=a;smithi173=b;count:2 2024-03-12T03:33:36.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:36 smithi169 ceph-mon[39949]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi169=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:36.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:36 smithi169 ceph-mon[39949]: Saving service node-exporter spec with placement smithi169=a;smithi173=b;count:2 2024-03-12T03:33:36.434 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:33:36.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:36 smithi173 ceph-mon[33544]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi169=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:36.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:36 smithi173 ceph-mon[33544]: Saving service node-exporter spec with placement smithi169=a;smithi173=b;count:2 2024-03-12T03:33:37.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:37.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:37.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:37.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:37.698 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:37 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:37.698 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:37 smithi173 ceph-mon[33544]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 4.3 KiB/s wr, 191 op/s 2024-03-12T03:33:37.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:37.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:37.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:37.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:37.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:37 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:37.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:37 smithi169 ceph-mon[39949]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 4.3 KiB/s wr, 191 op/s 2024-03-12T03:33:37.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:37.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:37.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:37.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:37.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:37 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:37.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:37 smithi169 ceph-mon[29344]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 4.3 KiB/s wr, 191 op/s 2024-03-12T03:33:38.308 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled alertmanager update... 2024-03-12T03:33:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:38 smithi173 ceph-mon[33544]: Deploying daemon prometheus.a on smithi173 2024-03-12T03:33:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:38 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:38.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:38 smithi169 ceph-mon[29344]: Deploying daemon prometheus.a on smithi173 2024-03-12T03:33:38.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:38 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:38.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:38 smithi169 ceph-mon[39949]: Deploying daemon prometheus.a on smithi173 2024-03-12T03:33:38.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:38 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:39.265 DEBUG:teuthology.orchestra.run.smithi169:alertmanager.a> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-12T03:33:39.268 INFO:tasks.cephadm:Adding grafana.a on smithi173 2024-03-12T03:33:39.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi173=a' 2024-03-12T03:33:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:39 smithi173 ceph-mon[33544]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi169=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:39 smithi173 ceph-mon[33544]: Saving service alertmanager spec with placement smithi169=a;count:1 2024-03-12T03:33:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:39 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:33:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:39 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:33:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:39 smithi173 ceph-mon[33544]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 3.6 KiB/s wr, 161 op/s 2024-03-12T03:33:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:39 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:39 smithi173 ceph-mon[33544]: osdmap e76: 8 total, 8 up, 8 in 2024-03-12T03:33:39.509 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:33:39.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[39949]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi169=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:39.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[39949]: Saving service alertmanager spec with placement smithi169=a;count:1 2024-03-12T03:33:39.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:33:39.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:33:39.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[39949]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 3.6 KiB/s wr, 161 op/s 2024-03-12T03:33:39.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:39.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[39949]: osdmap e76: 8 total, 8 up, 8 in 2024-03-12T03:33:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[29344]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi169=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[29344]: Saving service alertmanager spec with placement smithi169=a;count:1 2024-03-12T03:33:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:33:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:33:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[29344]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 3.6 KiB/s wr, 161 op/s 2024-03-12T03:33:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:39 smithi169 ceph-mon[29344]: osdmap e76: 8 total, 8 up, 8 in 2024-03-12T03:33:41.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:41 smithi173 ceph-mon[33544]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:33:41.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:41 smithi169 ceph-mon[29344]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:33:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:41 smithi169 ceph-mon[39949]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:33:42.264 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled grafana update... 2024-03-12T03:33:42.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:33:43.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:43 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:43.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:43 smithi169 ceph-mon[39949]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:43 smithi169 ceph-mon[39949]: Saving service grafana spec with placement smithi173=a;count:1 2024-03-12T03:33:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:43 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:43 smithi169 ceph-mon[39949]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:33:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:43 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:43 smithi169 ceph-mon[29344]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:43 smithi169 ceph-mon[29344]: Saving service grafana spec with placement smithi173=a;count:1 2024-03-12T03:33:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:43 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:43 smithi169 ceph-mon[29344]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:33:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:43 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:33:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:43 smithi173 ceph-mon[33544]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:43 smithi173 ceph-mon[33544]: Saving service grafana spec with placement smithi173=a;count:1 2024-03-12T03:33:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:43 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:43 smithi173 ceph-mon[33544]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:33:44.512 DEBUG:teuthology.orchestra.run.smithi173:grafana.a> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@grafana.a.service 2024-03-12T03:33:44.515 INFO:tasks.cephadm:Setting up client nodes... 2024-03-12T03:33:44.515 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T03:33:44.767 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:33:45.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:45 smithi169 ceph-mon[29344]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:33:45.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:45 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:33:45.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:45 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:33:45.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:45 smithi169 ceph-mon[39949]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:33:45.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:45 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:33:45.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:45 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:33:45.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:45 smithi173 ceph-mon[33544]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:33:45.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:45 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:33:45.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:45 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:33:46.806 INFO:teuthology.orchestra.run.smithi169.stdout:[client.0] 2024-03-12T03:33:46.806 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAaze9lEnLZLxAAVH8i8a82HgaAvyEfKpioqQ== 2024-03-12T03:33:46.807 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:46 smithi169 ceph-mon[29344]: osdmap e77: 8 total, 8 up, 8 in 2024-03-12T03:33:46.807 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:46 smithi169 ceph-mon[39949]: osdmap e77: 8 total, 8 up, 8 in 2024-03-12T03:33:46.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:46 smithi173 ceph-mon[33544]: osdmap e77: 8 total, 8 up, 8 in 2024-03-12T03:33:47.414 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-12T03:33:47.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-12T03:33:47.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-12T03:33:47.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T03:33:47.760 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:47 smithi173 ceph-mon[33544]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:33:47.760 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:47 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1742973295' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:33:47.760 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:47 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1742973295' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:33:47.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:47 smithi169 ceph-mon[39949]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:33:47.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:47 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1742973295' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:33:47.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:47 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1742973295' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:33:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:47 smithi169 ceph-mon[29344]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:33:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:47 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1742973295' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:33:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:47 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1742973295' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:33:49.272 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:33:49.395 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:49 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-12T03:33:49.395 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:49 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-12T03:33:49.395 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:49 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-12T03:33:49.395 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:49 smithi173 ceph-mon[33544]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:33:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:49 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-12T03:33:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:49 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-12T03:33:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:49 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-12T03:33:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:49 smithi169 ceph-mon[29344]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:33:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:49 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-12T03:33:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:49 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-12T03:33:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:49 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-12T03:33:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:49 smithi169 ceph-mon[39949]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:33:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:50 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-12T03:33:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:50 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-12T03:33:50.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:50 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-12T03:33:50.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:50 smithi169 ceph-mon[29344]: osdmap e78: 8 total, 8 up, 8 in 2024-03-12T03:33:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:50 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-12T03:33:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:50 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-12T03:33:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:50 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-12T03:33:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:50 smithi169 ceph-mon[39949]: osdmap e78: 8 total, 8 up, 8 in 2024-03-12T03:33:50.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:50 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-12T03:33:50.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:50 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-12T03:33:50.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:50 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-12T03:33:50.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:50 smithi173 ceph-mon[33544]: osdmap e78: 8 total, 8 up, 8 in 2024-03-12T03:33:51.319 INFO:teuthology.orchestra.run.smithi173.stdout:[client.1] 2024-03-12T03:33:51.319 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAfze9lJAyiEhAAkxR1wgOt/Nc18zUE/7YAiQ== 2024-03-12T03:33:51.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:51 smithi169 ceph-mon[29344]: osdmap e79: 8 total, 8 up, 8 in 2024-03-12T03:33:51.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:51 smithi169 ceph-mon[29344]: pgmap v220: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:51.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:51 smithi169 ceph-mon[39949]: osdmap e79: 8 total, 8 up, 8 in 2024-03-12T03:33:51.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:51 smithi169 ceph-mon[39949]: pgmap v220: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:51.434 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:51 smithi173 ceph-mon[33544]: osdmap e79: 8 total, 8 up, 8 in 2024-03-12T03:33:51.434 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:51 smithi173 ceph-mon[33544]: pgmap v220: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:33:52.168 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-12T03:33:52.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-12T03:33:52.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-12T03:33:52.211 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-12T03:33:52.211 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-12T03:33:52.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-12T03:33:52.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[29344]: from='client.? 172.21.15.173:0/2311741300' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:33:52.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:33:52.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:33:52.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:52.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:52.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:52.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:33:52.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[39949]: from='client.? 172.21.15.173:0/2311741300' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:33:52.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:33:52.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:33:52.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:52.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:52.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:52.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:33:52.462 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:33:52.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:33:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:33:52.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:52 smithi173 ceph-mon[33544]: from='client.? 172.21.15.173:0/2311741300' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:33:52.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:52 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:33:52.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:52 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:33:52.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:52 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:52.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:52 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:52.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:52 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' 2024-03-12T03:33:52.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:52 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:33:52.815 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ignoring --setuser ceph since I am not root 2024-03-12T03:33:52.815 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:52 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ignoring --setgroup ceph since I am not root 2024-03-12T03:33:52.986 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: ignoring --setuser ceph since I am not root 2024-03-12T03:33:52.986 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: ignoring --setgroup ceph since I am not root 2024-03-12T03:33:52.986 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:52 smithi173 ceph-mgr[35413]: -- 172.21.15.173:0/3147852823 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5647be2665a0 con 0x5647be23ac00 2024-03-12T03:33:53.486 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:53.190+0000 7fa2d334b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:33:53.486 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:53.283+0000 7fa2d334b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:33:53.566 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:53 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:53.210+0000 7fb3a8931200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:33:53.566 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:53 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:53.305+0000 7fb3a8931200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:33:53.862 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:53 smithi169 ceph-mon[29344]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:33:53.862 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:53 smithi169 ceph-mon[29344]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-12T03:33:53.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:53 smithi169 ceph-mon[29344]: osdmap e80: 8 total, 8 up, 8 in 2024-03-12T03:33:53.863 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:53 smithi169 ceph-mon[39949]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:33:53.863 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:53 smithi169 ceph-mon[39949]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-12T03:33:53.863 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:53 smithi169 ceph-mon[39949]: osdmap e80: 8 total, 8 up, 8 in 2024-03-12T03:33:53.920 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:53 smithi173 ceph-mon[33544]: from='mgr.14150 172.21.15.169:0/3261400510' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:33:53.920 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:53 smithi173 ceph-mon[33544]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-12T03:33:53.920 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:53 smithi173 ceph-mon[33544]: osdmap e80: 8 total, 8 up, 8 in 2024-03-12T03:33:54.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:53 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:53.967+0000 7fb3a8931200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:33:54.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:54.063+0000 7fb3a8931200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:33:54.236 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:53.916+0000 7fa2d334b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:33:54.236 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:54.008+0000 7fa2d334b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:33:54.236 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:54.150+0000 7fa2d334b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:33:54.365 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:33:54.593 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:54.245+0000 7fa2d334b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:33:54.594 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:54.329+0000 7fa2d334b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:33:54.632 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:54.213+0000 7fb3a8931200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:33:54.632 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:54.312+0000 7fb3a8931200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:33:54.633 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:54.402+0000 7fb3a8931200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:33:54.880 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:54 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2097992491' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:33:54.880 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:54.590+0000 7fa2d334b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:33:54.880 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:54.700+0000 7fa2d334b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:33:54.880 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:54.785+0000 7fa2d334b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:33:54.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:54 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2097992491' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:33:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:54 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2097992491' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:33:54.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:54.672+0000 7fb3a8931200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:33:54.924 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:54.783+0000 7fb3a8931200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:33:54.924 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:54.869+0000 7fb3a8931200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:33:55.102 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":2779395456}]},"active_addr":"172.21.15.169:6800/2779395456","active_change":"2024-03-12T03:28:48.518852+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.169:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":4041747161}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":342292643}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":197210445}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":555866474}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":1430250501}]}]} 2024-03-12T03:33:55.106 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-12T03:33:55.107 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-12T03:33:55.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:33:55.236 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:54.877+0000 7fa2d334b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:33:55.236 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:55 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:55.054+0000 7fa2d334b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:33:55.362 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:33:55.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:54.965+0000 7fb3a8931200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:33:55.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:55.147+0000 7fb3a8931200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:33:56.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:56 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:56.153+0000 7fb3a8931200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:33:56.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:56 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:56.285+0000 7fb3a8931200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:33:56.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:56 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:56.013+0000 7fa2d334b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:33:56.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:56 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:56.145+0000 7fa2d334b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:33:56.734 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:56 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:56.637+0000 7fb3a8931200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:33:56.734 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:56 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:56.731+0000 7fb3a8931200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:33:56.736 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:56 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:56.479+0000 7fa2d334b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:33:56.736 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:56 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:56.572+0000 7fa2d334b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:33:57.131 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:56 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:56.937+0000 7fb3a8931200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:33:57.131 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:57 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:57.128+0000 7fb3a8931200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:33:57.149 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:33:57.149 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":81,"fsid":"677d624a-e020-11ee-95c8-87774f69a715","created":"2024-03-12T03:27:53.319986+0000","modified":"2024-03-12T03:33:56.431655+0000","last_up_change":"2024-03-12T03:33:07.282669+0000","last_in_change":"2024-03-12T03:32:49.642694+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:31:04.863493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-12T03:33:18.983144+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-12T03:33:19.343301+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-12T03:33:21.864621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-12T03:33:23.416265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-12T03:33:25.476511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c44466b0-3112-44dc-b5f3-d6494b958761","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6801","nonce":676779916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":676779916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":676779916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6803","nonce":676779916}]},"public_addr":"172.21.15.169:6801/676779916","cluster_addr":"172.21.15.169:6802/676779916","heartbeat_back_addr":"172.21.15.169:6804/676779916","heartbeat_front_addr":"172.21.15.169:6803/676779916","state":["exists","up"]},{"osd":1,"uuid":"184c5c97-a992-4be9-bcd3-c08cb58a0e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":130042595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":130042595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":130042595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6807","nonce":130042595}]},"public_addr":"172.21.15.169:6805/130042595","cluster_addr":"172.21.15.169:6806/130042595","heartbeat_back_addr":"172.21.15.169:6808/130042595","heartbeat_front_addr":"172.21.15.169:6807/130042595","state":["exists","up"]},{"osd":2,"uuid":"beb15e06-1ca4-46a7-b640-efd1991367e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":2751816702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":2751816702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":2751816702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6811","nonce":2751816702}]},"public_addr":"172.21.15.169:6809/2751816702","cluster_addr":"172.21.15.169:6810/2751816702","heartbeat_back_addr":"172.21.15.169:6812/2751816702","heartbeat_front_addr":"172.21.15.169:6811/2751816702","state":["exists","up"]},{"osd":3,"uuid":"a00b4c50-e064-4c9e-85dc-b0a75ea5bf37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6813","nonce":195572266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":195572266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":195572266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6815","nonce":195572266}]},"public_addr":"172.21.15.169:6813/195572266","cluster_addr":"172.21.15.169:6814/195572266","heartbeat_back_addr":"172.21.15.169:6816/195572266","heartbeat_front_addr":"172.21.15.169:6815/195572266","state":["exists","up"]},{"osd":4,"uuid":"3a6121ef-6309-41b7-a964-6772c010b3a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1397212305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":1397212305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":1397212305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1397212305}]},"public_addr":"172.21.15.173:6800/1397212305","cluster_addr":"172.21.15.173:6801/1397212305","heartbeat_back_addr":"172.21.15.173:6803/1397212305","heartbeat_front_addr":"172.21.15.173:6802/1397212305","state":["exists","up"]},{"osd":5,"uuid":"3c2f6dd9-9784-4a07-8710-55ac3de82964","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3525888417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":3525888417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":3525888417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3525888417}]},"public_addr":"172.21.15.173:6804/3525888417","cluster_addr":"172.21.15.173:6805/3525888417","heartbeat_back_addr":"172.21.15.173:6807/3525888417","heartbeat_front_addr":"172.21.15.173:6806/3525888417","state":["exists","up"]},{"osd":6,"uuid":"d7759df7-1978-4d70-94a2-721540804f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2147192629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":2147192629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":2147192629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2147192629}]},"public_addr":"172.21.15.173:6808/2147192629","cluster_addr":"172.21.15.173:6809/2147192629","heartbeat_back_addr":"172.21.15.173:6811/2147192629","heartbeat_front_addr":"172.21.15.173:6810/2147192629","state":["exists","up"]},{"osd":7,"uuid":"8843039e-2b17-4230-9cca-5788a2294d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":4014976876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":4014976876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6815","nonce":4014976876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":4014976876}]},"public_addr":"172.21.15.173:6812/4014976876","cluster_addr":"172.21.15.173:6813/4014976876","heartbeat_back_addr":"172.21.15.173:6815/4014976876","heartbeat_front_addr":"172.21.15.173:6814/4014976876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:38.337984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:17.103074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:19.742089+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:56.380847+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:05.474553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:25.002739+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:44.675338+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:05.582338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/3918553041":"2024-03-13T03:28:48.518732+0000","172.21.15.169:0/1020514981":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/284191239":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/1212459137":"2024-03-13T03:28:48.518732+0000","172.21.15.169:6800/204892396":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/2751007072":"2024-03-13T03:28:48.518732+0000","172.21.15.169:0/2349376510":"2024-03-13T03:28:21.868923+0000","172.21.15.169:6800/108891420":"2024-03-13T03:28:48.518732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:33:57.236 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:56 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:56.767+0000 7fa2d334b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:33:57.236 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:56 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:56.951+0000 7fa2d334b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:33:57.501 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:57 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:57.236+0000 7fa2d334b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:33:57.502 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:57 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:33:57.502 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:57 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:33:57.502 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:57 smithi173 ceph-mon[33544]: osdmap e81: 8 total, 8 up, 8 in 2024-03-12T03:33:57.502 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:57 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2929899422' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:33:57.711 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:57 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:33:57.711 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:57 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:33:57.711 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:57 smithi169 ceph-mon[29344]: osdmap e81: 8 total, 8 up, 8 in 2024-03-12T03:33:57.711 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:57 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2929899422' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:33:57.711 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:57 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:33:57.711 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:57 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:33:57.711 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:57 smithi169 ceph-mon[39949]: osdmap e81: 8 total, 8 up, 8 in 2024-03-12T03:33:57.711 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:57 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2929899422' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:33:57.712 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:57 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:57.428+0000 7fb3a8931200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:33:57.832 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-12T03:33:57.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:33:57.986 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:57 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:57.499+0000 7fa2d334b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:33:57.986 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:57 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:57.736+0000 7fa2d334b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:33:58.081 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:33:58.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:57 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:57.708+0000 7fb3a8931200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:33:58.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:57 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:57.958+0000 7fb3a8931200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:33:58.986 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:58 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:58.538+0000 7fa2d334b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:33:59.134 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:58 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:58.798+0000 7fb3a8931200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:33:59.391 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:33:59 smithi169 ceph-mon[29344]: osdmap e82: 8 total, 8 up, 8 in 2024-03-12T03:33:59.392 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:33:59 smithi169 ceph-mon[39949]: osdmap e82: 8 total, 8 up, 8 in 2024-03-12T03:33:59.431 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:33:59 smithi173 ceph-mon[33544]: osdmap e82: 8 total, 8 up, 8 in 2024-03-12T03:33:59.432 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:59 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:59.111+0000 7fa2d334b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:33:59.432 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:59 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:59.211+0000 7fa2d334b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:33:59.673 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:59 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:59.388+0000 7fb3a8931200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:33:59.673 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:59 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:59.489+0000 7fb3a8931200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:33:59.691 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:59 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:59.429+0000 7fa2d334b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:33:59.691 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:59 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: [12/Mar/2024:03:33:59] ENGINE Bus STARTING 2024-03-12T03:33:59.691 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:59 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: CherryPy Checker: 2024-03-12T03:33:59.691 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:59 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: The Application mounted at '' has an empty config. 2024-03-12T03:33:59.691 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:59 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: 2024-03-12T03:33:59.691 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:59 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: [12/Mar/2024:03:33:59] ENGINE Serving on http://:::9283 2024-03-12T03:33:59.692 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:33:59 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x[35409]: [12/Mar/2024:03:33:59] ENGINE Bus STARTED 2024-03-12T03:33:59.973 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:59 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:33:59.717+0000 7fb3a8931200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:33:59.973 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:59 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:33:59] ENGINE Bus STARTING 2024-03-12T03:33:59.988 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:33:59.989 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":83,"fsid":"677d624a-e020-11ee-95c8-87774f69a715","created":"2024-03-12T03:27:53.319986+0000","modified":"2024-03-12T03:33:59.735113+0000","last_up_change":"2024-03-12T03:33:07.282669+0000","last_in_change":"2024-03-12T03:32:49.642694+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:31:04.863493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-12T03:33:18.983144+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-12T03:33:19.343301+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-12T03:33:21.864621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-12T03:33:23.416265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-12T03:33:25.476511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c44466b0-3112-44dc-b5f3-d6494b958761","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6801","nonce":676779916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":676779916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":676779916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6803","nonce":676779916}]},"public_addr":"172.21.15.169:6801/676779916","cluster_addr":"172.21.15.169:6802/676779916","heartbeat_back_addr":"172.21.15.169:6804/676779916","heartbeat_front_addr":"172.21.15.169:6803/676779916","state":["exists","up"]},{"osd":1,"uuid":"184c5c97-a992-4be9-bcd3-c08cb58a0e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":130042595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":130042595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":130042595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6807","nonce":130042595}]},"public_addr":"172.21.15.169:6805/130042595","cluster_addr":"172.21.15.169:6806/130042595","heartbeat_back_addr":"172.21.15.169:6808/130042595","heartbeat_front_addr":"172.21.15.169:6807/130042595","state":["exists","up"]},{"osd":2,"uuid":"beb15e06-1ca4-46a7-b640-efd1991367e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":2751816702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":2751816702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":2751816702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6811","nonce":2751816702}]},"public_addr":"172.21.15.169:6809/2751816702","cluster_addr":"172.21.15.169:6810/2751816702","heartbeat_back_addr":"172.21.15.169:6812/2751816702","heartbeat_front_addr":"172.21.15.169:6811/2751816702","state":["exists","up"]},{"osd":3,"uuid":"a00b4c50-e064-4c9e-85dc-b0a75ea5bf37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6813","nonce":195572266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":195572266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":195572266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6815","nonce":195572266}]},"public_addr":"172.21.15.169:6813/195572266","cluster_addr":"172.21.15.169:6814/195572266","heartbeat_back_addr":"172.21.15.169:6816/195572266","heartbeat_front_addr":"172.21.15.169:6815/195572266","state":["exists","up"]},{"osd":4,"uuid":"3a6121ef-6309-41b7-a964-6772c010b3a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1397212305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":1397212305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":1397212305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1397212305}]},"public_addr":"172.21.15.173:6800/1397212305","cluster_addr":"172.21.15.173:6801/1397212305","heartbeat_back_addr":"172.21.15.173:6803/1397212305","heartbeat_front_addr":"172.21.15.173:6802/1397212305","state":["exists","up"]},{"osd":5,"uuid":"3c2f6dd9-9784-4a07-8710-55ac3de82964","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3525888417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":3525888417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":3525888417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3525888417}]},"public_addr":"172.21.15.173:6804/3525888417","cluster_addr":"172.21.15.173:6805/3525888417","heartbeat_back_addr":"172.21.15.173:6807/3525888417","heartbeat_front_addr":"172.21.15.173:6806/3525888417","state":["exists","up"]},{"osd":6,"uuid":"d7759df7-1978-4d70-94a2-721540804f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2147192629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":2147192629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":2147192629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2147192629}]},"public_addr":"172.21.15.173:6808/2147192629","cluster_addr":"172.21.15.173:6809/2147192629","heartbeat_back_addr":"172.21.15.173:6811/2147192629","heartbeat_front_addr":"172.21.15.173:6810/2147192629","state":["exists","up"]},{"osd":7,"uuid":"8843039e-2b17-4230-9cca-5788a2294d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":4014976876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":4014976876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6815","nonce":4014976876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":4014976876}]},"public_addr":"172.21.15.173:6812/4014976876","cluster_addr":"172.21.15.173:6813/4014976876","heartbeat_back_addr":"172.21.15.173:6815/4014976876","heartbeat_front_addr":"172.21.15.173:6814/4014976876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:38.337984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:17.103074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:19.742089+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:56.380847+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:05.474553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:25.002739+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:44.675338+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:05.582338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/4041747161":"2024-03-13T03:33:59.734991+0000","172.21.15.169:6800/2779395456":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/555866474":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/342292643":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/197210445":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/1430250501":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/3918553041":"2024-03-13T03:28:48.518732+0000","172.21.15.169:0/1020514981":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/284191239":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/1212459137":"2024-03-13T03:28:48.518732+0000","172.21.15.169:6800/204892396":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/2751007072":"2024-03-13T03:28:48.518732+0000","172.21.15.169:0/2349376510":"2024-03-13T03:28:21.868923+0000","172.21.15.169:6800/108891420":"2024-03-13T03:28:48.518732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:34:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: Standby manager daemon x restarted 2024-03-12T03:34:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: Standby manager daemon x started 2024-03-12T03:34:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.? 172.21.15.173:0/2221033380' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:34:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.? 172.21.15.173:0/2221033380' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:34:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.? 172.21.15.173:0/2221033380' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:34:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.? 172.21.15.173:0/2221033380' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:34:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: Active manager daemon y restarted 2024-03-12T03:34:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: Activating manager daemon y 2024-03-12T03:34:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: osdmap e83: 8 total, 8 up, 8 in 2024-03-12T03:34:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: mgrmap e18: y(active, starting, since 0.0109715s), standbys: x 2024-03-12T03:34:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:34:00.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:34:00.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:34:00.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: Manager daemon y is now available 2024-03-12T03:34:00.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:00.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:00.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:00.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:00.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:00.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:00.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:00.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:00.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:00 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3913711069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:34:00.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: Standby manager daemon x restarted 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: Standby manager daemon x started 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.? 172.21.15.173:0/2221033380' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.? 172.21.15.173:0/2221033380' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.? 172.21.15.173:0/2221033380' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.? 172.21.15.173:0/2221033380' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: Active manager daemon y restarted 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: Activating manager daemon y 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: osdmap e83: 8 total, 8 up, 8 in 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: mgrmap e18: y(active, starting, since 0.0109715s), standbys: x 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:34:00.279 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:34:00.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:34:00.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:34:00.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:34:00.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:34:00.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:00.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:00.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:34:00.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:34:00.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:34:00.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: Manager daemon y is now available 2024-03-12T03:34:00.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:00.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: Standby manager daemon x restarted 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: Standby manager daemon x started 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.? 172.21.15.173:0/2221033380' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.? 172.21.15.173:0/2221033380' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.? 172.21.15.173:0/2221033380' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.? 172.21.15.173:0/2221033380' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: Active manager daemon y restarted 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: Activating manager daemon y 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: osdmap e83: 8 total, 8 up, 8 in 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: mgrmap e18: y(active, starting, since 0.0109715s), standbys: x 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:34:00.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: Manager daemon y is now available 2024-03-12T03:34:00.282 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:00.283 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:00.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:00.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:00.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:00.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:00.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:00.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:00.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3913711069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:34:00.283 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:59 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: CherryPy Checker: 2024-03-12T03:34:00.284 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:59 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: The Application mounted at '' has an empty config. 2024-03-12T03:34:00.284 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:59 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:34:00.284 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:59 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:33:59] ENGINE Serving on http://:::9283 2024-03-12T03:34:00.284 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:33:59 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:33:59] ENGINE Bus STARTED 2024-03-12T03:34:00.284 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:00.284 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:00.284 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:00.284 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:00.284 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:00.284 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:00.284 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:00 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3913711069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:34:00.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-12T03:34:00.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-12T03:34:00.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-12T03:34:00.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-12T03:34:00.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-12T03:34:00.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-12T03:34:00.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-12T03:34:00.693 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-12T03:34:01.890 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:01.890 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:01.891 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:01.891 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:02.018 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:01 smithi169 ceph-mon[29344]: mgrmap e19: y(active, since 1.04474s), standbys: x 2024-03-12T03:34:02.019 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:01 smithi169 ceph-mon[29344]: [12/Mar/2024:03:34:00] ENGINE Bus STARTING 2024-03-12T03:34:02.019 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:01 smithi169 ceph-mon[29344]: [12/Mar/2024:03:34:00] ENGINE Serving on http://172.21.15.169:8765 2024-03-12T03:34:02.019 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:01 smithi169 ceph-mon[29344]: [12/Mar/2024:03:34:01] ENGINE Serving on https://172.21.15.169:7150 2024-03-12T03:34:02.019 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:01 smithi169 ceph-mon[29344]: [12/Mar/2024:03:34:01] ENGINE Bus STARTED 2024-03-12T03:34:02.019 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:34:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:34:02.019 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:01 smithi169 ceph-mon[39949]: mgrmap e19: y(active, since 1.04474s), standbys: x 2024-03-12T03:34:02.020 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:01 smithi169 ceph-mon[39949]: [12/Mar/2024:03:34:00] ENGINE Bus STARTING 2024-03-12T03:34:02.020 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:01 smithi169 ceph-mon[39949]: [12/Mar/2024:03:34:00] ENGINE Serving on http://172.21.15.169:8765 2024-03-12T03:34:02.020 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:01 smithi169 ceph-mon[39949]: [12/Mar/2024:03:34:01] ENGINE Serving on https://172.21.15.169:7150 2024-03-12T03:34:02.020 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:01 smithi169 ceph-mon[39949]: [12/Mar/2024:03:34:01] ENGINE Bus STARTED 2024-03-12T03:34:02.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:01 smithi173 ceph-mon[33544]: mgrmap e19: y(active, since 1.04474s), standbys: x 2024-03-12T03:34:02.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:01 smithi173 ceph-mon[33544]: [12/Mar/2024:03:34:00] ENGINE Bus STARTING 2024-03-12T03:34:02.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:01 smithi173 ceph-mon[33544]: [12/Mar/2024:03:34:00] ENGINE Serving on http://172.21.15.169:8765 2024-03-12T03:34:02.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:01 smithi173 ceph-mon[33544]: [12/Mar/2024:03:34:01] ENGINE Serving on https://172.21.15.169:7150 2024-03-12T03:34:02.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:01 smithi173 ceph-mon[33544]: [12/Mar/2024:03:34:01] ENGINE Bus STARTED 2024-03-12T03:34:02.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:34:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:34:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:02 smithi173 ceph-mon[33544]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:02 smithi173 ceph-mon[33544]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-12T03:34:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:02 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:03.101 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:02 smithi169 ceph-mon[29344]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:03.101 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:02 smithi169 ceph-mon[29344]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-12T03:34:03.101 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:02 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:03.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:02 smithi169 ceph-mon[39949]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:03.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:02 smithi169 ceph-mon[39949]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-12T03:34:03.102 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:02 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:04.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:03 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:04.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:03 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:04.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:03 smithi173 ceph-mon[33544]: osdmap e84: 8 total, 8 up, 8 in 2024-03-12T03:34:04.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:03 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:04.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:03 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:04.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:03 smithi169 ceph-mon[29344]: osdmap e84: 8 total, 8 up, 8 in 2024-03-12T03:34:04.283 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:03 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:04.283 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:03 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:04.284 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:03 smithi169 ceph-mon[39949]: osdmap e84: 8 total, 8 up, 8 in 2024-03-12T03:34:05.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: Adjusting osd_memory_target on smithi173 to 3760M 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[29344]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-12T03:34:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: Adjusting osd_memory_target on smithi173 to 3760M 2024-03-12T03:34:05.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:05.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:05 smithi169 ceph-mon[39949]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-12T03:34:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: Adjusting osd_memory_target on smithi173 to 3760M 2024-03-12T03:34:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:05 smithi173 ceph-mon[33544]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-12T03:34:06.314 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:06.314 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:06.315 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:06.315 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:07.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:07 smithi169 ceph-mon[29344]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:07.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:07 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:07.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:07 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:07.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:07 smithi169 ceph-mon[39949]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:07.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:07 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:07.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:07 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:07 smithi173 ceph-mon[33544]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:07 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:07 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.832 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: Adjusting osd_memory_target on smithi169 to 3760M 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:09.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: Adjusting osd_memory_target on smithi169 to 3760M 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: Adjusting osd_memory_target on smithi169 to 3760M 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:09.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:09.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-12T03:34:09.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-12T03:34:09.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:09.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:10.266 INFO:teuthology.orchestra.run.smithi169.stdout:34359738417 2024-03-12T03:34:10.267 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-12T03:34:10.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[29344]: Updating smithi169:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:34:10.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[29344]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:34:10.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[29344]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:34:10.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[29344]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:34:10.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[29344]: Updating smithi169:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:34:10.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[29344]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:34:10.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[29344]: Deploying daemon alertmanager.a on smithi169 2024-03-12T03:34:10.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[39949]: Updating smithi169:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:34:10.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[39949]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:34:10.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[39949]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:34:10.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[39949]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:34:10.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[39949]: Updating smithi169:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:34:10.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[39949]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:34:10.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:10 smithi169 ceph-mon[39949]: Deploying daemon alertmanager.a on smithi169 2024-03-12T03:34:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:10 smithi173 ceph-mon[33544]: Updating smithi169:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:34:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:10 smithi173 ceph-mon[33544]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:34:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:10 smithi173 ceph-mon[33544]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:34:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:10 smithi173 ceph-mon[33544]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:34:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:10 smithi173 ceph-mon[33544]: Updating smithi169:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:34:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:10 smithi173 ceph-mon[33544]: Updating smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:34:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:10 smithi173 ceph-mon[33544]: Deploying daemon alertmanager.a on smithi169 2024-03-12T03:34:11.642 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:11 smithi169 ceph-mon[29344]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:34:11.643 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:11 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:34:11.643 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:11 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:34:11.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:11 smithi169 ceph-mon[39949]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:34:11.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:11 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:34:11.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:11 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:34:11.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:34:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:34:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:11 smithi173 ceph-mon[33544]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:34:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:11 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:34:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:11 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:34:12.436 INFO:teuthology.orchestra.run.smithi169.stdout:55834574892 2024-03-12T03:34:12.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-12T03:34:12.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:34:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:34:12.671 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:12 smithi169 ceph-mon[29344]: osdmap e85: 8 total, 8 up, 8 in 2024-03-12T03:34:12.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:12 smithi169 ceph-mon[39949]: osdmap e85: 8 total, 8 up, 8 in 2024-03-12T03:34:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:12 smithi173 ceph-mon[33544]: osdmap e85: 8 total, 8 up, 8 in 2024-03-12T03:34:13.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:13 smithi169 ceph-mon[39949]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-12T03:34:13.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:13 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:13.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:13 smithi169 ceph-mon[39949]: osdmap e86: 8 total, 8 up, 8 in 2024-03-12T03:34:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:13 smithi169 ceph-mon[29344]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-12T03:34:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:13 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:13 smithi169 ceph-mon[29344]: osdmap e86: 8 total, 8 up, 8 in 2024-03-12T03:34:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:13 smithi173 ceph-mon[33544]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-12T03:34:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:13 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:13 smithi173 ceph-mon[33544]: osdmap e86: 8 total, 8 up, 8 in 2024-03-12T03:34:15.264 INFO:teuthology.orchestra.run.smithi169.stdout:176093659161 2024-03-12T03:34:15.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-12T03:34:15.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:15 smithi169 ceph-mon[29344]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-12T03:34:15.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:15.816 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:15 smithi169 ceph-mon[39949]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-12T03:34:15.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:15 smithi173 ceph-mon[33544]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-12T03:34:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:17.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:17 smithi169 ceph-mon[29344]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-03-12T03:34:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:17 smithi169 ceph-mon[39949]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-03-12T03:34:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:17 smithi173 ceph-mon[33544]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-03-12T03:34:18.450 INFO:teuthology.orchestra.run.smithi169.stdout:137438953502 2024-03-12T03:34:18.450 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-12T03:34:19.393 INFO:teuthology.orchestra.run.smithi169.stdout:77309411368 2024-03-12T03:34:19.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-12T03:34:19.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:19 smithi169 ceph-mon[29344]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:19.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:19 smithi169 ceph-mon[39949]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:19 smithi173 ceph-mon[33544]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:20.169 INFO:teuthology.orchestra.run.smithi169.stdout:253403070479 2024-03-12T03:34:20.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-12T03:34:20.531 INFO:teuthology.orchestra.run.smithi169.stdout:115964117026 2024-03-12T03:34:20.531 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-12T03:34:21.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:21 smithi169 ceph-mon[29344]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 622 B/s rd, 0 op/s 2024-03-12T03:34:21.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:21 smithi169 ceph-mon[39949]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 622 B/s rd, 0 op/s 2024-03-12T03:34:21.908 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:34:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:34:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:21 smithi173 ceph-mon[33544]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 622 B/s rd, 0 op/s 2024-03-12T03:34:22.443 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:22.445 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:22.445 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:22.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:34:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:34:23.912 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:23 smithi169 ceph-mon[29344]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:34:23.912 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:23 smithi169 ceph-mon[29344]: 2.0 scrub starts 2024-03-12T03:34:23.913 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:23 smithi169 ceph-mon[29344]: 2.0 scrub ok 2024-03-12T03:34:23.913 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:23 smithi169 ceph-mon[29344]: 2.2 scrub starts 2024-03-12T03:34:23.913 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:23 smithi169 ceph-mon[29344]: 2.2 scrub ok 2024-03-12T03:34:23.913 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:23 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:23.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:23 smithi169 ceph-mon[39949]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:34:23.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:23 smithi169 ceph-mon[39949]: 2.0 scrub starts 2024-03-12T03:34:23.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:23 smithi169 ceph-mon[39949]: 2.0 scrub ok 2024-03-12T03:34:23.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:23 smithi169 ceph-mon[39949]: 2.2 scrub starts 2024-03-12T03:34:23.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:23 smithi169 ceph-mon[39949]: 2.2 scrub ok 2024-03-12T03:34:23.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:23 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:23 smithi173 ceph-mon[33544]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:34:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:23 smithi173 ceph-mon[33544]: 2.0 scrub starts 2024-03-12T03:34:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:23 smithi173 ceph-mon[33544]: 2.0 scrub ok 2024-03-12T03:34:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:23 smithi173 ceph-mon[33544]: 2.2 scrub starts 2024-03-12T03:34:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:23 smithi173 ceph-mon[33544]: 2.2 scrub ok 2024-03-12T03:34:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:23 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:24 smithi169 ceph-mon[29344]: 3.a deep-scrub starts 2024-03-12T03:34:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:24 smithi169 ceph-mon[29344]: 3.a deep-scrub ok 2024-03-12T03:34:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:24 smithi169 ceph-mon[39949]: 3.a deep-scrub starts 2024-03-12T03:34:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:24 smithi169 ceph-mon[39949]: 3.a deep-scrub ok 2024-03-12T03:34:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:24 smithi173 ceph-mon[33544]: 3.a deep-scrub starts 2024-03-12T03:34:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:24 smithi173 ceph-mon[33544]: 3.a deep-scrub ok 2024-03-12T03:34:25.291 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:25.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:25 smithi169 ceph-mon[29344]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-03-12T03:34:25.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:25 smithi169 ceph-mon[29344]: 4.1b deep-scrub starts 2024-03-12T03:34:25.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:25 smithi169 ceph-mon[29344]: 4.1b deep-scrub ok 2024-03-12T03:34:25.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:25 smithi169 ceph-mon[29344]: 3.e scrub starts 2024-03-12T03:34:25.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:25 smithi169 ceph-mon[29344]: 3.e scrub ok 2024-03-12T03:34:25.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:25 smithi169 ceph-mon[39949]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-03-12T03:34:25.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:25 smithi169 ceph-mon[39949]: 4.1b deep-scrub starts 2024-03-12T03:34:25.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:25 smithi169 ceph-mon[39949]: 4.1b deep-scrub ok 2024-03-12T03:34:25.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:25 smithi169 ceph-mon[39949]: 3.e scrub starts 2024-03-12T03:34:25.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:25 smithi169 ceph-mon[39949]: 3.e scrub ok 2024-03-12T03:34:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:25 smithi173 ceph-mon[33544]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-03-12T03:34:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:25 smithi173 ceph-mon[33544]: 4.1b deep-scrub starts 2024-03-12T03:34:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:25 smithi173 ceph-mon[33544]: 4.1b deep-scrub ok 2024-03-12T03:34:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:25 smithi173 ceph-mon[33544]: 3.e scrub starts 2024-03-12T03:34:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:25 smithi173 ceph-mon[33544]: 3.e scrub ok 2024-03-12T03:34:26.319 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:26.496 INFO:teuthology.orchestra.run.smithi169.stdout:55834574895 2024-03-12T03:34:26.772 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[29344]: 5.0 scrub starts 2024-03-12T03:34:26.772 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[29344]: 5.0 scrub ok 2024-03-12T03:34:26.772 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[29344]: 3.16 scrub starts 2024-03-12T03:34:26.772 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[29344]: 3.16 scrub ok 2024-03-12T03:34:26.772 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:34:26.772 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:34:26.773 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[29344]: osdmap e87: 8 total, 8 up, 8 in 2024-03-12T03:34:26.773 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[29344]: 5.1d scrub starts 2024-03-12T03:34:26.773 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[29344]: 5.1d scrub ok 2024-03-12T03:34:26.773 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4023414468' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:34:26.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[39949]: 5.0 scrub starts 2024-03-12T03:34:26.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[39949]: 5.0 scrub ok 2024-03-12T03:34:26.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[39949]: 3.16 scrub starts 2024-03-12T03:34:26.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[39949]: 3.16 scrub ok 2024-03-12T03:34:26.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:34:26.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:34:26.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[39949]: osdmap e87: 8 total, 8 up, 8 in 2024-03-12T03:34:26.774 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[39949]: 5.1d scrub starts 2024-03-12T03:34:26.774 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[39949]: 5.1d scrub ok 2024-03-12T03:34:26.774 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:26 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4023414468' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:34:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:26 smithi173 ceph-mon[33544]: 5.0 scrub starts 2024-03-12T03:34:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:26 smithi173 ceph-mon[33544]: 5.0 scrub ok 2024-03-12T03:34:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:26 smithi173 ceph-mon[33544]: 3.16 scrub starts 2024-03-12T03:34:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:26 smithi173 ceph-mon[33544]: 3.16 scrub ok 2024-03-12T03:34:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:26 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:34:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:26 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:34:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:26 smithi173 ceph-mon[33544]: osdmap e87: 8 total, 8 up, 8 in 2024-03-12T03:34:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:26 smithi173 ceph-mon[33544]: 5.1d scrub starts 2024-03-12T03:34:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:26 smithi173 ceph-mon[33544]: 5.1d scrub ok 2024-03-12T03:34:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:26 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4023414468' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:34:27.908 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:27 smithi169 ceph-mon[29344]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-12T03:34:27.909 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:27 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:34:27.909 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:27 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:34:27.909 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:27 smithi169 ceph-mon[29344]: 3.10 scrub starts 2024-03-12T03:34:27.909 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:27 smithi169 ceph-mon[29344]: 3.10 scrub ok 2024-03-12T03:34:27.909 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:27 smithi169 ceph-mon[29344]: osdmap e88: 8 total, 8 up, 8 in 2024-03-12T03:34:27.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:27 smithi169 ceph-mon[39949]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-12T03:34:27.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:27 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:34:27.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:27 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:34:27.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:27 smithi169 ceph-mon[39949]: 3.10 scrub starts 2024-03-12T03:34:27.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:27 smithi169 ceph-mon[39949]: 3.10 scrub ok 2024-03-12T03:34:27.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:27 smithi169 ceph-mon[39949]: osdmap e88: 8 total, 8 up, 8 in 2024-03-12T03:34:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:27 smithi173 ceph-mon[33544]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-12T03:34:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:27 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:34:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:27 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:34:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:27 smithi173 ceph-mon[33544]: 3.10 scrub starts 2024-03-12T03:34:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:27 smithi173 ceph-mon[33544]: 3.10 scrub ok 2024-03-12T03:34:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:27 smithi173 ceph-mon[33544]: osdmap e88: 8 total, 8 up, 8 in 2024-03-12T03:34:29.281 INFO:teuthology.orchestra.run.smithi169.stdout:176093659164 2024-03-12T03:34:29.781 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:29 smithi169 ceph-mon[29344]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:29.781 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:29 smithi169 ceph-mon[29344]: 5.2 scrub starts 2024-03-12T03:34:29.781 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:29 smithi169 ceph-mon[29344]: 5.2 scrub ok 2024-03-12T03:34:29.781 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:29 smithi169 ceph-mon[29344]: 5.19 scrub starts 2024-03-12T03:34:29.781 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:29 smithi169 ceph-mon[29344]: 5.19 scrub ok 2024-03-12T03:34:29.781 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:29 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/630621071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:34:29.781 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:29 smithi169 ceph-mon[39949]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:29.781 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:29 smithi169 ceph-mon[39949]: 5.2 scrub starts 2024-03-12T03:34:29.781 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:29 smithi169 ceph-mon[39949]: 5.2 scrub ok 2024-03-12T03:34:29.782 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:29 smithi169 ceph-mon[39949]: 5.19 scrub starts 2024-03-12T03:34:29.782 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:29 smithi169 ceph-mon[39949]: 5.19 scrub ok 2024-03-12T03:34:29.782 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:29 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/630621071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:34:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:29 smithi173 ceph-mon[33544]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:29 smithi173 ceph-mon[33544]: 5.2 scrub starts 2024-03-12T03:34:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:29 smithi173 ceph-mon[33544]: 5.2 scrub ok 2024-03-12T03:34:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:29 smithi173 ceph-mon[33544]: 5.19 scrub starts 2024-03-12T03:34:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:29 smithi173 ceph-mon[33544]: 5.19 scrub ok 2024-03-12T03:34:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:29 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/630621071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:34:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:34:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:34:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[29344]: 4.15 scrub starts 2024-03-12T03:34:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[29344]: 4.15 scrub ok 2024-03-12T03:34:30.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[29344]: osdmap e89: 8 total, 8 up, 8 in 2024-03-12T03:34:30.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:30.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[29344]: 5.c scrub starts 2024-03-12T03:34:30.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[29344]: 5.c scrub ok 2024-03-12T03:34:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:34:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:34:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[39949]: 4.15 scrub starts 2024-03-12T03:34:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[39949]: 4.15 scrub ok 2024-03-12T03:34:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[39949]: osdmap e89: 8 total, 8 up, 8 in 2024-03-12T03:34:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[39949]: 5.c scrub starts 2024-03-12T03:34:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:30 smithi169 ceph-mon[39949]: 5.c scrub ok 2024-03-12T03:34:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:30 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:34:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:30 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:34:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:30 smithi173 ceph-mon[33544]: 4.15 scrub starts 2024-03-12T03:34:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:30 smithi173 ceph-mon[33544]: 4.15 scrub ok 2024-03-12T03:34:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:30 smithi173 ceph-mon[33544]: osdmap e89: 8 total, 8 up, 8 in 2024-03-12T03:34:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:30 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:30 smithi173 ceph-mon[33544]: 5.c scrub starts 2024-03-12T03:34:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:30 smithi173 ceph-mon[33544]: 5.c scrub ok 2024-03-12T03:34:31.265 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659161 got 176093659164 for osd.5 2024-03-12T03:34:31.265 DEBUG:teuthology.parallel:result is None 2024-03-12T03:34:31.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:31 smithi169 ceph-mon[29344]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:31.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:31 smithi169 ceph-mon[29344]: 5.3 scrub starts 2024-03-12T03:34:31.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:31 smithi169 ceph-mon[29344]: 5.3 scrub ok 2024-03-12T03:34:31.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:31 smithi169 ceph-mon[39949]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:31.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:31 smithi169 ceph-mon[39949]: 5.3 scrub starts 2024-03-12T03:34:31.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:31 smithi169 ceph-mon[39949]: 5.3 scrub ok 2024-03-12T03:34:31.901 INFO:teuthology.orchestra.run.smithi169.stdout:219043332118 2024-03-12T03:34:31.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-12T03:34:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:31 smithi173 ceph-mon[33544]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:31 smithi173 ceph-mon[33544]: 5.3 scrub starts 2024-03-12T03:34:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:31 smithi173 ceph-mon[33544]: 5.3 scrub ok 2024-03-12T03:34:32.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:34:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:34:32.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:34:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:34:32.887 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:32 smithi169 ceph-mon[39949]: 3.d scrub starts 2024-03-12T03:34:32.887 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:32 smithi169 ceph-mon[39949]: 3.d scrub ok 2024-03-12T03:34:32.887 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:32 smithi169 ceph-mon[39949]: 6.f scrub starts 2024-03-12T03:34:32.887 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:32 smithi169 ceph-mon[39949]: 6.f scrub ok 2024-03-12T03:34:32.887 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:32 smithi169 ceph-mon[29344]: 3.d scrub starts 2024-03-12T03:34:32.887 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:32 smithi169 ceph-mon[29344]: 3.d scrub ok 2024-03-12T03:34:32.887 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:32 smithi169 ceph-mon[29344]: 6.f scrub starts 2024-03-12T03:34:32.888 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:32 smithi169 ceph-mon[29344]: 6.f scrub ok 2024-03-12T03:34:32.888 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:32 smithi169 podman[74090]: 2024-03-12 03:34:32.885341293 +0000 UTC m=+3.366834773 container create f1777cd585edeb9b5af26e3b828ac507f32e733e80dbf71901ad9e47001d40a0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:34:32.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:32 smithi173 ceph-mon[33544]: 3.d scrub starts 2024-03-12T03:34:32.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:32 smithi173 ceph-mon[33544]: 3.d scrub ok 2024-03-12T03:34:32.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:32 smithi173 ceph-mon[33544]: 6.f scrub starts 2024-03-12T03:34:32.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:32 smithi173 ceph-mon[33544]: 6.f scrub ok 2024-03-12T03:34:33.687 INFO:teuthology.orchestra.run.smithi169.stdout:253403070483 2024-03-12T03:34:33.878 INFO:teuthology.orchestra.run.smithi169.stdout:34359738422 2024-03-12T03:34:33.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[39949]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-12T03:34:33.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:33.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[39949]: 3.17 scrub starts 2024-03-12T03:34:33.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[39949]: 3.17 scrub ok 2024-03-12T03:34:33.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[39949]: osdmap e90: 8 total, 8 up, 8 in 2024-03-12T03:34:33.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[39949]: 5.12 scrub starts 2024-03-12T03:34:33.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[39949]: 5.12 scrub ok 2024-03-12T03:34:33.880 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[29344]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-12T03:34:33.880 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:33.880 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[29344]: 3.17 scrub starts 2024-03-12T03:34:33.880 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[29344]: 3.17 scrub ok 2024-03-12T03:34:33.880 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[29344]: osdmap e90: 8 total, 8 up, 8 in 2024-03-12T03:34:33.880 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[29344]: 5.12 scrub starts 2024-03-12T03:34:33.880 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:33 smithi169 ceph-mon[29344]: 5.12 scrub ok 2024-03-12T03:34:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:33 smithi173 ceph-mon[33544]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-12T03:34:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:33 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:33 smithi173 ceph-mon[33544]: 3.17 scrub starts 2024-03-12T03:34:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:33 smithi173 ceph-mon[33544]: 3.17 scrub ok 2024-03-12T03:34:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:33 smithi173 ceph-mon[33544]: osdmap e90: 8 total, 8 up, 8 in 2024-03-12T03:34:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:33 smithi173 ceph-mon[33544]: 5.12 scrub starts 2024-03-12T03:34:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:33 smithi173 ceph-mon[33544]: 5.12 scrub ok 2024-03-12T03:34:34.174 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:34 smithi169 podman[74090]: 2024-03-12 03:34:34.011536471 +0000 UTC m=+4.493029951 container init f1777cd585edeb9b5af26e3b828ac507f32e733e80dbf71901ad9e47001d40a0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:34:34.174 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:34 smithi169 podman[74090]: 2024-03-12 03:34:34.014788867 +0000 UTC m=+4.496282356 container start f1777cd585edeb9b5af26e3b828ac507f32e733e80dbf71901ad9e47001d40a0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:34:34.174 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[74234]: ts=2024-03-12T03:34:34.030Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-12T03:34:34.174 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[74234]: ts=2024-03-12T03:34:34.030Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-12T03:34:34.174 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[74234]: ts=2024-03-12T03:34:34.032Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.169 port=9094 2024-03-12T03:34:34.174 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[74234]: ts=2024-03-12T03:34:34.033Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-12T03:34:34.174 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[74234]: ts=2024-03-12T03:34:34.060Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:34:34.174 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[74234]: ts=2024-03-12T03:34:34.060Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:34:34.174 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[74234]: ts=2024-03-12T03:34:34.063Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-12T03:34:34.174 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[74234]: ts=2024-03-12T03:34:34.063Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-12T03:34:34.175 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:34 smithi169 bash[74090]: f1777cd585edeb9b5af26e3b828ac507f32e733e80dbf71901ad9e47001d40a0 2024-03-12T03:34:34.175 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:34 smithi169 systemd[1]: Started Ceph alertmanager.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:34:34.604 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:34:34] ENGINE Bus STOPPING 2024-03-12T03:34:34.604 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:34:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:34:34.604 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:34:34] ENGINE Bus STOPPED 2024-03-12T03:34:34.604 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:34:34] ENGINE Bus STARTING 2024-03-12T03:34:34.605 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:34:34] ENGINE Serving on http://:::9283 2024-03-12T03:34:34.605 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:34:34] ENGINE Bus STARTED 2024-03-12T03:34:34.607 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4167714788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T03:34:34.607 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: 5.1f scrub starts 2024-03-12T03:34:34.607 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: 5.1f scrub ok 2024-03-12T03:34:34.607 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2367595914' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:34:34.607 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.608 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.608 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.608 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.608 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.608 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.608 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:34:34.608 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.608 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: 3.4 scrub starts 2024-03-12T03:34:34.608 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[29344]: 3.4 scrub ok 2024-03-12T03:34:34.608 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4167714788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T03:34:34.608 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: 5.1f scrub starts 2024-03-12T03:34:34.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: 5.1f scrub ok 2024-03-12T03:34:34.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2367595914' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:34:34.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:34:34.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: 3.4 scrub starts 2024-03-12T03:34:34.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:34 smithi169 ceph-mon[39949]: 3.4 scrub ok 2024-03-12T03:34:34.685 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738422 for osd.0 2024-03-12T03:34:34.685 DEBUG:teuthology.parallel:result is None 2024-03-12T03:34:34.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4167714788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T03:34:34.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: 5.1f scrub starts 2024-03-12T03:34:34.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: 5.1f scrub ok 2024-03-12T03:34:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2367595914' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:34:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:34:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: 3.4 scrub starts 2024-03-12T03:34:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:34 smithi173 ceph-mon[33544]: 3.4 scrub ok 2024-03-12T03:34:35.267 INFO:teuthology.orchestra.run.smithi169.stdout:115964117030 2024-03-12T03:34:35.490 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070479 got 253403070483 for osd.7 2024-03-12T03:34:35.491 DEBUG:teuthology.parallel:result is None 2024-03-12T03:34:35.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 711 B/s rd, 0 op/s 2024-03-12T03:34:35.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:34:35.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:34:35.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: Deploying daemon grafana.a on smithi173 2024-03-12T03:34:35.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: 4.10 deep-scrub starts 2024-03-12T03:34:35.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: 4.10 deep-scrub ok 2024-03-12T03:34:35.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: 6.16 scrub starts 2024-03-12T03:34:35.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: 6.16 scrub ok 2024-03-12T03:34:35.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:35.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: 4.13 scrub starts 2024-03-12T03:34:35.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: 4.13 scrub ok 2024-03-12T03:34:35.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3742398906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: 4.2 scrub starts 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[39949]: 4.2 scrub ok 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 711 B/s rd, 0 op/s 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: Deploying daemon grafana.a on smithi173 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: 4.10 deep-scrub starts 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: 4.10 deep-scrub ok 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: 6.16 scrub starts 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: 6.16 scrub ok 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: 4.13 scrub starts 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: 4.13 scrub ok 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3742398906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:34:35.828 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: 4.2 scrub starts 2024-03-12T03:34:35.829 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:35 smithi169 ceph-mon[29344]: 4.2 scrub ok 2024-03-12T03:34:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 711 B/s rd, 0 op/s 2024-03-12T03:34:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:34:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:34:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: Deploying daemon grafana.a on smithi173 2024-03-12T03:34:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: 4.10 deep-scrub starts 2024-03-12T03:34:35.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: 4.10 deep-scrub ok 2024-03-12T03:34:35.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: 6.16 scrub starts 2024-03-12T03:34:35.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: 6.16 scrub ok 2024-03-12T03:34:35.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:35.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: 4.13 scrub starts 2024-03-12T03:34:35.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: 4.13 scrub ok 2024-03-12T03:34:35.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3742398906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:34:35.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: 4.2 scrub starts 2024-03-12T03:34:35.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:35 smithi173 ceph-mon[33544]: 4.2 scrub ok 2024-03-12T03:34:36.140 INFO:teuthology.orchestra.run.smithi169.stdout:77309411371 2024-03-12T03:34:36.140 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:36 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[74234]: ts=2024-03-12T03:34:36.034Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000851046s 2024-03-12T03:34:36.883 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:36 smithi169 ceph-mon[29344]: 4.5 scrub starts 2024-03-12T03:34:36.883 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:36 smithi169 ceph-mon[29344]: 4.5 scrub ok 2024-03-12T03:34:36.883 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:36 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1005580103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:34:36.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:36 smithi169 ceph-mon[39949]: 4.5 scrub starts 2024-03-12T03:34:36.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:36 smithi169 ceph-mon[39949]: 4.5 scrub ok 2024-03-12T03:34:36.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:36 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1005580103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:34:36.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:36 smithi173 ceph-mon[33544]: 4.5 scrub starts 2024-03-12T03:34:36.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:36 smithi173 ceph-mon[33544]: 4.5 scrub ok 2024-03-12T03:34:36.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:36 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1005580103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:34:37.091 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117026 got 115964117030 for osd.3 2024-03-12T03:34:37.091 DEBUG:teuthology.parallel:result is None 2024-03-12T03:34:37.677 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411371 for osd.2 2024-03-12T03:34:37.678 DEBUG:teuthology.parallel:result is None 2024-03-12T03:34:37.832 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[39949]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:34:37.833 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[39949]: 5.f scrub starts 2024-03-12T03:34:37.833 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[39949]: 5.f scrub ok 2024-03-12T03:34:37.833 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[39949]: 5.11 scrub starts 2024-03-12T03:34:37.833 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[39949]: 5.11 scrub ok 2024-03-12T03:34:37.833 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[39949]: 5.1c scrub starts 2024-03-12T03:34:37.833 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[39949]: 5.1c scrub ok 2024-03-12T03:34:37.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[29344]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:34:37.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[29344]: 5.f scrub starts 2024-03-12T03:34:37.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[29344]: 5.f scrub ok 2024-03-12T03:34:37.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[29344]: 5.11 scrub starts 2024-03-12T03:34:37.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[29344]: 5.11 scrub ok 2024-03-12T03:34:37.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[29344]: 5.1c scrub starts 2024-03-12T03:34:37.834 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:37 smithi169 ceph-mon[29344]: 5.1c scrub ok 2024-03-12T03:34:37.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:37 smithi173 ceph-mon[33544]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:34:37.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:37 smithi173 ceph-mon[33544]: 5.f scrub starts 2024-03-12T03:34:37.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:37 smithi173 ceph-mon[33544]: 5.f scrub ok 2024-03-12T03:34:37.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:37 smithi173 ceph-mon[33544]: 5.11 scrub starts 2024-03-12T03:34:37.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:37 smithi173 ceph-mon[33544]: 5.11 scrub ok 2024-03-12T03:34:37.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:37 smithi173 ceph-mon[33544]: 5.1c scrub starts 2024-03-12T03:34:37.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:37 smithi173 ceph-mon[33544]: 5.1c scrub ok 2024-03-12T03:34:38.141 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574895 for osd.1 2024-03-12T03:34:38.141 DEBUG:teuthology.parallel:result is None 2024-03-12T03:34:38.147 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:38.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:38 smithi169 ceph-mon[29344]: 6.9 scrub starts 2024-03-12T03:34:38.827 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:38 smithi169 ceph-mon[29344]: 6.9 scrub ok 2024-03-12T03:34:38.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:38 smithi169 ceph-mon[39949]: 6.9 scrub starts 2024-03-12T03:34:38.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:38 smithi169 ceph-mon[39949]: 6.9 scrub ok 2024-03-12T03:34:38.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:38 smithi173 ceph-mon[33544]: 6.9 scrub starts 2024-03-12T03:34:38.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:38 smithi173 ceph-mon[33544]: 6.9 scrub ok 2024-03-12T03:34:39.897 INFO:teuthology.orchestra.run.smithi169.stdout:137438953508 2024-03-12T03:34:39.897 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[29344]: 6.1c scrub starts 2024-03-12T03:34:39.897 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[29344]: 6.1c scrub ok 2024-03-12T03:34:39.897 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[29344]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:39.897 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[29344]: 5.15 scrub starts 2024-03-12T03:34:39.897 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[29344]: 5.15 scrub ok 2024-03-12T03:34:39.898 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[29344]: 5.5 scrub starts 2024-03-12T03:34:39.898 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[29344]: 5.5 scrub ok 2024-03-12T03:34:39.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[39949]: 6.1c scrub starts 2024-03-12T03:34:39.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[39949]: 6.1c scrub ok 2024-03-12T03:34:39.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[39949]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:39.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[39949]: 5.15 scrub starts 2024-03-12T03:34:39.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[39949]: 5.15 scrub ok 2024-03-12T03:34:39.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[39949]: 5.5 scrub starts 2024-03-12T03:34:39.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:39 smithi169 ceph-mon[39949]: 5.5 scrub ok 2024-03-12T03:34:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:39 smithi173 ceph-mon[33544]: 6.1c scrub starts 2024-03-12T03:34:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:39 smithi173 ceph-mon[33544]: 6.1c scrub ok 2024-03-12T03:34:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:39 smithi173 ceph-mon[33544]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:39 smithi173 ceph-mon[33544]: 5.15 scrub starts 2024-03-12T03:34:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:39 smithi173 ceph-mon[33544]: 5.15 scrub ok 2024-03-12T03:34:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:39 smithi173 ceph-mon[33544]: 5.5 scrub starts 2024-03-12T03:34:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:39 smithi173 ceph-mon[33544]: 5.5 scrub ok 2024-03-12T03:34:40.859 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[29344]: 3.13 scrub starts 2024-03-12T03:34:40.859 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[29344]: 3.13 scrub ok 2024-03-12T03:34:40.859 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[29344]: 4.8 deep-scrub starts 2024-03-12T03:34:40.859 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[29344]: 4.8 deep-scrub ok 2024-03-12T03:34:40.859 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[29344]: 4.6 scrub starts 2024-03-12T03:34:40.860 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[29344]: 4.6 scrub ok 2024-03-12T03:34:40.860 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4037114381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:34:40.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[39949]: 3.13 scrub starts 2024-03-12T03:34:40.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[39949]: 3.13 scrub ok 2024-03-12T03:34:40.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[39949]: 4.8 deep-scrub starts 2024-03-12T03:34:40.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[39949]: 4.8 deep-scrub ok 2024-03-12T03:34:40.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[39949]: 4.6 scrub starts 2024-03-12T03:34:40.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[39949]: 4.6 scrub ok 2024-03-12T03:34:40.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:40 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4037114381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:34:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:40 smithi173 ceph-mon[33544]: 3.13 scrub starts 2024-03-12T03:34:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:40 smithi173 ceph-mon[33544]: 3.13 scrub ok 2024-03-12T03:34:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:40 smithi173 ceph-mon[33544]: 4.8 deep-scrub starts 2024-03-12T03:34:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:40 smithi173 ceph-mon[33544]: 4.8 deep-scrub ok 2024-03-12T03:34:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:40 smithi173 ceph-mon[33544]: 4.6 scrub starts 2024-03-12T03:34:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:40 smithi173 ceph-mon[33544]: 4.6 scrub ok 2024-03-12T03:34:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:40 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4037114381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:34:41.066 INFO:teuthology.orchestra.run.smithi169.stdout:219043332121 2024-03-12T03:34:41.496 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953502 got 137438953508 for osd.4 2024-03-12T03:34:41.496 DEBUG:teuthology.parallel:result is None 2024-03-12T03:34:41.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[29344]: 5.9 scrub starts 2024-03-12T03:34:41.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[29344]: 5.9 scrub ok 2024-03-12T03:34:41.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[29344]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:34:41.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[29344]: 6.13 deep-scrub starts 2024-03-12T03:34:41.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[29344]: 6.13 deep-scrub ok 2024-03-12T03:34:41.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[29344]: 3.12 deep-scrub starts 2024-03-12T03:34:41.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[29344]: 3.12 deep-scrub ok 2024-03-12T03:34:41.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/950670836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T03:34:41.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:34:41.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:34:41.816 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[39949]: 5.9 scrub starts 2024-03-12T03:34:41.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[39949]: 5.9 scrub ok 2024-03-12T03:34:41.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[39949]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:34:41.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[39949]: 6.13 deep-scrub starts 2024-03-12T03:34:41.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[39949]: 6.13 deep-scrub ok 2024-03-12T03:34:41.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[39949]: 3.12 deep-scrub starts 2024-03-12T03:34:41.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[39949]: 3.12 deep-scrub ok 2024-03-12T03:34:41.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/950670836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T03:34:41.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:34:41.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:34:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:41 smithi173 ceph-mon[33544]: 5.9 scrub starts 2024-03-12T03:34:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:41 smithi173 ceph-mon[33544]: 5.9 scrub ok 2024-03-12T03:34:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:41 smithi173 ceph-mon[33544]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:34:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:41 smithi173 ceph-mon[33544]: 6.13 deep-scrub starts 2024-03-12T03:34:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:41 smithi173 ceph-mon[33544]: 6.13 deep-scrub ok 2024-03-12T03:34:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:41 smithi173 ceph-mon[33544]: 3.12 deep-scrub starts 2024-03-12T03:34:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:41 smithi173 ceph-mon[33544]: 3.12 deep-scrub ok 2024-03-12T03:34:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:41 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/950670836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T03:34:41.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:41 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:34:41.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:41 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:34:42.134 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:34:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:34:42.161 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332118 got 219043332121 for osd.6 2024-03-12T03:34:42.161 DEBUG:teuthology.parallel:result is None 2024-03-12T03:34:42.161 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-12T03:34:42.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:34:42.406 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:42.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:34:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:34:42.829 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[29344]: 6.5 deep-scrub starts 2024-03-12T03:34:42.829 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[29344]: 6.5 deep-scrub ok 2024-03-12T03:34:42.829 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[29344]: 3.1e deep-scrub starts 2024-03-12T03:34:42.829 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[29344]: 3.1e deep-scrub ok 2024-03-12T03:34:42.829 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[29344]: 3.5 scrub starts 2024-03-12T03:34:42.829 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[29344]: 3.5 scrub ok 2024-03-12T03:34:42.829 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[29344]: osdmap e91: 8 total, 8 up, 8 in 2024-03-12T03:34:42.829 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[39949]: 6.5 deep-scrub starts 2024-03-12T03:34:42.829 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[39949]: 6.5 deep-scrub ok 2024-03-12T03:34:42.829 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[39949]: 3.1e deep-scrub starts 2024-03-12T03:34:42.830 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[39949]: 3.1e deep-scrub ok 2024-03-12T03:34:42.830 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[39949]: 3.5 scrub starts 2024-03-12T03:34:42.830 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[39949]: 3.5 scrub ok 2024-03-12T03:34:42.830 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:42 smithi169 ceph-mon[39949]: osdmap e91: 8 total, 8 up, 8 in 2024-03-12T03:34:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:42 smithi173 ceph-mon[33544]: 6.5 deep-scrub starts 2024-03-12T03:34:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:42 smithi173 ceph-mon[33544]: 6.5 deep-scrub ok 2024-03-12T03:34:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:42 smithi173 ceph-mon[33544]: 3.1e deep-scrub starts 2024-03-12T03:34:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:42 smithi173 ceph-mon[33544]: 3.1e deep-scrub ok 2024-03-12T03:34:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:42 smithi173 ceph-mon[33544]: 3.5 scrub starts 2024-03-12T03:34:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:42 smithi173 ceph-mon[33544]: 3.5 scrub ok 2024-03-12T03:34:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:42 smithi173 ceph-mon[33544]: osdmap e91: 8 total, 8 up, 8 in 2024-03-12T03:34:43.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[39949]: 3.15 scrub starts 2024-03-12T03:34:43.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[39949]: 3.15 scrub ok 2024-03-12T03:34:43.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[39949]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:43.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[39949]: 5.6 scrub starts 2024-03-12T03:34:43.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[39949]: 5.6 scrub ok 2024-03-12T03:34:43.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:43.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[39949]: osdmap e92: 8 total, 8 up, 8 in 2024-03-12T03:34:43.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[39949]: 3.19 scrub starts 2024-03-12T03:34:43.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[39949]: 3.19 scrub ok 2024-03-12T03:34:43.867 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[29344]: 3.15 scrub starts 2024-03-12T03:34:43.867 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[29344]: 3.15 scrub ok 2024-03-12T03:34:43.867 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[29344]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:43.867 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[29344]: 5.6 scrub starts 2024-03-12T03:34:43.867 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[29344]: 5.6 scrub ok 2024-03-12T03:34:43.868 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:43.868 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[29344]: osdmap e92: 8 total, 8 up, 8 in 2024-03-12T03:34:43.868 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[29344]: 3.19 scrub starts 2024-03-12T03:34:43.868 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:43 smithi169 ceph-mon[29344]: 3.19 scrub ok 2024-03-12T03:34:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:43 smithi173 ceph-mon[33544]: 3.15 scrub starts 2024-03-12T03:34:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:43 smithi173 ceph-mon[33544]: 3.15 scrub ok 2024-03-12T03:34:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:43 smithi173 ceph-mon[33544]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:43 smithi173 ceph-mon[33544]: 5.6 scrub starts 2024-03-12T03:34:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:43 smithi173 ceph-mon[33544]: 5.6 scrub ok 2024-03-12T03:34:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:43 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:43 smithi173 ceph-mon[33544]: osdmap e92: 8 total, 8 up, 8 in 2024-03-12T03:34:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:43 smithi173 ceph-mon[33544]: 3.19 scrub starts 2024-03-12T03:34:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:43 smithi173 ceph-mon[33544]: 3.19 scrub ok 2024-03-12T03:34:44.173 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[74234]: ts=2024-03-12T03:34:44.035Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002220644s 2024-03-12T03:34:44.283 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:34:44.285 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:34:44.883 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[29344]: 6.14 scrub starts 2024-03-12T03:34:44.883 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[29344]: 6.14 scrub ok 2024-03-12T03:34:44.884 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[29344]: 6.8 scrub starts 2024-03-12T03:34:44.884 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[29344]: 6.8 scrub ok 2024-03-12T03:34:44.884 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[29344]: 3.1a scrub starts 2024-03-12T03:34:44.884 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[29344]: 3.1a scrub ok 2024-03-12T03:34:44.884 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[29344]: 4.7 scrub starts 2024-03-12T03:34:44.884 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[29344]: 4.7 scrub ok 2024-03-12T03:34:44.884 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:34:44.884 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:34:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[39949]: 6.14 scrub starts 2024-03-12T03:34:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[39949]: 6.14 scrub ok 2024-03-12T03:34:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[39949]: 6.8 scrub starts 2024-03-12T03:34:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[39949]: 6.8 scrub ok 2024-03-12T03:34:44.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[39949]: 3.1a scrub starts 2024-03-12T03:34:44.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[39949]: 3.1a scrub ok 2024-03-12T03:34:44.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[39949]: 4.7 scrub starts 2024-03-12T03:34:44.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[39949]: 4.7 scrub ok 2024-03-12T03:34:44.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:34:44.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:44 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:34:44.910 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:44 smithi173 ceph-mon[33544]: 6.14 scrub starts 2024-03-12T03:34:44.910 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:44 smithi173 ceph-mon[33544]: 6.14 scrub ok 2024-03-12T03:34:44.910 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:44 smithi173 ceph-mon[33544]: 6.8 scrub starts 2024-03-12T03:34:44.910 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:44 smithi173 ceph-mon[33544]: 6.8 scrub ok 2024-03-12T03:34:44.910 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:44 smithi173 ceph-mon[33544]: 3.1a scrub starts 2024-03-12T03:34:44.910 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:44 smithi173 ceph-mon[33544]: 3.1a scrub ok 2024-03-12T03:34:44.910 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:44 smithi173 ceph-mon[33544]: 4.7 scrub starts 2024-03-12T03:34:44.910 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:44 smithi173 ceph-mon[33544]: 4.7 scrub ok 2024-03-12T03:34:44.910 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:44 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:34:44.910 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:44 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:34:44.913 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-12T03:34:43.764239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1296,"num_read_kb":1110,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":810,"ondisk_log_size":810,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":232608,"kb_used_data":8156,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749728608,"statfs":{"total":767960285184,"available":767722094592,"internally_reserved":0,"allocated":8351744,"data_stored":4630624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001994"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.690774+0000","last_change":"2024-03-12T03:33:27.183162+0000","last_active":"2024-03-12T03:34:33.690774+0000","last_peered":"2024-03-12T03:34:33.690774+0000","last_clean":"2024-03-12T03:34:33.690774+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:34:33.690774+0000","last_undegraded":"2024-03-12T03:34:33.690774+0000","last_fullsized":"2024-03-12T03:34:33.690774+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:37.952931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257264+0000","last_change":"2024-03-12T03:34:29.244162+0000","last_active":"2024-03-12T03:34:33.257264+0000","last_peered":"2024-03-12T03:34:33.257264+0000","last_clean":"2024-03-12T03:34:33.257264+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:34:33.257264+0000","last_undegraded":"2024-03-12T03:34:33.257264+0000","last_fullsized":"2024-03-12T03:34:33.257264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:29.244078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:29.244078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.593696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141067+0000","last_change":"2024-03-12T03:33:21.573453+0000","last_active":"2024-03-12T03:34:33.141067+0000","last_peered":"2024-03-12T03:34:33.141067+0000","last_clean":"2024-03-12T03:34:33.141067+0000","last_became_active":"2024-03-12T03:33:21.573080+0000","last_became_peered":"2024-03-12T03:33:21.573080+0000","last_unstale":"2024-03-12T03:34:33.141067+0000","last_undegraded":"2024-03-12T03:34:33.141067+0000","last_fullsized":"2024-03-12T03:34:33.141067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:37.186503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.690632+0000","last_change":"2024-03-12T03:33:23.164433+0000","last_active":"2024-03-12T03:34:33.690632+0000","last_peered":"2024-03-12T03:34:33.690632+0000","last_clean":"2024-03-12T03:34:33.690632+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:34:33.690632+0000","last_undegraded":"2024-03-12T03:34:33.690632+0000","last_fullsized":"2024-03-12T03:34:33.690632+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:41.633750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141856+0000","last_change":"2024-03-12T03:33:27.382583+0000","last_active":"2024-03-12T03:34:33.141856+0000","last_peered":"2024-03-12T03:34:33.141856+0000","last_clean":"2024-03-12T03:34:33.141856+0000","last_became_active":"2024-03-12T03:33:27.382050+0000","last_became_peered":"2024-03-12T03:33:27.382050+0000","last_unstale":"2024-03-12T03:34:33.141856+0000","last_undegraded":"2024-03-12T03:34:33.141856+0000","last_fullsized":"2024-03-12T03:34:33.141856+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:47.150721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.690640+0000","last_change":"2024-03-12T03:33:25.180282+0000","last_active":"2024-03-12T03:34:33.690640+0000","last_peered":"2024-03-12T03:34:33.690640+0000","last_clean":"2024-03-12T03:34:33.690640+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:34:33.690640+0000","last_undegraded":"2024-03-12T03:34:33.690640+0000","last_fullsized":"2024-03-12T03:34:33.690640+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141935+0000","last_change":"2024-03-12T03:33:21.573256+0000","last_active":"2024-03-12T03:34:33.141935+0000","last_peered":"2024-03-12T03:34:33.141935+0000","last_clean":"2024-03-12T03:34:33.141935+0000","last_became_active":"2024-03-12T03:33:21.572876+0000","last_became_peered":"2024-03-12T03:33:21.572876+0000","last_unstale":"2024-03-12T03:34:33.141935+0000","last_undegraded":"2024-03-12T03:34:33.141935+0000","last_fullsized":"2024-03-12T03:34:33.141935+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":126,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142016+0000","last_change":"2024-03-12T03:33:23.161838+0000","last_active":"2024-03-12T03:34:33.142016+0000","last_peered":"2024-03-12T03:34:33.142016+0000","last_clean":"2024-03-12T03:34:33.142016+0000","last_became_active":"2024-03-12T03:33:23.161549+0000","last_became_peered":"2024-03-12T03:33:23.161549+0000","last_unstale":"2024-03-12T03:34:33.142016+0000","last_undegraded":"2024-03-12T03:34:33.142016+0000","last_fullsized":"2024-03-12T03:34:33.142016+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:37.915866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140629+0000","last_change":"2024-03-12T03:33:27.189571+0000","last_active":"2024-03-12T03:34:33.140629+0000","last_peered":"2024-03-12T03:34:33.140629+0000","last_clean":"2024-03-12T03:34:33.140629+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:34:33.140629+0000","last_undegraded":"2024-03-12T03:34:33.140629+0000","last_fullsized":"2024-03-12T03:34:33.140629+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:46.083461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259342+0000","last_change":"2024-03-12T03:33:25.176211+0000","last_active":"2024-03-12T03:34:33.259342+0000","last_peered":"2024-03-12T03:34:33.259342+0000","last_clean":"2024-03-12T03:34:33.259342+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:34:33.259342+0000","last_undegraded":"2024-03-12T03:34:33.259342+0000","last_fullsized":"2024-03-12T03:34:33.259342+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:40.095375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259703+0000","last_change":"2024-03-12T03:33:21.102752+0000","last_active":"2024-03-12T03:34:33.259703+0000","last_peered":"2024-03-12T03:34:33.259703+0000","last_clean":"2024-03-12T03:34:33.259703+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:34:33.259703+0000","last_undegraded":"2024-03-12T03:34:33.259703+0000","last_fullsized":"2024-03-12T03:34:33.259703+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:48.338428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'21","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.690983+0000","last_change":"2024-03-12T03:33:23.163157+0000","last_active":"2024-03-12T03:34:33.690983+0000","last_peered":"2024-03-12T03:34:33.690983+0000","last_clean":"2024-03-12T03:34:33.690983+0000","last_became_active":"2024-03-12T03:33:23.162749+0000","last_became_peered":"2024-03-12T03:33:23.162749+0000","last_unstale":"2024-03-12T03:34:33.690983+0000","last_undegraded":"2024-03-12T03:34:33.690983+0000","last_fullsized":"2024-03-12T03:34:33.690983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:34.859449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259214+0000","last_change":"2024-03-12T03:33:27.188261+0000","last_active":"2024-03-12T03:34:33.259214+0000","last_peered":"2024-03-12T03:34:33.259214+0000","last_clean":"2024-03-12T03:34:33.259214+0000","last_became_active":"2024-03-12T03:33:27.187943+0000","last_became_peered":"2024-03-12T03:33:27.187943+0000","last_unstale":"2024-03-12T03:34:33.259214+0000","last_undegraded":"2024-03-12T03:34:33.259214+0000","last_fullsized":"2024-03-12T03:34:33.259214+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.815395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693540+0000","last_change":"2024-03-12T03:33:25.182503+0000","last_active":"2024-03-12T03:34:33.693540+0000","last_peered":"2024-03-12T03:34:33.693540+0000","last_clean":"2024-03-12T03:34:33.693540+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:34:33.693540+0000","last_undegraded":"2024-03-12T03:34:33.693540+0000","last_fullsized":"2024-03-12T03:34:33.693540+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:47.480158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259799+0000","last_change":"2024-03-12T03:33:21.098805+0000","last_active":"2024-03-12T03:34:33.259799+0000","last_peered":"2024-03-12T03:34:33.259799+0000","last_clean":"2024-03-12T03:34:33.259799+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:34:33.259799+0000","last_undegraded":"2024-03-12T03:34:33.259799+0000","last_fullsized":"2024-03-12T03:34:33.259799+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:35.479958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"86'14","reported_seq":100,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.691148+0000","last_change":"2024-03-12T03:34:23.795189+0000","last_active":"2024-03-12T03:34:33.691148+0000","last_peered":"2024-03-12T03:34:33.691148+0000","last_clean":"2024-03-12T03:34:33.691148+0000","last_became_active":"2024-03-12T03:33:50.155908+0000","last_became_peered":"2024-03-12T03:33:50.155908+0000","last_unstale":"2024-03-12T03:34:33.691148+0000","last_undegraded":"2024-03-12T03:34:33.691148+0000","last_fullsized":"2024-03-12T03:34:33.691148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.795040+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:35.995794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.691429+0000","last_change":"2024-03-12T03:33:27.380906+0000","last_active":"2024-03-12T03:34:33.691429+0000","last_peered":"2024-03-12T03:34:33.691429+0000","last_clean":"2024-03-12T03:34:33.691429+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:34:33.691429+0000","last_undegraded":"2024-03-12T03:34:33.691429+0000","last_fullsized":"2024-03-12T03:34:33.691429+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:40.585476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257865+0000","last_change":"2024-03-12T03:34:26.233162+0000","last_active":"2024-03-12T03:34:33.257865+0000","last_peered":"2024-03-12T03:34:33.257865+0000","last_clean":"2024-03-12T03:34:33.257865+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:34:33.257865+0000","last_undegraded":"2024-03-12T03:34:33.257865+0000","last_fullsized":"2024-03-12T03:34:33.257865+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:26.233072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:26.233072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.096475+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141454+0000","last_change":"2024-03-12T03:33:21.103671+0000","last_active":"2024-03-12T03:34:33.141454+0000","last_peered":"2024-03-12T03:34:33.141454+0000","last_clean":"2024-03-12T03:34:33.141454+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:34:33.141454+0000","last_undegraded":"2024-03-12T03:34:33.141454+0000","last_fullsized":"2024-03-12T03:34:33.141454+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:39.146149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561674+0000","last_change":"2024-03-12T03:33:23.161120+0000","last_active":"2024-03-12T03:34:33.561674+0000","last_peered":"2024-03-12T03:34:33.561674+0000","last_clean":"2024-03-12T03:34:33.561674+0000","last_became_active":"2024-03-12T03:33:23.160702+0000","last_became_peered":"2024-03-12T03:33:23.160702+0000","last_unstale":"2024-03-12T03:34:33.561674+0000","last_undegraded":"2024-03-12T03:34:33.561674+0000","last_fullsized":"2024-03-12T03:34:33.561674+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:39.664690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142268+0000","last_change":"2024-03-12T03:33:27.381440+0000","last_active":"2024-03-12T03:34:33.142268+0000","last_peered":"2024-03-12T03:34:33.142268+0000","last_clean":"2024-03-12T03:34:33.142268+0000","last_became_active":"2024-03-12T03:33:27.380880+0000","last_became_peered":"2024-03-12T03:33:27.380880+0000","last_unstale":"2024-03-12T03:34:33.142268+0000","last_undegraded":"2024-03-12T03:34:33.142268+0000","last_fullsized":"2024-03-12T03:34:33.142268+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.898172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:36.949185+0000","last_change":"2024-03-12T03:34:36.949207+0000","last_active":"2024-03-12T03:34:36.949185+0000","last_peered":"2024-03-12T03:34:36.949185+0000","last_clean":"2024-03-12T03:34:36.949185+0000","last_became_active":"2024-03-12T03:33:25.181087+0000","last_became_peered":"2024-03-12T03:33:25.181087+0000","last_unstale":"2024-03-12T03:34:36.949185+0000","last_undegraded":"2024-03-12T03:34:36.949185+0000","last_fullsized":"2024-03-12T03:34:36.949185+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.949124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.949124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:57.461161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.690461+0000","last_change":"2024-03-12T03:33:21.573553+0000","last_active":"2024-03-12T03:34:33.690461+0000","last_peered":"2024-03-12T03:34:33.690461+0000","last_clean":"2024-03-12T03:34:33.690461+0000","last_became_active":"2024-03-12T03:33:21.573218+0000","last_became_peered":"2024-03-12T03:33:21.573218+0000","last_unstale":"2024-03-12T03:34:33.690461+0000","last_undegraded":"2024-03-12T03:34:33.690461+0000","last_fullsized":"2024-03-12T03:34:33.690461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141485+0000","last_change":"2024-03-12T03:33:23.162704+0000","last_active":"2024-03-12T03:34:33.141485+0000","last_peered":"2024-03-12T03:34:33.141485+0000","last_clean":"2024-03-12T03:34:33.141485+0000","last_became_active":"2024-03-12T03:33:23.162427+0000","last_became_peered":"2024-03-12T03:33:23.162427+0000","last_unstale":"2024-03-12T03:34:33.141485+0000","last_undegraded":"2024-03-12T03:34:33.141485+0000","last_fullsized":"2024-03-12T03:34:33.141485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.329189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:37.658672+0000","last_change":"2024-03-12T03:34:37.658704+0000","last_active":"2024-03-12T03:34:37.658672+0000","last_peered":"2024-03-12T03:34:37.658672+0000","last_clean":"2024-03-12T03:34:37.658672+0000","last_became_active":"2024-03-12T03:33:27.189152+0000","last_became_peered":"2024-03-12T03:33:27.189152+0000","last_unstale":"2024-03-12T03:34:37.658672+0000","last_undegraded":"2024-03-12T03:34:37.658672+0000","last_fullsized":"2024-03-12T03:34:37.658672+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:34:37.658602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:37.658602+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:38.697166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.696207+0000","last_change":"2024-03-12T03:34:33.696207+0000","last_active":"2024-03-12T03:34:33.696207+0000","last_peered":"2024-03-12T03:34:33.696207+0000","last_clean":"2024-03-12T03:34:33.696207+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:34:33.696207+0000","last_undegraded":"2024-03-12T03:34:33.696207+0000","last_fullsized":"2024-03-12T03:34:33.696207+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:33.696102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:33.696102+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.429139+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257322+0000","last_change":"2024-03-12T03:33:21.097524+0000","last_active":"2024-03-12T03:34:33.257322+0000","last_peered":"2024-03-12T03:34:33.257322+0000","last_clean":"2024-03-12T03:34:33.257322+0000","last_became_active":"2024-03-12T03:33:21.097218+0000","last_became_peered":"2024-03-12T03:33:21.097218+0000","last_unstale":"2024-03-12T03:34:33.257322+0000","last_undegraded":"2024-03-12T03:34:33.257322+0000","last_fullsized":"2024-03-12T03:34:33.257322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'17","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140984+0000","last_change":"2024-03-12T03:33:23.553096+0000","last_active":"2024-03-12T03:34:33.140984+0000","last_peered":"2024-03-12T03:34:33.140984+0000","last_clean":"2024-03-12T03:34:33.140984+0000","last_became_active":"2024-03-12T03:33:23.552812+0000","last_became_peered":"2024-03-12T03:33:23.552812+0000","last_unstale":"2024-03-12T03:34:33.140984+0000","last_undegraded":"2024-03-12T03:34:33.140984+0000","last_fullsized":"2024-03-12T03:34:33.140984+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:48.664041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257952+0000","last_change":"2024-03-12T03:33:27.184475+0000","last_active":"2024-03-12T03:34:33.257952+0000","last_peered":"2024-03-12T03:34:33.257952+0000","last_clean":"2024-03-12T03:34:33.257952+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:34:33.257952+0000","last_undegraded":"2024-03-12T03:34:33.257952+0000","last_fullsized":"2024-03-12T03:34:33.257952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:52.246462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141241+0000","last_change":"2024-03-12T03:33:25.179324+0000","last_active":"2024-03-12T03:34:33.141241+0000","last_peered":"2024-03-12T03:34:33.141241+0000","last_clean":"2024-03-12T03:34:33.141241+0000","last_became_active":"2024-03-12T03:33:25.178346+0000","last_became_peered":"2024-03-12T03:33:25.178346+0000","last_unstale":"2024-03-12T03:34:33.141241+0000","last_undegraded":"2024-03-12T03:34:33.141241+0000","last_fullsized":"2024-03-12T03:34:33.141241+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:36.290977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141686+0000","last_change":"2024-03-12T03:33:21.095860+0000","last_active":"2024-03-12T03:34:33.141686+0000","last_peered":"2024-03-12T03:34:33.141686+0000","last_clean":"2024-03-12T03:34:33.141686+0000","last_became_active":"2024-03-12T03:33:21.095556+0000","last_became_peered":"2024-03-12T03:33:21.095556+0000","last_unstale":"2024-03-12T03:34:33.141686+0000","last_undegraded":"2024-03-12T03:34:33.141686+0000","last_fullsized":"2024-03-12T03:34:33.141686+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.291634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142598+0000","last_change":"2024-03-12T03:33:23.159103+0000","last_active":"2024-03-12T03:34:33.142598+0000","last_peered":"2024-03-12T03:34:33.142598+0000","last_clean":"2024-03-12T03:34:33.142598+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:34:33.142598+0000","last_undegraded":"2024-03-12T03:34:33.142598+0000","last_fullsized":"2024-03-12T03:34:33.142598+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:47.268938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693506+0000","last_change":"2024-03-12T03:33:27.190023+0000","last_active":"2024-03-12T03:34:33.693506+0000","last_peered":"2024-03-12T03:34:33.693506+0000","last_clean":"2024-03-12T03:34:33.693506+0000","last_became_active":"2024-03-12T03:33:27.189696+0000","last_became_peered":"2024-03-12T03:33:27.189696+0000","last_unstale":"2024-03-12T03:34:33.693506+0000","last_undegraded":"2024-03-12T03:34:33.693506+0000","last_fullsized":"2024-03-12T03:34:33.693506+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.530688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:36.701671+0000","last_change":"2024-03-12T03:34:36.701692+0000","last_active":"2024-03-12T03:34:36.701671+0000","last_peered":"2024-03-12T03:34:36.701671+0000","last_clean":"2024-03-12T03:34:36.701671+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:34:36.701671+0000","last_undegraded":"2024-03-12T03:34:36.701671+0000","last_fullsized":"2024-03-12T03:34:36.701671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.701611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.701611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.280217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141208+0000","last_change":"2024-03-12T03:34:32.746113+0000","last_active":"2024-03-12T03:34:33.141208+0000","last_peered":"2024-03-12T03:34:33.141208+0000","last_clean":"2024-03-12T03:34:33.141208+0000","last_became_active":"2024-03-12T03:33:21.096560+0000","last_became_peered":"2024-03-12T03:33:21.096560+0000","last_unstale":"2024-03-12T03:34:33.141208+0000","last_undegraded":"2024-03-12T03:34:33.141208+0000","last_fullsized":"2024-03-12T03:34:33.141208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:32.746027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:32.746027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:33.564397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"90'6","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:34.274429+0000","last_change":"2024-03-12T03:34:34.274478+0000","last_active":"2024-03-12T03:34:34.274429+0000","last_peered":"2024-03-12T03:34:34.274429+0000","last_clean":"2024-03-12T03:34:34.274429+0000","last_became_active":"2024-03-12T03:33:23.161801+0000","last_became_peered":"2024-03-12T03:33:23.161801+0000","last_unstale":"2024-03-12T03:34:34.274429+0000","last_undegraded":"2024-03-12T03:34:34.274429+0000","last_fullsized":"2024-03-12T03:34:34.274429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.274353+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.042399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:40.191598+0000","last_change":"2024-03-12T03:34:40.191618+0000","last_active":"2024-03-12T03:34:40.191598+0000","last_peered":"2024-03-12T03:34:40.191598+0000","last_clean":"2024-03-12T03:34:40.191598+0000","last_became_active":"2024-03-12T03:33:27.381110+0000","last_became_peered":"2024-03-12T03:33:27.381110+0000","last_unstale":"2024-03-12T03:34:40.191598+0000","last_undegraded":"2024-03-12T03:34:40.191598+0000","last_fullsized":"2024-03-12T03:34:40.191598+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.191538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.480916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.694028+0000","last_change":"2024-03-12T03:33:25.178948+0000","last_active":"2024-03-12T03:34:33.694028+0000","last_peered":"2024-03-12T03:34:33.694028+0000","last_clean":"2024-03-12T03:34:33.694028+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:34:33.694028+0000","last_undegraded":"2024-03-12T03:34:33.694028+0000","last_fullsized":"2024-03-12T03:34:33.694028+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:44.813464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259793+0000","last_change":"2024-03-12T03:34:25.171761+0000","last_active":"2024-03-12T03:34:33.259793+0000","last_peered":"2024-03-12T03:34:33.259793+0000","last_clean":"2024-03-12T03:34:33.259793+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:34:33.259793+0000","last_undegraded":"2024-03-12T03:34:33.259793+0000","last_fullsized":"2024-03-12T03:34:33.259793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:25.171622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.171622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:37.280670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'18","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141921+0000","last_change":"2024-03-12T03:33:23.552489+0000","last_active":"2024-03-12T03:34:33.141921+0000","last_peered":"2024-03-12T03:34:33.141921+0000","last_clean":"2024-03-12T03:34:33.141921+0000","last_became_active":"2024-03-12T03:33:23.552210+0000","last_became_peered":"2024-03-12T03:33:23.552210+0000","last_unstale":"2024-03-12T03:34:33.141921+0000","last_undegraded":"2024-03-12T03:34:33.141921+0000","last_fullsized":"2024-03-12T03:34:33.141921+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:36.029074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140452+0000","last_change":"2024-03-12T03:33:27.193134+0000","last_active":"2024-03-12T03:34:33.140452+0000","last_peered":"2024-03-12T03:34:33.140452+0000","last_clean":"2024-03-12T03:34:33.140452+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:34:33.140452+0000","last_undegraded":"2024-03-12T03:34:33.140452+0000","last_fullsized":"2024-03-12T03:34:33.140452+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:43.487708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141983+0000","last_change":"2024-03-12T03:33:25.184266+0000","last_active":"2024-03-12T03:34:33.141983+0000","last_peered":"2024-03-12T03:34:33.141983+0000","last_clean":"2024-03-12T03:34:33.141983+0000","last_became_active":"2024-03-12T03:33:25.183725+0000","last_became_peered":"2024-03-12T03:33:25.183725+0000","last_unstale":"2024-03-12T03:34:33.141983+0000","last_undegraded":"2024-03-12T03:34:33.141983+0000","last_fullsized":"2024-03-12T03:34:33.141983+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:33.221144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693992+0000","last_change":"2024-03-12T03:33:21.107392+0000","last_active":"2024-03-12T03:34:33.693992+0000","last_peered":"2024-03-12T03:34:33.693992+0000","last_clean":"2024-03-12T03:34:33.693992+0000","last_became_active":"2024-03-12T03:33:21.107037+0000","last_became_peered":"2024-03-12T03:33:21.107037+0000","last_unstale":"2024-03-12T03:34:33.693992+0000","last_undegraded":"2024-03-12T03:34:33.693992+0000","last_fullsized":"2024-03-12T03:34:33.693992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:34.083565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257758+0000","last_change":"2024-03-12T03:33:23.161686+0000","last_active":"2024-03-12T03:34:33.257758+0000","last_peered":"2024-03-12T03:34:33.257758+0000","last_clean":"2024-03-12T03:34:33.257758+0000","last_became_active":"2024-03-12T03:33:23.160818+0000","last_became_peered":"2024-03-12T03:33:23.160818+0000","last_unstale":"2024-03-12T03:34:33.257758+0000","last_undegraded":"2024-03-12T03:34:33.257758+0000","last_fullsized":"2024-03-12T03:34:33.257758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.366562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142135+0000","last_change":"2024-03-12T03:33:27.192215+0000","last_active":"2024-03-12T03:34:33.142135+0000","last_peered":"2024-03-12T03:34:33.142135+0000","last_clean":"2024-03-12T03:34:33.142135+0000","last_became_active":"2024-03-12T03:33:27.191710+0000","last_became_peered":"2024-03-12T03:33:27.191710+0000","last_unstale":"2024-03-12T03:34:33.142135+0000","last_undegraded":"2024-03-12T03:34:33.142135+0000","last_fullsized":"2024-03-12T03:34:33.142135+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:45.524065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.258695+0000","last_change":"2024-03-12T03:34:33.258733+0000","last_active":"2024-03-12T03:34:33.258695+0000","last_peered":"2024-03-12T03:34:33.258695+0000","last_clean":"2024-03-12T03:34:33.258695+0000","last_became_active":"2024-03-12T03:33:25.182004+0000","last_became_peered":"2024-03-12T03:33:25.182004+0000","last_unstale":"2024-03-12T03:34:33.258695+0000","last_undegraded":"2024-03-12T03:34:33.258695+0000","last_fullsized":"2024-03-12T03:34:33.258695+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:33.258626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:33.258626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.420748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.691234+0000","last_change":"2024-03-12T03:33:21.109429+0000","last_active":"2024-03-12T03:34:33.691234+0000","last_peered":"2024-03-12T03:34:33.691234+0000","last_clean":"2024-03-12T03:34:33.691234+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:34:33.691234+0000","last_undegraded":"2024-03-12T03:34:33.691234+0000","last_fullsized":"2024-03-12T03:34:33.691234+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:34.953625+0000","last_change":"2024-03-12T03:34:34.953658+0000","last_active":"2024-03-12T03:34:34.953625+0000","last_peered":"2024-03-12T03:34:34.953625+0000","last_clean":"2024-03-12T03:34:34.953625+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:34:34.953625+0000","last_undegraded":"2024-03-12T03:34:34.953625+0000","last_fullsized":"2024-03-12T03:34:34.953625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:34:34.953565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.953565+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.161677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:34.718570+0000","last_change":"2024-03-12T03:34:34.718591+0000","last_active":"2024-03-12T03:34:34.718570+0000","last_peered":"2024-03-12T03:34:34.718570+0000","last_clean":"2024-03-12T03:34:34.718570+0000","last_became_active":"2024-03-12T03:33:27.194094+0000","last_became_peered":"2024-03-12T03:33:27.194094+0000","last_unstale":"2024-03-12T03:34:34.718570+0000","last_undegraded":"2024-03-12T03:34:34.718570+0000","last_fullsized":"2024-03-12T03:34:34.718570+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:34.718509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.718509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.921678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:38.139665+0000","last_change":"2024-03-12T03:34:38.139709+0000","last_active":"2024-03-12T03:34:38.139665+0000","last_peered":"2024-03-12T03:34:38.139665+0000","last_clean":"2024-03-12T03:34:38.139665+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:34:38.139665+0000","last_undegraded":"2024-03-12T03:34:38.139665+0000","last_fullsized":"2024-03-12T03:34:38.139665+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:38.139582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.139582+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:39.751323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:38.663608+0000","last_change":"2024-03-12T03:34:38.663634+0000","last_active":"2024-03-12T03:34:38.663608+0000","last_peered":"2024-03-12T03:34:38.663608+0000","last_clean":"2024-03-12T03:34:38.663608+0000","last_became_active":"2024-03-12T03:33:21.573244+0000","last_became_peered":"2024-03-12T03:33:21.573244+0000","last_unstale":"2024-03-12T03:34:38.663608+0000","last_undegraded":"2024-03-12T03:34:38.663608+0000","last_fullsized":"2024-03-12T03:34:38.663608+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:34:38.663542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.663542+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:59.053797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141451+0000","last_change":"2024-03-12T03:33:23.552006+0000","last_active":"2024-03-12T03:34:33.141451+0000","last_peered":"2024-03-12T03:34:33.141451+0000","last_clean":"2024-03-12T03:34:33.141451+0000","last_became_active":"2024-03-12T03:33:23.551706+0000","last_became_peered":"2024-03-12T03:33:23.551706+0000","last_unstale":"2024-03-12T03:34:33.141451+0000","last_undegraded":"2024-03-12T03:34:33.141451+0000","last_fullsized":"2024-03-12T03:34:33.141451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:45.152617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.691671+0000","last_change":"2024-03-12T03:33:27.185209+0000","last_active":"2024-03-12T03:34:33.691671+0000","last_peered":"2024-03-12T03:34:33.691671+0000","last_clean":"2024-03-12T03:34:33.691671+0000","last_became_active":"2024-03-12T03:33:27.184916+0000","last_became_peered":"2024-03-12T03:33:27.184916+0000","last_unstale":"2024-03-12T03:34:33.691671+0000","last_undegraded":"2024-03-12T03:34:33.691671+0000","last_fullsized":"2024-03-12T03:34:33.691671+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141598+0000","last_change":"2024-03-12T03:33:25.179522+0000","last_active":"2024-03-12T03:34:33.141598+0000","last_peered":"2024-03-12T03:34:33.141598+0000","last_clean":"2024-03-12T03:34:33.141598+0000","last_became_active":"2024-03-12T03:33:25.179158+0000","last_became_peered":"2024-03-12T03:33:25.179158+0000","last_unstale":"2024-03-12T03:34:33.141598+0000","last_undegraded":"2024-03-12T03:34:33.141598+0000","last_fullsized":"2024-03-12T03:34:33.141598+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:41.537531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:40.753211+0000","last_change":"2024-03-12T03:34:40.753233+0000","last_active":"2024-03-12T03:34:40.753211+0000","last_peered":"2024-03-12T03:34:40.753211+0000","last_clean":"2024-03-12T03:34:40.753211+0000","last_became_active":"2024-03-12T03:33:21.101819+0000","last_became_peered":"2024-03-12T03:33:21.101819+0000","last_unstale":"2024-03-12T03:34:40.753211+0000","last_undegraded":"2024-03-12T03:34:40.753211+0000","last_fullsized":"2024-03-12T03:34:40.753211+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.753153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.816911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259938+0000","last_change":"2024-03-12T03:34:29.083183+0000","last_active":"2024-03-12T03:34:33.259938+0000","last_peered":"2024-03-12T03:34:33.259938+0000","last_clean":"2024-03-12T03:34:33.259938+0000","last_became_active":"2024-03-12T03:33:23.553480+0000","last_became_peered":"2024-03-12T03:33:23.553480+0000","last_unstale":"2024-03-12T03:34:33.259938+0000","last_undegraded":"2024-03-12T03:34:33.259938+0000","last_fullsized":"2024-03-12T03:34:33.259938+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:34:29.083063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:29.083063+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:45.174021+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.562207+0000","last_change":"2024-03-12T03:33:27.188065+0000","last_active":"2024-03-12T03:34:33.562207+0000","last_peered":"2024-03-12T03:34:33.562207+0000","last_clean":"2024-03-12T03:34:33.562207+0000","last_became_active":"2024-03-12T03:33:27.187657+0000","last_became_peered":"2024-03-12T03:33:27.187657+0000","last_unstale":"2024-03-12T03:34:33.562207+0000","last_undegraded":"2024-03-12T03:34:33.562207+0000","last_fullsized":"2024-03-12T03:34:33.562207+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142236+0000","last_change":"2024-03-12T03:33:25.175313+0000","last_active":"2024-03-12T03:34:33.142236+0000","last_peered":"2024-03-12T03:34:33.142236+0000","last_clean":"2024-03-12T03:34:33.142236+0000","last_became_active":"2024-03-12T03:33:25.174953+0000","last_became_peered":"2024-03-12T03:33:25.174953+0000","last_unstale":"2024-03-12T03:34:33.142236+0000","last_undegraded":"2024-03-12T03:34:33.142236+0000","last_fullsized":"2024-03-12T03:34:33.142236+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:37.703054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693529+0000","last_change":"2024-03-12T03:33:21.106130+0000","last_active":"2024-03-12T03:34:33.693529+0000","last_peered":"2024-03-12T03:34:33.693529+0000","last_clean":"2024-03-12T03:34:33.693529+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:34:33.693529+0000","last_undegraded":"2024-03-12T03:34:33.693529+0000","last_fullsized":"2024-03-12T03:34:33.693529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:43.636698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141096+0000","last_change":"2024-03-12T03:33:23.552425+0000","last_active":"2024-03-12T03:34:33.141096+0000","last_peered":"2024-03-12T03:34:33.141096+0000","last_clean":"2024-03-12T03:34:33.141096+0000","last_became_active":"2024-03-12T03:33:23.552164+0000","last_became_peered":"2024-03-12T03:33:23.552164+0000","last_unstale":"2024-03-12T03:34:33.141096+0000","last_undegraded":"2024-03-12T03:34:33.141096+0000","last_fullsized":"2024-03-12T03:34:33.141096+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.480291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693972+0000","last_change":"2024-03-12T03:33:27.383521+0000","last_active":"2024-03-12T03:34:33.693972+0000","last_peered":"2024-03-12T03:34:33.693972+0000","last_clean":"2024-03-12T03:34:33.693972+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:34:33.693972+0000","last_undegraded":"2024-03-12T03:34:33.693972+0000","last_fullsized":"2024-03-12T03:34:33.693972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.404863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259842+0000","last_change":"2024-03-12T03:33:25.181046+0000","last_active":"2024-03-12T03:34:33.259842+0000","last_peered":"2024-03-12T03:34:33.259842+0000","last_clean":"2024-03-12T03:34:33.259842+0000","last_became_active":"2024-03-12T03:33:25.180752+0000","last_became_peered":"2024-03-12T03:33:25.180752+0000","last_unstale":"2024-03-12T03:34:33.259842+0000","last_undegraded":"2024-03-12T03:34:33.259842+0000","last_fullsized":"2024-03-12T03:34:33.259842+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:52.874476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141670+0000","last_change":"2024-03-12T03:34:26.565344+0000","last_active":"2024-03-12T03:34:33.141670+0000","last_peered":"2024-03-12T03:34:33.141670+0000","last_clean":"2024-03-12T03:34:33.141670+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:34:33.141670+0000","last_undegraded":"2024-03-12T03:34:33.141670+0000","last_fullsized":"2024-03-12T03:34:33.141670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:26.565245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:26.565245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:28.167100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142318+0000","last_change":"2024-03-12T03:33:23.552149+0000","last_active":"2024-03-12T03:34:33.142318+0000","last_peered":"2024-03-12T03:34:33.142318+0000","last_clean":"2024-03-12T03:34:33.142318+0000","last_became_active":"2024-03-12T03:33:23.551773+0000","last_became_peered":"2024-03-12T03:33:23.551773+0000","last_unstale":"2024-03-12T03:34:33.142318+0000","last_undegraded":"2024-03-12T03:34:33.142318+0000","last_fullsized":"2024-03-12T03:34:33.142318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:35.193863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259399+0000","last_change":"2024-03-12T03:33:27.382928+0000","last_active":"2024-03-12T03:34:33.259399+0000","last_peered":"2024-03-12T03:34:33.259399+0000","last_clean":"2024-03-12T03:34:33.259399+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:34:33.259399+0000","last_undegraded":"2024-03-12T03:34:33.259399+0000","last_fullsized":"2024-03-12T03:34:33.259399+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.104295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.694256+0000","last_change":"2024-03-12T03:33:21.107807+0000","last_active":"2024-03-12T03:34:33.694256+0000","last_peered":"2024-03-12T03:34:33.694256+0000","last_clean":"2024-03-12T03:34:33.694256+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:34:33.694256+0000","last_undegraded":"2024-03-12T03:34:33.694256+0000","last_fullsized":"2024-03-12T03:34:33.694256+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:48.638486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"90'36","reported_seq":119,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:39.184041+0000","last_change":"2024-03-12T03:34:39.184089+0000","last_active":"2024-03-12T03:34:39.184041+0000","last_peered":"2024-03-12T03:34:39.184041+0000","last_clean":"2024-03-12T03:34:39.184041+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:34:39.184041+0000","last_undegraded":"2024-03-12T03:34:39.184041+0000","last_fullsized":"2024-03-12T03:34:39.184041+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.183984+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.564093+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:39.659357+0000","last_change":"2024-03-12T03:34:39.659403+0000","last_active":"2024-03-12T03:34:39.659357+0000","last_peered":"2024-03-12T03:34:39.659357+0000","last_clean":"2024-03-12T03:34:39.659357+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:34:39.659357+0000","last_undegraded":"2024-03-12T03:34:39.659357+0000","last_fullsized":"2024-03-12T03:34:39.659357+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:39.659277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.659277+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:08.903088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141754+0000","last_change":"2024-03-12T03:33:27.191091+0000","last_active":"2024-03-12T03:34:33.141754+0000","last_peered":"2024-03-12T03:34:33.141754+0000","last_clean":"2024-03-12T03:34:33.141754+0000","last_became_active":"2024-03-12T03:33:27.190753+0000","last_became_peered":"2024-03-12T03:33:27.190753+0000","last_unstale":"2024-03-12T03:34:33.141754+0000","last_undegraded":"2024-03-12T03:34:33.141754+0000","last_fullsized":"2024-03-12T03:34:33.141754+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.061395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.694206+0000","last_change":"2024-03-12T03:34:23.842758+0000","last_active":"2024-03-12T03:34:33.694206+0000","last_peered":"2024-03-12T03:34:33.694206+0000","last_clean":"2024-03-12T03:34:33.694206+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:34:33.694206+0000","last_undegraded":"2024-03-12T03:34:33.694206+0000","last_fullsized":"2024-03-12T03:34:33.694206+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:23.842628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.842628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.823849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":100,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.691130+0000","last_change":"2024-03-12T03:33:23.163665+0000","last_active":"2024-03-12T03:34:33.691130+0000","last_peered":"2024-03-12T03:34:33.691130+0000","last_clean":"2024-03-12T03:34:33.691130+0000","last_became_active":"2024-03-12T03:33:23.163351+0000","last_became_peered":"2024-03-12T03:33:23.163351+0000","last_unstale":"2024-03-12T03:34:33.691130+0000","last_undegraded":"2024-03-12T03:34:33.691130+0000","last_fullsized":"2024-03-12T03:34:33.691130+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:43.519988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561362+0000","last_change":"2024-03-12T03:33:25.185933+0000","last_active":"2024-03-12T03:34:33.561362+0000","last_peered":"2024-03-12T03:34:33.561362+0000","last_clean":"2024-03-12T03:34:33.561362+0000","last_became_active":"2024-03-12T03:33:25.185596+0000","last_became_peered":"2024-03-12T03:33:25.185596+0000","last_unstale":"2024-03-12T03:34:33.561362+0000","last_undegraded":"2024-03-12T03:34:33.561362+0000","last_fullsized":"2024-03-12T03:34:33.561362+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:34.341308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693346+0000","last_change":"2024-03-12T03:33:27.188268+0000","last_active":"2024-03-12T03:34:33.693346+0000","last_peered":"2024-03-12T03:34:33.693346+0000","last_clean":"2024-03-12T03:34:33.693346+0000","last_became_active":"2024-03-12T03:33:27.187955+0000","last_became_peered":"2024-03-12T03:33:27.187955+0000","last_unstale":"2024-03-12T03:34:33.693346+0000","last_undegraded":"2024-03-12T03:34:33.693346+0000","last_fullsized":"2024-03-12T03:34:33.693346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:37.416612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561861+0000","last_change":"2024-03-12T03:33:25.180331+0000","last_active":"2024-03-12T03:34:33.561861+0000","last_peered":"2024-03-12T03:34:33.561861+0000","last_clean":"2024-03-12T03:34:33.561861+0000","last_became_active":"2024-03-12T03:33:25.179593+0000","last_became_peered":"2024-03-12T03:33:25.179593+0000","last_unstale":"2024-03-12T03:34:33.561861+0000","last_undegraded":"2024-03-12T03:34:33.561861+0000","last_fullsized":"2024-03-12T03:34:33.561861+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.050267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.694238+0000","last_change":"2024-03-12T03:34:30.792781+0000","last_active":"2024-03-12T03:34:33.694238+0000","last_peered":"2024-03-12T03:34:33.694238+0000","last_clean":"2024-03-12T03:34:33.694238+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:34:33.694238+0000","last_undegraded":"2024-03-12T03:34:33.694238+0000","last_fullsized":"2024-03-12T03:34:33.694238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:30.792702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.792702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:50.132706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":96,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142060+0000","last_change":"2024-03-12T03:33:23.551366+0000","last_active":"2024-03-12T03:34:33.142060+0000","last_peered":"2024-03-12T03:34:33.142060+0000","last_clean":"2024-03-12T03:34:33.142060+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:34:33.142060+0000","last_undegraded":"2024-03-12T03:34:33.142060+0000","last_fullsized":"2024-03-12T03:34:33.142060+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:44.149594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:37.737449+0000","last_change":"2024-03-12T03:34:37.737471+0000","last_active":"2024-03-12T03:34:37.737449+0000","last_peered":"2024-03-12T03:34:37.737449+0000","last_clean":"2024-03-12T03:34:37.737449+0000","last_became_active":"2024-03-12T03:33:27.189948+0000","last_became_peered":"2024-03-12T03:33:27.189948+0000","last_unstale":"2024-03-12T03:34:37.737449+0000","last_undegraded":"2024-03-12T03:34:37.737449+0000","last_fullsized":"2024-03-12T03:34:37.737449+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:37.737386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:37.737386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.387230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.562079+0000","last_change":"2024-03-12T03:33:25.184715+0000","last_active":"2024-03-12T03:34:33.562079+0000","last_peered":"2024-03-12T03:34:33.562079+0000","last_clean":"2024-03-12T03:34:33.562079+0000","last_became_active":"2024-03-12T03:33:25.184289+0000","last_became_peered":"2024-03-12T03:33:25.184289+0000","last_unstale":"2024-03-12T03:34:33.562079+0000","last_undegraded":"2024-03-12T03:34:33.562079+0000","last_fullsized":"2024-03-12T03:34:33.562079+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:41.396354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259558+0000","last_change":"2024-03-12T03:33:21.102971+0000","last_active":"2024-03-12T03:34:33.259558+0000","last_peered":"2024-03-12T03:34:33.259558+0000","last_clean":"2024-03-12T03:34:33.259558+0000","last_became_active":"2024-03-12T03:33:21.102410+0000","last_became_peered":"2024-03-12T03:33:21.102410+0000","last_unstale":"2024-03-12T03:34:33.259558+0000","last_undegraded":"2024-03-12T03:34:33.259558+0000","last_fullsized":"2024-03-12T03:34:33.259558+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:33.557063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141352+0000","last_change":"2024-03-12T03:33:23.164442+0000","last_active":"2024-03-12T03:34:33.141352+0000","last_peered":"2024-03-12T03:34:33.141352+0000","last_clean":"2024-03-12T03:34:33.141352+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:34:33.141352+0000","last_undegraded":"2024-03-12T03:34:33.141352+0000","last_fullsized":"2024-03-12T03:34:33.141352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:50.796888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141003+0000","last_change":"2024-03-12T03:33:27.185374+0000","last_active":"2024-03-12T03:34:33.141003+0000","last_peered":"2024-03-12T03:34:33.141003+0000","last_clean":"2024-03-12T03:34:33.141003+0000","last_became_active":"2024-03-12T03:33:27.185041+0000","last_became_peered":"2024-03-12T03:33:27.185041+0000","last_unstale":"2024-03-12T03:34:33.141003+0000","last_undegraded":"2024-03-12T03:34:33.141003+0000","last_fullsized":"2024-03-12T03:34:33.141003+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:34.953585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259525+0000","last_change":"2024-03-12T03:34:22.168794+0000","last_active":"2024-03-12T03:34:33.259525+0000","last_peered":"2024-03-12T03:34:33.259525+0000","last_clean":"2024-03-12T03:34:33.259525+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:34:33.259525+0000","last_undegraded":"2024-03-12T03:34:33.259525+0000","last_fullsized":"2024-03-12T03:34:33.259525+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:34:22.168654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:22.168654+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:48.338217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.690974+0000","last_change":"2024-03-12T03:33:21.107222+0000","last_active":"2024-03-12T03:34:33.690974+0000","last_peered":"2024-03-12T03:34:33.690974+0000","last_clean":"2024-03-12T03:34:33.690974+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:34:33.690974+0000","last_undegraded":"2024-03-12T03:34:33.690974+0000","last_fullsized":"2024-03-12T03:34:33.690974+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'35","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259363+0000","last_change":"2024-03-12T03:33:50.166841+0000","last_active":"2024-03-12T03:34:33.259363+0000","last_peered":"2024-03-12T03:34:33.259363+0000","last_clean":"2024-03-12T03:34:33.259363+0000","last_became_active":"2024-03-12T03:33:50.151664+0000","last_became_peered":"2024-03-12T03:33:50.151664+0000","last_unstale":"2024-03-12T03:34:33.259363+0000","last_undegraded":"2024-03-12T03:34:33.259363+0000","last_fullsized":"2024-03-12T03:34:33.259363+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:48.926653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:38.697982+0000","last_change":"2024-03-12T03:34:38.698005+0000","last_active":"2024-03-12T03:34:38.697982+0000","last_peered":"2024-03-12T03:34:38.697982+0000","last_clean":"2024-03-12T03:34:38.697982+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:34:38.697982+0000","last_undegraded":"2024-03-12T03:34:38.697982+0000","last_fullsized":"2024-03-12T03:34:38.697982+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:38.697923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.697923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:07.086680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:40.663056+0000","last_change":"2024-03-12T03:34:40.663082+0000","last_active":"2024-03-12T03:34:40.663056+0000","last_peered":"2024-03-12T03:34:40.663056+0000","last_clean":"2024-03-12T03:34:40.663056+0000","last_became_active":"2024-03-12T03:33:27.380389+0000","last_became_peered":"2024-03-12T03:33:27.380389+0000","last_unstale":"2024-03-12T03:34:40.663056+0000","last_undegraded":"2024-03-12T03:34:40.663056+0000","last_fullsized":"2024-03-12T03:34:40.663056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.663002+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.064026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257463+0000","last_change":"2024-03-12T03:33:21.573748+0000","last_active":"2024-03-12T03:34:33.257463+0000","last_peered":"2024-03-12T03:34:33.257463+0000","last_clean":"2024-03-12T03:34:33.257463+0000","last_became_active":"2024-03-12T03:33:21.572911+0000","last_became_peered":"2024-03-12T03:33:21.572911+0000","last_unstale":"2024-03-12T03:34:33.257463+0000","last_undegraded":"2024-03-12T03:34:33.257463+0000","last_fullsized":"2024-03-12T03:34:33.257463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:45.983059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561323+0000","last_change":"2024-03-12T03:33:24.161865+0000","last_active":"2024-03-12T03:34:33.561323+0000","last_peered":"2024-03-12T03:34:33.561323+0000","last_clean":"2024-03-12T03:34:33.561323+0000","last_became_active":"2024-03-12T03:33:20.093064+0000","last_became_peered":"2024-03-12T03:33:20.093064+0000","last_unstale":"2024-03-12T03:34:33.561323+0000","last_undegraded":"2024-03-12T03:34:33.561323+0000","last_fullsized":"2024-03-12T03:34:33.561323+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:33:19.077015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000516994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.258062+0000","last_change":"2024-03-12T03:33:23.163705+0000","last_active":"2024-03-12T03:34:33.258062+0000","last_peered":"2024-03-12T03:34:33.258062+0000","last_clean":"2024-03-12T03:34:33.258062+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:34:33.258062+0000","last_undegraded":"2024-03-12T03:34:33.258062+0000","last_fullsized":"2024-03-12T03:34:33.258062+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:40.056709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561574+0000","last_change":"2024-03-12T03:33:25.179406+0000","last_active":"2024-03-12T03:34:33.561574+0000","last_peered":"2024-03-12T03:34:33.561574+0000","last_clean":"2024-03-12T03:34:33.561574+0000","last_became_active":"2024-03-12T03:33:25.179022+0000","last_became_peered":"2024-03-12T03:33:25.179022+0000","last_unstale":"2024-03-12T03:34:33.561574+0000","last_undegraded":"2024-03-12T03:34:33.561574+0000","last_fullsized":"2024-03-12T03:34:33.561574+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259883+0000","last_change":"2024-03-12T03:33:27.188886+0000","last_active":"2024-03-12T03:34:33.259883+0000","last_peered":"2024-03-12T03:34:33.259883+0000","last_clean":"2024-03-12T03:34:33.259883+0000","last_became_active":"2024-03-12T03:33:27.188551+0000","last_became_peered":"2024-03-12T03:33:27.188551+0000","last_unstale":"2024-03-12T03:34:33.259883+0000","last_undegraded":"2024-03-12T03:34:33.259883+0000","last_fullsized":"2024-03-12T03:34:33.259883+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:50.069549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":312,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693459+0000","last_change":"2024-03-12T03:33:29.565156+0000","last_active":"2024-03-12T03:34:33.693459+0000","last_peered":"2024-03-12T03:34:33.693459+0000","last_clean":"2024-03-12T03:34:33.693459+0000","last_became_active":"2024-03-12T03:33:11.319888+0000","last_became_peered":"2024-03-12T03:33:11.319888+0000","last_unstale":"2024-03-12T03:34:33.693459+0000","last_undegraded":"2024-03-12T03:34:33.693459+0000","last_fullsized":"2024-03-12T03:34:33.693459+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-12T03:33:29.565069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:33:29.565069+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:47.469193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141670+0000","last_change":"2024-03-12T03:33:21.101239+0000","last_active":"2024-03-12T03:34:33.141670+0000","last_peered":"2024-03-12T03:34:33.141670+0000","last_clean":"2024-03-12T03:34:33.141670+0000","last_became_active":"2024-03-12T03:33:21.100900+0000","last_became_peered":"2024-03-12T03:33:21.100900+0000","last_unstale":"2024-03-12T03:34:33.141670+0000","last_undegraded":"2024-03-12T03:34:33.141670+0000","last_fullsized":"2024-03-12T03:34:33.141670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.038220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":99,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:35.664996+0000","last_change":"2024-03-12T03:34:35.664996+0000","last_active":"2024-03-12T03:34:35.664996+0000","last_peered":"2024-03-12T03:34:35.664996+0000","last_clean":"2024-03-12T03:34:35.664996+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:34:35.664996+0000","last_undegraded":"2024-03-12T03:34:35.664996+0000","last_fullsized":"2024-03-12T03:34:35.664996+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:34:35.664876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:35.664876+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:54.461746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693937+0000","last_change":"2024-03-12T03:33:25.178070+0000","last_active":"2024-03-12T03:34:33.693937+0000","last_peered":"2024-03-12T03:34:33.693937+0000","last_clean":"2024-03-12T03:34:33.693937+0000","last_became_active":"2024-03-12T03:33:25.177762+0000","last_became_peered":"2024-03-12T03:33:25.177762+0000","last_unstale":"2024-03-12T03:34:33.693937+0000","last_undegraded":"2024-03-12T03:34:33.693937+0000","last_fullsized":"2024-03-12T03:34:33.693937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.594234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257389+0000","last_change":"2024-03-12T03:33:27.192203+0000","last_active":"2024-03-12T03:34:33.257389+0000","last_peered":"2024-03-12T03:34:33.257389+0000","last_clean":"2024-03-12T03:34:33.257389+0000","last_became_active":"2024-03-12T03:33:27.191878+0000","last_became_peered":"2024-03-12T03:33:27.191878+0000","last_unstale":"2024-03-12T03:34:33.257389+0000","last_undegraded":"2024-03-12T03:34:33.257389+0000","last_fullsized":"2024-03-12T03:34:33.257389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:43.704977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140893+0000","last_change":"2024-03-12T03:33:21.097466+0000","last_active":"2024-03-12T03:34:33.140893+0000","last_peered":"2024-03-12T03:34:33.140893+0000","last_clean":"2024-03-12T03:34:33.140893+0000","last_became_active":"2024-03-12T03:33:21.097086+0000","last_became_peered":"2024-03-12T03:33:21.097086+0000","last_unstale":"2024-03-12T03:34:33.140893+0000","last_undegraded":"2024-03-12T03:34:33.140893+0000","last_fullsized":"2024-03-12T03:34:33.140893+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.630139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":151,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:39.239320+0000","last_change":"2024-03-12T03:34:21.799658+0000","last_active":"2024-03-12T03:34:39.239320+0000","last_peered":"2024-03-12T03:34:39.239320+0000","last_clean":"2024-03-12T03:34:39.239320+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:34:39.239320+0000","last_undegraded":"2024-03-12T03:34:39.239320+0000","last_fullsized":"2024-03-12T03:34:39.239320+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:34:21.799534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:21.799534+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.973279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:39.707339+0000","last_change":"2024-03-12T03:34:39.707369+0000","last_active":"2024-03-12T03:34:39.707339+0000","last_peered":"2024-03-12T03:34:39.707339+0000","last_clean":"2024-03-12T03:34:39.707339+0000","last_became_active":"2024-03-12T03:33:23.168750+0000","last_became_peered":"2024-03-12T03:33:23.168750+0000","last_unstale":"2024-03-12T03:34:39.707339+0000","last_undegraded":"2024-03-12T03:34:39.707339+0000","last_fullsized":"2024-03-12T03:34:39.707339+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:34:39.707264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.707264+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:43.684441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.260112+0000","last_change":"2024-03-12T03:33:25.177321+0000","last_active":"2024-03-12T03:34:33.260112+0000","last_peered":"2024-03-12T03:34:33.260112+0000","last_clean":"2024-03-12T03:34:33.260112+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:34:33.260112+0000","last_undegraded":"2024-03-12T03:34:33.260112+0000","last_fullsized":"2024-03-12T03:34:33.260112+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.569538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257204+0000","last_change":"2024-03-12T03:33:27.380812+0000","last_active":"2024-03-12T03:34:33.257204+0000","last_peered":"2024-03-12T03:34:33.257204+0000","last_clean":"2024-03-12T03:34:33.257204+0000","last_became_active":"2024-03-12T03:33:27.380458+0000","last_became_peered":"2024-03-12T03:33:27.380458+0000","last_unstale":"2024-03-12T03:34:33.257204+0000","last_undegraded":"2024-03-12T03:34:33.257204+0000","last_fullsized":"2024-03-12T03:34:33.257204+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:41.294255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:34.272859+0000","last_change":"2024-03-12T03:34:34.272895+0000","last_active":"2024-03-12T03:34:34.272859+0000","last_peered":"2024-03-12T03:34:34.272859+0000","last_clean":"2024-03-12T03:34:34.272859+0000","last_became_active":"2024-03-12T03:33:21.573003+0000","last_became_peered":"2024-03-12T03:33:21.573003+0000","last_unstale":"2024-03-12T03:34:34.272859+0000","last_undegraded":"2024-03-12T03:34:34.272859+0000","last_fullsized":"2024-03-12T03:34:34.272859+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:34.272759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.272759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:47.242996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":101,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141481+0000","last_change":"2024-03-12T03:33:23.551923+0000","last_active":"2024-03-12T03:34:33.141481+0000","last_peered":"2024-03-12T03:34:33.141481+0000","last_clean":"2024-03-12T03:34:33.141481+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:34:33.141481+0000","last_undegraded":"2024-03-12T03:34:33.141481+0000","last_fullsized":"2024-03-12T03:34:33.141481+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:50.306884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142106+0000","last_change":"2024-03-12T03:34:28.572496+0000","last_active":"2024-03-12T03:34:33.142106+0000","last_peered":"2024-03-12T03:34:33.142106+0000","last_clean":"2024-03-12T03:34:33.142106+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:34:33.142106+0000","last_undegraded":"2024-03-12T03:34:33.142106+0000","last_fullsized":"2024-03-12T03:34:33.142106+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:28.572379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:28.572379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:58.234927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140452+0000","last_change":"2024-03-12T03:33:27.194720+0000","last_active":"2024-03-12T03:34:33.140452+0000","last_peered":"2024-03-12T03:34:33.140452+0000","last_clean":"2024-03-12T03:34:33.140452+0000","last_became_active":"2024-03-12T03:33:27.194266+0000","last_became_peered":"2024-03-12T03:33:27.194266+0000","last_unstale":"2024-03-12T03:34:33.140452+0000","last_undegraded":"2024-03-12T03:34:33.140452+0000","last_fullsized":"2024-03-12T03:34:33.140452+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.549699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259665+0000","last_change":"2024-03-12T03:33:21.573568+0000","last_active":"2024-03-12T03:34:33.259665+0000","last_peered":"2024-03-12T03:34:33.259665+0000","last_clean":"2024-03-12T03:34:33.259665+0000","last_became_active":"2024-03-12T03:33:21.573251+0000","last_became_peered":"2024-03-12T03:33:21.573251+0000","last_unstale":"2024-03-12T03:34:33.259665+0000","last_undegraded":"2024-03-12T03:34:33.259665+0000","last_fullsized":"2024-03-12T03:34:33.259665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:35.282277+0000","last_change":"2024-03-12T03:34:35.282309+0000","last_active":"2024-03-12T03:34:35.282277+0000","last_peered":"2024-03-12T03:34:35.282277+0000","last_clean":"2024-03-12T03:34:35.282277+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:34:35.282277+0000","last_undegraded":"2024-03-12T03:34:35.282277+0000","last_fullsized":"2024-03-12T03:34:35.282277+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:34:35.282203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:35.282203+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.121154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140999+0000","last_change":"2024-03-12T03:34:30.767076+0000","last_active":"2024-03-12T03:34:33.140999+0000","last_peered":"2024-03-12T03:34:33.140999+0000","last_clean":"2024-03-12T03:34:33.140999+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:34:33.140999+0000","last_undegraded":"2024-03-12T03:34:33.140999+0000","last_fullsized":"2024-03-12T03:34:33.140999+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:30.766975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.766975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:45.275798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693659+0000","last_change":"2024-03-12T03:33:27.383686+0000","last_active":"2024-03-12T03:34:33.693659+0000","last_peered":"2024-03-12T03:34:33.693659+0000","last_clean":"2024-03-12T03:34:33.693659+0000","last_became_active":"2024-03-12T03:33:27.383249+0000","last_became_peered":"2024-03-12T03:33:27.383249+0000","last_unstale":"2024-03-12T03:34:33.693659+0000","last_undegraded":"2024-03-12T03:34:33.693659+0000","last_fullsized":"2024-03-12T03:34:33.693659+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:40.138363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140884+0000","last_change":"2024-03-12T03:33:21.098280+0000","last_active":"2024-03-12T03:34:33.140884+0000","last_peered":"2024-03-12T03:34:33.140884+0000","last_clean":"2024-03-12T03:34:33.140884+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:34:33.140884+0000","last_undegraded":"2024-03-12T03:34:33.140884+0000","last_fullsized":"2024-03-12T03:34:33.140884+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.640150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.690888+0000","last_change":"2024-03-12T03:33:23.156886+0000","last_active":"2024-03-12T03:34:33.690888+0000","last_peered":"2024-03-12T03:34:33.690888+0000","last_clean":"2024-03-12T03:34:33.690888+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:34:33.690888+0000","last_undegraded":"2024-03-12T03:34:33.690888+0000","last_fullsized":"2024-03-12T03:34:33.690888+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141938+0000","last_change":"2024-03-12T03:34:25.168128+0000","last_active":"2024-03-12T03:34:33.141938+0000","last_peered":"2024-03-12T03:34:33.141938+0000","last_clean":"2024-03-12T03:34:33.141938+0000","last_became_active":"2024-03-12T03:33:25.177311+0000","last_became_peered":"2024-03-12T03:33:25.177311+0000","last_unstale":"2024-03-12T03:34:33.141938+0000","last_undegraded":"2024-03-12T03:34:33.141938+0000","last_fullsized":"2024-03-12T03:34:33.141938+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:25.168002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.168002+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.036057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.690827+0000","last_change":"2024-03-12T03:33:27.188212+0000","last_active":"2024-03-12T03:34:33.690827+0000","last_peered":"2024-03-12T03:34:33.690827+0000","last_clean":"2024-03-12T03:34:33.690827+0000","last_became_active":"2024-03-12T03:33:27.187864+0000","last_became_peered":"2024-03-12T03:33:27.187864+0000","last_unstale":"2024-03-12T03:34:33.690827+0000","last_undegraded":"2024-03-12T03:34:33.690827+0000","last_fullsized":"2024-03-12T03:34:33.690827+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142153+0000","last_change":"2024-03-12T03:33:21.104718+0000","last_active":"2024-03-12T03:34:33.142153+0000","last_peered":"2024-03-12T03:34:33.142153+0000","last_clean":"2024-03-12T03:34:33.142153+0000","last_became_active":"2024-03-12T03:33:21.104430+0000","last_became_peered":"2024-03-12T03:33:21.104430+0000","last_unstale":"2024-03-12T03:34:33.142153+0000","last_undegraded":"2024-03-12T03:34:33.142153+0000","last_fullsized":"2024-03-12T03:34:33.142153+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:33.711229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":109,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141829+0000","last_change":"2024-03-12T03:33:23.551957+0000","last_active":"2024-03-12T03:34:33.141829+0000","last_peered":"2024-03-12T03:34:33.141829+0000","last_clean":"2024-03-12T03:34:33.141829+0000","last_became_active":"2024-03-12T03:33:23.551623+0000","last_became_peered":"2024-03-12T03:33:23.551623+0000","last_unstale":"2024-03-12T03:34:33.141829+0000","last_undegraded":"2024-03-12T03:34:33.141829+0000","last_fullsized":"2024-03-12T03:34:33.141829+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:45.740434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.690676+0000","last_change":"2024-03-12T03:33:25.182046+0000","last_active":"2024-03-12T03:34:33.690676+0000","last_peered":"2024-03-12T03:34:33.690676+0000","last_clean":"2024-03-12T03:34:33.690676+0000","last_became_active":"2024-03-12T03:33:25.181020+0000","last_became_peered":"2024-03-12T03:33:25.181020+0000","last_unstale":"2024-03-12T03:34:33.690676+0000","last_undegraded":"2024-03-12T03:34:33.690676+0000","last_fullsized":"2024-03-12T03:34:33.690676+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259564+0000","last_change":"2024-03-12T03:33:27.190398+0000","last_active":"2024-03-12T03:34:33.259564+0000","last_peered":"2024-03-12T03:34:33.259564+0000","last_clean":"2024-03-12T03:34:33.259564+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:34:33.259564+0000","last_undegraded":"2024-03-12T03:34:33.259564+0000","last_fullsized":"2024-03-12T03:34:33.259564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:53.538609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.691680+0000","last_change":"2024-03-12T03:33:25.176357+0000","last_active":"2024-03-12T03:34:33.691680+0000","last_peered":"2024-03-12T03:34:33.691680+0000","last_clean":"2024-03-12T03:34:33.691680+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:34:33.691680+0000","last_undegraded":"2024-03-12T03:34:33.691680+0000","last_fullsized":"2024-03-12T03:34:33.691680+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141783+0000","last_change":"2024-03-12T03:33:21.104557+0000","last_active":"2024-03-12T03:34:33.141783+0000","last_peered":"2024-03-12T03:34:33.141783+0000","last_clean":"2024-03-12T03:34:33.141783+0000","last_became_active":"2024-03-12T03:33:21.104293+0000","last_became_peered":"2024-03-12T03:33:21.104293+0000","last_unstale":"2024-03-12T03:34:33.141783+0000","last_undegraded":"2024-03-12T03:34:33.141783+0000","last_fullsized":"2024-03-12T03:34:33.141783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.027134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561565+0000","last_change":"2024-03-12T03:33:50.166158+0000","last_active":"2024-03-12T03:34:33.561565+0000","last_peered":"2024-03-12T03:34:33.561565+0000","last_clean":"2024-03-12T03:34:33.561565+0000","last_became_active":"2024-03-12T03:33:50.153240+0000","last_became_peered":"2024-03-12T03:33:50.153240+0000","last_unstale":"2024-03-12T03:34:33.561565+0000","last_undegraded":"2024-03-12T03:34:33.561565+0000","last_fullsized":"2024-03-12T03:34:33.561565+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:46.362489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141294+0000","last_change":"2024-03-12T03:33:27.381253+0000","last_active":"2024-03-12T03:34:33.141294+0000","last_peered":"2024-03-12T03:34:33.141294+0000","last_clean":"2024-03-12T03:34:33.141294+0000","last_became_active":"2024-03-12T03:33:27.380896+0000","last_became_peered":"2024-03-12T03:33:27.380896+0000","last_unstale":"2024-03-12T03:34:33.141294+0000","last_undegraded":"2024-03-12T03:34:33.141294+0000","last_fullsized":"2024-03-12T03:34:33.141294+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:46.257478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:36.138928+0000","last_change":"2024-03-12T03:34:36.138975+0000","last_active":"2024-03-12T03:34:36.138928+0000","last_peered":"2024-03-12T03:34:36.138928+0000","last_clean":"2024-03-12T03:34:36.138928+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:34:36.138928+0000","last_undegraded":"2024-03-12T03:34:36.138928+0000","last_fullsized":"2024-03-12T03:34:36.138928+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.138847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.138847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.950469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.690446+0000","last_change":"2024-03-12T03:33:21.573917+0000","last_active":"2024-03-12T03:34:33.690446+0000","last_peered":"2024-03-12T03:34:33.690446+0000","last_clean":"2024-03-12T03:34:33.690446+0000","last_became_active":"2024-03-12T03:33:21.573257+0000","last_became_peered":"2024-03-12T03:33:21.573257+0000","last_unstale":"2024-03-12T03:34:33.690446+0000","last_undegraded":"2024-03-12T03:34:33.690446+0000","last_fullsized":"2024-03-12T03:34:33.690446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.737276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.691591+0000","last_change":"2024-03-12T03:33:23.168956+0000","last_active":"2024-03-12T03:34:33.691591+0000","last_peered":"2024-03-12T03:34:33.691591+0000","last_clean":"2024-03-12T03:34:33.691591+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:34:33.691591+0000","last_undegraded":"2024-03-12T03:34:33.691591+0000","last_fullsized":"2024-03-12T03:34:33.691591+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561942+0000","last_change":"2024-03-12T03:34:31.142859+0000","last_active":"2024-03-12T03:34:33.561942+0000","last_peered":"2024-03-12T03:34:33.561942+0000","last_clean":"2024-03-12T03:34:33.561942+0000","last_became_active":"2024-03-12T03:33:27.187815+0000","last_became_peered":"2024-03-12T03:33:27.187815+0000","last_unstale":"2024-03-12T03:34:33.561942+0000","last_undegraded":"2024-03-12T03:34:33.561942+0000","last_fullsized":"2024-03-12T03:34:33.561942+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:31.142765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:31.142765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:54.545296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257712+0000","last_change":"2024-03-12T03:34:30.280067+0000","last_active":"2024-03-12T03:34:33.257712+0000","last_peered":"2024-03-12T03:34:33.257712+0000","last_clean":"2024-03-12T03:34:33.257712+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:34:33.257712+0000","last_undegraded":"2024-03-12T03:34:33.257712+0000","last_fullsized":"2024-03-12T03:34:33.257712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:30.279966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.279966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.784484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141881+0000","last_change":"2024-03-12T03:34:23.646936+0000","last_active":"2024-03-12T03:34:33.141881+0000","last_peered":"2024-03-12T03:34:33.141881+0000","last_clean":"2024-03-12T03:34:33.141881+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:34:33.141881+0000","last_undegraded":"2024-03-12T03:34:33.141881+0000","last_fullsized":"2024-03-12T03:34:33.141881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.646819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:33.240250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.691546+0000","last_change":"2024-03-12T03:33:23.164300+0000","last_active":"2024-03-12T03:34:33.691546+0000","last_peered":"2024-03-12T03:34:33.691546+0000","last_clean":"2024-03-12T03:34:33.691546+0000","last_became_active":"2024-03-12T03:33:23.163871+0000","last_became_peered":"2024-03-12T03:33:23.163871+0000","last_unstale":"2024-03-12T03:34:33.691546+0000","last_undegraded":"2024-03-12T03:34:33.691546+0000","last_fullsized":"2024-03-12T03:34:33.691546+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:38.948742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.691729+0000","last_change":"2024-03-12T03:33:27.184691+0000","last_active":"2024-03-12T03:34:33.691729+0000","last_peered":"2024-03-12T03:34:33.691729+0000","last_clean":"2024-03-12T03:34:33.691729+0000","last_became_active":"2024-03-12T03:33:27.184405+0000","last_became_peered":"2024-03-12T03:33:27.184405+0000","last_unstale":"2024-03-12T03:34:33.691729+0000","last_undegraded":"2024-03-12T03:34:33.691729+0000","last_fullsized":"2024-03-12T03:34:33.691729+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.120291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561857+0000","last_change":"2024-03-12T03:33:25.178413+0000","last_active":"2024-03-12T03:34:33.561857+0000","last_peered":"2024-03-12T03:34:33.561857+0000","last_clean":"2024-03-12T03:34:33.561857+0000","last_became_active":"2024-03-12T03:33:25.177613+0000","last_became_peered":"2024-03-12T03:33:25.177613+0000","last_unstale":"2024-03-12T03:34:33.561857+0000","last_undegraded":"2024-03-12T03:34:33.561857+0000","last_fullsized":"2024-03-12T03:34:33.561857+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142039+0000","last_change":"2024-03-12T03:33:21.097634+0000","last_active":"2024-03-12T03:34:33.142039+0000","last_peered":"2024-03-12T03:34:33.142039+0000","last_clean":"2024-03-12T03:34:33.142039+0000","last_became_active":"2024-03-12T03:33:21.097304+0000","last_became_peered":"2024-03-12T03:33:21.097304+0000","last_unstale":"2024-03-12T03:34:33.142039+0000","last_undegraded":"2024-03-12T03:34:33.142039+0000","last_fullsized":"2024-03-12T03:34:33.142039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:43.765898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.691099+0000","last_change":"2024-03-12T03:33:23.163246+0000","last_active":"2024-03-12T03:34:33.691099+0000","last_peered":"2024-03-12T03:34:33.691099+0000","last_clean":"2024-03-12T03:34:33.691099+0000","last_became_active":"2024-03-12T03:33:23.162657+0000","last_became_peered":"2024-03-12T03:33:23.162657+0000","last_unstale":"2024-03-12T03:34:33.691099+0000","last_undegraded":"2024-03-12T03:34:33.691099+0000","last_fullsized":"2024-03-12T03:34:33.691099+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:38.622721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":602,"ondisk_log_size":602,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070485,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28400,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716752,"statfs":{"total":95995035648,"available":95965954048,"internally_reserved":0,"allocated":1417216,"data_stored":948578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332122,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28376,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716776,"statfs":{"total":95995035648,"available":95965978624,"internally_reserved":0,"allocated":1392640,"data_stored":946615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":357877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32560,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712592,"statfs":{"total":95995035648,"available":95961694208,"internally_reserved":0,"allocated":1417216,"data_stored":947832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":802816,"data_stored":357480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117032,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":839680,"data_stored":357574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953509,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27824,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717328,"statfs":{"total":95995035648,"available":95966543872,"internally_reserved":0,"allocated":827392,"data_stored":357750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659167,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":815104,"data_stored":356918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:34:44.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:34:45.168 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:45 smithi173 ceph-mon[33544]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:45 smithi173 ceph-mon[33544]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:34:45.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:45 smithi173 ceph-mon[33544]: osdmap e93: 8 total, 8 up, 8 in 2024-03-12T03:34:45.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:45 smithi173 ceph-mon[33544]: 6.3 scrub starts 2024-03-12T03:34:45.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:45 smithi173 ceph-mon[33544]: 6.3 scrub ok 2024-03-12T03:34:45.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:45 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:45.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:45 smithi169 ceph-mon[39949]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:45 smithi169 ceph-mon[39949]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:34:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:45 smithi169 ceph-mon[39949]: osdmap e93: 8 total, 8 up, 8 in 2024-03-12T03:34:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:45 smithi169 ceph-mon[39949]: 6.3 scrub starts 2024-03-12T03:34:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:45 smithi169 ceph-mon[39949]: 6.3 scrub ok 2024-03-12T03:34:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:45 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:45.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:45 smithi169 ceph-mon[29344]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:45.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:45 smithi169 ceph-mon[29344]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:34:45.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:45 smithi169 ceph-mon[29344]: osdmap e93: 8 total, 8 up, 8 in 2024-03-12T03:34:45.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:45 smithi169 ceph-mon[29344]: 6.3 scrub starts 2024-03-12T03:34:45.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:45 smithi169 ceph-mon[29344]: 6.3 scrub ok 2024-03-12T03:34:45.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:45 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:34:46.802 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:46 smithi169 ceph-mon[39949]: 5.13 scrub starts 2024-03-12T03:34:46.802 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:46 smithi169 ceph-mon[39949]: 5.13 scrub ok 2024-03-12T03:34:46.802 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:46 smithi169 ceph-mon[39949]: 6.1 scrub starts 2024-03-12T03:34:46.802 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:46 smithi169 ceph-mon[39949]: 6.1 scrub ok 2024-03-12T03:34:46.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:46 smithi169 ceph-mon[29344]: 5.13 scrub starts 2024-03-12T03:34:46.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:46 smithi169 ceph-mon[29344]: 5.13 scrub ok 2024-03-12T03:34:46.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:46 smithi169 ceph-mon[29344]: 6.1 scrub starts 2024-03-12T03:34:46.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:46 smithi169 ceph-mon[29344]: 6.1 scrub ok 2024-03-12T03:34:46.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:46 smithi173 ceph-mon[33544]: 5.13 scrub starts 2024-03-12T03:34:46.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:46 smithi173 ceph-mon[33544]: 5.13 scrub ok 2024-03-12T03:34:46.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:46 smithi173 ceph-mon[33544]: 6.1 scrub starts 2024-03-12T03:34:46.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:46 smithi173 ceph-mon[33544]: 6.1 scrub ok 2024-03-12T03:34:47.008 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:34:47.010 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:34:47.715 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[29344]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:47.715 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[29344]: 4.a scrub starts 2024-03-12T03:34:47.715 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[29344]: 4.a scrub ok 2024-03-12T03:34:47.715 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[29344]: 6.15 scrub starts 2024-03-12T03:34:47.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[29344]: 6.15 scrub ok 2024-03-12T03:34:47.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[29344]: 4.1 scrub starts 2024-03-12T03:34:47.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[29344]: 4.1 scrub ok 2024-03-12T03:34:47.716 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[39949]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:47.716 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[39949]: 4.a scrub starts 2024-03-12T03:34:47.716 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[39949]: 4.a scrub ok 2024-03-12T03:34:47.716 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[39949]: 6.15 scrub starts 2024-03-12T03:34:47.716 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[39949]: 6.15 scrub ok 2024-03-12T03:34:47.716 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[39949]: 4.1 scrub starts 2024-03-12T03:34:47.716 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:47 smithi169 ceph-mon[39949]: 4.1 scrub ok 2024-03-12T03:34:47.741 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-12T03:34:45.764721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1296,"num_read_kb":1110,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":810,"ondisk_log_size":810,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":232692,"kb_used_data":8228,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749728524,"statfs":{"total":767960285184,"available":767722008576,"internally_reserved":0,"allocated":8425472,"data_stored":4669578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001385"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639195+0000","last_change":"2024-03-12T03:33:27.183162+0000","last_active":"2024-03-12T03:34:44.639195+0000","last_peered":"2024-03-12T03:34:44.639195+0000","last_clean":"2024-03-12T03:34:44.639195+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:34:44.639195+0000","last_undegraded":"2024-03-12T03:34:44.639195+0000","last_fullsized":"2024-03-12T03:34:44.639195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257264+0000","last_change":"2024-03-12T03:34:29.244162+0000","last_active":"2024-03-12T03:34:33.257264+0000","last_peered":"2024-03-12T03:34:33.257264+0000","last_clean":"2024-03-12T03:34:33.257264+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:34:33.257264+0000","last_undegraded":"2024-03-12T03:34:33.257264+0000","last_fullsized":"2024-03-12T03:34:33.257264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:29.244078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:29.244078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.593696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141067+0000","last_change":"2024-03-12T03:33:21.573453+0000","last_active":"2024-03-12T03:34:33.141067+0000","last_peered":"2024-03-12T03:34:33.141067+0000","last_clean":"2024-03-12T03:34:33.141067+0000","last_became_active":"2024-03-12T03:33:21.573080+0000","last_became_peered":"2024-03-12T03:33:21.573080+0000","last_unstale":"2024-03-12T03:34:33.141067+0000","last_undegraded":"2024-03-12T03:34:33.141067+0000","last_fullsized":"2024-03-12T03:34:33.141067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:37.186503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639104+0000","last_change":"2024-03-12T03:33:23.164433+0000","last_active":"2024-03-12T03:34:44.639104+0000","last_peered":"2024-03-12T03:34:44.639104+0000","last_clean":"2024-03-12T03:34:44.639104+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:34:44.639104+0000","last_undegraded":"2024-03-12T03:34:44.639104+0000","last_fullsized":"2024-03-12T03:34:44.639104+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141856+0000","last_change":"2024-03-12T03:33:27.382583+0000","last_active":"2024-03-12T03:34:33.141856+0000","last_peered":"2024-03-12T03:34:33.141856+0000","last_clean":"2024-03-12T03:34:33.141856+0000","last_became_active":"2024-03-12T03:33:27.382050+0000","last_became_peered":"2024-03-12T03:33:27.382050+0000","last_unstale":"2024-03-12T03:34:33.141856+0000","last_undegraded":"2024-03-12T03:34:33.141856+0000","last_fullsized":"2024-03-12T03:34:33.141856+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:47.150721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639095+0000","last_change":"2024-03-12T03:33:25.180282+0000","last_active":"2024-03-12T03:34:44.639095+0000","last_peered":"2024-03-12T03:34:44.639095+0000","last_clean":"2024-03-12T03:34:44.639095+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:34:44.639095+0000","last_undegraded":"2024-03-12T03:34:44.639095+0000","last_fullsized":"2024-03-12T03:34:44.639095+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141935+0000","last_change":"2024-03-12T03:33:21.573256+0000","last_active":"2024-03-12T03:34:33.141935+0000","last_peered":"2024-03-12T03:34:33.141935+0000","last_clean":"2024-03-12T03:34:33.141935+0000","last_became_active":"2024-03-12T03:33:21.572876+0000","last_became_peered":"2024-03-12T03:33:21.572876+0000","last_unstale":"2024-03-12T03:34:33.141935+0000","last_undegraded":"2024-03-12T03:34:33.141935+0000","last_fullsized":"2024-03-12T03:34:33.141935+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":126,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142016+0000","last_change":"2024-03-12T03:33:23.161838+0000","last_active":"2024-03-12T03:34:33.142016+0000","last_peered":"2024-03-12T03:34:33.142016+0000","last_clean":"2024-03-12T03:34:33.142016+0000","last_became_active":"2024-03-12T03:33:23.161549+0000","last_became_peered":"2024-03-12T03:33:23.161549+0000","last_unstale":"2024-03-12T03:34:33.142016+0000","last_undegraded":"2024-03-12T03:34:33.142016+0000","last_fullsized":"2024-03-12T03:34:33.142016+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:37.915866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140629+0000","last_change":"2024-03-12T03:33:27.189571+0000","last_active":"2024-03-12T03:34:33.140629+0000","last_peered":"2024-03-12T03:34:33.140629+0000","last_clean":"2024-03-12T03:34:33.140629+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:34:33.140629+0000","last_undegraded":"2024-03-12T03:34:33.140629+0000","last_fullsized":"2024-03-12T03:34:33.140629+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:46.083461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259342+0000","last_change":"2024-03-12T03:33:25.176211+0000","last_active":"2024-03-12T03:34:33.259342+0000","last_peered":"2024-03-12T03:34:33.259342+0000","last_clean":"2024-03-12T03:34:33.259342+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:34:33.259342+0000","last_undegraded":"2024-03-12T03:34:33.259342+0000","last_fullsized":"2024-03-12T03:34:33.259342+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:40.095375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259703+0000","last_change":"2024-03-12T03:33:21.102752+0000","last_active":"2024-03-12T03:34:33.259703+0000","last_peered":"2024-03-12T03:34:33.259703+0000","last_clean":"2024-03-12T03:34:33.259703+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:34:33.259703+0000","last_undegraded":"2024-03-12T03:34:33.259703+0000","last_fullsized":"2024-03-12T03:34:33.259703+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:48.338428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'21","reported_seq":99,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639377+0000","last_change":"2024-03-12T03:33:23.163157+0000","last_active":"2024-03-12T03:34:44.639377+0000","last_peered":"2024-03-12T03:34:44.639377+0000","last_clean":"2024-03-12T03:34:44.639377+0000","last_became_active":"2024-03-12T03:33:23.162749+0000","last_became_peered":"2024-03-12T03:33:23.162749+0000","last_unstale":"2024-03-12T03:34:44.639377+0000","last_undegraded":"2024-03-12T03:34:44.639377+0000","last_fullsized":"2024-03-12T03:34:44.639377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259214+0000","last_change":"2024-03-12T03:33:27.188261+0000","last_active":"2024-03-12T03:34:33.259214+0000","last_peered":"2024-03-12T03:34:33.259214+0000","last_clean":"2024-03-12T03:34:33.259214+0000","last_became_active":"2024-03-12T03:33:27.187943+0000","last_became_peered":"2024-03-12T03:33:27.187943+0000","last_unstale":"2024-03-12T03:34:33.259214+0000","last_undegraded":"2024-03-12T03:34:33.259214+0000","last_fullsized":"2024-03-12T03:34:33.259214+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.815395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693540+0000","last_change":"2024-03-12T03:33:25.182503+0000","last_active":"2024-03-12T03:34:33.693540+0000","last_peered":"2024-03-12T03:34:33.693540+0000","last_clean":"2024-03-12T03:34:33.693540+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:34:33.693540+0000","last_undegraded":"2024-03-12T03:34:33.693540+0000","last_fullsized":"2024-03-12T03:34:33.693540+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:47.480158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259799+0000","last_change":"2024-03-12T03:33:21.098805+0000","last_active":"2024-03-12T03:34:33.259799+0000","last_peered":"2024-03-12T03:34:33.259799+0000","last_clean":"2024-03-12T03:34:33.259799+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:34:33.259799+0000","last_undegraded":"2024-03-12T03:34:33.259799+0000","last_fullsized":"2024-03-12T03:34:33.259799+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:35.479958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"86'14","reported_seq":106,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639509+0000","last_change":"2024-03-12T03:34:23.795189+0000","last_active":"2024-03-12T03:34:44.639509+0000","last_peered":"2024-03-12T03:34:44.639509+0000","last_clean":"2024-03-12T03:34:44.639509+0000","last_became_active":"2024-03-12T03:33:50.155908+0000","last_became_peered":"2024-03-12T03:33:50.155908+0000","last_unstale":"2024-03-12T03:34:44.639509+0000","last_undegraded":"2024-03-12T03:34:44.639509+0000","last_fullsized":"2024-03-12T03:34:44.639509+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.795040+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:35.995794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639783+0000","last_change":"2024-03-12T03:33:27.380906+0000","last_active":"2024-03-12T03:34:44.639783+0000","last_peered":"2024-03-12T03:34:44.639783+0000","last_clean":"2024-03-12T03:34:44.639783+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:34:44.639783+0000","last_undegraded":"2024-03-12T03:34:44.639783+0000","last_fullsized":"2024-03-12T03:34:44.639783+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257865+0000","last_change":"2024-03-12T03:34:26.233162+0000","last_active":"2024-03-12T03:34:33.257865+0000","last_peered":"2024-03-12T03:34:33.257865+0000","last_clean":"2024-03-12T03:34:33.257865+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:34:33.257865+0000","last_undegraded":"2024-03-12T03:34:33.257865+0000","last_fullsized":"2024-03-12T03:34:33.257865+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:26.233072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:26.233072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.096475+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141454+0000","last_change":"2024-03-12T03:33:21.103671+0000","last_active":"2024-03-12T03:34:33.141454+0000","last_peered":"2024-03-12T03:34:33.141454+0000","last_clean":"2024-03-12T03:34:33.141454+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:34:33.141454+0000","last_undegraded":"2024-03-12T03:34:33.141454+0000","last_fullsized":"2024-03-12T03:34:33.141454+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:39.146149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561674+0000","last_change":"2024-03-12T03:33:23.161120+0000","last_active":"2024-03-12T03:34:33.561674+0000","last_peered":"2024-03-12T03:34:33.561674+0000","last_clean":"2024-03-12T03:34:33.561674+0000","last_became_active":"2024-03-12T03:33:23.160702+0000","last_became_peered":"2024-03-12T03:33:23.160702+0000","last_unstale":"2024-03-12T03:34:33.561674+0000","last_undegraded":"2024-03-12T03:34:33.561674+0000","last_fullsized":"2024-03-12T03:34:33.561674+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:39.664690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142268+0000","last_change":"2024-03-12T03:33:27.381440+0000","last_active":"2024-03-12T03:34:33.142268+0000","last_peered":"2024-03-12T03:34:33.142268+0000","last_clean":"2024-03-12T03:34:33.142268+0000","last_became_active":"2024-03-12T03:33:27.380880+0000","last_became_peered":"2024-03-12T03:33:27.380880+0000","last_unstale":"2024-03-12T03:34:33.142268+0000","last_undegraded":"2024-03-12T03:34:33.142268+0000","last_fullsized":"2024-03-12T03:34:33.142268+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.898172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639525+0000","last_change":"2024-03-12T03:34:36.949207+0000","last_active":"2024-03-12T03:34:44.639525+0000","last_peered":"2024-03-12T03:34:44.639525+0000","last_clean":"2024-03-12T03:34:44.639525+0000","last_became_active":"2024-03-12T03:33:25.181087+0000","last_became_peered":"2024-03-12T03:33:25.181087+0000","last_unstale":"2024-03-12T03:34:44.639525+0000","last_undegraded":"2024-03-12T03:34:44.639525+0000","last_fullsized":"2024-03-12T03:34:44.639525+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.949124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.949124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:57.461161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639452+0000","last_change":"2024-03-12T03:34:44.029203+0000","last_active":"2024-03-12T03:34:44.639452+0000","last_peered":"2024-03-12T03:34:44.639452+0000","last_clean":"2024-03-12T03:34:44.639452+0000","last_became_active":"2024-03-12T03:33:21.573218+0000","last_became_peered":"2024-03-12T03:33:21.573218+0000","last_unstale":"2024-03-12T03:34:44.639452+0000","last_undegraded":"2024-03-12T03:34:44.639452+0000","last_fullsized":"2024-03-12T03:34:44.639452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:44.029060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.029060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.635332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141485+0000","last_change":"2024-03-12T03:33:23.162704+0000","last_active":"2024-03-12T03:34:33.141485+0000","last_peered":"2024-03-12T03:34:33.141485+0000","last_clean":"2024-03-12T03:34:33.141485+0000","last_became_active":"2024-03-12T03:33:23.162427+0000","last_became_peered":"2024-03-12T03:33:23.162427+0000","last_unstale":"2024-03-12T03:34:33.141485+0000","last_undegraded":"2024-03-12T03:34:33.141485+0000","last_fullsized":"2024-03-12T03:34:33.141485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.329189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:37.658672+0000","last_change":"2024-03-12T03:34:37.658704+0000","last_active":"2024-03-12T03:34:37.658672+0000","last_peered":"2024-03-12T03:34:37.658672+0000","last_clean":"2024-03-12T03:34:37.658672+0000","last_became_active":"2024-03-12T03:33:27.189152+0000","last_became_peered":"2024-03-12T03:33:27.189152+0000","last_unstale":"2024-03-12T03:34:37.658672+0000","last_undegraded":"2024-03-12T03:34:37.658672+0000","last_fullsized":"2024-03-12T03:34:37.658672+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:34:37.658602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:37.658602+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:38.697166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":91,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:45.342287+0000","last_change":"2024-03-12T03:34:33.696207+0000","last_active":"2024-03-12T03:34:45.342287+0000","last_peered":"2024-03-12T03:34:45.342287+0000","last_clean":"2024-03-12T03:34:45.342287+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:34:45.342287+0000","last_undegraded":"2024-03-12T03:34:45.342287+0000","last_fullsized":"2024-03-12T03:34:45.342287+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:33.696102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:33.696102+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.429139+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257322+0000","last_change":"2024-03-12T03:33:21.097524+0000","last_active":"2024-03-12T03:34:33.257322+0000","last_peered":"2024-03-12T03:34:33.257322+0000","last_clean":"2024-03-12T03:34:33.257322+0000","last_became_active":"2024-03-12T03:33:21.097218+0000","last_became_peered":"2024-03-12T03:33:21.097218+0000","last_unstale":"2024-03-12T03:34:33.257322+0000","last_undegraded":"2024-03-12T03:34:33.257322+0000","last_fullsized":"2024-03-12T03:34:33.257322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'17","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140984+0000","last_change":"2024-03-12T03:33:23.553096+0000","last_active":"2024-03-12T03:34:33.140984+0000","last_peered":"2024-03-12T03:34:33.140984+0000","last_clean":"2024-03-12T03:34:33.140984+0000","last_became_active":"2024-03-12T03:33:23.552812+0000","last_became_peered":"2024-03-12T03:33:23.552812+0000","last_unstale":"2024-03-12T03:34:33.140984+0000","last_undegraded":"2024-03-12T03:34:33.140984+0000","last_fullsized":"2024-03-12T03:34:33.140984+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:48.664041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257952+0000","last_change":"2024-03-12T03:33:27.184475+0000","last_active":"2024-03-12T03:34:33.257952+0000","last_peered":"2024-03-12T03:34:33.257952+0000","last_clean":"2024-03-12T03:34:33.257952+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:34:33.257952+0000","last_undegraded":"2024-03-12T03:34:33.257952+0000","last_fullsized":"2024-03-12T03:34:33.257952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:52.246462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141241+0000","last_change":"2024-03-12T03:33:25.179324+0000","last_active":"2024-03-12T03:34:33.141241+0000","last_peered":"2024-03-12T03:34:33.141241+0000","last_clean":"2024-03-12T03:34:33.141241+0000","last_became_active":"2024-03-12T03:33:25.178346+0000","last_became_peered":"2024-03-12T03:33:25.178346+0000","last_unstale":"2024-03-12T03:34:33.141241+0000","last_undegraded":"2024-03-12T03:34:33.141241+0000","last_fullsized":"2024-03-12T03:34:33.141241+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:36.290977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141686+0000","last_change":"2024-03-12T03:33:21.095860+0000","last_active":"2024-03-12T03:34:33.141686+0000","last_peered":"2024-03-12T03:34:33.141686+0000","last_clean":"2024-03-12T03:34:33.141686+0000","last_became_active":"2024-03-12T03:33:21.095556+0000","last_became_peered":"2024-03-12T03:33:21.095556+0000","last_unstale":"2024-03-12T03:34:33.141686+0000","last_undegraded":"2024-03-12T03:34:33.141686+0000","last_fullsized":"2024-03-12T03:34:33.141686+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.291634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:45.343212+0000","last_change":"2024-03-12T03:33:23.159103+0000","last_active":"2024-03-12T03:34:45.343212+0000","last_peered":"2024-03-12T03:34:45.343212+0000","last_clean":"2024-03-12T03:34:45.343212+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:34:45.343212+0000","last_undegraded":"2024-03-12T03:34:45.343212+0000","last_fullsized":"2024-03-12T03:34:45.343212+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:47.268938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693506+0000","last_change":"2024-03-12T03:33:27.190023+0000","last_active":"2024-03-12T03:34:33.693506+0000","last_peered":"2024-03-12T03:34:33.693506+0000","last_clean":"2024-03-12T03:34:33.693506+0000","last_became_active":"2024-03-12T03:33:27.189696+0000","last_became_peered":"2024-03-12T03:33:27.189696+0000","last_unstale":"2024-03-12T03:34:33.693506+0000","last_undegraded":"2024-03-12T03:34:33.693506+0000","last_fullsized":"2024-03-12T03:34:33.693506+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.530688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:45.342225+0000","last_change":"2024-03-12T03:34:36.701692+0000","last_active":"2024-03-12T03:34:45.342225+0000","last_peered":"2024-03-12T03:34:45.342225+0000","last_clean":"2024-03-12T03:34:45.342225+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:34:45.342225+0000","last_undegraded":"2024-03-12T03:34:45.342225+0000","last_fullsized":"2024-03-12T03:34:45.342225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.701611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.701611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.280217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141208+0000","last_change":"2024-03-12T03:34:32.746113+0000","last_active":"2024-03-12T03:34:33.141208+0000","last_peered":"2024-03-12T03:34:33.141208+0000","last_clean":"2024-03-12T03:34:33.141208+0000","last_became_active":"2024-03-12T03:33:21.096560+0000","last_became_peered":"2024-03-12T03:33:21.096560+0000","last_unstale":"2024-03-12T03:34:33.141208+0000","last_undegraded":"2024-03-12T03:34:33.141208+0000","last_fullsized":"2024-03-12T03:34:33.141208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:32.746027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:32.746027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:33.564397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"90'6","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:34.274429+0000","last_change":"2024-03-12T03:34:34.274478+0000","last_active":"2024-03-12T03:34:34.274429+0000","last_peered":"2024-03-12T03:34:34.274429+0000","last_clean":"2024-03-12T03:34:34.274429+0000","last_became_active":"2024-03-12T03:33:23.161801+0000","last_became_peered":"2024-03-12T03:33:23.161801+0000","last_unstale":"2024-03-12T03:34:34.274429+0000","last_undegraded":"2024-03-12T03:34:34.274429+0000","last_fullsized":"2024-03-12T03:34:34.274429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.274353+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.042399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:40.191598+0000","last_change":"2024-03-12T03:34:40.191618+0000","last_active":"2024-03-12T03:34:40.191598+0000","last_peered":"2024-03-12T03:34:40.191598+0000","last_clean":"2024-03-12T03:34:40.191598+0000","last_became_active":"2024-03-12T03:33:27.381110+0000","last_became_peered":"2024-03-12T03:33:27.381110+0000","last_unstale":"2024-03-12T03:34:40.191598+0000","last_undegraded":"2024-03-12T03:34:40.191598+0000","last_fullsized":"2024-03-12T03:34:40.191598+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.191538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.480916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.694028+0000","last_change":"2024-03-12T03:33:25.178948+0000","last_active":"2024-03-12T03:34:33.694028+0000","last_peered":"2024-03-12T03:34:33.694028+0000","last_clean":"2024-03-12T03:34:33.694028+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:34:33.694028+0000","last_undegraded":"2024-03-12T03:34:33.694028+0000","last_fullsized":"2024-03-12T03:34:33.694028+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:44.813464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259793+0000","last_change":"2024-03-12T03:34:25.171761+0000","last_active":"2024-03-12T03:34:33.259793+0000","last_peered":"2024-03-12T03:34:33.259793+0000","last_clean":"2024-03-12T03:34:33.259793+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:34:33.259793+0000","last_undegraded":"2024-03-12T03:34:33.259793+0000","last_fullsized":"2024-03-12T03:34:33.259793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:25.171622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.171622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:37.280670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'18","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141921+0000","last_change":"2024-03-12T03:33:23.552489+0000","last_active":"2024-03-12T03:34:33.141921+0000","last_peered":"2024-03-12T03:34:33.141921+0000","last_clean":"2024-03-12T03:34:33.141921+0000","last_became_active":"2024-03-12T03:33:23.552210+0000","last_became_peered":"2024-03-12T03:33:23.552210+0000","last_unstale":"2024-03-12T03:34:33.141921+0000","last_undegraded":"2024-03-12T03:34:33.141921+0000","last_fullsized":"2024-03-12T03:34:33.141921+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:36.029074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140452+0000","last_change":"2024-03-12T03:33:27.193134+0000","last_active":"2024-03-12T03:34:33.140452+0000","last_peered":"2024-03-12T03:34:33.140452+0000","last_clean":"2024-03-12T03:34:33.140452+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:34:33.140452+0000","last_undegraded":"2024-03-12T03:34:33.140452+0000","last_fullsized":"2024-03-12T03:34:33.140452+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:43.487708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141983+0000","last_change":"2024-03-12T03:33:25.184266+0000","last_active":"2024-03-12T03:34:33.141983+0000","last_peered":"2024-03-12T03:34:33.141983+0000","last_clean":"2024-03-12T03:34:33.141983+0000","last_became_active":"2024-03-12T03:33:25.183725+0000","last_became_peered":"2024-03-12T03:33:25.183725+0000","last_unstale":"2024-03-12T03:34:33.141983+0000","last_undegraded":"2024-03-12T03:34:33.141983+0000","last_fullsized":"2024-03-12T03:34:33.141983+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:33.221144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693992+0000","last_change":"2024-03-12T03:33:21.107392+0000","last_active":"2024-03-12T03:34:33.693992+0000","last_peered":"2024-03-12T03:34:33.693992+0000","last_clean":"2024-03-12T03:34:33.693992+0000","last_became_active":"2024-03-12T03:33:21.107037+0000","last_became_peered":"2024-03-12T03:33:21.107037+0000","last_unstale":"2024-03-12T03:34:33.693992+0000","last_undegraded":"2024-03-12T03:34:33.693992+0000","last_fullsized":"2024-03-12T03:34:33.693992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:34.083565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257758+0000","last_change":"2024-03-12T03:33:23.161686+0000","last_active":"2024-03-12T03:34:33.257758+0000","last_peered":"2024-03-12T03:34:33.257758+0000","last_clean":"2024-03-12T03:34:33.257758+0000","last_became_active":"2024-03-12T03:33:23.160818+0000","last_became_peered":"2024-03-12T03:33:23.160818+0000","last_unstale":"2024-03-12T03:34:33.257758+0000","last_undegraded":"2024-03-12T03:34:33.257758+0000","last_fullsized":"2024-03-12T03:34:33.257758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.366562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142135+0000","last_change":"2024-03-12T03:33:27.192215+0000","last_active":"2024-03-12T03:34:33.142135+0000","last_peered":"2024-03-12T03:34:33.142135+0000","last_clean":"2024-03-12T03:34:33.142135+0000","last_became_active":"2024-03-12T03:33:27.191710+0000","last_became_peered":"2024-03-12T03:33:27.191710+0000","last_unstale":"2024-03-12T03:34:33.142135+0000","last_undegraded":"2024-03-12T03:34:33.142135+0000","last_fullsized":"2024-03-12T03:34:33.142135+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:45.524065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.258695+0000","last_change":"2024-03-12T03:34:33.258733+0000","last_active":"2024-03-12T03:34:33.258695+0000","last_peered":"2024-03-12T03:34:33.258695+0000","last_clean":"2024-03-12T03:34:33.258695+0000","last_became_active":"2024-03-12T03:33:25.182004+0000","last_became_peered":"2024-03-12T03:33:25.182004+0000","last_unstale":"2024-03-12T03:34:33.258695+0000","last_undegraded":"2024-03-12T03:34:33.258695+0000","last_fullsized":"2024-03-12T03:34:33.258695+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:33.258626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:33.258626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.420748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639689+0000","last_change":"2024-03-12T03:33:21.109429+0000","last_active":"2024-03-12T03:34:44.639689+0000","last_peered":"2024-03-12T03:34:44.639689+0000","last_clean":"2024-03-12T03:34:44.639689+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:34:44.639689+0000","last_undegraded":"2024-03-12T03:34:44.639689+0000","last_fullsized":"2024-03-12T03:34:44.639689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639634+0000","last_change":"2024-03-12T03:34:34.953658+0000","last_active":"2024-03-12T03:34:44.639634+0000","last_peered":"2024-03-12T03:34:44.639634+0000","last_clean":"2024-03-12T03:34:44.639634+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:34:44.639634+0000","last_undegraded":"2024-03-12T03:34:44.639634+0000","last_fullsized":"2024-03-12T03:34:44.639634+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:34:34.953565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.953565+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.161677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:34.718570+0000","last_change":"2024-03-12T03:34:34.718591+0000","last_active":"2024-03-12T03:34:34.718570+0000","last_peered":"2024-03-12T03:34:34.718570+0000","last_clean":"2024-03-12T03:34:34.718570+0000","last_became_active":"2024-03-12T03:33:27.194094+0000","last_became_peered":"2024-03-12T03:33:27.194094+0000","last_unstale":"2024-03-12T03:34:34.718570+0000","last_undegraded":"2024-03-12T03:34:34.718570+0000","last_fullsized":"2024-03-12T03:34:34.718570+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:34.718509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.718509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.921678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:38.139665+0000","last_change":"2024-03-12T03:34:38.139709+0000","last_active":"2024-03-12T03:34:38.139665+0000","last_peered":"2024-03-12T03:34:38.139665+0000","last_clean":"2024-03-12T03:34:38.139665+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:34:38.139665+0000","last_undegraded":"2024-03-12T03:34:38.139665+0000","last_fullsized":"2024-03-12T03:34:38.139665+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:38.139582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.139582+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:39.751323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:38.663608+0000","last_change":"2024-03-12T03:34:38.663634+0000","last_active":"2024-03-12T03:34:38.663608+0000","last_peered":"2024-03-12T03:34:38.663608+0000","last_clean":"2024-03-12T03:34:38.663608+0000","last_became_active":"2024-03-12T03:33:21.573244+0000","last_became_peered":"2024-03-12T03:33:21.573244+0000","last_unstale":"2024-03-12T03:34:38.663608+0000","last_undegraded":"2024-03-12T03:34:38.663608+0000","last_fullsized":"2024-03-12T03:34:38.663608+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:34:38.663542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.663542+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:59.053797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141451+0000","last_change":"2024-03-12T03:33:23.552006+0000","last_active":"2024-03-12T03:34:33.141451+0000","last_peered":"2024-03-12T03:34:33.141451+0000","last_clean":"2024-03-12T03:34:33.141451+0000","last_became_active":"2024-03-12T03:33:23.551706+0000","last_became_peered":"2024-03-12T03:33:23.551706+0000","last_unstale":"2024-03-12T03:34:33.141451+0000","last_undegraded":"2024-03-12T03:34:33.141451+0000","last_fullsized":"2024-03-12T03:34:33.141451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:45.152617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.640053+0000","last_change":"2024-03-12T03:33:27.185209+0000","last_active":"2024-03-12T03:34:44.640053+0000","last_peered":"2024-03-12T03:34:44.640053+0000","last_clean":"2024-03-12T03:34:44.640053+0000","last_became_active":"2024-03-12T03:33:27.184916+0000","last_became_peered":"2024-03-12T03:33:27.184916+0000","last_unstale":"2024-03-12T03:34:44.640053+0000","last_undegraded":"2024-03-12T03:34:44.640053+0000","last_fullsized":"2024-03-12T03:34:44.640053+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141598+0000","last_change":"2024-03-12T03:33:25.179522+0000","last_active":"2024-03-12T03:34:33.141598+0000","last_peered":"2024-03-12T03:34:33.141598+0000","last_clean":"2024-03-12T03:34:33.141598+0000","last_became_active":"2024-03-12T03:33:25.179158+0000","last_became_peered":"2024-03-12T03:33:25.179158+0000","last_unstale":"2024-03-12T03:34:33.141598+0000","last_undegraded":"2024-03-12T03:34:33.141598+0000","last_fullsized":"2024-03-12T03:34:33.141598+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:41.537531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:40.753211+0000","last_change":"2024-03-12T03:34:40.753233+0000","last_active":"2024-03-12T03:34:40.753211+0000","last_peered":"2024-03-12T03:34:40.753211+0000","last_clean":"2024-03-12T03:34:40.753211+0000","last_became_active":"2024-03-12T03:33:21.101819+0000","last_became_peered":"2024-03-12T03:33:21.101819+0000","last_unstale":"2024-03-12T03:34:40.753211+0000","last_undegraded":"2024-03-12T03:34:40.753211+0000","last_fullsized":"2024-03-12T03:34:40.753211+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.753153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.816911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259938+0000","last_change":"2024-03-12T03:34:29.083183+0000","last_active":"2024-03-12T03:34:33.259938+0000","last_peered":"2024-03-12T03:34:33.259938+0000","last_clean":"2024-03-12T03:34:33.259938+0000","last_became_active":"2024-03-12T03:33:23.553480+0000","last_became_peered":"2024-03-12T03:33:23.553480+0000","last_unstale":"2024-03-12T03:34:33.259938+0000","last_undegraded":"2024-03-12T03:34:33.259938+0000","last_fullsized":"2024-03-12T03:34:33.259938+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:34:29.083063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:29.083063+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:45.174021+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.562207+0000","last_change":"2024-03-12T03:33:27.188065+0000","last_active":"2024-03-12T03:34:33.562207+0000","last_peered":"2024-03-12T03:34:33.562207+0000","last_clean":"2024-03-12T03:34:33.562207+0000","last_became_active":"2024-03-12T03:33:27.187657+0000","last_became_peered":"2024-03-12T03:33:27.187657+0000","last_unstale":"2024-03-12T03:34:33.562207+0000","last_undegraded":"2024-03-12T03:34:33.562207+0000","last_fullsized":"2024-03-12T03:34:33.562207+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142236+0000","last_change":"2024-03-12T03:33:25.175313+0000","last_active":"2024-03-12T03:34:33.142236+0000","last_peered":"2024-03-12T03:34:33.142236+0000","last_clean":"2024-03-12T03:34:33.142236+0000","last_became_active":"2024-03-12T03:33:25.174953+0000","last_became_peered":"2024-03-12T03:33:25.174953+0000","last_unstale":"2024-03-12T03:34:33.142236+0000","last_undegraded":"2024-03-12T03:34:33.142236+0000","last_fullsized":"2024-03-12T03:34:33.142236+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:37.703054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693529+0000","last_change":"2024-03-12T03:33:21.106130+0000","last_active":"2024-03-12T03:34:33.693529+0000","last_peered":"2024-03-12T03:34:33.693529+0000","last_clean":"2024-03-12T03:34:33.693529+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:34:33.693529+0000","last_undegraded":"2024-03-12T03:34:33.693529+0000","last_fullsized":"2024-03-12T03:34:33.693529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:43.636698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141096+0000","last_change":"2024-03-12T03:33:23.552425+0000","last_active":"2024-03-12T03:34:33.141096+0000","last_peered":"2024-03-12T03:34:33.141096+0000","last_clean":"2024-03-12T03:34:33.141096+0000","last_became_active":"2024-03-12T03:33:23.552164+0000","last_became_peered":"2024-03-12T03:33:23.552164+0000","last_unstale":"2024-03-12T03:34:33.141096+0000","last_undegraded":"2024-03-12T03:34:33.141096+0000","last_fullsized":"2024-03-12T03:34:33.141096+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.480291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693972+0000","last_change":"2024-03-12T03:33:27.383521+0000","last_active":"2024-03-12T03:34:33.693972+0000","last_peered":"2024-03-12T03:34:33.693972+0000","last_clean":"2024-03-12T03:34:33.693972+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:34:33.693972+0000","last_undegraded":"2024-03-12T03:34:33.693972+0000","last_fullsized":"2024-03-12T03:34:33.693972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.404863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259842+0000","last_change":"2024-03-12T03:33:25.181046+0000","last_active":"2024-03-12T03:34:33.259842+0000","last_peered":"2024-03-12T03:34:33.259842+0000","last_clean":"2024-03-12T03:34:33.259842+0000","last_became_active":"2024-03-12T03:33:25.180752+0000","last_became_peered":"2024-03-12T03:33:25.180752+0000","last_unstale":"2024-03-12T03:34:33.259842+0000","last_undegraded":"2024-03-12T03:34:33.259842+0000","last_fullsized":"2024-03-12T03:34:33.259842+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:52.874476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:45.342435+0000","last_change":"2024-03-12T03:34:26.565344+0000","last_active":"2024-03-12T03:34:45.342435+0000","last_peered":"2024-03-12T03:34:45.342435+0000","last_clean":"2024-03-12T03:34:45.342435+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:34:45.342435+0000","last_undegraded":"2024-03-12T03:34:45.342435+0000","last_fullsized":"2024-03-12T03:34:45.342435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:26.565245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:26.565245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:28.167100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142318+0000","last_change":"2024-03-12T03:33:23.552149+0000","last_active":"2024-03-12T03:34:33.142318+0000","last_peered":"2024-03-12T03:34:33.142318+0000","last_clean":"2024-03-12T03:34:33.142318+0000","last_became_active":"2024-03-12T03:33:23.551773+0000","last_became_peered":"2024-03-12T03:33:23.551773+0000","last_unstale":"2024-03-12T03:34:33.142318+0000","last_undegraded":"2024-03-12T03:34:33.142318+0000","last_fullsized":"2024-03-12T03:34:33.142318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:35.193863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259399+0000","last_change":"2024-03-12T03:33:27.382928+0000","last_active":"2024-03-12T03:34:33.259399+0000","last_peered":"2024-03-12T03:34:33.259399+0000","last_clean":"2024-03-12T03:34:33.259399+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:34:33.259399+0000","last_undegraded":"2024-03-12T03:34:33.259399+0000","last_fullsized":"2024-03-12T03:34:33.259399+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.104295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.694256+0000","last_change":"2024-03-12T03:33:21.107807+0000","last_active":"2024-03-12T03:34:33.694256+0000","last_peered":"2024-03-12T03:34:33.694256+0000","last_clean":"2024-03-12T03:34:33.694256+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:34:33.694256+0000","last_undegraded":"2024-03-12T03:34:33.694256+0000","last_fullsized":"2024-03-12T03:34:33.694256+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:48.638486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"90'36","reported_seq":119,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:39.184041+0000","last_change":"2024-03-12T03:34:39.184089+0000","last_active":"2024-03-12T03:34:39.184041+0000","last_peered":"2024-03-12T03:34:39.184041+0000","last_clean":"2024-03-12T03:34:39.184041+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:34:39.184041+0000","last_undegraded":"2024-03-12T03:34:39.184041+0000","last_fullsized":"2024-03-12T03:34:39.184041+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.183984+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.564093+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:39.659357+0000","last_change":"2024-03-12T03:34:39.659403+0000","last_active":"2024-03-12T03:34:39.659357+0000","last_peered":"2024-03-12T03:34:39.659357+0000","last_clean":"2024-03-12T03:34:39.659357+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:34:39.659357+0000","last_undegraded":"2024-03-12T03:34:39.659357+0000","last_fullsized":"2024-03-12T03:34:39.659357+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:39.659277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.659277+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:08.903088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141754+0000","last_change":"2024-03-12T03:33:27.191091+0000","last_active":"2024-03-12T03:34:33.141754+0000","last_peered":"2024-03-12T03:34:33.141754+0000","last_clean":"2024-03-12T03:34:33.141754+0000","last_became_active":"2024-03-12T03:33:27.190753+0000","last_became_peered":"2024-03-12T03:33:27.190753+0000","last_unstale":"2024-03-12T03:34:33.141754+0000","last_undegraded":"2024-03-12T03:34:33.141754+0000","last_fullsized":"2024-03-12T03:34:33.141754+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.061395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.694206+0000","last_change":"2024-03-12T03:34:23.842758+0000","last_active":"2024-03-12T03:34:33.694206+0000","last_peered":"2024-03-12T03:34:33.694206+0000","last_clean":"2024-03-12T03:34:33.694206+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:34:33.694206+0000","last_undegraded":"2024-03-12T03:34:33.694206+0000","last_fullsized":"2024-03-12T03:34:33.694206+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:23.842628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.842628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.823849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":106,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639472+0000","last_change":"2024-03-12T03:33:23.163665+0000","last_active":"2024-03-12T03:34:44.639472+0000","last_peered":"2024-03-12T03:34:44.639472+0000","last_clean":"2024-03-12T03:34:44.639472+0000","last_became_active":"2024-03-12T03:33:23.163351+0000","last_became_peered":"2024-03-12T03:33:23.163351+0000","last_unstale":"2024-03-12T03:34:44.639472+0000","last_undegraded":"2024-03-12T03:34:44.639472+0000","last_fullsized":"2024-03-12T03:34:44.639472+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561362+0000","last_change":"2024-03-12T03:33:25.185933+0000","last_active":"2024-03-12T03:34:33.561362+0000","last_peered":"2024-03-12T03:34:33.561362+0000","last_clean":"2024-03-12T03:34:33.561362+0000","last_became_active":"2024-03-12T03:33:25.185596+0000","last_became_peered":"2024-03-12T03:33:25.185596+0000","last_unstale":"2024-03-12T03:34:33.561362+0000","last_undegraded":"2024-03-12T03:34:33.561362+0000","last_fullsized":"2024-03-12T03:34:33.561362+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:34.341308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693346+0000","last_change":"2024-03-12T03:33:27.188268+0000","last_active":"2024-03-12T03:34:33.693346+0000","last_peered":"2024-03-12T03:34:33.693346+0000","last_clean":"2024-03-12T03:34:33.693346+0000","last_became_active":"2024-03-12T03:33:27.187955+0000","last_became_peered":"2024-03-12T03:33:27.187955+0000","last_unstale":"2024-03-12T03:34:33.693346+0000","last_undegraded":"2024-03-12T03:34:33.693346+0000","last_fullsized":"2024-03-12T03:34:33.693346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:37.416612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561861+0000","last_change":"2024-03-12T03:33:25.180331+0000","last_active":"2024-03-12T03:34:33.561861+0000","last_peered":"2024-03-12T03:34:33.561861+0000","last_clean":"2024-03-12T03:34:33.561861+0000","last_became_active":"2024-03-12T03:33:25.179593+0000","last_became_peered":"2024-03-12T03:33:25.179593+0000","last_unstale":"2024-03-12T03:34:33.561861+0000","last_undegraded":"2024-03-12T03:34:33.561861+0000","last_fullsized":"2024-03-12T03:34:33.561861+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.050267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.694238+0000","last_change":"2024-03-12T03:34:30.792781+0000","last_active":"2024-03-12T03:34:33.694238+0000","last_peered":"2024-03-12T03:34:33.694238+0000","last_clean":"2024-03-12T03:34:33.694238+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:34:33.694238+0000","last_undegraded":"2024-03-12T03:34:33.694238+0000","last_fullsized":"2024-03-12T03:34:33.694238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:30.792702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.792702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:50.132706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":102,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:45.342840+0000","last_change":"2024-03-12T03:33:23.551366+0000","last_active":"2024-03-12T03:34:45.342840+0000","last_peered":"2024-03-12T03:34:45.342840+0000","last_clean":"2024-03-12T03:34:45.342840+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:34:45.342840+0000","last_undegraded":"2024-03-12T03:34:45.342840+0000","last_fullsized":"2024-03-12T03:34:45.342840+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:37.737449+0000","last_change":"2024-03-12T03:34:37.737471+0000","last_active":"2024-03-12T03:34:37.737449+0000","last_peered":"2024-03-12T03:34:37.737449+0000","last_clean":"2024-03-12T03:34:37.737449+0000","last_became_active":"2024-03-12T03:33:27.189948+0000","last_became_peered":"2024-03-12T03:33:27.189948+0000","last_unstale":"2024-03-12T03:34:37.737449+0000","last_undegraded":"2024-03-12T03:34:37.737449+0000","last_fullsized":"2024-03-12T03:34:37.737449+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:37.737386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:37.737386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.387230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.562079+0000","last_change":"2024-03-12T03:33:25.184715+0000","last_active":"2024-03-12T03:34:33.562079+0000","last_peered":"2024-03-12T03:34:33.562079+0000","last_clean":"2024-03-12T03:34:33.562079+0000","last_became_active":"2024-03-12T03:33:25.184289+0000","last_became_peered":"2024-03-12T03:33:25.184289+0000","last_unstale":"2024-03-12T03:34:33.562079+0000","last_undegraded":"2024-03-12T03:34:33.562079+0000","last_fullsized":"2024-03-12T03:34:33.562079+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:41.396354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259558+0000","last_change":"2024-03-12T03:33:21.102971+0000","last_active":"2024-03-12T03:34:33.259558+0000","last_peered":"2024-03-12T03:34:33.259558+0000","last_clean":"2024-03-12T03:34:33.259558+0000","last_became_active":"2024-03-12T03:33:21.102410+0000","last_became_peered":"2024-03-12T03:33:21.102410+0000","last_unstale":"2024-03-12T03:34:33.259558+0000","last_undegraded":"2024-03-12T03:34:33.259558+0000","last_fullsized":"2024-03-12T03:34:33.259558+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:33.557063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141352+0000","last_change":"2024-03-12T03:33:23.164442+0000","last_active":"2024-03-12T03:34:33.141352+0000","last_peered":"2024-03-12T03:34:33.141352+0000","last_clean":"2024-03-12T03:34:33.141352+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:34:33.141352+0000","last_undegraded":"2024-03-12T03:34:33.141352+0000","last_fullsized":"2024-03-12T03:34:33.141352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:50.796888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141003+0000","last_change":"2024-03-12T03:33:27.185374+0000","last_active":"2024-03-12T03:34:33.141003+0000","last_peered":"2024-03-12T03:34:33.141003+0000","last_clean":"2024-03-12T03:34:33.141003+0000","last_became_active":"2024-03-12T03:33:27.185041+0000","last_became_peered":"2024-03-12T03:33:27.185041+0000","last_unstale":"2024-03-12T03:34:33.141003+0000","last_undegraded":"2024-03-12T03:34:33.141003+0000","last_fullsized":"2024-03-12T03:34:33.141003+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:34.953585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259525+0000","last_change":"2024-03-12T03:34:22.168794+0000","last_active":"2024-03-12T03:34:33.259525+0000","last_peered":"2024-03-12T03:34:33.259525+0000","last_clean":"2024-03-12T03:34:33.259525+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:34:33.259525+0000","last_undegraded":"2024-03-12T03:34:33.259525+0000","last_fullsized":"2024-03-12T03:34:33.259525+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:34:22.168654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:22.168654+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:48.338217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639347+0000","last_change":"2024-03-12T03:33:21.107222+0000","last_active":"2024-03-12T03:34:44.639347+0000","last_peered":"2024-03-12T03:34:44.639347+0000","last_clean":"2024-03-12T03:34:44.639347+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:34:44.639347+0000","last_undegraded":"2024-03-12T03:34:44.639347+0000","last_fullsized":"2024-03-12T03:34:44.639347+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'35","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259363+0000","last_change":"2024-03-12T03:33:50.166841+0000","last_active":"2024-03-12T03:34:33.259363+0000","last_peered":"2024-03-12T03:34:33.259363+0000","last_clean":"2024-03-12T03:34:33.259363+0000","last_became_active":"2024-03-12T03:33:50.151664+0000","last_became_peered":"2024-03-12T03:33:50.151664+0000","last_unstale":"2024-03-12T03:34:33.259363+0000","last_undegraded":"2024-03-12T03:34:33.259363+0000","last_fullsized":"2024-03-12T03:34:33.259363+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:48.926653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:38.697982+0000","last_change":"2024-03-12T03:34:38.698005+0000","last_active":"2024-03-12T03:34:38.697982+0000","last_peered":"2024-03-12T03:34:38.697982+0000","last_clean":"2024-03-12T03:34:38.697982+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:34:38.697982+0000","last_undegraded":"2024-03-12T03:34:38.697982+0000","last_fullsized":"2024-03-12T03:34:38.697982+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:38.697923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.697923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:07.086680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:40.663056+0000","last_change":"2024-03-12T03:34:40.663082+0000","last_active":"2024-03-12T03:34:40.663056+0000","last_peered":"2024-03-12T03:34:40.663056+0000","last_clean":"2024-03-12T03:34:40.663056+0000","last_became_active":"2024-03-12T03:33:27.380389+0000","last_became_peered":"2024-03-12T03:33:27.380389+0000","last_unstale":"2024-03-12T03:34:40.663056+0000","last_undegraded":"2024-03-12T03:34:40.663056+0000","last_fullsized":"2024-03-12T03:34:40.663056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.663002+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.064026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257463+0000","last_change":"2024-03-12T03:33:21.573748+0000","last_active":"2024-03-12T03:34:33.257463+0000","last_peered":"2024-03-12T03:34:33.257463+0000","last_clean":"2024-03-12T03:34:33.257463+0000","last_became_active":"2024-03-12T03:33:21.572911+0000","last_became_peered":"2024-03-12T03:33:21.572911+0000","last_unstale":"2024-03-12T03:34:33.257463+0000","last_undegraded":"2024-03-12T03:34:33.257463+0000","last_fullsized":"2024-03-12T03:34:33.257463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:45.983059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561323+0000","last_change":"2024-03-12T03:33:24.161865+0000","last_active":"2024-03-12T03:34:33.561323+0000","last_peered":"2024-03-12T03:34:33.561323+0000","last_clean":"2024-03-12T03:34:33.561323+0000","last_became_active":"2024-03-12T03:33:20.093064+0000","last_became_peered":"2024-03-12T03:33:20.093064+0000","last_unstale":"2024-03-12T03:34:33.561323+0000","last_undegraded":"2024-03-12T03:34:33.561323+0000","last_fullsized":"2024-03-12T03:34:33.561323+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:33:19.077015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000516994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.258062+0000","last_change":"2024-03-12T03:33:23.163705+0000","last_active":"2024-03-12T03:34:33.258062+0000","last_peered":"2024-03-12T03:34:33.258062+0000","last_clean":"2024-03-12T03:34:33.258062+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:34:33.258062+0000","last_undegraded":"2024-03-12T03:34:33.258062+0000","last_fullsized":"2024-03-12T03:34:33.258062+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:40.056709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561574+0000","last_change":"2024-03-12T03:33:25.179406+0000","last_active":"2024-03-12T03:34:33.561574+0000","last_peered":"2024-03-12T03:34:33.561574+0000","last_clean":"2024-03-12T03:34:33.561574+0000","last_became_active":"2024-03-12T03:33:25.179022+0000","last_became_peered":"2024-03-12T03:33:25.179022+0000","last_unstale":"2024-03-12T03:34:33.561574+0000","last_undegraded":"2024-03-12T03:34:33.561574+0000","last_fullsized":"2024-03-12T03:34:33.561574+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259883+0000","last_change":"2024-03-12T03:33:27.188886+0000","last_active":"2024-03-12T03:34:33.259883+0000","last_peered":"2024-03-12T03:34:33.259883+0000","last_clean":"2024-03-12T03:34:33.259883+0000","last_became_active":"2024-03-12T03:33:27.188551+0000","last_became_peered":"2024-03-12T03:33:27.188551+0000","last_unstale":"2024-03-12T03:34:33.259883+0000","last_undegraded":"2024-03-12T03:34:33.259883+0000","last_fullsized":"2024-03-12T03:34:33.259883+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:50.069549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":312,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693459+0000","last_change":"2024-03-12T03:33:29.565156+0000","last_active":"2024-03-12T03:34:33.693459+0000","last_peered":"2024-03-12T03:34:33.693459+0000","last_clean":"2024-03-12T03:34:33.693459+0000","last_became_active":"2024-03-12T03:33:11.319888+0000","last_became_peered":"2024-03-12T03:33:11.319888+0000","last_unstale":"2024-03-12T03:34:33.693459+0000","last_undegraded":"2024-03-12T03:34:33.693459+0000","last_fullsized":"2024-03-12T03:34:33.693459+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-12T03:33:29.565069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:33:29.565069+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:47.469193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141670+0000","last_change":"2024-03-12T03:33:21.101239+0000","last_active":"2024-03-12T03:34:33.141670+0000","last_peered":"2024-03-12T03:34:33.141670+0000","last_clean":"2024-03-12T03:34:33.141670+0000","last_became_active":"2024-03-12T03:33:21.100900+0000","last_became_peered":"2024-03-12T03:33:21.100900+0000","last_unstale":"2024-03-12T03:34:33.141670+0000","last_undegraded":"2024-03-12T03:34:33.141670+0000","last_fullsized":"2024-03-12T03:34:33.141670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.038220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":106,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:45.343075+0000","last_change":"2024-03-12T03:34:35.664996+0000","last_active":"2024-03-12T03:34:45.343075+0000","last_peered":"2024-03-12T03:34:45.343075+0000","last_clean":"2024-03-12T03:34:45.343075+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:34:45.343075+0000","last_undegraded":"2024-03-12T03:34:45.343075+0000","last_fullsized":"2024-03-12T03:34:45.343075+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:34:35.664876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:35.664876+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:54.461746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693937+0000","last_change":"2024-03-12T03:33:25.178070+0000","last_active":"2024-03-12T03:34:33.693937+0000","last_peered":"2024-03-12T03:34:33.693937+0000","last_clean":"2024-03-12T03:34:33.693937+0000","last_became_active":"2024-03-12T03:33:25.177762+0000","last_became_peered":"2024-03-12T03:33:25.177762+0000","last_unstale":"2024-03-12T03:34:33.693937+0000","last_undegraded":"2024-03-12T03:34:33.693937+0000","last_fullsized":"2024-03-12T03:34:33.693937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.594234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257389+0000","last_change":"2024-03-12T03:33:27.192203+0000","last_active":"2024-03-12T03:34:33.257389+0000","last_peered":"2024-03-12T03:34:33.257389+0000","last_clean":"2024-03-12T03:34:33.257389+0000","last_became_active":"2024-03-12T03:33:27.191878+0000","last_became_peered":"2024-03-12T03:33:27.191878+0000","last_unstale":"2024-03-12T03:34:33.257389+0000","last_undegraded":"2024-03-12T03:34:33.257389+0000","last_fullsized":"2024-03-12T03:34:33.257389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:43.704977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140893+0000","last_change":"2024-03-12T03:33:21.097466+0000","last_active":"2024-03-12T03:34:33.140893+0000","last_peered":"2024-03-12T03:34:33.140893+0000","last_clean":"2024-03-12T03:34:33.140893+0000","last_became_active":"2024-03-12T03:33:21.097086+0000","last_became_peered":"2024-03-12T03:33:21.097086+0000","last_unstale":"2024-03-12T03:34:33.140893+0000","last_undegraded":"2024-03-12T03:34:33.140893+0000","last_fullsized":"2024-03-12T03:34:33.140893+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.630139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":151,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:39.239320+0000","last_change":"2024-03-12T03:34:21.799658+0000","last_active":"2024-03-12T03:34:39.239320+0000","last_peered":"2024-03-12T03:34:39.239320+0000","last_clean":"2024-03-12T03:34:39.239320+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:34:39.239320+0000","last_undegraded":"2024-03-12T03:34:39.239320+0000","last_fullsized":"2024-03-12T03:34:39.239320+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:34:21.799534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:21.799534+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.973279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:39.707339+0000","last_change":"2024-03-12T03:34:39.707369+0000","last_active":"2024-03-12T03:34:39.707339+0000","last_peered":"2024-03-12T03:34:39.707339+0000","last_clean":"2024-03-12T03:34:39.707339+0000","last_became_active":"2024-03-12T03:33:23.168750+0000","last_became_peered":"2024-03-12T03:33:23.168750+0000","last_unstale":"2024-03-12T03:34:39.707339+0000","last_undegraded":"2024-03-12T03:34:39.707339+0000","last_fullsized":"2024-03-12T03:34:39.707339+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:34:39.707264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.707264+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:43.684441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.260112+0000","last_change":"2024-03-12T03:33:25.177321+0000","last_active":"2024-03-12T03:34:33.260112+0000","last_peered":"2024-03-12T03:34:33.260112+0000","last_clean":"2024-03-12T03:34:33.260112+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:34:33.260112+0000","last_undegraded":"2024-03-12T03:34:33.260112+0000","last_fullsized":"2024-03-12T03:34:33.260112+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.569538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257204+0000","last_change":"2024-03-12T03:33:27.380812+0000","last_active":"2024-03-12T03:34:33.257204+0000","last_peered":"2024-03-12T03:34:33.257204+0000","last_clean":"2024-03-12T03:34:33.257204+0000","last_became_active":"2024-03-12T03:33:27.380458+0000","last_became_peered":"2024-03-12T03:33:27.380458+0000","last_unstale":"2024-03-12T03:34:33.257204+0000","last_undegraded":"2024-03-12T03:34:33.257204+0000","last_fullsized":"2024-03-12T03:34:33.257204+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:41.294255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:34.272859+0000","last_change":"2024-03-12T03:34:34.272895+0000","last_active":"2024-03-12T03:34:34.272859+0000","last_peered":"2024-03-12T03:34:34.272859+0000","last_clean":"2024-03-12T03:34:34.272859+0000","last_became_active":"2024-03-12T03:33:21.573003+0000","last_became_peered":"2024-03-12T03:33:21.573003+0000","last_unstale":"2024-03-12T03:34:34.272859+0000","last_undegraded":"2024-03-12T03:34:34.272859+0000","last_fullsized":"2024-03-12T03:34:34.272859+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:34.272759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.272759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:47.242996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":101,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141481+0000","last_change":"2024-03-12T03:33:23.551923+0000","last_active":"2024-03-12T03:34:33.141481+0000","last_peered":"2024-03-12T03:34:33.141481+0000","last_clean":"2024-03-12T03:34:33.141481+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:34:33.141481+0000","last_undegraded":"2024-03-12T03:34:33.141481+0000","last_fullsized":"2024-03-12T03:34:33.141481+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:50.306884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:45.342214+0000","last_change":"2024-03-12T03:34:28.572496+0000","last_active":"2024-03-12T03:34:45.342214+0000","last_peered":"2024-03-12T03:34:45.342214+0000","last_clean":"2024-03-12T03:34:45.342214+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:34:45.342214+0000","last_undegraded":"2024-03-12T03:34:45.342214+0000","last_fullsized":"2024-03-12T03:34:45.342214+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:28.572379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:28.572379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:58.234927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140452+0000","last_change":"2024-03-12T03:33:27.194720+0000","last_active":"2024-03-12T03:34:33.140452+0000","last_peered":"2024-03-12T03:34:33.140452+0000","last_clean":"2024-03-12T03:34:33.140452+0000","last_became_active":"2024-03-12T03:33:27.194266+0000","last_became_peered":"2024-03-12T03:33:27.194266+0000","last_unstale":"2024-03-12T03:34:33.140452+0000","last_undegraded":"2024-03-12T03:34:33.140452+0000","last_fullsized":"2024-03-12T03:34:33.140452+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.549699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259665+0000","last_change":"2024-03-12T03:33:21.573568+0000","last_active":"2024-03-12T03:34:33.259665+0000","last_peered":"2024-03-12T03:34:33.259665+0000","last_clean":"2024-03-12T03:34:33.259665+0000","last_became_active":"2024-03-12T03:33:21.573251+0000","last_became_peered":"2024-03-12T03:33:21.573251+0000","last_unstale":"2024-03-12T03:34:33.259665+0000","last_undegraded":"2024-03-12T03:34:33.259665+0000","last_fullsized":"2024-03-12T03:34:33.259665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:35.282277+0000","last_change":"2024-03-12T03:34:35.282309+0000","last_active":"2024-03-12T03:34:35.282277+0000","last_peered":"2024-03-12T03:34:35.282277+0000","last_clean":"2024-03-12T03:34:35.282277+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:34:35.282277+0000","last_undegraded":"2024-03-12T03:34:35.282277+0000","last_fullsized":"2024-03-12T03:34:35.282277+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:34:35.282203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:35.282203+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.121154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140999+0000","last_change":"2024-03-12T03:34:30.767076+0000","last_active":"2024-03-12T03:34:33.140999+0000","last_peered":"2024-03-12T03:34:33.140999+0000","last_clean":"2024-03-12T03:34:33.140999+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:34:33.140999+0000","last_undegraded":"2024-03-12T03:34:33.140999+0000","last_fullsized":"2024-03-12T03:34:33.140999+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:30.766975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.766975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:45.275798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.693659+0000","last_change":"2024-03-12T03:33:27.383686+0000","last_active":"2024-03-12T03:34:33.693659+0000","last_peered":"2024-03-12T03:34:33.693659+0000","last_clean":"2024-03-12T03:34:33.693659+0000","last_became_active":"2024-03-12T03:33:27.383249+0000","last_became_peered":"2024-03-12T03:33:27.383249+0000","last_unstale":"2024-03-12T03:34:33.693659+0000","last_undegraded":"2024-03-12T03:34:33.693659+0000","last_fullsized":"2024-03-12T03:34:33.693659+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:40.138363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.140884+0000","last_change":"2024-03-12T03:33:21.098280+0000","last_active":"2024-03-12T03:34:33.140884+0000","last_peered":"2024-03-12T03:34:33.140884+0000","last_clean":"2024-03-12T03:34:33.140884+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:34:33.140884+0000","last_undegraded":"2024-03-12T03:34:33.140884+0000","last_fullsized":"2024-03-12T03:34:33.140884+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.640150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":92,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639355+0000","last_change":"2024-03-12T03:33:23.156886+0000","last_active":"2024-03-12T03:34:44.639355+0000","last_peered":"2024-03-12T03:34:44.639355+0000","last_clean":"2024-03-12T03:34:44.639355+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:34:44.639355+0000","last_undegraded":"2024-03-12T03:34:44.639355+0000","last_fullsized":"2024-03-12T03:34:44.639355+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141938+0000","last_change":"2024-03-12T03:34:25.168128+0000","last_active":"2024-03-12T03:34:33.141938+0000","last_peered":"2024-03-12T03:34:33.141938+0000","last_clean":"2024-03-12T03:34:33.141938+0000","last_became_active":"2024-03-12T03:33:25.177311+0000","last_became_peered":"2024-03-12T03:33:25.177311+0000","last_unstale":"2024-03-12T03:34:33.141938+0000","last_undegraded":"2024-03-12T03:34:33.141938+0000","last_fullsized":"2024-03-12T03:34:33.141938+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:25.168002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.168002+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.036057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639266+0000","last_change":"2024-03-12T03:33:27.188212+0000","last_active":"2024-03-12T03:34:44.639266+0000","last_peered":"2024-03-12T03:34:44.639266+0000","last_clean":"2024-03-12T03:34:44.639266+0000","last_became_active":"2024-03-12T03:33:27.187864+0000","last_became_peered":"2024-03-12T03:33:27.187864+0000","last_unstale":"2024-03-12T03:34:44.639266+0000","last_undegraded":"2024-03-12T03:34:44.639266+0000","last_fullsized":"2024-03-12T03:34:44.639266+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142153+0000","last_change":"2024-03-12T03:33:21.104718+0000","last_active":"2024-03-12T03:34:33.142153+0000","last_peered":"2024-03-12T03:34:33.142153+0000","last_clean":"2024-03-12T03:34:33.142153+0000","last_became_active":"2024-03-12T03:33:21.104430+0000","last_became_peered":"2024-03-12T03:33:21.104430+0000","last_unstale":"2024-03-12T03:34:33.142153+0000","last_undegraded":"2024-03-12T03:34:33.142153+0000","last_fullsized":"2024-03-12T03:34:33.142153+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:33.711229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":109,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141829+0000","last_change":"2024-03-12T03:33:23.551957+0000","last_active":"2024-03-12T03:34:33.141829+0000","last_peered":"2024-03-12T03:34:33.141829+0000","last_clean":"2024-03-12T03:34:33.141829+0000","last_became_active":"2024-03-12T03:33:23.551623+0000","last_became_peered":"2024-03-12T03:33:23.551623+0000","last_unstale":"2024-03-12T03:34:33.141829+0000","last_undegraded":"2024-03-12T03:34:33.141829+0000","last_fullsized":"2024-03-12T03:34:33.141829+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:45.740434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639166+0000","last_change":"2024-03-12T03:33:25.182046+0000","last_active":"2024-03-12T03:34:44.639166+0000","last_peered":"2024-03-12T03:34:44.639166+0000","last_clean":"2024-03-12T03:34:44.639166+0000","last_became_active":"2024-03-12T03:33:25.181020+0000","last_became_peered":"2024-03-12T03:33:25.181020+0000","last_unstale":"2024-03-12T03:34:44.639166+0000","last_undegraded":"2024-03-12T03:34:44.639166+0000","last_fullsized":"2024-03-12T03:34:44.639166+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.259564+0000","last_change":"2024-03-12T03:33:27.190398+0000","last_active":"2024-03-12T03:34:33.259564+0000","last_peered":"2024-03-12T03:34:33.259564+0000","last_clean":"2024-03-12T03:34:33.259564+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:34:33.259564+0000","last_undegraded":"2024-03-12T03:34:33.259564+0000","last_fullsized":"2024-03-12T03:34:33.259564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:53.538609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":80,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.640046+0000","last_change":"2024-03-12T03:33:25.176357+0000","last_active":"2024-03-12T03:34:44.640046+0000","last_peered":"2024-03-12T03:34:44.640046+0000","last_clean":"2024-03-12T03:34:44.640046+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:34:44.640046+0000","last_undegraded":"2024-03-12T03:34:44.640046+0000","last_fullsized":"2024-03-12T03:34:44.640046+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141783+0000","last_change":"2024-03-12T03:33:21.104557+0000","last_active":"2024-03-12T03:34:33.141783+0000","last_peered":"2024-03-12T03:34:33.141783+0000","last_clean":"2024-03-12T03:34:33.141783+0000","last_became_active":"2024-03-12T03:33:21.104293+0000","last_became_peered":"2024-03-12T03:33:21.104293+0000","last_unstale":"2024-03-12T03:34:33.141783+0000","last_undegraded":"2024-03-12T03:34:33.141783+0000","last_fullsized":"2024-03-12T03:34:33.141783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:42.027134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561565+0000","last_change":"2024-03-12T03:33:50.166158+0000","last_active":"2024-03-12T03:34:33.561565+0000","last_peered":"2024-03-12T03:34:33.561565+0000","last_clean":"2024-03-12T03:34:33.561565+0000","last_became_active":"2024-03-12T03:33:50.153240+0000","last_became_peered":"2024-03-12T03:33:50.153240+0000","last_unstale":"2024-03-12T03:34:33.561565+0000","last_undegraded":"2024-03-12T03:34:33.561565+0000","last_fullsized":"2024-03-12T03:34:33.561565+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:46.362489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.141294+0000","last_change":"2024-03-12T03:33:27.381253+0000","last_active":"2024-03-12T03:34:33.141294+0000","last_peered":"2024-03-12T03:34:33.141294+0000","last_clean":"2024-03-12T03:34:33.141294+0000","last_became_active":"2024-03-12T03:33:27.380896+0000","last_became_peered":"2024-03-12T03:33:27.380896+0000","last_unstale":"2024-03-12T03:34:33.141294+0000","last_undegraded":"2024-03-12T03:34:33.141294+0000","last_fullsized":"2024-03-12T03:34:33.141294+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:46.257478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:36.138928+0000","last_change":"2024-03-12T03:34:36.138975+0000","last_active":"2024-03-12T03:34:36.138928+0000","last_peered":"2024-03-12T03:34:36.138928+0000","last_clean":"2024-03-12T03:34:36.138928+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:34:36.138928+0000","last_undegraded":"2024-03-12T03:34:36.138928+0000","last_fullsized":"2024-03-12T03:34:36.138928+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.138847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.138847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.950469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639172+0000","last_change":"2024-03-12T03:33:21.573917+0000","last_active":"2024-03-12T03:34:44.639172+0000","last_peered":"2024-03-12T03:34:44.639172+0000","last_clean":"2024-03-12T03:34:44.639172+0000","last_became_active":"2024-03-12T03:33:21.573257+0000","last_became_peered":"2024-03-12T03:33:21.573257+0000","last_unstale":"2024-03-12T03:34:44.639172+0000","last_undegraded":"2024-03-12T03:34:44.639172+0000","last_fullsized":"2024-03-12T03:34:44.639172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.737276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639991+0000","last_change":"2024-03-12T03:33:23.168956+0000","last_active":"2024-03-12T03:34:44.639991+0000","last_peered":"2024-03-12T03:34:44.639991+0000","last_clean":"2024-03-12T03:34:44.639991+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:34:44.639991+0000","last_undegraded":"2024-03-12T03:34:44.639991+0000","last_fullsized":"2024-03-12T03:34:44.639991+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561942+0000","last_change":"2024-03-12T03:34:31.142859+0000","last_active":"2024-03-12T03:34:33.561942+0000","last_peered":"2024-03-12T03:34:33.561942+0000","last_clean":"2024-03-12T03:34:33.561942+0000","last_became_active":"2024-03-12T03:33:27.187815+0000","last_became_peered":"2024-03-12T03:33:27.187815+0000","last_unstale":"2024-03-12T03:34:33.561942+0000","last_undegraded":"2024-03-12T03:34:33.561942+0000","last_fullsized":"2024-03-12T03:34:33.561942+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:31.142765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:31.142765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:54.545296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.257712+0000","last_change":"2024-03-12T03:34:30.280067+0000","last_active":"2024-03-12T03:34:33.257712+0000","last_peered":"2024-03-12T03:34:33.257712+0000","last_clean":"2024-03-12T03:34:33.257712+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:34:33.257712+0000","last_undegraded":"2024-03-12T03:34:33.257712+0000","last_fullsized":"2024-03-12T03:34:33.257712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:30.279966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.279966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.784484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:45.342696+0000","last_change":"2024-03-12T03:34:23.646936+0000","last_active":"2024-03-12T03:34:45.342696+0000","last_peered":"2024-03-12T03:34:45.342696+0000","last_clean":"2024-03-12T03:34:45.342696+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:34:45.342696+0000","last_undegraded":"2024-03-12T03:34:45.342696+0000","last_fullsized":"2024-03-12T03:34:45.342696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.646819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:33.240250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639968+0000","last_change":"2024-03-12T03:33:23.164300+0000","last_active":"2024-03-12T03:34:44.639968+0000","last_peered":"2024-03-12T03:34:44.639968+0000","last_clean":"2024-03-12T03:34:44.639968+0000","last_became_active":"2024-03-12T03:33:23.163871+0000","last_became_peered":"2024-03-12T03:33:23.163871+0000","last_unstale":"2024-03-12T03:34:44.639968+0000","last_undegraded":"2024-03-12T03:34:44.639968+0000","last_fullsized":"2024-03-12T03:34:44.639968+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.640103+0000","last_change":"2024-03-12T03:33:27.184691+0000","last_active":"2024-03-12T03:34:44.640103+0000","last_peered":"2024-03-12T03:34:44.640103+0000","last_clean":"2024-03-12T03:34:44.640103+0000","last_became_active":"2024-03-12T03:33:27.184405+0000","last_became_peered":"2024-03-12T03:33:27.184405+0000","last_unstale":"2024-03-12T03:34:44.640103+0000","last_undegraded":"2024-03-12T03:34:44.640103+0000","last_fullsized":"2024-03-12T03:34:44.640103+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.561857+0000","last_change":"2024-03-12T03:33:25.178413+0000","last_active":"2024-03-12T03:34:33.561857+0000","last_peered":"2024-03-12T03:34:33.561857+0000","last_clean":"2024-03-12T03:34:33.561857+0000","last_became_active":"2024-03-12T03:33:25.177613+0000","last_became_peered":"2024-03-12T03:33:25.177613+0000","last_unstale":"2024-03-12T03:34:33.561857+0000","last_undegraded":"2024-03-12T03:34:33.561857+0000","last_fullsized":"2024-03-12T03:34:33.561857+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-12T03:34:33.142039+0000","last_change":"2024-03-12T03:33:21.097634+0000","last_active":"2024-03-12T03:34:33.142039+0000","last_peered":"2024-03-12T03:34:33.142039+0000","last_clean":"2024-03-12T03:34:33.142039+0000","last_became_active":"2024-03-12T03:33:21.097304+0000","last_became_peered":"2024-03-12T03:33:21.097304+0000","last_unstale":"2024-03-12T03:34:33.142039+0000","last_undegraded":"2024-03-12T03:34:33.142039+0000","last_fullsized":"2024-03-12T03:34:33.142039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:43.765898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-12T03:34:44.639624+0000","last_change":"2024-03-12T03:33:23.163246+0000","last_active":"2024-03-12T03:34:44.639624+0000","last_peered":"2024-03-12T03:34:44.639624+0000","last_clean":"2024-03-12T03:34:44.639624+0000","last_became_active":"2024-03-12T03:33:23.162657+0000","last_became_peered":"2024-03-12T03:33:23.162657+0000","last_unstale":"2024-03-12T03:34:44.639624+0000","last_undegraded":"2024-03-12T03:34:44.639624+0000","last_fullsized":"2024-03-12T03:34:44.639624+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":602,"ondisk_log_size":602,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070485,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28400,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716752,"statfs":{"total":95995035648,"available":95965954048,"internally_reserved":0,"allocated":1417216,"data_stored":948578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332123,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28416,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716736,"statfs":{"total":95995035648,"available":95965937664,"internally_reserved":0,"allocated":1429504,"data_stored":966092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":357877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32560,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712592,"statfs":{"total":95995035648,"available":95961694208,"internally_reserved":0,"allocated":1417216,"data_stored":947832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":802816,"data_stored":357480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117032,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":839680,"data_stored":357574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953510,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":864256,"data_stored":377227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659167,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":815104,"data_stored":356918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:34:47.746 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-12T03:34:47.746 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-12T03:34:47.746 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-12T03:34:47.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-12T03:34:47.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:47 smithi173 ceph-mon[33544]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:34:47.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:47 smithi173 ceph-mon[33544]: 4.a scrub starts 2024-03-12T03:34:47.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:47 smithi173 ceph-mon[33544]: 4.a scrub ok 2024-03-12T03:34:47.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:47 smithi173 ceph-mon[33544]: 6.15 scrub starts 2024-03-12T03:34:47.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:47 smithi173 ceph-mon[33544]: 6.15 scrub ok 2024-03-12T03:34:47.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:47 smithi173 ceph-mon[33544]: 4.1 scrub starts 2024-03-12T03:34:47.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:47 smithi173 ceph-mon[33544]: 4.1 scrub ok 2024-03-12T03:34:47.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:47 smithi173 podman[63942]: 2024-03-12 03:34:47.89679554 +0000 UTC m=+0.148378598 container create 93c2f834c6423dd67068acdfbce512d1be78c1bd64dab0139fdebf7e3f4bd6aa (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, io.openshift.expose-services=, vendor=Red Hat, Inc., description=Ceph Grafana Container, version=8.5, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, vcs-type=git, release=239.1651231664, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-12T03:34:48.024 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:48.292 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 podman[63942]: 2024-03-12 03:34:48.133571678 +0000 UTC m=+0.385154694 container init 93c2f834c6423dd67068acdfbce512d1be78c1bd64dab0139fdebf7e3f4bd6aa (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, maintainer=Paul Cuzner , io.buildah.version=1.32.0, vendor=Red Hat, Inc., release=239.1651231664, architecture=x86_64, version=8.5, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8) 2024-03-12T03:34:48.292 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 podman[63942]: 2024-03-12 03:34:48.136356042 +0000 UTC m=+0.387939060 container start 93c2f834c6423dd67068acdfbce512d1be78c1bd64dab0139fdebf7e3f4bd6aa (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a, io.buildah.version=1.32.0, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, version=8.5, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, maintainer=Paul Cuzner ) 2024-03-12T03:34:48.292 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 bash[63942]: 93c2f834c6423dd67068acdfbce512d1be78c1bd64dab0139fdebf7e3f4bd6aa 2024-03-12T03:34:48.292 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 systemd[1]: Started Ceph grafana.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:34:48.543 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.290669808Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-12T03:34:48.543 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.290891453Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-12T03:34:48.543 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.290956325Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-12T03:34:48.543 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.291007825Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-12T03:34:48.543 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.291020889Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-12T03:34:48.544 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.291030906Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-12T03:34:48.544 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.291040139Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-12T03:34:48.544 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.29104954Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-12T03:34:48.544 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.291058864Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-12T03:34:48.544 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.291068845Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-12T03:34:48.544 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.291077543Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-12T03:34:48.544 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.291085962Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-12T03:34:48.544 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.291098648Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-12T03:34:48.545 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=settings t=2024-03-12T03:34:48.29110933Z level=info msg="App mode production" 2024-03-12T03:34:48.545 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=sqlstore t=2024-03-12T03:34:48.291174901Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-12T03:34:48.545 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=sqlstore t=2024-03-12T03:34:48.291202106Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-12T03:34:48.545 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.292530838Z level=info msg="Starting DB migrations" 2024-03-12T03:34:48.545 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.292635169Z level=info msg="Executing migration" id="create migration_log table" 2024-03-12T03:34:48.545 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.293345501Z level=info msg="Executing migration" id="create user table" 2024-03-12T03:34:48.545 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.293888453Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-12T03:34:48.545 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.294671368Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-12T03:34:48.546 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.295594229Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-12T03:34:48.546 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.296355794Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-12T03:34:48.546 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.297057084Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-12T03:34:48.546 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.298029979Z level=info msg="Executing migration" id="create user table v2" 2024-03-12T03:34:48.546 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.298580583Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-12T03:34:48.546 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.299344628Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-12T03:34:48.546 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.299809467Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-12T03:34:48.546 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.30021187Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-12T03:34:48.547 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.301090344Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-12T03:34:48.547 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.301844129Z level=info msg="Executing migration" id="Update user table charset" 2024-03-12T03:34:48.547 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.302695977Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-12T03:34:48.547 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.303651831Z level=info msg="Executing migration" id="Add missing user data" 2024-03-12T03:34:48.547 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.304422202Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-12T03:34:48.547 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.30534508Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-12T03:34:48.547 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.30590558Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-12T03:34:48.547 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.306321375Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-12T03:34:48.547 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.30742095Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-12T03:34:48.550 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.308215743Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-12T03:34:48.550 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.309213839Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-12T03:34:48.550 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.309776799Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-12T03:34:48.550 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.310622184Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-12T03:34:48.550 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.311056391Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-12T03:34:48.551 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.312079525Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-12T03:34:48.551 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.312736851Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-12T03:34:48.551 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.313586002Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-12T03:34:48.551 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.314491854Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-12T03:34:48.551 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.317167054Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-12T03:34:48.551 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.319911328Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-12T03:34:48.551 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.322629945Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-12T03:34:48.552 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.325450691Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-12T03:34:48.552 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.32806865Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-12T03:34:48.552 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.330614076Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-12T03:34:48.552 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.33350207Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-12T03:34:48.552 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.336086831Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-12T03:34:48.552 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.336424178Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-12T03:34:48.552 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.339257492Z level=info msg="Executing migration" id="create star table" 2024-03-12T03:34:48.552 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.339669952Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-12T03:34:48.552 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.342016951Z level=info msg="Executing migration" id="create org table v1" 2024-03-12T03:34:48.553 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.344710897Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-12T03:34:48.553 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.347463361Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-12T03:34:48.553 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.350050907Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-12T03:34:48.553 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.352785649Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-12T03:34:48.553 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.35324329Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-12T03:34:48.553 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.355956783Z level=info msg="Executing migration" id="Update org table charset" 2024-03-12T03:34:48.553 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.35848652Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-12T03:34:48.553 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.361143165Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-12T03:34:48.554 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.361613454Z level=info msg="Executing migration" id="create dashboard table" 2024-03-12T03:34:48.554 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.364249622Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-12T03:34:48.554 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.366952813Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-12T03:34:48.554 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.36934641Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-12T03:34:48.554 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.369752947Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-12T03:34:48.554 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.372518936Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-12T03:34:48.554 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.374918424Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-12T03:34:48.554 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.377609646Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-12T03:34:48.555 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.378040573Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-12T03:34:48.555 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.380663685Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-12T03:34:48.555 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.383301404Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-12T03:34:48.555 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.385880001Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-12T03:34:48.555 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.38631382Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-12T03:34:48.555 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.389075416Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-12T03:34:48.555 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.391722522Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-12T03:34:48.555 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.392535649Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-12T03:34:48.556 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.393037163Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-12T03:34:48.556 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.393943891Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-12T03:34:48.556 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.394696664Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-12T03:34:48.556 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.395476999Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-12T03:34:48.556 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.395907379Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-12T03:34:48.556 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.396750122Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-12T03:34:48.556 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.397025346Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-12T03:34:48.556 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.397542311Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-12T03:34:48.556 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.398444874Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-12T03:34:48.557 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.399089206Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-12T03:34:48.557 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.399587581Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-12T03:34:48.557 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.400492011Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-12T03:34:48.557 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.401081436Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-12T03:34:48.557 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.401482894Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-12T03:34:48.557 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.402383181Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-12T03:34:48.557 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.402822852Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-12T03:34:48.557 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.403212337Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-12T03:34:48.557 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.404110786Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-12T03:34:48.558 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.405126504Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-12T03:34:48.558 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.405826959Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-12T03:34:48.558 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.406264653Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-12T03:34:48.558 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.407120992Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-12T03:34:48.558 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.408036906Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-12T03:34:48.558 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.410883683Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-12T03:34:48.558 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.41141952Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-12T03:34:48.559 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.414365392Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-12T03:34:48.559 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.41472762Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-12T03:34:48.559 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.417208891Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-12T03:34:48.559 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.42012407Z level=info msg="Executing migration" id="create data_source table" 2024-03-12T03:34:48.559 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.420559459Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-12T03:34:48.559 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.423377953Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-12T03:34:48.559 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.423802389Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-12T03:34:48.559 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.426717009Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-12T03:34:48.559 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.427146424Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-12T03:34:48.560 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.430072094Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-12T03:34:48.560 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.432994533Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-12T03:34:48.560 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.435629405Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-12T03:34:48.560 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.436109823Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-12T03:34:48.560 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.438982873Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-12T03:34:48.560 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.44209709Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-12T03:34:48.560 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.444827644Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-12T03:34:48.560 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.445155403Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-12T03:34:48.560 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.447987596Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-12T03:34:48.561 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.450655477Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-12T03:34:48.561 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.453285169Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-12T03:34:48.561 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.453797482Z level=info msg="Executing migration" id="Add uid column" 2024-03-12T03:34:48.561 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.456551116Z level=info msg="Executing migration" id="Update uid value" 2024-03-12T03:34:48.561 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.459281112Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-12T03:34:48.561 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.462050561Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-12T03:34:48.561 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.462491023Z level=info msg="Executing migration" id="create api_key table" 2024-03-12T03:34:48.561 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.465123755Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-12T03:34:48.561 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.467861651Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-12T03:34:48.562 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.470682213Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-12T03:34:48.562 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.471158163Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-12T03:34:48.562 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.473850863Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-12T03:34:48.562 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.476497417Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-12T03:34:48.562 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.479348867Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-12T03:34:48.562 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.482036957Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-12T03:34:48.562 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.48471727Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-12T03:34:48.562 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.485444285Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-12T03:34:48.563 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.486286127Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-12T03:34:48.563 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.487171336Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-12T03:34:48.563 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.48786729Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-12T03:34:48.563 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.488226934Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-12T03:34:48.563 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.489079679Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-12T03:34:48.563 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.489930486Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-12T03:34:48.563 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.49072785Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-12T03:34:48.563 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.491627806Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-12T03:34:48.563 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.492193897Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-12T03:34:48.564 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.49317995Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-12T03:34:48.564 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.493841385Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-12T03:34:48.564 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.494883324Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-12T03:34:48.564 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.495513947Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-12T03:34:48.564 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.496569266Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-12T03:34:48.564 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.497223766Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-12T03:34:48.564 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.498092745Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-12T03:34:48.564 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.49904048Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-12T03:34:48.564 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.499804934Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-12T03:34:48.565 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.500659212Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-12T03:34:48.565 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.501564581Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-12T03:34:48.565 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.502341708Z level=info msg="Executing migration" id="create quota table v1" 2024-03-12T03:34:48.565 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.50334972Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-12T03:34:48.565 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.504055112Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-12T03:34:48.565 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.504843777Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-12T03:34:48.565 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.505246518Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-12T03:34:48.565 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.506104108Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-12T03:34:48.565 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.507015504Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-12T03:34:48.566 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.507693417Z level=info msg="Executing migration" id="create session table" 2024-03-12T03:34:48.566 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.508117741Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-12T03:34:48.566 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.5089993Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-12T03:34:48.566 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.509288406Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-12T03:34:48.566 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.509671882Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-12T03:34:48.566 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.510608207Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-12T03:34:48.566 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.510897235Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-12T03:34:48.566 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.511185654Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-12T03:34:48.567 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.511467314Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-12T03:34:48.567 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.511748684Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-12T03:34:48.567 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.51230219Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-12T03:34:48.567 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.512811305Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-12T03:34:48.567 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.513477136Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-12T03:34:48.567 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.514287431Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-12T03:34:48.567 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.514946737Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-12T03:34:48.567 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.515812679Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-12T03:34:48.567 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.516734397Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-12T03:34:48.568 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.517325703Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-12T03:34:48.568 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.518118219Z level=info msg="Executing migration" id="create alert table v1" 2024-03-12T03:34:48.568 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.520948242Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-12T03:34:48.568 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.523763179Z level=info msg="Executing migration" id="add index alert state" 2024-03-12T03:34:48.568 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.526652658Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-12T03:34:48.568 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.527078047Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-12T03:34:48.568 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.527452646Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-12T03:34:48.568 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.527897739Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-12T03:34:48.569 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.528320507Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-12T03:34:48.569 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.529716876Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-12T03:34:48.569 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.530107603Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-12T03:34:48.569 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.53054442Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-12T03:34:48.569 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.530823842Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-12T03:34:48.569 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.531182313Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-12T03:34:48.569 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.531584592Z level=info msg="Executing migration" id="Add column is_default" 2024-03-12T03:34:48.569 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.532199164Z level=info msg="Executing migration" id="Add column frequency" 2024-03-12T03:34:48.569 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.53283446Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-12T03:34:48.569 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.533494096Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-12T03:34:48.570 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.534124544Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-12T03:34:48.570 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.534571626Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-12T03:34:48.570 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.534821569Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-12T03:34:48.570 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.535126352Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-12T03:34:48.570 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.535523002Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-12T03:34:48.570 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.535938334Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-12T03:34:48.570 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.536379199Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-12T03:34:48.570 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.53674853Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-12T03:34:48.571 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.537179346Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-12T03:34:48.571 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.537821702Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-12T03:34:48.571 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.538491294Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-12T03:34:48.571 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.538806607Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-12T03:34:48.571 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.539239428Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-12T03:34:48.571 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.539664869Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-12T03:34:48.571 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.540341901Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-12T03:34:48.571 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.540633583Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-12T03:34:48.571 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.541028006Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-12T03:34:48.572 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.541505445Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-12T03:34:48.572 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.541808254Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-12T03:34:48.572 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.54229837Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-12T03:34:48.572 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.544830228Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-12T03:34:48.572 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.54529099Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-12T03:34:48.572 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.546082613Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-12T03:34:48.754 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:34:48.754 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: 3.7 scrub starts 2024-03-12T03:34:48.754 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: 3.7 scrub ok 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: 6.4 scrub starts 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: 6.4 scrub ok 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: 5.4 scrub starts 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: 5.4 scrub ok 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: 5.1e scrub starts 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: 5.1e scrub ok 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: osdmap e94: 8 total, 8 up, 8 in 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:34:48.755 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: 3.7 scrub starts 2024-03-12T03:34:48.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: 3.7 scrub ok 2024-03-12T03:34:48.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: 6.4 scrub starts 2024-03-12T03:34:48.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: 6.4 scrub ok 2024-03-12T03:34:48.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: 5.4 scrub starts 2024-03-12T03:34:48.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: 5.4 scrub ok 2024-03-12T03:34:48.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: 5.1e scrub starts 2024-03-12T03:34:48.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: 5.1e scrub ok 2024-03-12T03:34:48.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: osdmap e94: 8 total, 8 up, 8 in 2024-03-12T03:34:48.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:48.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:48.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:48.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:48 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:48.793 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:34:48.793 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: 3.7 scrub starts 2024-03-12T03:34:48.793 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: 3.7 scrub ok 2024-03-12T03:34:48.793 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: 6.4 scrub starts 2024-03-12T03:34:48.793 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: 6.4 scrub ok 2024-03-12T03:34:48.794 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: 5.4 scrub starts 2024-03-12T03:34:48.794 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: 5.4 scrub ok 2024-03-12T03:34:48.794 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: 5.1e scrub starts 2024-03-12T03:34:48.794 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: 5.1e scrub ok 2024-03-12T03:34:48.794 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: osdmap e94: 8 total, 8 up, 8 in 2024-03-12T03:34:48.794 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:48.794 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:48.794 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:48.794 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:48.795 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.546933235Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-12T03:34:48.795 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.54771777Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-12T03:34:48.795 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.548467624Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-12T03:34:48.795 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.549171264Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-12T03:34:48.795 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.549611821Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-12T03:34:48.795 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.550419441Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-12T03:34:48.795 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.551246396Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-12T03:34:48.795 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.552018844Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-12T03:34:48.795 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.552805318Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.554384016Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.555205722Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.556040001Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.556974228Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.55770653Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.558517032Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.559265586Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.5600504Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.56087881Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.561722714Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.562557608Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.563347009Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.564191754Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.565087511Z level=info msg="Executing migration" id="Move region to single row" 2024-03-12T03:34:48.796 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.565864268Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.56672106Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.567524632Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.568324279Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.569300619Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.570071553Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.572897028Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.575447278Z level=info msg="Executing migration" id="create test_data table" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.578322174Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.580898147Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.583547063Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.586201962Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.588893789Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.591487853Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.594273452Z level=info msg="Executing migration" id="create team table" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.597512658Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-12T03:34:48.797 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.600188766Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-12T03:34:48.798 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.602881877Z level=info msg="Executing migration" id="create team member table" 2024-03-12T03:34:48.798 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.605735221Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-12T03:34:48.798 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.608287781Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-12T03:34:48.798 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.61178103Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-12T03:34:48.798 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.614344894Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-12T03:34:48.798 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.61722106Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-12T03:34:48.798 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.619951063Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-12T03:34:48.798 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.622670076Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-12T03:34:48.798 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.625085746Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-12T03:34:48.798 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.627676244Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-12T03:34:48.798 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.630482156Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-12T03:34:48.798 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.633111268Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.635696593Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.638363833Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.640902284Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.643336188Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.643749951Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.64633317Z level=info msg="Executing migration" id="create tag table" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.646741128Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.64898959Z level=info msg="Executing migration" id="create login attempt table" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.651651613Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.654117944Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.656952708Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.6600335Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.660789069Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.661635762Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.662461226Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-12T03:34:48.799 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.663238823Z level=info msg="Executing migration" id="create user auth table" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.664077471Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.665015729Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.665645642Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.666439976Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.667255935Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.668047647Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.668843089Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.669684905Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.670473907Z level=info msg="Executing migration" id="create server_lock table" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.671398802Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.672296677Z level=info msg="Executing migration" id="create user auth token table" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.67308493Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.673725663Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.674693306Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.675372413Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-12T03:34:48.800 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.676218536Z level=info msg="Executing migration" id="create cache_data table" 2024-03-12T03:34:48.801 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.677182881Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-12T03:34:48.801 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.677934512Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-12T03:34:48.801 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.678904975Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-12T03:34:48.801 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.679543146Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-12T03:34:48.801 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.680352472Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-12T03:34:48.801 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.680647951Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-12T03:34:48.801 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.681260114Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-12T03:34:48.801 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.682152841Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-12T03:34:48.801 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.682975899Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-12T03:34:48.801 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.683792611Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-12T03:34:48.802 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.684690257Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-12T03:34:48.802 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.685395175Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-12T03:34:48.802 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.686329043Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-12T03:34:48.802 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.68705623Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-12T03:34:48.802 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.687944448Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-12T03:34:48.802 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.690710022Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-12T03:34:48.802 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.693374982Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-12T03:34:48.802 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.69611287Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-12T03:34:48.802 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.699119955Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.701670345Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.70433354Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.707090099Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.709820058Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.712669096Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.715466267Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.717999182Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.720809631Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.723599266Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.727160473Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.731078212Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.733900867Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.736398376Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.739098699Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-12T03:34:48.803 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.741683108Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.744146093Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.746910108Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.749373784Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.749693092Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.750546422Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.751401366Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.752210295Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.752681559Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.753122425Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.753989057Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.754842112Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.755326855Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.756147227Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.756989425Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.757334917Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-12T03:34:48.804 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.757810107Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.760224881Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.760749987Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.761018515Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.761896055Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.764697191Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.76556889Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.7664695Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.767316227Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.767622155Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.768022747Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.768847906Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.769163894Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.770034942Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.77291049Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.775458029Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-12T03:34:48.805 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.778190221Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-12T03:34:48.806 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.780820836Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-12T03:34:48.806 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.783597323Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-12T03:34:48.806 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.786308886Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-12T03:34:48.806 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.789133097Z level=info msg="Executing migration" id="create alert_image table" 2024-03-12T03:34:48.806 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.791794201Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-12T03:34:48.806 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.794489888Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-12T03:34:49.051 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.797150389Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-12T03:34:49.051 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.797588563Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-12T03:34:49.051 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.800274702Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-12T03:34:49.051 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.800380388Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-12T03:34:49.051 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.800674022Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-12T03:34:49.051 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.803005265Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-12T03:34:49.052 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.803466791Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-12T03:34:49.052 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.803586931Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-12T03:34:49.052 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.803705526Z level=info msg="alerts found to migrate" alerts=0 2024-03-12T03:34:49.052 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.804125528Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-12T03:34:49.052 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.805150878Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-12T03:34:49.052 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.805783736Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-12T03:34:49.052 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.806637355Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-12T03:34:49.052 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.807606027Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-12T03:34:49.052 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.808432573Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-12T03:34:49.052 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.809112934Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-12T03:34:49.052 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.810084458Z level=info msg="Executing migration" id="create data_keys table" 2024-03-12T03:34:49.052 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.810839487Z level=info msg="Executing migration" id="create secrets table" 2024-03-12T03:34:49.053 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.811839371Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-12T03:34:49.053 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.816138076Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-12T03:34:49.053 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.81704752Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-12T03:34:49.053 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.817906447Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-12T03:34:49.053 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.82274851Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-12T03:34:49.053 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.827171248Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.82809406Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.82886521Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.829717544Z level=info msg="Executing migration" id="create permission table" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.830557112Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.831416791Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.832288015Z level=info msg="Executing migration" id="create role table" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.833154644Z level=info msg="Executing migration" id="add column display_name" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.834221596Z level=info msg="Executing migration" id="add column group_name" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.835355003Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.836152486Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.837153835Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.837868393Z level=info msg="Executing migration" id="create team role table" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.838855959Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-12T03:34:49.054 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.83959271Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.840569583Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.841294044Z level=info msg="Executing migration" id="create user role table" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.842142642Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.843158979Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.843821696Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.8465687Z level=info msg="Executing migration" id="create builtin role table" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.849467893Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.852252223Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.854835749Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.857540592Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.858062291Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.861110967Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.861668795Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.862123848Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.864774908Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-12T03:34:49.055 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.865265389Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.866250269Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.866682379Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.867146177Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.867456181Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.867860115Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.868355603Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.868733196Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.869019099Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.869401363Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.869824485Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.8703348Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.871298571Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.871601094Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.872026598Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.872476215Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-12T03:34:49.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.872917712Z level=info msg="Executing migration" id="add correlation config column" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.873896207Z level=info msg="Executing migration" id="create entity_events table" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.874365748Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.87480415Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.874893988Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.875190783Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.875275391Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.875549632Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.875939167Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.876385513Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.876831668Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.877312914Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.877734122Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.878181571Z level=info msg="Executing migration" id="Drop public config table" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.878652297Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.87904489Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-12T03:34:49.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.879536339Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.879978383Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.880475859Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.883490226Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.884545856Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.885573066Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.885892171Z level=info msg="Executing migration" id="add share column" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.886902327Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.887393219Z level=info msg="Executing migration" id="create file table" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.887843386Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.890706647Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.891407041Z level=info msg="Executing migration" id="create file_meta table" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.894306643Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.897207224Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.900027389Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-12T03:34:49.058 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.902761895Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.905554601Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.90830996Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.911043291Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.913793075Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.916610981Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.917194635Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.919880956Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.922439502Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.925293216Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.927815708Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.930376477Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.939766181Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.942565038Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.94501445Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-12T03:34:49.059 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.948348005Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.950911626Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.953388512Z level=info msg="Executing migration" id="create folder table" 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.956184543Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.958646048Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.96124674Z level=info msg="Executing migration" id="Update folder title length" 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.963956978Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=migrator t=2024-03-12T03:34:48.966740256Z level=info msg="migrations completed" performed=485 skipped=0 duration=674.136557ms 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:48 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=sqlstore t=2024-03-12T03:34:48.967164166Z level=info msg="Created default organization" 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=plugin.loader t=2024-03-12T03:34:49.021781971Z level=info msg="Plugin registered" pluginID=input 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=plugin.loader t=2024-03-12T03:34:49.049453274Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=plugin.loader t=2024-03-12T03:34:49.049494764Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=secrets t=2024-03-12T03:34:49.049541288Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-12T03:34:49.060 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=query_data t=2024-03-12T03:34:49.051675853Z level=info msg="Query Service initialization" 2024-03-12T03:34:49.307 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=live.push_http t=2024-03-12T03:34:49.05417431Z level=info msg="Live Push Gateway initialization" 2024-03-12T03:34:49.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:49 smithi169 ceph-mon[29344]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-03-12T03:34:49.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:49 smithi169 ceph-mon[29344]: 3.c deep-scrub starts 2024-03-12T03:34:49.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:49 smithi169 ceph-mon[29344]: 3.c deep-scrub ok 2024-03-12T03:34:49.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:49 smithi169 ceph-mon[29344]: Deploying daemon node-exporter.a on smithi169 2024-03-12T03:34:49.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:49 smithi169 ceph-mon[29344]: 3.1f scrub starts 2024-03-12T03:34:49.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:49 smithi169 ceph-mon[29344]: 3.1f scrub ok 2024-03-12T03:34:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:49 smithi169 ceph-mon[39949]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-03-12T03:34:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:49 smithi169 ceph-mon[39949]: 3.c deep-scrub starts 2024-03-12T03:34:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:49 smithi169 ceph-mon[39949]: 3.c deep-scrub ok 2024-03-12T03:34:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:49 smithi169 ceph-mon[39949]: Deploying daemon node-exporter.a on smithi169 2024-03-12T03:34:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:49 smithi169 ceph-mon[39949]: 3.1f scrub starts 2024-03-12T03:34:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:49 smithi169 ceph-mon[39949]: 3.1f scrub ok 2024-03-12T03:34:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:34:49.967 INFO:teuthology.orchestra.run.smithi169.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-12T03:34:49.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-mon[33544]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-03-12T03:34:49.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-mon[33544]: 3.c deep-scrub starts 2024-03-12T03:34:49.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-mon[33544]: 3.c deep-scrub ok 2024-03-12T03:34:49.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-mon[33544]: Deploying daemon node-exporter.a on smithi169 2024-03-12T03:34:49.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-mon[33544]: 3.1f scrub starts 2024-03-12T03:34:49.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-mon[33544]: 3.1f scrub ok 2024-03-12T03:34:49.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=infra.usagestats.collector t=2024-03-12T03:34:49.693478284Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-12T03:34:49.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=provisioning.datasources t=2024-03-12T03:34:49.694457389Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-12T03:34:49.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=provisioning.datasources t=2024-03-12T03:34:49.705798441Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-12T03:34:49.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=provisioning.alerting t=2024-03-12T03:34:49.716884536Z level=info msg="starting to provision alerting" 2024-03-12T03:34:49.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=provisioning.alerting t=2024-03-12T03:34:49.716904511Z level=info msg="finished to provision alerting" 2024-03-12T03:34:49.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=grafanaStorageLogger t=2024-03-12T03:34:49.717017808Z level=info msg="storage starting" 2024-03-12T03:34:49.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=ngalert.state.manager t=2024-03-12T03:34:49.717183855Z level=info msg="Warming state cache for startup" 2024-03-12T03:34:49.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=http.server t=2024-03-12T03:34:49.719492356Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-12T03:34:49.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=ngalert.state.manager t=2024-03-12T03:34:49.723760921Z level=info msg="State cache has been initialized" states=0 duration=6.576341ms 2024-03-12T03:34:49.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=ticker t=2024-03-12T03:34:49.723814784Z level=info msg=starting first_tick=2024-03-12T03:34:50Z 2024-03-12T03:34:49.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=ngalert.multiorg.alertmanager t=2024-03-12T03:34:49.723839392Z level=info msg="starting MultiOrg Alertmanager" 2024-03-12T03:34:49.987 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:34:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=sqlstore.transactions t=2024-03-12T03:34:49.746263421Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-12T03:34:50.737 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-12T03:34:50.738 INFO:tasks.cephadm:Setup complete, yielding 2024-03-12T03:34:50.738 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-12T03:34:50.748 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-12T03:34:50.749 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '1c349451176cc5b4ebfb24b22eaaa754e05cff6c'}} 2024-03-12T03:34:50.749 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-12T03:34:50.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-12T03:34:50.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[29344]: 6.6 scrub starts 2024-03-12T03:34:50.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[29344]: 6.6 scrub ok 2024-03-12T03:34:50.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[29344]: 3.1c scrub starts 2024-03-12T03:34:50.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[29344]: 3.1c scrub ok 2024-03-12T03:34:50.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:50.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2344213137' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:34:50.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[29344]: 3.14 scrub starts 2024-03-12T03:34:50.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[29344]: 3.14 scrub ok 2024-03-12T03:34:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[39949]: 6.6 scrub starts 2024-03-12T03:34:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[39949]: 6.6 scrub ok 2024-03-12T03:34:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[39949]: 3.1c scrub starts 2024-03-12T03:34:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[39949]: 3.1c scrub ok 2024-03-12T03:34:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2344213137' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:34:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[39949]: 3.14 scrub starts 2024-03-12T03:34:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:50 smithi169 ceph-mon[39949]: 3.14 scrub ok 2024-03-12T03:34:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:50 smithi173 ceph-mon[33544]: 6.6 scrub starts 2024-03-12T03:34:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:50 smithi173 ceph-mon[33544]: 6.6 scrub ok 2024-03-12T03:34:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:50 smithi173 ceph-mon[33544]: 3.1c scrub starts 2024-03-12T03:34:50.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:50 smithi173 ceph-mon[33544]: 3.1c scrub ok 2024-03-12T03:34:50.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:50 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:50.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:50 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2344213137' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:34:50.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:50 smithi173 ceph-mon[33544]: 3.14 scrub starts 2024-03-12T03:34:50.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:50 smithi173 ceph-mon[33544]: 3.14 scrub ok 2024-03-12T03:34:51.035 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:51.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[29344]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-03-12T03:34:51.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[29344]: 2.1 scrub starts 2024-03-12T03:34:51.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[29344]: 2.1 scrub ok 2024-03-12T03:34:51.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[29344]: 5.1b scrub starts 2024-03-12T03:34:51.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[29344]: 5.1b scrub ok 2024-03-12T03:34:51.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[29344]: 3.1b scrub starts 2024-03-12T03:34:51.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[29344]: 3.1b scrub ok 2024-03-12T03:34:51.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[39949]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-03-12T03:34:51.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[39949]: 2.1 scrub starts 2024-03-12T03:34:51.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[39949]: 2.1 scrub ok 2024-03-12T03:34:51.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[39949]: 5.1b scrub starts 2024-03-12T03:34:51.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[39949]: 5.1b scrub ok 2024-03-12T03:34:51.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[39949]: 3.1b scrub starts 2024-03-12T03:34:51.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-mon[39949]: 3.1b scrub ok 2024-03-12T03:34:51.925 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:34:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:34:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:51 smithi173 ceph-mon[33544]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-03-12T03:34:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:51 smithi173 ceph-mon[33544]: 2.1 scrub starts 2024-03-12T03:34:51.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:51 smithi173 ceph-mon[33544]: 2.1 scrub ok 2024-03-12T03:34:51.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:51 smithi173 ceph-mon[33544]: 5.1b scrub starts 2024-03-12T03:34:51.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:51 smithi173 ceph-mon[33544]: 5.1b scrub ok 2024-03-12T03:34:51.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:51 smithi173 ceph-mon[33544]: 3.1b scrub starts 2024-03-12T03:34:51.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:51 smithi173 ceph-mon[33544]: 3.1b scrub ok 2024-03-12T03:34:52.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:34:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:34:52.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:52 smithi173 ceph-mon[33544]: 3.1d deep-scrub starts 2024-03-12T03:34:52.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:52 smithi173 ceph-mon[33544]: 3.1d deep-scrub ok 2024-03-12T03:34:52.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:52 smithi173 ceph-mon[33544]: 3.0 scrub starts 2024-03-12T03:34:52.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:52 smithi173 ceph-mon[33544]: 3.0 scrub ok 2024-03-12T03:34:52.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:52 smithi173 ceph-mon[33544]: 6.0 scrub starts 2024-03-12T03:34:52.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:52 smithi173 ceph-mon[33544]: 6.0 scrub ok 2024-03-12T03:34:52.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:52 smithi173 ceph-mon[33544]: 5.8 scrub starts 2024-03-12T03:34:52.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:52 smithi173 ceph-mon[33544]: 5.8 scrub ok 2024-03-12T03:34:53.095 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[29344]: 3.1d deep-scrub starts 2024-03-12T03:34:53.095 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[29344]: 3.1d deep-scrub ok 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[29344]: 3.0 scrub starts 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[29344]: 3.0 scrub ok 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[29344]: 6.0 scrub starts 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[29344]: 6.0 scrub ok 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[29344]: 5.8 scrub starts 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[29344]: 5.8 scrub ok 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[39949]: 3.1d deep-scrub starts 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[39949]: 3.1d deep-scrub ok 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[39949]: 3.0 scrub starts 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[39949]: 3.0 scrub ok 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[39949]: 6.0 scrub starts 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[39949]: 6.0 scrub ok 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[39949]: 5.8 scrub starts 2024-03-12T03:34:53.096 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:52 smithi169 ceph-mon[39949]: 5.8 scrub ok 2024-03-12T03:34:53.379 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:53 smithi169 bash[75381]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-12T03:34:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:53 smithi169 ceph-mon[29344]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:53 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:53 smithi169 ceph-mon[29344]: 4.1f deep-scrub starts 2024-03-12T03:34:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:53 smithi169 ceph-mon[29344]: 4.1f deep-scrub ok 2024-03-12T03:34:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:53 smithi169 ceph-mon[29344]: 3.11 scrub starts 2024-03-12T03:34:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:53 smithi169 ceph-mon[29344]: 3.11 scrub ok 2024-03-12T03:34:53.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:53 smithi169 ceph-mon[39949]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:53.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:53 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:53 smithi169 ceph-mon[39949]: 4.1f deep-scrub starts 2024-03-12T03:34:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:53 smithi169 ceph-mon[39949]: 4.1f deep-scrub ok 2024-03-12T03:34:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:53 smithi169 ceph-mon[39949]: 3.11 scrub starts 2024-03-12T03:34:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:53 smithi169 ceph-mon[39949]: 3.11 scrub ok 2024-03-12T03:34:53.925 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:53 smithi169 bash[75381]: Writing manifest to image destination 2024-03-12T03:34:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:53 smithi173 ceph-mon[33544]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:34:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:53 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:34:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:53 smithi173 ceph-mon[33544]: 4.1f deep-scrub starts 2024-03-12T03:34:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:53 smithi173 ceph-mon[33544]: 4.1f deep-scrub ok 2024-03-12T03:34:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:53 smithi173 ceph-mon[33544]: 3.11 scrub starts 2024-03-12T03:34:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:53 smithi173 ceph-mon[33544]: 3.11 scrub ok 2024-03-12T03:34:54.276 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 podman[75381]: 2024-03-12 03:34:54.055960681 +0000 UTC m=+3.216382643 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-12T03:34:54.277 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 podman[75381]: 2024-03-12 03:34:54.171056719 +0000 UTC m=+3.331478677 container create 07069691a7352c6e3d11e30cea279d9aacd1fbe482a708d2f151cf62c251fe8c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:34:54.277 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:34:54] ENGINE Bus STOPPING 2024-03-12T03:34:54.277 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:34:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:34:54.277 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:34:54] ENGINE Bus STOPPED 2024-03-12T03:34:54.277 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:34:54] ENGINE Bus STARTING 2024-03-12T03:34:54.277 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:34:54] ENGINE Serving on http://:::9283 2024-03-12T03:34:54.277 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:34:54] ENGINE Bus STARTED 2024-03-12T03:34:54.925 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 podman[75381]: 2024-03-12 03:34:54.63389214 +0000 UTC m=+3.794314106 container init 07069691a7352c6e3d11e30cea279d9aacd1fbe482a708d2f151cf62c251fe8c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 podman[75381]: 2024-03-12 03:34:54.63661306 +0000 UTC m=+3.797035019 container start 07069691a7352c6e3d11e30cea279d9aacd1fbe482a708d2f151cf62c251fe8c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.640Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.640Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.640Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.640Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.640Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.640Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=arp 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-12T03:34:54.926 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=edac 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-12T03:34:54.927 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=os 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=stat 2024-03-12T03:34:54.928 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-12T03:34:54.929 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-12T03:34:54.929 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-12T03:34:54.929 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=time 2024-03-12T03:34:54.929 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-12T03:34:54.929 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=uname 2024-03-12T03:34:54.929 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-12T03:34:54.929 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-12T03:34:54.929 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-12T03:34:54.929 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-12T03:34:54.929 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:54 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a[75580]: ts=2024-03-12T03:34:54.641Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-12T03:34:55.205 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-12T03:34:55.205 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-12T03:34:55.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:34:55.423 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:55 smithi169 bash[75381]: 07069691a7352c6e3d11e30cea279d9aacd1fbe482a708d2f151cf62c251fe8c 2024-03-12T03:34:55.423 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:34:55 smithi169 systemd[1]: Started Ceph node-exporter.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:34:55.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:55 smithi169 ceph-mon[29344]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:34:55.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:55 smithi169 ceph-mon[29344]: 5.10 scrub starts 2024-03-12T03:34:55.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:55 smithi169 ceph-mon[29344]: 5.10 scrub ok 2024-03-12T03:34:55.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:55 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1613113547' entity='client.admin' 2024-03-12T03:34:55.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:55 smithi169 ceph-mon[29344]: 4.4 scrub starts 2024-03-12T03:34:55.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:55 smithi169 ceph-mon[29344]: 4.4 scrub ok 2024-03-12T03:34:55.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:55 smithi169 ceph-mon[39949]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:34:55.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:55 smithi169 ceph-mon[39949]: 5.10 scrub starts 2024-03-12T03:34:55.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:55 smithi169 ceph-mon[39949]: 5.10 scrub ok 2024-03-12T03:34:55.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:55 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1613113547' entity='client.admin' 2024-03-12T03:34:55.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:55 smithi169 ceph-mon[39949]: 4.4 scrub starts 2024-03-12T03:34:55.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:55 smithi169 ceph-mon[39949]: 4.4 scrub ok 2024-03-12T03:34:55.461 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:55.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:55 smithi173 ceph-mon[33544]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:34:55.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:55 smithi173 ceph-mon[33544]: 5.10 scrub starts 2024-03-12T03:34:55.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:55 smithi173 ceph-mon[33544]: 5.10 scrub ok 2024-03-12T03:34:55.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:55 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1613113547' entity='client.admin' 2024-03-12T03:34:55.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:55 smithi173 ceph-mon[33544]: 4.4 scrub starts 2024-03-12T03:34:55.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:55 smithi173 ceph-mon[33544]: 4.4 scrub ok 2024-03-12T03:34:56.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:56 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:56.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:56 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:56.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:56 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:56.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:56 smithi169 ceph-mon[29344]: Deploying daemon node-exporter.b on smithi173 2024-03-12T03:34:56.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:56 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:56.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:56 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:56.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:56 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:56.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:56 smithi169 ceph-mon[39949]: Deploying daemon node-exporter.b on smithi173 2024-03-12T03:34:56.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:56 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:56.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:56 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:56.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:56 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:34:56.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:56 smithi173 ceph-mon[33544]: Deploying daemon node-exporter.b on smithi173 2024-03-12T03:34:57.441 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:34:57.443 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:34:57.448 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:34:57 smithi173 bash[64222]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-12T03:34:57.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:57 smithi173 ceph-mon[33544]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:34:57.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:57 smithi173 ceph-mon[33544]: 5.d scrub starts 2024-03-12T03:34:57.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:57 smithi173 ceph-mon[33544]: 5.d scrub ok 2024-03-12T03:34:57.765 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:57 smithi169 ceph-mon[29344]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:34:57.765 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:57 smithi169 ceph-mon[29344]: 5.d scrub starts 2024-03-12T03:34:57.765 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:57 smithi169 ceph-mon[29344]: 5.d scrub ok 2024-03-12T03:34:57.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:57 smithi169 ceph-mon[39949]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:34:57.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:57 smithi169 ceph-mon[39949]: 5.d scrub starts 2024-03-12T03:34:57.766 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:57 smithi169 ceph-mon[39949]: 5.d scrub ok 2024-03-12T03:34:58.094 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-03-12T03:34:55.766985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1306,"num_read_kb":1120,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":816,"ondisk_log_size":816,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":233008,"kb_used_data":8540,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749728208,"statfs":{"total":767960285184,"available":767721684992,"internally_reserved":0,"allocated":8744960,"data_stored":4838112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002264"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144456+0000","last_change":"2024-03-12T03:33:27.183162+0000","last_active":"2024-03-12T03:34:48.144456+0000","last_peered":"2024-03-12T03:34:48.144456+0000","last_clean":"2024-03-12T03:34:48.144456+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:34:48.144456+0000","last_undegraded":"2024-03-12T03:34:48.144456+0000","last_fullsized":"2024-03-12T03:34:48.144456+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819661+0000","last_change":"2024-03-12T03:34:29.244162+0000","last_active":"2024-03-12T03:34:48.819661+0000","last_peered":"2024-03-12T03:34:48.819661+0000","last_clean":"2024-03-12T03:34:48.819661+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:34:48.819661+0000","last_undegraded":"2024-03-12T03:34:48.819661+0000","last_fullsized":"2024-03-12T03:34:48.819661+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:29.244078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:29.244078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.593696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.828981+0000","last_change":"2024-03-12T03:34:48.829003+0000","last_active":"2024-03-12T03:34:48.828981+0000","last_peered":"2024-03-12T03:34:48.828981+0000","last_clean":"2024-03-12T03:34:48.828981+0000","last_became_active":"2024-03-12T03:33:21.573080+0000","last_became_peered":"2024-03-12T03:33:21.573080+0000","last_unstale":"2024-03-12T03:34:48.828981+0000","last_undegraded":"2024-03-12T03:34:48.828981+0000","last_fullsized":"2024-03-12T03:34:48.828981+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:48.828926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:48.828926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:15.319549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144350+0000","last_change":"2024-03-12T03:33:23.164433+0000","last_active":"2024-03-12T03:34:48.144350+0000","last_peered":"2024-03-12T03:34:48.144350+0000","last_clean":"2024-03-12T03:34:48.144350+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:34:48.144350+0000","last_undegraded":"2024-03-12T03:34:48.144350+0000","last_fullsized":"2024-03-12T03:34:48.144350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147609+0000","last_change":"2024-03-12T03:33:27.382583+0000","last_active":"2024-03-12T03:34:48.147609+0000","last_peered":"2024-03-12T03:34:48.147609+0000","last_clean":"2024-03-12T03:34:48.147609+0000","last_became_active":"2024-03-12T03:33:27.382050+0000","last_became_peered":"2024-03-12T03:33:27.382050+0000","last_unstale":"2024-03-12T03:34:48.147609+0000","last_undegraded":"2024-03-12T03:34:48.147609+0000","last_fullsized":"2024-03-12T03:34:48.147609+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144353+0000","last_change":"2024-03-12T03:33:25.180282+0000","last_active":"2024-03-12T03:34:48.144353+0000","last_peered":"2024-03-12T03:34:48.144353+0000","last_clean":"2024-03-12T03:34:48.144353+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:34:48.144353+0000","last_undegraded":"2024-03-12T03:34:48.144353+0000","last_fullsized":"2024-03-12T03:34:48.144353+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148220+0000","last_change":"2024-03-12T03:34:41.143075+0000","last_active":"2024-03-12T03:34:48.148220+0000","last_peered":"2024-03-12T03:34:48.148220+0000","last_clean":"2024-03-12T03:34:48.148220+0000","last_became_active":"2024-03-12T03:33:21.572876+0000","last_became_peered":"2024-03-12T03:33:21.572876+0000","last_unstale":"2024-03-12T03:34:48.148220+0000","last_undegraded":"2024-03-12T03:34:48.148220+0000","last_fullsized":"2024-03-12T03:34:48.148220+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.142994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.165643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":134,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147913+0000","last_change":"2024-03-12T03:33:23.161838+0000","last_active":"2024-03-12T03:34:48.147913+0000","last_peered":"2024-03-12T03:34:48.147913+0000","last_clean":"2024-03-12T03:34:48.147913+0000","last_became_active":"2024-03-12T03:33:23.161549+0000","last_became_peered":"2024-03-12T03:33:23.161549+0000","last_unstale":"2024-03-12T03:34:48.147913+0000","last_undegraded":"2024-03-12T03:34:48.147913+0000","last_fullsized":"2024-03-12T03:34:48.147913+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818555+0000","last_change":"2024-03-12T03:33:27.189571+0000","last_active":"2024-03-12T03:34:48.818555+0000","last_peered":"2024-03-12T03:34:48.818555+0000","last_clean":"2024-03-12T03:34:48.818555+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:34:48.818555+0000","last_undegraded":"2024-03-12T03:34:48.818555+0000","last_fullsized":"2024-03-12T03:34:48.818555+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:50.189316+0000","last_change":"2024-03-12T03:34:50.189337+0000","last_active":"2024-03-12T03:34:50.189316+0000","last_peered":"2024-03-12T03:34:50.189316+0000","last_clean":"2024-03-12T03:34:50.189316+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:34:50.189316+0000","last_undegraded":"2024-03-12T03:34:50.189316+0000","last_fullsized":"2024-03-12T03:34:50.189316+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:50.189256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:50.189256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:52.054548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147657+0000","last_change":"2024-03-12T03:33:21.102752+0000","last_active":"2024-03-12T03:34:48.147657+0000","last_peered":"2024-03-12T03:34:48.147657+0000","last_clean":"2024-03-12T03:34:48.147657+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:34:48.147657+0000","last_undegraded":"2024-03-12T03:34:48.147657+0000","last_fullsized":"2024-03-12T03:34:48.147657+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:48.338428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'21","reported_seq":101,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144620+0000","last_change":"2024-03-12T03:33:23.163157+0000","last_active":"2024-03-12T03:34:48.144620+0000","last_peered":"2024-03-12T03:34:48.144620+0000","last_clean":"2024-03-12T03:34:48.144620+0000","last_became_active":"2024-03-12T03:33:23.162749+0000","last_became_peered":"2024-03-12T03:33:23.162749+0000","last_unstale":"2024-03-12T03:34:48.144620+0000","last_undegraded":"2024-03-12T03:34:48.144620+0000","last_fullsized":"2024-03-12T03:34:48.144620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147170+0000","last_change":"2024-03-12T03:33:27.188261+0000","last_active":"2024-03-12T03:34:48.147170+0000","last_peered":"2024-03-12T03:34:48.147170+0000","last_clean":"2024-03-12T03:34:48.147170+0000","last_became_active":"2024-03-12T03:33:27.187943+0000","last_became_peered":"2024-03-12T03:33:27.187943+0000","last_unstale":"2024-03-12T03:34:48.147170+0000","last_undegraded":"2024-03-12T03:34:48.147170+0000","last_fullsized":"2024-03-12T03:34:48.147170+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.815395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818972+0000","last_change":"2024-03-12T03:33:25.182503+0000","last_active":"2024-03-12T03:34:48.818972+0000","last_peered":"2024-03-12T03:34:48.818972+0000","last_clean":"2024-03-12T03:34:48.818972+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:34:48.818972+0000","last_undegraded":"2024-03-12T03:34:48.818972+0000","last_fullsized":"2024-03-12T03:34:48.818972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:49.175148+0000","last_change":"2024-03-12T03:34:49.175167+0000","last_active":"2024-03-12T03:34:49.175148+0000","last_peered":"2024-03-12T03:34:49.175148+0000","last_clean":"2024-03-12T03:34:49.175148+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:34:49.175148+0000","last_undegraded":"2024-03-12T03:34:49.175148+0000","last_fullsized":"2024-03-12T03:34:49.175148+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.175091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.175091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:59.268389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"86'14","reported_seq":108,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144629+0000","last_change":"2024-03-12T03:34:23.795189+0000","last_active":"2024-03-12T03:34:48.144629+0000","last_peered":"2024-03-12T03:34:48.144629+0000","last_clean":"2024-03-12T03:34:48.144629+0000","last_became_active":"2024-03-12T03:33:50.155908+0000","last_became_peered":"2024-03-12T03:33:50.155908+0000","last_unstale":"2024-03-12T03:34:48.144629+0000","last_undegraded":"2024-03-12T03:34:48.144629+0000","last_fullsized":"2024-03-12T03:34:48.144629+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.795040+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:35.995794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145032+0000","last_change":"2024-03-12T03:33:27.380906+0000","last_active":"2024-03-12T03:34:48.145032+0000","last_peered":"2024-03-12T03:34:48.145032+0000","last_clean":"2024-03-12T03:34:48.145032+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:34:48.145032+0000","last_undegraded":"2024-03-12T03:34:48.145032+0000","last_fullsized":"2024-03-12T03:34:48.145032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819849+0000","last_change":"2024-03-12T03:34:26.233162+0000","last_active":"2024-03-12T03:34:48.819849+0000","last_peered":"2024-03-12T03:34:48.819849+0000","last_clean":"2024-03-12T03:34:48.819849+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:34:48.819849+0000","last_undegraded":"2024-03-12T03:34:48.819849+0000","last_fullsized":"2024-03-12T03:34:48.819849+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:26.233072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:26.233072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.096475+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819563+0000","last_change":"2024-03-12T03:33:21.103671+0000","last_active":"2024-03-12T03:34:48.819563+0000","last_peered":"2024-03-12T03:34:48.819563+0000","last_clean":"2024-03-12T03:34:48.819563+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:34:48.819563+0000","last_undegraded":"2024-03-12T03:34:48.819563+0000","last_fullsized":"2024-03-12T03:34:48.819563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818791+0000","last_change":"2024-03-12T03:33:23.161120+0000","last_active":"2024-03-12T03:34:48.818791+0000","last_peered":"2024-03-12T03:34:48.818791+0000","last_clean":"2024-03-12T03:34:48.818791+0000","last_became_active":"2024-03-12T03:33:23.160702+0000","last_became_peered":"2024-03-12T03:33:23.160702+0000","last_unstale":"2024-03-12T03:34:48.818791+0000","last_undegraded":"2024-03-12T03:34:48.818791+0000","last_fullsized":"2024-03-12T03:34:48.818791+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148426+0000","last_change":"2024-03-12T03:33:27.381440+0000","last_active":"2024-03-12T03:34:48.148426+0000","last_peered":"2024-03-12T03:34:48.148426+0000","last_clean":"2024-03-12T03:34:48.148426+0000","last_became_active":"2024-03-12T03:33:27.380880+0000","last_became_peered":"2024-03-12T03:33:27.380880+0000","last_unstale":"2024-03-12T03:34:48.148426+0000","last_undegraded":"2024-03-12T03:34:48.148426+0000","last_fullsized":"2024-03-12T03:34:48.148426+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.898172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144737+0000","last_change":"2024-03-12T03:34:36.949207+0000","last_active":"2024-03-12T03:34:48.144737+0000","last_peered":"2024-03-12T03:34:48.144737+0000","last_clean":"2024-03-12T03:34:48.144737+0000","last_became_active":"2024-03-12T03:33:25.181087+0000","last_became_peered":"2024-03-12T03:33:25.181087+0000","last_unstale":"2024-03-12T03:34:48.144737+0000","last_undegraded":"2024-03-12T03:34:48.144737+0000","last_fullsized":"2024-03-12T03:34:48.144737+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.949124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.949124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:57.461161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144627+0000","last_change":"2024-03-12T03:34:44.029203+0000","last_active":"2024-03-12T03:34:48.144627+0000","last_peered":"2024-03-12T03:34:48.144627+0000","last_clean":"2024-03-12T03:34:48.144627+0000","last_became_active":"2024-03-12T03:33:21.573218+0000","last_became_peered":"2024-03-12T03:33:21.573218+0000","last_unstale":"2024-03-12T03:34:48.144627+0000","last_undegraded":"2024-03-12T03:34:48.144627+0000","last_fullsized":"2024-03-12T03:34:48.144627+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:44.029060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.029060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.635332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":96,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147725+0000","last_change":"2024-03-12T03:33:23.162704+0000","last_active":"2024-03-12T03:34:48.147725+0000","last_peered":"2024-03-12T03:34:48.147725+0000","last_clean":"2024-03-12T03:34:48.147725+0000","last_became_active":"2024-03-12T03:33:23.162427+0000","last_became_peered":"2024-03-12T03:33:23.162427+0000","last_unstale":"2024-03-12T03:34:48.147725+0000","last_undegraded":"2024-03-12T03:34:48.147725+0000","last_fullsized":"2024-03-12T03:34:48.147725+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.329189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819143+0000","last_change":"2024-03-12T03:34:37.658704+0000","last_active":"2024-03-12T03:34:48.819143+0000","last_peered":"2024-03-12T03:34:48.819143+0000","last_clean":"2024-03-12T03:34:48.819143+0000","last_became_active":"2024-03-12T03:33:27.189152+0000","last_became_peered":"2024-03-12T03:33:27.189152+0000","last_unstale":"2024-03-12T03:34:48.819143+0000","last_undegraded":"2024-03-12T03:34:48.819143+0000","last_fullsized":"2024-03-12T03:34:48.819143+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:34:37.658602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:37.658602+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:38.697166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":95,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.159237+0000","last_change":"2024-03-12T03:34:33.696207+0000","last_active":"2024-03-12T03:34:48.159237+0000","last_peered":"2024-03-12T03:34:48.159237+0000","last_clean":"2024-03-12T03:34:48.159237+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:34:48.159237+0000","last_undegraded":"2024-03-12T03:34:48.159237+0000","last_fullsized":"2024-03-12T03:34:48.159237+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:33.696102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:33.696102+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.429139+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819746+0000","last_change":"2024-03-12T03:34:43.324075+0000","last_active":"2024-03-12T03:34:48.819746+0000","last_peered":"2024-03-12T03:34:48.819746+0000","last_clean":"2024-03-12T03:34:48.819746+0000","last_became_active":"2024-03-12T03:33:21.097218+0000","last_became_peered":"2024-03-12T03:33:21.097218+0000","last_unstale":"2024-03-12T03:34:48.819746+0000","last_undegraded":"2024-03-12T03:34:48.819746+0000","last_fullsized":"2024-03-12T03:34:48.819746+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:34:43.323991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.323991+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:05.428737+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'17","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819891+0000","last_change":"2024-03-12T03:33:23.553096+0000","last_active":"2024-03-12T03:34:48.819891+0000","last_peered":"2024-03-12T03:34:48.819891+0000","last_clean":"2024-03-12T03:34:48.819891+0000","last_became_active":"2024-03-12T03:33:23.552812+0000","last_became_peered":"2024-03-12T03:33:23.552812+0000","last_unstale":"2024-03-12T03:34:48.819891+0000","last_undegraded":"2024-03-12T03:34:48.819891+0000","last_fullsized":"2024-03-12T03:34:48.819891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820035+0000","last_change":"2024-03-12T03:33:27.184475+0000","last_active":"2024-03-12T03:34:48.820035+0000","last_peered":"2024-03-12T03:34:48.820035+0000","last_clean":"2024-03-12T03:34:48.820035+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:34:48.820035+0000","last_undegraded":"2024-03-12T03:34:48.820035+0000","last_fullsized":"2024-03-12T03:34:48.820035+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:52.246462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820105+0000","last_change":"2024-03-12T03:34:47.832795+0000","last_active":"2024-03-12T03:34:48.820105+0000","last_peered":"2024-03-12T03:34:48.820105+0000","last_clean":"2024-03-12T03:34:48.820105+0000","last_became_active":"2024-03-12T03:33:25.178346+0000","last_became_peered":"2024-03-12T03:33:25.178346+0000","last_unstale":"2024-03-12T03:34:48.820105+0000","last_undegraded":"2024-03-12T03:34:48.820105+0000","last_fullsized":"2024-03-12T03:34:48.820105+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.832716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.832716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.392333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147332+0000","last_change":"2024-03-12T03:33:21.095860+0000","last_active":"2024-03-12T03:34:48.147332+0000","last_peered":"2024-03-12T03:34:48.147332+0000","last_clean":"2024-03-12T03:34:48.147332+0000","last_became_active":"2024-03-12T03:33:21.095556+0000","last_became_peered":"2024-03-12T03:33:21.095556+0000","last_unstale":"2024-03-12T03:34:48.147332+0000","last_undegraded":"2024-03-12T03:34:48.147332+0000","last_fullsized":"2024-03-12T03:34:48.147332+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.291634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"94'23","reported_seq":106,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:52.730057+0000","last_change":"2024-03-12T03:34:52.730098+0000","last_active":"2024-03-12T03:34:52.730057+0000","last_peered":"2024-03-12T03:34:52.730057+0000","last_clean":"2024-03-12T03:34:52.730057+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:34:52.730057+0000","last_undegraded":"2024-03-12T03:34:52.730057+0000","last_fullsized":"2024-03-12T03:34:52.730057+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'23","last_scrub_stamp":"2024-03-12T03:34:52.729983+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:34:52.729983+0000","last_clean_scrub_stamp":"2024-03-12T03:34:52.729983+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.107241+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818843+0000","last_change":"2024-03-12T03:33:27.190023+0000","last_active":"2024-03-12T03:34:48.818843+0000","last_peered":"2024-03-12T03:34:48.818843+0000","last_clean":"2024-03-12T03:34:48.818843+0000","last_became_active":"2024-03-12T03:33:27.189696+0000","last_became_peered":"2024-03-12T03:33:27.189696+0000","last_unstale":"2024-03-12T03:34:48.818843+0000","last_undegraded":"2024-03-12T03:34:48.818843+0000","last_fullsized":"2024-03-12T03:34:48.818843+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.530688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.158608+0000","last_change":"2024-03-12T03:34:36.701692+0000","last_active":"2024-03-12T03:34:48.158608+0000","last_peered":"2024-03-12T03:34:48.158608+0000","last_clean":"2024-03-12T03:34:48.158608+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:34:48.158608+0000","last_undegraded":"2024-03-12T03:34:48.158608+0000","last_fullsized":"2024-03-12T03:34:48.158608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.701611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.701611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.280217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819714+0000","last_change":"2024-03-12T03:34:32.746113+0000","last_active":"2024-03-12T03:34:48.819714+0000","last_peered":"2024-03-12T03:34:48.819714+0000","last_clean":"2024-03-12T03:34:48.819714+0000","last_became_active":"2024-03-12T03:33:21.096560+0000","last_became_peered":"2024-03-12T03:33:21.096560+0000","last_unstale":"2024-03-12T03:34:48.819714+0000","last_undegraded":"2024-03-12T03:34:48.819714+0000","last_fullsized":"2024-03-12T03:34:48.819714+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:32.746027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:32.746027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:33.564397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"90'6","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147341+0000","last_change":"2024-03-12T03:34:34.274478+0000","last_active":"2024-03-12T03:34:48.147341+0000","last_peered":"2024-03-12T03:34:48.147341+0000","last_clean":"2024-03-12T03:34:48.147341+0000","last_became_active":"2024-03-12T03:33:23.161801+0000","last_became_peered":"2024-03-12T03:33:23.161801+0000","last_unstale":"2024-03-12T03:34:48.147341+0000","last_undegraded":"2024-03-12T03:34:48.147341+0000","last_fullsized":"2024-03-12T03:34:48.147341+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.274353+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.042399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147514+0000","last_change":"2024-03-12T03:34:40.191618+0000","last_active":"2024-03-12T03:34:48.147514+0000","last_peered":"2024-03-12T03:34:48.147514+0000","last_clean":"2024-03-12T03:34:48.147514+0000","last_became_active":"2024-03-12T03:33:27.381110+0000","last_became_peered":"2024-03-12T03:33:27.381110+0000","last_unstale":"2024-03-12T03:34:48.147514+0000","last_undegraded":"2024-03-12T03:34:48.147514+0000","last_fullsized":"2024-03-12T03:34:48.147514+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.191538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.480916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818979+0000","last_change":"2024-03-12T03:33:25.178948+0000","last_active":"2024-03-12T03:34:48.818979+0000","last_peered":"2024-03-12T03:34:48.818979+0000","last_clean":"2024-03-12T03:34:48.818979+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:34:48.818979+0000","last_undegraded":"2024-03-12T03:34:48.818979+0000","last_fullsized":"2024-03-12T03:34:48.818979+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147736+0000","last_change":"2024-03-12T03:34:25.171761+0000","last_active":"2024-03-12T03:34:48.147736+0000","last_peered":"2024-03-12T03:34:48.147736+0000","last_clean":"2024-03-12T03:34:48.147736+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:34:48.147736+0000","last_undegraded":"2024-03-12T03:34:48.147736+0000","last_fullsized":"2024-03-12T03:34:48.147736+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:25.171622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.171622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:37.280670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'18","reported_seq":93,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147525+0000","last_change":"2024-03-12T03:33:23.552489+0000","last_active":"2024-03-12T03:34:48.147525+0000","last_peered":"2024-03-12T03:34:48.147525+0000","last_clean":"2024-03-12T03:34:48.147525+0000","last_became_active":"2024-03-12T03:33:23.552210+0000","last_became_peered":"2024-03-12T03:33:23.552210+0000","last_unstale":"2024-03-12T03:34:48.147525+0000","last_undegraded":"2024-03-12T03:34:48.147525+0000","last_fullsized":"2024-03-12T03:34:48.147525+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818700+0000","last_change":"2024-03-12T03:33:27.193134+0000","last_active":"2024-03-12T03:34:48.818700+0000","last_peered":"2024-03-12T03:34:48.818700+0000","last_clean":"2024-03-12T03:34:48.818700+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:34:48.818700+0000","last_undegraded":"2024-03-12T03:34:48.818700+0000","last_fullsized":"2024-03-12T03:34:48.818700+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147678+0000","last_change":"2024-03-12T03:34:45.089928+0000","last_active":"2024-03-12T03:34:48.147678+0000","last_peered":"2024-03-12T03:34:48.147678+0000","last_clean":"2024-03-12T03:34:48.147678+0000","last_became_active":"2024-03-12T03:33:25.183725+0000","last_became_peered":"2024-03-12T03:33:25.183725+0000","last_unstale":"2024-03-12T03:34:48.147678+0000","last_undegraded":"2024-03-12T03:34:48.147678+0000","last_fullsized":"2024-03-12T03:34:48.147678+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:45.089762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:45.089762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.500637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819348+0000","last_change":"2024-03-12T03:34:41.698397+0000","last_active":"2024-03-12T03:34:48.819348+0000","last_peered":"2024-03-12T03:34:48.819348+0000","last_clean":"2024-03-12T03:34:48.819348+0000","last_became_active":"2024-03-12T03:33:21.107037+0000","last_became_peered":"2024-03-12T03:33:21.107037+0000","last_unstale":"2024-03-12T03:34:48.819348+0000","last_undegraded":"2024-03-12T03:34:48.819348+0000","last_fullsized":"2024-03-12T03:34:48.819348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.698268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.698268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:05.134034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820148+0000","last_change":"2024-03-12T03:33:23.161686+0000","last_active":"2024-03-12T03:34:48.820148+0000","last_peered":"2024-03-12T03:34:48.820148+0000","last_clean":"2024-03-12T03:34:48.820148+0000","last_became_active":"2024-03-12T03:33:23.160818+0000","last_became_peered":"2024-03-12T03:33:23.160818+0000","last_unstale":"2024-03-12T03:34:48.820148+0000","last_undegraded":"2024-03-12T03:34:48.820148+0000","last_fullsized":"2024-03-12T03:34:48.820148+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.366562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147313+0000","last_change":"2024-03-12T03:33:27.192215+0000","last_active":"2024-03-12T03:34:48.147313+0000","last_peered":"2024-03-12T03:34:48.147313+0000","last_clean":"2024-03-12T03:34:48.147313+0000","last_became_active":"2024-03-12T03:33:27.191710+0000","last_became_peered":"2024-03-12T03:33:27.191710+0000","last_unstale":"2024-03-12T03:34:48.147313+0000","last_undegraded":"2024-03-12T03:34:48.147313+0000","last_fullsized":"2024-03-12T03:34:48.147313+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819961+0000","last_change":"2024-03-12T03:34:33.258733+0000","last_active":"2024-03-12T03:34:48.819961+0000","last_peered":"2024-03-12T03:34:48.819961+0000","last_clean":"2024-03-12T03:34:48.819961+0000","last_became_active":"2024-03-12T03:33:25.182004+0000","last_became_peered":"2024-03-12T03:33:25.182004+0000","last_unstale":"2024-03-12T03:34:48.819961+0000","last_undegraded":"2024-03-12T03:34:48.819961+0000","last_fullsized":"2024-03-12T03:34:48.819961+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:33.258626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:33.258626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.420748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:50.076875+0000","last_change":"2024-03-12T03:34:50.076895+0000","last_active":"2024-03-12T03:34:50.076875+0000","last_peered":"2024-03-12T03:34:50.076875+0000","last_clean":"2024-03-12T03:34:50.076875+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:34:50.076875+0000","last_undegraded":"2024-03-12T03:34:50.076875+0000","last_fullsized":"2024-03-12T03:34:50.076875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:50.076814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:50.076814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:58.283856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144708+0000","last_change":"2024-03-12T03:34:34.953658+0000","last_active":"2024-03-12T03:34:48.144708+0000","last_peered":"2024-03-12T03:34:48.144708+0000","last_clean":"2024-03-12T03:34:48.144708+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:34:48.144708+0000","last_undegraded":"2024-03-12T03:34:48.144708+0000","last_fullsized":"2024-03-12T03:34:48.144708+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:34:34.953565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.953565+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.161677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819583+0000","last_change":"2024-03-12T03:34:34.718591+0000","last_active":"2024-03-12T03:34:48.819583+0000","last_peered":"2024-03-12T03:34:48.819583+0000","last_clean":"2024-03-12T03:34:48.819583+0000","last_became_active":"2024-03-12T03:33:27.194094+0000","last_became_peered":"2024-03-12T03:33:27.194094+0000","last_unstale":"2024-03-12T03:34:48.819583+0000","last_undegraded":"2024-03-12T03:34:48.819583+0000","last_fullsized":"2024-03-12T03:34:48.819583+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:34.718509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.718509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.921678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":91,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147723+0000","last_change":"2024-03-12T03:34:38.139709+0000","last_active":"2024-03-12T03:34:48.147723+0000","last_peered":"2024-03-12T03:34:48.147723+0000","last_clean":"2024-03-12T03:34:48.147723+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:34:48.147723+0000","last_undegraded":"2024-03-12T03:34:48.147723+0000","last_fullsized":"2024-03-12T03:34:48.147723+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:38.139582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.139582+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:39.751323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819095+0000","last_change":"2024-03-12T03:34:38.663634+0000","last_active":"2024-03-12T03:34:48.819095+0000","last_peered":"2024-03-12T03:34:48.819095+0000","last_clean":"2024-03-12T03:34:48.819095+0000","last_became_active":"2024-03-12T03:33:21.573244+0000","last_became_peered":"2024-03-12T03:33:21.573244+0000","last_unstale":"2024-03-12T03:34:48.819095+0000","last_undegraded":"2024-03-12T03:34:48.819095+0000","last_fullsized":"2024-03-12T03:34:48.819095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:34:38.663542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.663542+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:59.053797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":100,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147785+0000","last_change":"2024-03-12T03:33:23.552006+0000","last_active":"2024-03-12T03:34:48.147785+0000","last_peered":"2024-03-12T03:34:48.147785+0000","last_clean":"2024-03-12T03:34:48.147785+0000","last_became_active":"2024-03-12T03:33:23.551706+0000","last_became_peered":"2024-03-12T03:33:23.551706+0000","last_unstale":"2024-03-12T03:34:48.147785+0000","last_undegraded":"2024-03-12T03:34:48.147785+0000","last_fullsized":"2024-03-12T03:34:48.147785+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145312+0000","last_change":"2024-03-12T03:33:27.185209+0000","last_active":"2024-03-12T03:34:48.145312+0000","last_peered":"2024-03-12T03:34:48.145312+0000","last_clean":"2024-03-12T03:34:48.145312+0000","last_became_active":"2024-03-12T03:33:27.184916+0000","last_became_peered":"2024-03-12T03:33:27.184916+0000","last_unstale":"2024-03-12T03:34:48.145312+0000","last_undegraded":"2024-03-12T03:34:48.145312+0000","last_fullsized":"2024-03-12T03:34:48.145312+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148098+0000","last_change":"2024-03-12T03:33:25.179522+0000","last_active":"2024-03-12T03:34:48.148098+0000","last_peered":"2024-03-12T03:34:48.148098+0000","last_clean":"2024-03-12T03:34:48.148098+0000","last_became_active":"2024-03-12T03:33:25.179158+0000","last_became_peered":"2024-03-12T03:33:25.179158+0000","last_unstale":"2024-03-12T03:34:48.148098+0000","last_undegraded":"2024-03-12T03:34:48.148098+0000","last_fullsized":"2024-03-12T03:34:48.148098+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819199+0000","last_change":"2024-03-12T03:34:40.753233+0000","last_active":"2024-03-12T03:34:48.819199+0000","last_peered":"2024-03-12T03:34:48.819199+0000","last_clean":"2024-03-12T03:34:48.819199+0000","last_became_active":"2024-03-12T03:33:21.101819+0000","last_became_peered":"2024-03-12T03:33:21.101819+0000","last_unstale":"2024-03-12T03:34:48.819199+0000","last_undegraded":"2024-03-12T03:34:48.819199+0000","last_fullsized":"2024-03-12T03:34:48.819199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.753153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.816911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":97,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147821+0000","last_change":"2024-03-12T03:34:29.083183+0000","last_active":"2024-03-12T03:34:48.147821+0000","last_peered":"2024-03-12T03:34:48.147821+0000","last_clean":"2024-03-12T03:34:48.147821+0000","last_became_active":"2024-03-12T03:33:23.553480+0000","last_became_peered":"2024-03-12T03:33:23.553480+0000","last_unstale":"2024-03-12T03:34:48.147821+0000","last_undegraded":"2024-03-12T03:34:48.147821+0000","last_fullsized":"2024-03-12T03:34:48.147821+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:34:29.083063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:29.083063+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:45.174021+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819151+0000","last_change":"2024-03-12T03:34:43.039900+0000","last_active":"2024-03-12T03:34:48.819151+0000","last_peered":"2024-03-12T03:34:48.819151+0000","last_clean":"2024-03-12T03:34:48.819151+0000","last_became_active":"2024-03-12T03:33:27.187657+0000","last_became_peered":"2024-03-12T03:33:27.187657+0000","last_unstale":"2024-03-12T03:34:48.819151+0000","last_undegraded":"2024-03-12T03:34:48.819151+0000","last_fullsized":"2024-03-12T03:34:48.819151+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:43.039799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.039799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:01.927684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148407+0000","last_change":"2024-03-12T03:33:25.175313+0000","last_active":"2024-03-12T03:34:48.148407+0000","last_peered":"2024-03-12T03:34:48.148407+0000","last_clean":"2024-03-12T03:34:48.148407+0000","last_became_active":"2024-03-12T03:33:25.174953+0000","last_became_peered":"2024-03-12T03:33:25.174953+0000","last_unstale":"2024-03-12T03:34:48.148407+0000","last_undegraded":"2024-03-12T03:34:48.148407+0000","last_fullsized":"2024-03-12T03:34:48.148407+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818837+0000","last_change":"2024-03-12T03:33:21.106130+0000","last_active":"2024-03-12T03:34:48.818837+0000","last_peered":"2024-03-12T03:34:48.818837+0000","last_clean":"2024-03-12T03:34:48.818837+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:34:48.818837+0000","last_undegraded":"2024-03-12T03:34:48.818837+0000","last_fullsized":"2024-03-12T03:34:48.818837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":101,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819952+0000","last_change":"2024-03-12T03:33:23.552425+0000","last_active":"2024-03-12T03:34:48.819952+0000","last_peered":"2024-03-12T03:34:48.819952+0000","last_clean":"2024-03-12T03:34:48.819952+0000","last_became_active":"2024-03-12T03:33:23.552164+0000","last_became_peered":"2024-03-12T03:33:23.552164+0000","last_unstale":"2024-03-12T03:34:48.819952+0000","last_undegraded":"2024-03-12T03:34:48.819952+0000","last_fullsized":"2024-03-12T03:34:48.819952+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.480291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819188+0000","last_change":"2024-03-12T03:34:46.767546+0000","last_active":"2024-03-12T03:34:48.819188+0000","last_peered":"2024-03-12T03:34:48.819188+0000","last_clean":"2024-03-12T03:34:48.819188+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:34:48.819188+0000","last_undegraded":"2024-03-12T03:34:48.819188+0000","last_fullsized":"2024-03-12T03:34:48.819188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:46.767453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:46.767453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:56.661402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147746+0000","last_change":"2024-03-12T03:33:25.181046+0000","last_active":"2024-03-12T03:34:48.147746+0000","last_peered":"2024-03-12T03:34:48.147746+0000","last_clean":"2024-03-12T03:34:48.147746+0000","last_became_active":"2024-03-12T03:33:25.180752+0000","last_became_peered":"2024-03-12T03:33:25.180752+0000","last_unstale":"2024-03-12T03:34:48.147746+0000","last_undegraded":"2024-03-12T03:34:48.147746+0000","last_fullsized":"2024-03-12T03:34:48.147746+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:52.874476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.158563+0000","last_change":"2024-03-12T03:34:26.565344+0000","last_active":"2024-03-12T03:34:48.158563+0000","last_peered":"2024-03-12T03:34:48.158563+0000","last_clean":"2024-03-12T03:34:48.158563+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:34:48.158563+0000","last_undegraded":"2024-03-12T03:34:48.158563+0000","last_fullsized":"2024-03-12T03:34:48.158563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:26.565245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:26.565245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:28.167100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148494+0000","last_change":"2024-03-12T03:33:23.552149+0000","last_active":"2024-03-12T03:34:48.148494+0000","last_peered":"2024-03-12T03:34:48.148494+0000","last_clean":"2024-03-12T03:34:48.148494+0000","last_became_active":"2024-03-12T03:33:23.551773+0000","last_became_peered":"2024-03-12T03:33:23.551773+0000","last_unstale":"2024-03-12T03:34:48.148494+0000","last_undegraded":"2024-03-12T03:34:48.148494+0000","last_fullsized":"2024-03-12T03:34:48.148494+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147234+0000","last_change":"2024-03-12T03:33:27.382928+0000","last_active":"2024-03-12T03:34:48.147234+0000","last_peered":"2024-03-12T03:34:48.147234+0000","last_clean":"2024-03-12T03:34:48.147234+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:34:48.147234+0000","last_undegraded":"2024-03-12T03:34:48.147234+0000","last_fullsized":"2024-03-12T03:34:48.147234+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.104295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819589+0000","last_change":"2024-03-12T03:33:21.107807+0000","last_active":"2024-03-12T03:34:48.819589+0000","last_peered":"2024-03-12T03:34:48.819589+0000","last_clean":"2024-03-12T03:34:48.819589+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:34:48.819589+0000","last_undegraded":"2024-03-12T03:34:48.819589+0000","last_fullsized":"2024-03-12T03:34:48.819589+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"90'36","reported_seq":127,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147179+0000","last_change":"2024-03-12T03:34:39.184089+0000","last_active":"2024-03-12T03:34:48.147179+0000","last_peered":"2024-03-12T03:34:48.147179+0000","last_clean":"2024-03-12T03:34:48.147179+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:34:48.147179+0000","last_undegraded":"2024-03-12T03:34:48.147179+0000","last_fullsized":"2024-03-12T03:34:48.147179+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.183984+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.564093+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":91,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818938+0000","last_change":"2024-03-12T03:34:39.659403+0000","last_active":"2024-03-12T03:34:48.818938+0000","last_peered":"2024-03-12T03:34:48.818938+0000","last_clean":"2024-03-12T03:34:48.818938+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:34:48.818938+0000","last_undegraded":"2024-03-12T03:34:48.818938+0000","last_fullsized":"2024-03-12T03:34:48.818938+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:39.659277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.659277+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:08.903088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147582+0000","last_change":"2024-03-12T03:33:27.191091+0000","last_active":"2024-03-12T03:34:48.147582+0000","last_peered":"2024-03-12T03:34:48.147582+0000","last_clean":"2024-03-12T03:34:48.147582+0000","last_became_active":"2024-03-12T03:33:27.190753+0000","last_became_peered":"2024-03-12T03:33:27.190753+0000","last_unstale":"2024-03-12T03:34:48.147582+0000","last_undegraded":"2024-03-12T03:34:48.147582+0000","last_fullsized":"2024-03-12T03:34:48.147582+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.061395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819447+0000","last_change":"2024-03-12T03:34:23.842758+0000","last_active":"2024-03-12T03:34:48.819447+0000","last_peered":"2024-03-12T03:34:48.819447+0000","last_clean":"2024-03-12T03:34:48.819447+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:34:48.819447+0000","last_undegraded":"2024-03-12T03:34:48.819447+0000","last_fullsized":"2024-03-12T03:34:48.819447+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:23.842628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.842628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.823849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":108,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145079+0000","last_change":"2024-03-12T03:33:23.163665+0000","last_active":"2024-03-12T03:34:48.145079+0000","last_peered":"2024-03-12T03:34:48.145079+0000","last_clean":"2024-03-12T03:34:48.145079+0000","last_became_active":"2024-03-12T03:33:23.163351+0000","last_became_peered":"2024-03-12T03:33:23.163351+0000","last_unstale":"2024-03-12T03:34:48.145079+0000","last_undegraded":"2024-03-12T03:34:48.145079+0000","last_fullsized":"2024-03-12T03:34:48.145079+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818888+0000","last_change":"2024-03-12T03:33:25.185933+0000","last_active":"2024-03-12T03:34:48.818888+0000","last_peered":"2024-03-12T03:34:48.818888+0000","last_clean":"2024-03-12T03:34:48.818888+0000","last_became_active":"2024-03-12T03:33:25.185596+0000","last_became_peered":"2024-03-12T03:33:25.185596+0000","last_unstale":"2024-03-12T03:34:48.818888+0000","last_undegraded":"2024-03-12T03:34:48.818888+0000","last_fullsized":"2024-03-12T03:34:48.818888+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818706+0000","last_change":"2024-03-12T03:34:43.716123+0000","last_active":"2024-03-12T03:34:48.818706+0000","last_peered":"2024-03-12T03:34:48.818706+0000","last_clean":"2024-03-12T03:34:48.818706+0000","last_became_active":"2024-03-12T03:33:27.187955+0000","last_became_peered":"2024-03-12T03:33:27.187955+0000","last_unstale":"2024-03-12T03:34:48.818706+0000","last_undegraded":"2024-03-12T03:34:48.818706+0000","last_fullsized":"2024-03-12T03:34:48.818706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:43.716021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.716021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:01.645227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818709+0000","last_change":"2024-03-12T03:33:25.180331+0000","last_active":"2024-03-12T03:34:48.818709+0000","last_peered":"2024-03-12T03:34:48.818709+0000","last_clean":"2024-03-12T03:34:48.818709+0000","last_became_active":"2024-03-12T03:33:25.179593+0000","last_became_peered":"2024-03-12T03:33:25.179593+0000","last_unstale":"2024-03-12T03:34:48.818709+0000","last_undegraded":"2024-03-12T03:34:48.818709+0000","last_fullsized":"2024-03-12T03:34:48.818709+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.050267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819470+0000","last_change":"2024-03-12T03:34:30.792781+0000","last_active":"2024-03-12T03:34:48.819470+0000","last_peered":"2024-03-12T03:34:48.819470+0000","last_clean":"2024-03-12T03:34:48.819470+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:34:48.819470+0000","last_undegraded":"2024-03-12T03:34:48.819470+0000","last_fullsized":"2024-03-12T03:34:48.819470+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:30.792702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.792702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:50.132706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":112,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.158771+0000","last_change":"2024-03-12T03:34:46.722408+0000","last_active":"2024-03-12T03:34:48.158771+0000","last_peered":"2024-03-12T03:34:48.158771+0000","last_clean":"2024-03-12T03:34:48.158771+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:34:48.158771+0000","last_undegraded":"2024-03-12T03:34:48.158771+0000","last_fullsized":"2024-03-12T03:34:48.158771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:34:46.722278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:46.722278+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.145172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818954+0000","last_change":"2024-03-12T03:34:37.737471+0000","last_active":"2024-03-12T03:34:48.818954+0000","last_peered":"2024-03-12T03:34:48.818954+0000","last_clean":"2024-03-12T03:34:48.818954+0000","last_became_active":"2024-03-12T03:33:27.189948+0000","last_became_peered":"2024-03-12T03:33:27.189948+0000","last_unstale":"2024-03-12T03:34:48.818954+0000","last_undegraded":"2024-03-12T03:34:48.818954+0000","last_fullsized":"2024-03-12T03:34:48.818954+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:37.737386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:37.737386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.387230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818751+0000","last_change":"2024-03-12T03:33:25.184715+0000","last_active":"2024-03-12T03:34:48.818751+0000","last_peered":"2024-03-12T03:34:48.818751+0000","last_clean":"2024-03-12T03:34:48.818751+0000","last_became_active":"2024-03-12T03:33:25.184289+0000","last_became_peered":"2024-03-12T03:33:25.184289+0000","last_unstale":"2024-03-12T03:34:48.818751+0000","last_undegraded":"2024-03-12T03:34:48.818751+0000","last_fullsized":"2024-03-12T03:34:48.818751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.160445+0000","last_change":"2024-03-12T03:34:48.160481+0000","last_active":"2024-03-12T03:34:48.160445+0000","last_peered":"2024-03-12T03:34:48.160445+0000","last_clean":"2024-03-12T03:34:48.160445+0000","last_became_active":"2024-03-12T03:33:21.102410+0000","last_became_peered":"2024-03-12T03:33:21.102410+0000","last_unstale":"2024-03-12T03:34:48.160445+0000","last_undegraded":"2024-03-12T03:34:48.160445+0000","last_fullsized":"2024-03-12T03:34:48.160445+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:34:48.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:17.839681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819359+0000","last_change":"2024-03-12T03:33:23.164442+0000","last_active":"2024-03-12T03:34:48.819359+0000","last_peered":"2024-03-12T03:34:48.819359+0000","last_clean":"2024-03-12T03:34:48.819359+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:34:48.819359+0000","last_undegraded":"2024-03-12T03:34:48.819359+0000","last_fullsized":"2024-03-12T03:34:48.819359+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:50.796888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:49.088162+0000","last_change":"2024-03-12T03:34:49.088183+0000","last_active":"2024-03-12T03:34:49.088162+0000","last_peered":"2024-03-12T03:34:49.088162+0000","last_clean":"2024-03-12T03:34:49.088162+0000","last_became_active":"2024-03-12T03:33:27.185041+0000","last_became_peered":"2024-03-12T03:33:27.185041+0000","last_unstale":"2024-03-12T03:34:49.088162+0000","last_undegraded":"2024-03-12T03:34:49.088162+0000","last_fullsized":"2024-03-12T03:34:49.088162+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.088098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.088098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.622130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147376+0000","last_change":"2024-03-12T03:34:22.168794+0000","last_active":"2024-03-12T03:34:48.147376+0000","last_peered":"2024-03-12T03:34:48.147376+0000","last_clean":"2024-03-12T03:34:48.147376+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:34:48.147376+0000","last_undegraded":"2024-03-12T03:34:48.147376+0000","last_fullsized":"2024-03-12T03:34:48.147376+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:34:22.168654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:22.168654+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:48.338217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144560+0000","last_change":"2024-03-12T03:33:21.107222+0000","last_active":"2024-03-12T03:34:48.144560+0000","last_peered":"2024-03-12T03:34:48.144560+0000","last_clean":"2024-03-12T03:34:48.144560+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:34:48.144560+0000","last_undegraded":"2024-03-12T03:34:48.144560+0000","last_fullsized":"2024-03-12T03:34:48.144560+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'35","reported_seq":78,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147340+0000","last_change":"2024-03-12T03:33:50.166841+0000","last_active":"2024-03-12T03:34:48.147340+0000","last_peered":"2024-03-12T03:34:48.147340+0000","last_clean":"2024-03-12T03:34:48.147340+0000","last_became_active":"2024-03-12T03:33:50.151664+0000","last_became_peered":"2024-03-12T03:33:50.151664+0000","last_unstale":"2024-03-12T03:34:48.147340+0000","last_undegraded":"2024-03-12T03:34:48.147340+0000","last_fullsized":"2024-03-12T03:34:48.147340+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:48.926653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819470+0000","last_change":"2024-03-12T03:34:38.698005+0000","last_active":"2024-03-12T03:34:48.819470+0000","last_peered":"2024-03-12T03:34:48.819470+0000","last_clean":"2024-03-12T03:34:48.819470+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:34:48.819470+0000","last_undegraded":"2024-03-12T03:34:48.819470+0000","last_fullsized":"2024-03-12T03:34:48.819470+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:38.697923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.697923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:07.086680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819534+0000","last_change":"2024-03-12T03:34:40.663082+0000","last_active":"2024-03-12T03:34:48.819534+0000","last_peered":"2024-03-12T03:34:48.819534+0000","last_clean":"2024-03-12T03:34:48.819534+0000","last_became_active":"2024-03-12T03:33:27.380389+0000","last_became_peered":"2024-03-12T03:33:27.380389+0000","last_unstale":"2024-03-12T03:34:48.819534+0000","last_undegraded":"2024-03-12T03:34:48.819534+0000","last_fullsized":"2024-03-12T03:34:48.819534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.663002+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.064026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820376+0000","last_change":"2024-03-12T03:33:21.573748+0000","last_active":"2024-03-12T03:34:48.820376+0000","last_peered":"2024-03-12T03:34:48.820376+0000","last_clean":"2024-03-12T03:34:48.820376+0000","last_became_active":"2024-03-12T03:33:21.572911+0000","last_became_peered":"2024-03-12T03:33:21.572911+0000","last_unstale":"2024-03-12T03:34:48.820376+0000","last_undegraded":"2024-03-12T03:34:48.820376+0000","last_fullsized":"2024-03-12T03:34:48.820376+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:49.918877+0000","last_change":"2024-03-12T03:34:49.918898+0000","last_active":"2024-03-12T03:34:49.918877+0000","last_peered":"2024-03-12T03:34:49.918877+0000","last_clean":"2024-03-12T03:34:49.918877+0000","last_became_active":"2024-03-12T03:33:20.093064+0000","last_became_peered":"2024-03-12T03:33:20.093064+0000","last_unstale":"2024-03-12T03:34:49.918877+0000","last_undegraded":"2024-03-12T03:34:49.918877+0000","last_fullsized":"2024-03-12T03:34:49.918877+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.918802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.918802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.157514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000516994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":108,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820127+0000","last_change":"2024-03-12T03:34:44.301692+0000","last_active":"2024-03-12T03:34:48.820127+0000","last_peered":"2024-03-12T03:34:48.820127+0000","last_clean":"2024-03-12T03:34:48.820127+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:34:48.820127+0000","last_undegraded":"2024-03-12T03:34:48.820127+0000","last_fullsized":"2024-03-12T03:34:48.820127+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-12T03:34:44.301556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.301556+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.466292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819079+0000","last_change":"2024-03-12T03:34:42.086021+0000","last_active":"2024-03-12T03:34:48.819079+0000","last_peered":"2024-03-12T03:34:48.819079+0000","last_clean":"2024-03-12T03:34:48.819079+0000","last_became_active":"2024-03-12T03:33:25.179022+0000","last_became_peered":"2024-03-12T03:33:25.179022+0000","last_unstale":"2024-03-12T03:34:48.819079+0000","last_undegraded":"2024-03-12T03:34:48.819079+0000","last_fullsized":"2024-03-12T03:34:48.819079+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:42.085936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:42.085936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.119017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147829+0000","last_change":"2024-03-12T03:33:27.188886+0000","last_active":"2024-03-12T03:34:48.147829+0000","last_peered":"2024-03-12T03:34:48.147829+0000","last_clean":"2024-03-12T03:34:48.147829+0000","last_became_active":"2024-03-12T03:33:27.188551+0000","last_became_peered":"2024-03-12T03:33:27.188551+0000","last_unstale":"2024-03-12T03:34:48.147829+0000","last_undegraded":"2024-03-12T03:34:48.147829+0000","last_fullsized":"2024-03-12T03:34:48.147829+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:50.069549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":320,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818814+0000","last_change":"2024-03-12T03:33:29.565156+0000","last_active":"2024-03-12T03:34:48.818814+0000","last_peered":"2024-03-12T03:34:48.818814+0000","last_clean":"2024-03-12T03:34:48.818814+0000","last_became_active":"2024-03-12T03:33:11.319888+0000","last_became_peered":"2024-03-12T03:33:11.319888+0000","last_unstale":"2024-03-12T03:34:48.818814+0000","last_undegraded":"2024-03-12T03:34:48.818814+0000","last_fullsized":"2024-03-12T03:34:48.818814+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-12T03:33:29.565069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:33:29.565069+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147698+0000","last_change":"2024-03-12T03:33:21.101239+0000","last_active":"2024-03-12T03:34:48.147698+0000","last_peered":"2024-03-12T03:34:48.147698+0000","last_clean":"2024-03-12T03:34:48.147698+0000","last_became_active":"2024-03-12T03:33:21.100900+0000","last_became_peered":"2024-03-12T03:33:21.100900+0000","last_unstale":"2024-03-12T03:34:48.147698+0000","last_undegraded":"2024-03-12T03:34:48.147698+0000","last_fullsized":"2024-03-12T03:34:48.147698+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":110,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.159037+0000","last_change":"2024-03-12T03:34:35.664996+0000","last_active":"2024-03-12T03:34:48.159037+0000","last_peered":"2024-03-12T03:34:48.159037+0000","last_clean":"2024-03-12T03:34:48.159037+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:34:48.159037+0000","last_undegraded":"2024-03-12T03:34:48.159037+0000","last_fullsized":"2024-03-12T03:34:48.159037+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:34:35.664876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:35.664876+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:54.461746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819213+0000","last_change":"2024-03-12T03:34:47.796670+0000","last_active":"2024-03-12T03:34:48.819213+0000","last_peered":"2024-03-12T03:34:48.819213+0000","last_clean":"2024-03-12T03:34:48.819213+0000","last_became_active":"2024-03-12T03:33:25.177762+0000","last_became_peered":"2024-03-12T03:33:25.177762+0000","last_unstale":"2024-03-12T03:34:48.819213+0000","last_undegraded":"2024-03-12T03:34:48.819213+0000","last_fullsized":"2024-03-12T03:34:48.819213+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.796561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.796561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.258919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820246+0000","last_change":"2024-03-12T03:34:47.352525+0000","last_active":"2024-03-12T03:34:48.820246+0000","last_peered":"2024-03-12T03:34:48.820246+0000","last_clean":"2024-03-12T03:34:48.820246+0000","last_became_active":"2024-03-12T03:33:27.191878+0000","last_became_peered":"2024-03-12T03:33:27.191878+0000","last_unstale":"2024-03-12T03:34:48.820246+0000","last_undegraded":"2024-03-12T03:34:48.820246+0000","last_fullsized":"2024-03-12T03:34:48.820246+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.352450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.352450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:01.666869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819126+0000","last_change":"2024-03-12T03:33:21.097466+0000","last_active":"2024-03-12T03:34:48.819126+0000","last_peered":"2024-03-12T03:34:48.819126+0000","last_clean":"2024-03-12T03:34:48.819126+0000","last_became_active":"2024-03-12T03:33:21.097086+0000","last_became_peered":"2024-03-12T03:33:21.097086+0000","last_unstale":"2024-03-12T03:34:48.819126+0000","last_undegraded":"2024-03-12T03:34:48.819126+0000","last_fullsized":"2024-03-12T03:34:48.819126+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.630139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:49.252431+0000","last_change":"2024-03-12T03:34:21.799658+0000","last_active":"2024-03-12T03:34:49.252431+0000","last_peered":"2024-03-12T03:34:49.252431+0000","last_clean":"2024-03-12T03:34:49.252431+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:34:49.252431+0000","last_undegraded":"2024-03-12T03:34:49.252431+0000","last_fullsized":"2024-03-12T03:34:49.252431+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:34:21.799534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:21.799534+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.973279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":79,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819685+0000","last_change":"2024-03-12T03:34:39.707369+0000","last_active":"2024-03-12T03:34:48.819685+0000","last_peered":"2024-03-12T03:34:48.819685+0000","last_clean":"2024-03-12T03:34:48.819685+0000","last_became_active":"2024-03-12T03:33:23.168750+0000","last_became_peered":"2024-03-12T03:33:23.168750+0000","last_unstale":"2024-03-12T03:34:48.819685+0000","last_undegraded":"2024-03-12T03:34:48.819685+0000","last_fullsized":"2024-03-12T03:34:48.819685+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:34:39.707264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.707264+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:43.684441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148076+0000","last_change":"2024-03-12T03:33:25.177321+0000","last_active":"2024-03-12T03:34:48.148076+0000","last_peered":"2024-03-12T03:34:48.148076+0000","last_clean":"2024-03-12T03:34:48.148076+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:34:48.148076+0000","last_undegraded":"2024-03-12T03:34:48.148076+0000","last_fullsized":"2024-03-12T03:34:48.148076+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.569538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819556+0000","last_change":"2024-03-12T03:34:45.344064+0000","last_active":"2024-03-12T03:34:48.819556+0000","last_peered":"2024-03-12T03:34:48.819556+0000","last_clean":"2024-03-12T03:34:48.819556+0000","last_became_active":"2024-03-12T03:33:27.380458+0000","last_became_peered":"2024-03-12T03:33:27.380458+0000","last_unstale":"2024-03-12T03:34:48.819556+0000","last_undegraded":"2024-03-12T03:34:48.819556+0000","last_fullsized":"2024-03-12T03:34:48.819556+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:45.343945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:45.343945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.292957+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820380+0000","last_change":"2024-03-12T03:34:34.272895+0000","last_active":"2024-03-12T03:34:48.820380+0000","last_peered":"2024-03-12T03:34:48.820380+0000","last_clean":"2024-03-12T03:34:48.820380+0000","last_became_active":"2024-03-12T03:33:21.573003+0000","last_became_peered":"2024-03-12T03:33:21.573003+0000","last_unstale":"2024-03-12T03:34:48.820380+0000","last_undegraded":"2024-03-12T03:34:48.820380+0000","last_fullsized":"2024-03-12T03:34:48.820380+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:34.272759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.272759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:47.242996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":109,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819841+0000","last_change":"2024-03-12T03:33:23.551923+0000","last_active":"2024-03-12T03:34:48.819841+0000","last_peered":"2024-03-12T03:34:48.819841+0000","last_clean":"2024-03-12T03:34:48.819841+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:34:48.819841+0000","last_undegraded":"2024-03-12T03:34:48.819841+0000","last_fullsized":"2024-03-12T03:34:48.819841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:50.306884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.159120+0000","last_change":"2024-03-12T03:34:28.572496+0000","last_active":"2024-03-12T03:34:48.159120+0000","last_peered":"2024-03-12T03:34:48.159120+0000","last_clean":"2024-03-12T03:34:48.159120+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:34:48.159120+0000","last_undegraded":"2024-03-12T03:34:48.159120+0000","last_fullsized":"2024-03-12T03:34:48.159120+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:28.572379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:28.572379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:58.234927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818383+0000","last_change":"2024-03-12T03:33:27.194720+0000","last_active":"2024-03-12T03:34:48.818383+0000","last_peered":"2024-03-12T03:34:48.818383+0000","last_clean":"2024-03-12T03:34:48.818383+0000","last_became_active":"2024-03-12T03:33:27.194266+0000","last_became_peered":"2024-03-12T03:33:27.194266+0000","last_unstale":"2024-03-12T03:34:48.818383+0000","last_undegraded":"2024-03-12T03:34:48.818383+0000","last_fullsized":"2024-03-12T03:34:48.818383+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147637+0000","last_change":"2024-03-12T03:34:41.152579+0000","last_active":"2024-03-12T03:34:48.147637+0000","last_peered":"2024-03-12T03:34:48.147637+0000","last_clean":"2024-03-12T03:34:48.147637+0000","last_became_active":"2024-03-12T03:33:21.573251+0000","last_became_peered":"2024-03-12T03:33:21.573251+0000","last_unstale":"2024-03-12T03:34:48.147637+0000","last_undegraded":"2024-03-12T03:34:48.147637+0000","last_fullsized":"2024-03-12T03:34:48.147637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.152440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.152440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.371887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":96,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820053+0000","last_change":"2024-03-12T03:34:35.282309+0000","last_active":"2024-03-12T03:34:48.820053+0000","last_peered":"2024-03-12T03:34:48.820053+0000","last_clean":"2024-03-12T03:34:48.820053+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:34:48.820053+0000","last_undegraded":"2024-03-12T03:34:48.820053+0000","last_fullsized":"2024-03-12T03:34:48.820053+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:34:35.282203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:35.282203+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.121154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":91,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819196+0000","last_change":"2024-03-12T03:34:30.767076+0000","last_active":"2024-03-12T03:34:48.819196+0000","last_peered":"2024-03-12T03:34:48.819196+0000","last_clean":"2024-03-12T03:34:48.819196+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:34:48.819196+0000","last_undegraded":"2024-03-12T03:34:48.819196+0000","last_fullsized":"2024-03-12T03:34:48.819196+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:30.766975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.766975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:45.275798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819077+0000","last_change":"2024-03-12T03:34:44.750269+0000","last_active":"2024-03-12T03:34:48.819077+0000","last_peered":"2024-03-12T03:34:48.819077+0000","last_clean":"2024-03-12T03:34:48.819077+0000","last_became_active":"2024-03-12T03:33:27.383249+0000","last_became_peered":"2024-03-12T03:33:27.383249+0000","last_unstale":"2024-03-12T03:34:48.819077+0000","last_undegraded":"2024-03-12T03:34:48.819077+0000","last_fullsized":"2024-03-12T03:34:48.819077+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:44.750135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.750135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.084645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":66,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819787+0000","last_change":"2024-03-12T03:33:21.098280+0000","last_active":"2024-03-12T03:34:48.819787+0000","last_peered":"2024-03-12T03:34:48.819787+0000","last_clean":"2024-03-12T03:34:48.819787+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:34:48.819787+0000","last_undegraded":"2024-03-12T03:34:48.819787+0000","last_fullsized":"2024-03-12T03:34:48.819787+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":102,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144849+0000","last_change":"2024-03-12T03:34:47.043566+0000","last_active":"2024-03-12T03:34:48.144849+0000","last_peered":"2024-03-12T03:34:48.144849+0000","last_clean":"2024-03-12T03:34:48.144849+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:34:48.144849+0000","last_undegraded":"2024-03-12T03:34:48.144849+0000","last_fullsized":"2024-03-12T03:34:48.144849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-03-12T03:34:47.043433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.043433+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.321445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148144+0000","last_change":"2024-03-12T03:34:25.168128+0000","last_active":"2024-03-12T03:34:48.148144+0000","last_peered":"2024-03-12T03:34:48.148144+0000","last_clean":"2024-03-12T03:34:48.148144+0000","last_became_active":"2024-03-12T03:33:25.177311+0000","last_became_peered":"2024-03-12T03:33:25.177311+0000","last_unstale":"2024-03-12T03:34:48.148144+0000","last_undegraded":"2024-03-12T03:34:48.148144+0000","last_fullsized":"2024-03-12T03:34:48.148144+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:25.168002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.168002+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.036057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144500+0000","last_change":"2024-03-12T03:33:27.188212+0000","last_active":"2024-03-12T03:34:48.144500+0000","last_peered":"2024-03-12T03:34:48.144500+0000","last_clean":"2024-03-12T03:34:48.144500+0000","last_became_active":"2024-03-12T03:33:27.187864+0000","last_became_peered":"2024-03-12T03:33:27.187864+0000","last_unstale":"2024-03-12T03:34:48.144500+0000","last_undegraded":"2024-03-12T03:34:48.144500+0000","last_fullsized":"2024-03-12T03:34:48.144500+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148221+0000","last_change":"2024-03-12T03:34:47.155013+0000","last_active":"2024-03-12T03:34:48.148221+0000","last_peered":"2024-03-12T03:34:48.148221+0000","last_clean":"2024-03-12T03:34:48.148221+0000","last_became_active":"2024-03-12T03:33:21.104430+0000","last_became_peered":"2024-03-12T03:33:21.104430+0000","last_unstale":"2024-03-12T03:34:48.148221+0000","last_undegraded":"2024-03-12T03:34:48.148221+0000","last_fullsized":"2024-03-12T03:34:48.148221+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.154925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.154925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:48.895337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":117,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147818+0000","last_change":"2024-03-12T03:33:23.551957+0000","last_active":"2024-03-12T03:34:48.147818+0000","last_peered":"2024-03-12T03:34:48.147818+0000","last_clean":"2024-03-12T03:34:48.147818+0000","last_became_active":"2024-03-12T03:33:23.551623+0000","last_became_peered":"2024-03-12T03:33:23.551623+0000","last_unstale":"2024-03-12T03:34:48.147818+0000","last_undegraded":"2024-03-12T03:34:48.147818+0000","last_fullsized":"2024-03-12T03:34:48.147818+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145182+0000","last_change":"2024-03-12T03:33:25.182046+0000","last_active":"2024-03-12T03:34:48.145182+0000","last_peered":"2024-03-12T03:34:48.145182+0000","last_clean":"2024-03-12T03:34:48.145182+0000","last_became_active":"2024-03-12T03:33:25.181020+0000","last_became_peered":"2024-03-12T03:33:25.181020+0000","last_unstale":"2024-03-12T03:34:48.145182+0000","last_undegraded":"2024-03-12T03:34:48.145182+0000","last_fullsized":"2024-03-12T03:34:48.145182+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147561+0000","last_change":"2024-03-12T03:33:27.190398+0000","last_active":"2024-03-12T03:34:48.147561+0000","last_peered":"2024-03-12T03:34:48.147561+0000","last_clean":"2024-03-12T03:34:48.147561+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:34:48.147561+0000","last_undegraded":"2024-03-12T03:34:48.147561+0000","last_fullsized":"2024-03-12T03:34:48.147561+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:53.538609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145398+0000","last_change":"2024-03-12T03:33:25.176357+0000","last_active":"2024-03-12T03:34:48.145398+0000","last_peered":"2024-03-12T03:34:48.145398+0000","last_clean":"2024-03-12T03:34:48.145398+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:34:48.145398+0000","last_undegraded":"2024-03-12T03:34:48.145398+0000","last_fullsized":"2024-03-12T03:34:48.145398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147466+0000","last_change":"2024-03-12T03:33:21.104557+0000","last_active":"2024-03-12T03:34:48.147466+0000","last_peered":"2024-03-12T03:34:48.147466+0000","last_clean":"2024-03-12T03:34:48.147466+0000","last_became_active":"2024-03-12T03:33:21.104293+0000","last_became_peered":"2024-03-12T03:33:21.104293+0000","last_unstale":"2024-03-12T03:34:48.147466+0000","last_undegraded":"2024-03-12T03:34:48.147466+0000","last_fullsized":"2024-03-12T03:34:48.147466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819014+0000","last_change":"2024-03-12T03:33:50.166158+0000","last_active":"2024-03-12T03:34:48.819014+0000","last_peered":"2024-03-12T03:34:48.819014+0000","last_clean":"2024-03-12T03:34:48.819014+0000","last_became_active":"2024-03-12T03:33:50.153240+0000","last_became_peered":"2024-03-12T03:33:50.153240+0000","last_unstale":"2024-03-12T03:34:48.819014+0000","last_undegraded":"2024-03-12T03:34:48.819014+0000","last_fullsized":"2024-03-12T03:34:48.819014+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147547+0000","last_change":"2024-03-12T03:33:27.381253+0000","last_active":"2024-03-12T03:34:48.147547+0000","last_peered":"2024-03-12T03:34:48.147547+0000","last_clean":"2024-03-12T03:34:48.147547+0000","last_became_active":"2024-03-12T03:33:27.380896+0000","last_became_peered":"2024-03-12T03:33:27.380896+0000","last_unstale":"2024-03-12T03:34:48.147547+0000","last_undegraded":"2024-03-12T03:34:48.147547+0000","last_fullsized":"2024-03-12T03:34:48.147547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148070+0000","last_change":"2024-03-12T03:34:36.138975+0000","last_active":"2024-03-12T03:34:48.148070+0000","last_peered":"2024-03-12T03:34:48.148070+0000","last_clean":"2024-03-12T03:34:48.148070+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:34:48.148070+0000","last_undegraded":"2024-03-12T03:34:48.148070+0000","last_fullsized":"2024-03-12T03:34:48.148070+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.138847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.138847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.950469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144623+0000","last_change":"2024-03-12T03:33:21.573917+0000","last_active":"2024-03-12T03:34:48.144623+0000","last_peered":"2024-03-12T03:34:48.144623+0000","last_clean":"2024-03-12T03:34:48.144623+0000","last_became_active":"2024-03-12T03:33:21.573257+0000","last_became_peered":"2024-03-12T03:33:21.573257+0000","last_unstale":"2024-03-12T03:34:48.144623+0000","last_undegraded":"2024-03-12T03:34:48.144623+0000","last_fullsized":"2024-03-12T03:34:48.144623+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.737276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145215+0000","last_change":"2024-03-12T03:33:23.168956+0000","last_active":"2024-03-12T03:34:48.145215+0000","last_peered":"2024-03-12T03:34:48.145215+0000","last_clean":"2024-03-12T03:34:48.145215+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:34:48.145215+0000","last_undegraded":"2024-03-12T03:34:48.145215+0000","last_fullsized":"2024-03-12T03:34:48.145215+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819236+0000","last_change":"2024-03-12T03:34:31.142859+0000","last_active":"2024-03-12T03:34:48.819236+0000","last_peered":"2024-03-12T03:34:48.819236+0000","last_clean":"2024-03-12T03:34:48.819236+0000","last_became_active":"2024-03-12T03:33:27.187815+0000","last_became_peered":"2024-03-12T03:33:27.187815+0000","last_unstale":"2024-03-12T03:34:48.819236+0000","last_undegraded":"2024-03-12T03:34:48.819236+0000","last_fullsized":"2024-03-12T03:34:48.819236+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:31.142765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:31.142765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:54.545296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820450+0000","last_change":"2024-03-12T03:34:30.280067+0000","last_active":"2024-03-12T03:34:48.820450+0000","last_peered":"2024-03-12T03:34:48.820450+0000","last_clean":"2024-03-12T03:34:48.820450+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:34:48.820450+0000","last_undegraded":"2024-03-12T03:34:48.820450+0000","last_fullsized":"2024-03-12T03:34:48.820450+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:30.279966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.279966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.784484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.158678+0000","last_change":"2024-03-12T03:34:23.646936+0000","last_active":"2024-03-12T03:34:48.158678+0000","last_peered":"2024-03-12T03:34:48.158678+0000","last_clean":"2024-03-12T03:34:48.158678+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:34:48.158678+0000","last_undegraded":"2024-03-12T03:34:48.158678+0000","last_fullsized":"2024-03-12T03:34:48.158678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.646819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:33.240250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145216+0000","last_change":"2024-03-12T03:33:23.164300+0000","last_active":"2024-03-12T03:34:48.145216+0000","last_peered":"2024-03-12T03:34:48.145216+0000","last_clean":"2024-03-12T03:34:48.145216+0000","last_became_active":"2024-03-12T03:33:23.163871+0000","last_became_peered":"2024-03-12T03:33:23.163871+0000","last_unstale":"2024-03-12T03:34:48.145216+0000","last_undegraded":"2024-03-12T03:34:48.145216+0000","last_fullsized":"2024-03-12T03:34:48.145216+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145400+0000","last_change":"2024-03-12T03:33:27.184691+0000","last_active":"2024-03-12T03:34:48.145400+0000","last_peered":"2024-03-12T03:34:48.145400+0000","last_clean":"2024-03-12T03:34:48.145400+0000","last_became_active":"2024-03-12T03:33:27.184405+0000","last_became_peered":"2024-03-12T03:33:27.184405+0000","last_unstale":"2024-03-12T03:34:48.145400+0000","last_undegraded":"2024-03-12T03:34:48.145400+0000","last_fullsized":"2024-03-12T03:34:48.145400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818742+0000","last_change":"2024-03-12T03:33:25.178413+0000","last_active":"2024-03-12T03:34:48.818742+0000","last_peered":"2024-03-12T03:34:48.818742+0000","last_clean":"2024-03-12T03:34:48.818742+0000","last_became_active":"2024-03-12T03:33:25.177613+0000","last_became_peered":"2024-03-12T03:33:25.177613+0000","last_unstale":"2024-03-12T03:34:48.818742+0000","last_undegraded":"2024-03-12T03:34:48.818742+0000","last_fullsized":"2024-03-12T03:34:48.818742+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147771+0000","last_change":"2024-03-12T03:33:21.097634+0000","last_active":"2024-03-12T03:34:48.147771+0000","last_peered":"2024-03-12T03:34:48.147771+0000","last_clean":"2024-03-12T03:34:48.147771+0000","last_became_active":"2024-03-12T03:33:21.097304+0000","last_became_peered":"2024-03-12T03:33:21.097304+0000","last_unstale":"2024-03-12T03:34:48.147771+0000","last_undegraded":"2024-03-12T03:34:48.147771+0000","last_fullsized":"2024-03-12T03:34:48.147771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":76,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144768+0000","last_change":"2024-03-12T03:33:23.163246+0000","last_active":"2024-03-12T03:34:48.144768+0000","last_peered":"2024-03-12T03:34:48.144768+0000","last_clean":"2024-03-12T03:34:48.144768+0000","last_became_active":"2024-03-12T03:33:23.162657+0000","last_became_peered":"2024-03-12T03:33:23.162657+0000","last_unstale":"2024-03-12T03:34:48.144768+0000","last_undegraded":"2024-03-12T03:34:48.144768+0000","last_fullsized":"2024-03-12T03:34:48.144768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":608,"ondisk_log_size":608,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":79,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070487,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28452,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716700,"statfs":{"total":95995035648,"available":95965900800,"internally_reserved":0,"allocated":1466368,"data_stored":974514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332125,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28424,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716728,"statfs":{"total":95995035648,"available":95965929472,"internally_reserved":0,"allocated":1441792,"data_stored":972551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":888832,"data_stored":383813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32612,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712540,"statfs":{"total":95995035648,"available":95961640960,"internally_reserved":0,"allocated":1466368,"data_stored":973768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717300,"statfs":{"total":95995035648,"available":95966515200,"internally_reserved":0,"allocated":851968,"data_stored":383416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117035,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":383510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953512,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":383686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659169,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":864256,"data_stored":382854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:34:58.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:34:58.236 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:34:57 smithi173 bash[64222]: Getting image source signatures 2024-03-12T03:34:58.236 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:34:57 smithi173 bash[64222]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-12T03:34:58.236 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:34:57 smithi173 bash[64222]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-12T03:34:58.236 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:34:57 smithi173 bash[64222]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-12T03:34:58.346 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:58.703 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:58 smithi169 ceph-mon[29344]: 3.6 scrub starts 2024-03-12T03:34:58.704 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:58 smithi169 ceph-mon[29344]: 3.6 scrub ok 2024-03-12T03:34:58.704 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:58 smithi169 ceph-mon[29344]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:34:58.704 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:58 smithi169 ceph-mon[39949]: 3.6 scrub starts 2024-03-12T03:34:58.704 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:58 smithi169 ceph-mon[39949]: 3.6 scrub ok 2024-03-12T03:34:58.704 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:58 smithi169 ceph-mon[39949]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:34:58.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:58 smithi173 ceph-mon[33544]: 3.6 scrub starts 2024-03-12T03:34:58.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:58 smithi173 ceph-mon[33544]: 3.6 scrub ok 2024-03-12T03:34:58.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:58 smithi173 ceph-mon[33544]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:34:59.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:59 smithi173 ceph-mon[33544]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:34:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:59 smithi173 ceph-mon[33544]: 6.10 scrub starts 2024-03-12T03:34:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:59 smithi173 ceph-mon[33544]: 6.10 scrub ok 2024-03-12T03:34:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:59 smithi173 ceph-mon[33544]: 6.2 scrub starts 2024-03-12T03:34:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:59 smithi173 ceph-mon[33544]: 6.2 scrub ok 2024-03-12T03:34:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:59 smithi173 ceph-mon[33544]: 6.7 scrub starts 2024-03-12T03:34:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:59 smithi173 ceph-mon[33544]: 6.7 scrub ok 2024-03-12T03:34:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:59 smithi173 ceph-mon[33544]: 4.12 scrub starts 2024-03-12T03:34:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:34:59 smithi173 ceph-mon[33544]: 4.12 scrub ok 2024-03-12T03:34:59.763 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[29344]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:34:59.763 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[29344]: 6.10 scrub starts 2024-03-12T03:34:59.763 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[29344]: 6.10 scrub ok 2024-03-12T03:34:59.763 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[29344]: 6.2 scrub starts 2024-03-12T03:34:59.763 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[29344]: 6.2 scrub ok 2024-03-12T03:34:59.763 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[29344]: 6.7 scrub starts 2024-03-12T03:34:59.763 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[29344]: 6.7 scrub ok 2024-03-12T03:34:59.764 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[29344]: 4.12 scrub starts 2024-03-12T03:34:59.764 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[29344]: 4.12 scrub ok 2024-03-12T03:34:59.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[39949]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:34:59.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[39949]: 6.10 scrub starts 2024-03-12T03:34:59.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[39949]: 6.10 scrub ok 2024-03-12T03:34:59.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[39949]: 6.2 scrub starts 2024-03-12T03:34:59.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[39949]: 6.2 scrub ok 2024-03-12T03:34:59.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[39949]: 6.7 scrub starts 2024-03-12T03:34:59.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[39949]: 6.7 scrub ok 2024-03-12T03:34:59.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[39949]: 4.12 scrub starts 2024-03-12T03:34:59.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:34:59 smithi169 ceph-mon[39949]: 4.12 scrub ok 2024-03-12T03:35:00.320 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:35:00.322 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:35:00.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[39949]: 4.1c scrub starts 2024-03-12T03:35:00.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[39949]: 4.1c scrub ok 2024-03-12T03:35:00.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[39949]: 1.0 scrub starts 2024-03-12T03:35:00.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[39949]: 1.0 scrub ok 2024-03-12T03:35:00.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[39949]: 3.3 scrub starts 2024-03-12T03:35:00.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[39949]: 3.3 scrub ok 2024-03-12T03:35:00.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[39949]: 6.1d scrub starts 2024-03-12T03:35:00.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[39949]: 6.1d scrub ok 2024-03-12T03:35:00.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:00.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[29344]: 4.1c scrub starts 2024-03-12T03:35:00.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[29344]: 4.1c scrub ok 2024-03-12T03:35:00.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[29344]: 1.0 scrub starts 2024-03-12T03:35:00.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[29344]: 1.0 scrub ok 2024-03-12T03:35:00.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[29344]: 3.3 scrub starts 2024-03-12T03:35:00.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[29344]: 3.3 scrub ok 2024-03-12T03:35:00.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[29344]: 6.1d scrub starts 2024-03-12T03:35:00.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[29344]: 6.1d scrub ok 2024-03-12T03:35:00.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:00.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:00 smithi173 ceph-mon[33544]: 4.1c scrub starts 2024-03-12T03:35:00.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:00 smithi173 ceph-mon[33544]: 4.1c scrub ok 2024-03-12T03:35:00.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:00 smithi173 ceph-mon[33544]: 1.0 scrub starts 2024-03-12T03:35:00.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:00 smithi173 ceph-mon[33544]: 1.0 scrub ok 2024-03-12T03:35:00.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:00 smithi173 ceph-mon[33544]: 3.3 scrub starts 2024-03-12T03:35:00.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:00 smithi173 ceph-mon[33544]: 3.3 scrub ok 2024-03-12T03:35:00.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:00 smithi173 ceph-mon[33544]: 6.1d scrub starts 2024-03-12T03:35:00.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:00 smithi173 ceph-mon[33544]: 6.1d scrub ok 2024-03-12T03:35:00.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:01.079 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-03-12T03:34:59.767901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1311,"num_read_kb":1125,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":816,"ondisk_log_size":816,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":232988,"kb_used_data":8540,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749728228,"statfs":{"total":767960285184,"available":767721705472,"internally_reserved":0,"allocated":8744960,"data_stored":4838112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.621502"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144456+0000","last_change":"2024-03-12T03:33:27.183162+0000","last_active":"2024-03-12T03:34:48.144456+0000","last_peered":"2024-03-12T03:34:48.144456+0000","last_clean":"2024-03-12T03:34:48.144456+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:34:48.144456+0000","last_undegraded":"2024-03-12T03:34:48.144456+0000","last_fullsized":"2024-03-12T03:34:48.144456+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819661+0000","last_change":"2024-03-12T03:34:29.244162+0000","last_active":"2024-03-12T03:34:48.819661+0000","last_peered":"2024-03-12T03:34:48.819661+0000","last_clean":"2024-03-12T03:34:48.819661+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:34:48.819661+0000","last_undegraded":"2024-03-12T03:34:48.819661+0000","last_fullsized":"2024-03-12T03:34:48.819661+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:29.244078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:29.244078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.593696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.828981+0000","last_change":"2024-03-12T03:34:48.829003+0000","last_active":"2024-03-12T03:34:48.828981+0000","last_peered":"2024-03-12T03:34:48.828981+0000","last_clean":"2024-03-12T03:34:48.828981+0000","last_became_active":"2024-03-12T03:33:21.573080+0000","last_became_peered":"2024-03-12T03:33:21.573080+0000","last_unstale":"2024-03-12T03:34:48.828981+0000","last_undegraded":"2024-03-12T03:34:48.828981+0000","last_fullsized":"2024-03-12T03:34:48.828981+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:48.828926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:48.828926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:15.319549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144350+0000","last_change":"2024-03-12T03:33:23.164433+0000","last_active":"2024-03-12T03:34:48.144350+0000","last_peered":"2024-03-12T03:34:48.144350+0000","last_clean":"2024-03-12T03:34:48.144350+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:34:48.144350+0000","last_undegraded":"2024-03-12T03:34:48.144350+0000","last_fullsized":"2024-03-12T03:34:48.144350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147609+0000","last_change":"2024-03-12T03:33:27.382583+0000","last_active":"2024-03-12T03:34:48.147609+0000","last_peered":"2024-03-12T03:34:48.147609+0000","last_clean":"2024-03-12T03:34:48.147609+0000","last_became_active":"2024-03-12T03:33:27.382050+0000","last_became_peered":"2024-03-12T03:33:27.382050+0000","last_unstale":"2024-03-12T03:34:48.147609+0000","last_undegraded":"2024-03-12T03:34:48.147609+0000","last_fullsized":"2024-03-12T03:34:48.147609+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144353+0000","last_change":"2024-03-12T03:33:25.180282+0000","last_active":"2024-03-12T03:34:48.144353+0000","last_peered":"2024-03-12T03:34:48.144353+0000","last_clean":"2024-03-12T03:34:48.144353+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:34:48.144353+0000","last_undegraded":"2024-03-12T03:34:48.144353+0000","last_fullsized":"2024-03-12T03:34:48.144353+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148220+0000","last_change":"2024-03-12T03:34:41.143075+0000","last_active":"2024-03-12T03:34:48.148220+0000","last_peered":"2024-03-12T03:34:48.148220+0000","last_clean":"2024-03-12T03:34:48.148220+0000","last_became_active":"2024-03-12T03:33:21.572876+0000","last_became_peered":"2024-03-12T03:33:21.572876+0000","last_unstale":"2024-03-12T03:34:48.148220+0000","last_undegraded":"2024-03-12T03:34:48.148220+0000","last_fullsized":"2024-03-12T03:34:48.148220+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.142994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.165643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":134,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147913+0000","last_change":"2024-03-12T03:33:23.161838+0000","last_active":"2024-03-12T03:34:48.147913+0000","last_peered":"2024-03-12T03:34:48.147913+0000","last_clean":"2024-03-12T03:34:48.147913+0000","last_became_active":"2024-03-12T03:33:23.161549+0000","last_became_peered":"2024-03-12T03:33:23.161549+0000","last_unstale":"2024-03-12T03:34:48.147913+0000","last_undegraded":"2024-03-12T03:34:48.147913+0000","last_fullsized":"2024-03-12T03:34:48.147913+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818555+0000","last_change":"2024-03-12T03:33:27.189571+0000","last_active":"2024-03-12T03:34:48.818555+0000","last_peered":"2024-03-12T03:34:48.818555+0000","last_clean":"2024-03-12T03:34:48.818555+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:34:48.818555+0000","last_undegraded":"2024-03-12T03:34:48.818555+0000","last_fullsized":"2024-03-12T03:34:48.818555+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:50.189316+0000","last_change":"2024-03-12T03:34:50.189337+0000","last_active":"2024-03-12T03:34:50.189316+0000","last_peered":"2024-03-12T03:34:50.189316+0000","last_clean":"2024-03-12T03:34:50.189316+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:34:50.189316+0000","last_undegraded":"2024-03-12T03:34:50.189316+0000","last_fullsized":"2024-03-12T03:34:50.189316+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:50.189256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:50.189256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:52.054548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:51.188040+0000","last_change":"2024-03-12T03:34:51.188063+0000","last_active":"2024-03-12T03:34:51.188040+0000","last_peered":"2024-03-12T03:34:51.188040+0000","last_clean":"2024-03-12T03:34:51.188040+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:34:51.188040+0000","last_undegraded":"2024-03-12T03:34:51.188040+0000","last_fullsized":"2024-03-12T03:34:51.188040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:34:51.187976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:08.066983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'21","reported_seq":101,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144620+0000","last_change":"2024-03-12T03:33:23.163157+0000","last_active":"2024-03-12T03:34:48.144620+0000","last_peered":"2024-03-12T03:34:48.144620+0000","last_clean":"2024-03-12T03:34:48.144620+0000","last_became_active":"2024-03-12T03:33:23.162749+0000","last_became_peered":"2024-03-12T03:33:23.162749+0000","last_unstale":"2024-03-12T03:34:48.144620+0000","last_undegraded":"2024-03-12T03:34:48.144620+0000","last_fullsized":"2024-03-12T03:34:48.144620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147170+0000","last_change":"2024-03-12T03:33:27.188261+0000","last_active":"2024-03-12T03:34:48.147170+0000","last_peered":"2024-03-12T03:34:48.147170+0000","last_clean":"2024-03-12T03:34:48.147170+0000","last_became_active":"2024-03-12T03:33:27.187943+0000","last_became_peered":"2024-03-12T03:33:27.187943+0000","last_unstale":"2024-03-12T03:34:48.147170+0000","last_undegraded":"2024-03-12T03:34:48.147170+0000","last_fullsized":"2024-03-12T03:34:48.147170+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.815395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818972+0000","last_change":"2024-03-12T03:33:25.182503+0000","last_active":"2024-03-12T03:34:48.818972+0000","last_peered":"2024-03-12T03:34:48.818972+0000","last_clean":"2024-03-12T03:34:48.818972+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:34:48.818972+0000","last_undegraded":"2024-03-12T03:34:48.818972+0000","last_fullsized":"2024-03-12T03:34:48.818972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:49.175148+0000","last_change":"2024-03-12T03:34:49.175167+0000","last_active":"2024-03-12T03:34:49.175148+0000","last_peered":"2024-03-12T03:34:49.175148+0000","last_clean":"2024-03-12T03:34:49.175148+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:34:49.175148+0000","last_undegraded":"2024-03-12T03:34:49.175148+0000","last_fullsized":"2024-03-12T03:34:49.175148+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.175091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.175091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:59.268389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"86'14","reported_seq":108,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144629+0000","last_change":"2024-03-12T03:34:23.795189+0000","last_active":"2024-03-12T03:34:48.144629+0000","last_peered":"2024-03-12T03:34:48.144629+0000","last_clean":"2024-03-12T03:34:48.144629+0000","last_became_active":"2024-03-12T03:33:50.155908+0000","last_became_peered":"2024-03-12T03:33:50.155908+0000","last_unstale":"2024-03-12T03:34:48.144629+0000","last_undegraded":"2024-03-12T03:34:48.144629+0000","last_fullsized":"2024-03-12T03:34:48.144629+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.795040+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:35.995794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145032+0000","last_change":"2024-03-12T03:33:27.380906+0000","last_active":"2024-03-12T03:34:48.145032+0000","last_peered":"2024-03-12T03:34:48.145032+0000","last_clean":"2024-03-12T03:34:48.145032+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:34:48.145032+0000","last_undegraded":"2024-03-12T03:34:48.145032+0000","last_fullsized":"2024-03-12T03:34:48.145032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819849+0000","last_change":"2024-03-12T03:34:26.233162+0000","last_active":"2024-03-12T03:34:48.819849+0000","last_peered":"2024-03-12T03:34:48.819849+0000","last_clean":"2024-03-12T03:34:48.819849+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:34:48.819849+0000","last_undegraded":"2024-03-12T03:34:48.819849+0000","last_fullsized":"2024-03-12T03:34:48.819849+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:26.233072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:26.233072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.096475+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:50.784030+0000","last_change":"2024-03-12T03:34:50.784072+0000","last_active":"2024-03-12T03:34:50.784030+0000","last_peered":"2024-03-12T03:34:50.784030+0000","last_clean":"2024-03-12T03:34:50.784030+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:34:50.784030+0000","last_undegraded":"2024-03-12T03:34:50.784030+0000","last_fullsized":"2024-03-12T03:34:50.784030+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:50.783950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:50.783950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.176860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818791+0000","last_change":"2024-03-12T03:33:23.161120+0000","last_active":"2024-03-12T03:34:48.818791+0000","last_peered":"2024-03-12T03:34:48.818791+0000","last_clean":"2024-03-12T03:34:48.818791+0000","last_became_active":"2024-03-12T03:33:23.160702+0000","last_became_peered":"2024-03-12T03:33:23.160702+0000","last_unstale":"2024-03-12T03:34:48.818791+0000","last_undegraded":"2024-03-12T03:34:48.818791+0000","last_fullsized":"2024-03-12T03:34:48.818791+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148426+0000","last_change":"2024-03-12T03:33:27.381440+0000","last_active":"2024-03-12T03:34:48.148426+0000","last_peered":"2024-03-12T03:34:48.148426+0000","last_clean":"2024-03-12T03:34:48.148426+0000","last_became_active":"2024-03-12T03:33:27.380880+0000","last_became_peered":"2024-03-12T03:33:27.380880+0000","last_unstale":"2024-03-12T03:34:48.148426+0000","last_undegraded":"2024-03-12T03:34:48.148426+0000","last_fullsized":"2024-03-12T03:34:48.148426+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.898172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144737+0000","last_change":"2024-03-12T03:34:36.949207+0000","last_active":"2024-03-12T03:34:48.144737+0000","last_peered":"2024-03-12T03:34:48.144737+0000","last_clean":"2024-03-12T03:34:48.144737+0000","last_became_active":"2024-03-12T03:33:25.181087+0000","last_became_peered":"2024-03-12T03:33:25.181087+0000","last_unstale":"2024-03-12T03:34:48.144737+0000","last_undegraded":"2024-03-12T03:34:48.144737+0000","last_fullsized":"2024-03-12T03:34:48.144737+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.949124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.949124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:57.461161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144627+0000","last_change":"2024-03-12T03:34:44.029203+0000","last_active":"2024-03-12T03:34:48.144627+0000","last_peered":"2024-03-12T03:34:48.144627+0000","last_clean":"2024-03-12T03:34:48.144627+0000","last_became_active":"2024-03-12T03:33:21.573218+0000","last_became_peered":"2024-03-12T03:33:21.573218+0000","last_unstale":"2024-03-12T03:34:48.144627+0000","last_undegraded":"2024-03-12T03:34:48.144627+0000","last_fullsized":"2024-03-12T03:34:48.144627+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:44.029060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.029060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.635332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":96,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147725+0000","last_change":"2024-03-12T03:33:23.162704+0000","last_active":"2024-03-12T03:34:48.147725+0000","last_peered":"2024-03-12T03:34:48.147725+0000","last_clean":"2024-03-12T03:34:48.147725+0000","last_became_active":"2024-03-12T03:33:23.162427+0000","last_became_peered":"2024-03-12T03:33:23.162427+0000","last_unstale":"2024-03-12T03:34:48.147725+0000","last_undegraded":"2024-03-12T03:34:48.147725+0000","last_fullsized":"2024-03-12T03:34:48.147725+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.329189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819143+0000","last_change":"2024-03-12T03:34:37.658704+0000","last_active":"2024-03-12T03:34:48.819143+0000","last_peered":"2024-03-12T03:34:48.819143+0000","last_clean":"2024-03-12T03:34:48.819143+0000","last_became_active":"2024-03-12T03:33:27.189152+0000","last_became_peered":"2024-03-12T03:33:27.189152+0000","last_unstale":"2024-03-12T03:34:48.819143+0000","last_undegraded":"2024-03-12T03:34:48.819143+0000","last_fullsized":"2024-03-12T03:34:48.819143+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:34:37.658602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:37.658602+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:38.697166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":95,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.159237+0000","last_change":"2024-03-12T03:34:33.696207+0000","last_active":"2024-03-12T03:34:48.159237+0000","last_peered":"2024-03-12T03:34:48.159237+0000","last_clean":"2024-03-12T03:34:48.159237+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:34:48.159237+0000","last_undegraded":"2024-03-12T03:34:48.159237+0000","last_fullsized":"2024-03-12T03:34:48.159237+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:33.696102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:33.696102+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.429139+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819746+0000","last_change":"2024-03-12T03:34:43.324075+0000","last_active":"2024-03-12T03:34:48.819746+0000","last_peered":"2024-03-12T03:34:48.819746+0000","last_clean":"2024-03-12T03:34:48.819746+0000","last_became_active":"2024-03-12T03:33:21.097218+0000","last_became_peered":"2024-03-12T03:33:21.097218+0000","last_unstale":"2024-03-12T03:34:48.819746+0000","last_undegraded":"2024-03-12T03:34:48.819746+0000","last_fullsized":"2024-03-12T03:34:48.819746+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:34:43.323991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.323991+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:05.428737+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'17","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819891+0000","last_change":"2024-03-12T03:33:23.553096+0000","last_active":"2024-03-12T03:34:48.819891+0000","last_peered":"2024-03-12T03:34:48.819891+0000","last_clean":"2024-03-12T03:34:48.819891+0000","last_became_active":"2024-03-12T03:33:23.552812+0000","last_became_peered":"2024-03-12T03:33:23.552812+0000","last_unstale":"2024-03-12T03:34:48.819891+0000","last_undegraded":"2024-03-12T03:34:48.819891+0000","last_fullsized":"2024-03-12T03:34:48.819891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820035+0000","last_change":"2024-03-12T03:33:27.184475+0000","last_active":"2024-03-12T03:34:48.820035+0000","last_peered":"2024-03-12T03:34:48.820035+0000","last_clean":"2024-03-12T03:34:48.820035+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:34:48.820035+0000","last_undegraded":"2024-03-12T03:34:48.820035+0000","last_fullsized":"2024-03-12T03:34:48.820035+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:52.246462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820105+0000","last_change":"2024-03-12T03:34:47.832795+0000","last_active":"2024-03-12T03:34:48.820105+0000","last_peered":"2024-03-12T03:34:48.820105+0000","last_clean":"2024-03-12T03:34:48.820105+0000","last_became_active":"2024-03-12T03:33:25.178346+0000","last_became_peered":"2024-03-12T03:33:25.178346+0000","last_unstale":"2024-03-12T03:34:48.820105+0000","last_undegraded":"2024-03-12T03:34:48.820105+0000","last_fullsized":"2024-03-12T03:34:48.820105+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.832716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.832716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.392333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147332+0000","last_change":"2024-03-12T03:33:21.095860+0000","last_active":"2024-03-12T03:34:48.147332+0000","last_peered":"2024-03-12T03:34:48.147332+0000","last_clean":"2024-03-12T03:34:48.147332+0000","last_became_active":"2024-03-12T03:33:21.095556+0000","last_became_peered":"2024-03-12T03:33:21.095556+0000","last_unstale":"2024-03-12T03:34:48.147332+0000","last_undegraded":"2024-03-12T03:34:48.147332+0000","last_fullsized":"2024-03-12T03:34:48.147332+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.291634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"94'23","reported_seq":106,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:52.730057+0000","last_change":"2024-03-12T03:34:52.730098+0000","last_active":"2024-03-12T03:34:52.730057+0000","last_peered":"2024-03-12T03:34:52.730057+0000","last_clean":"2024-03-12T03:34:52.730057+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:34:52.730057+0000","last_undegraded":"2024-03-12T03:34:52.730057+0000","last_fullsized":"2024-03-12T03:34:52.730057+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'23","last_scrub_stamp":"2024-03-12T03:34:52.729983+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:34:52.729983+0000","last_clean_scrub_stamp":"2024-03-12T03:34:52.729983+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.107241+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818843+0000","last_change":"2024-03-12T03:33:27.190023+0000","last_active":"2024-03-12T03:34:48.818843+0000","last_peered":"2024-03-12T03:34:48.818843+0000","last_clean":"2024-03-12T03:34:48.818843+0000","last_became_active":"2024-03-12T03:33:27.189696+0000","last_became_peered":"2024-03-12T03:33:27.189696+0000","last_unstale":"2024-03-12T03:34:48.818843+0000","last_undegraded":"2024-03-12T03:34:48.818843+0000","last_fullsized":"2024-03-12T03:34:48.818843+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.530688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.158608+0000","last_change":"2024-03-12T03:34:36.701692+0000","last_active":"2024-03-12T03:34:48.158608+0000","last_peered":"2024-03-12T03:34:48.158608+0000","last_clean":"2024-03-12T03:34:48.158608+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:34:48.158608+0000","last_undegraded":"2024-03-12T03:34:48.158608+0000","last_fullsized":"2024-03-12T03:34:48.158608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.701611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.701611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.280217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819714+0000","last_change":"2024-03-12T03:34:32.746113+0000","last_active":"2024-03-12T03:34:48.819714+0000","last_peered":"2024-03-12T03:34:48.819714+0000","last_clean":"2024-03-12T03:34:48.819714+0000","last_became_active":"2024-03-12T03:33:21.096560+0000","last_became_peered":"2024-03-12T03:33:21.096560+0000","last_unstale":"2024-03-12T03:34:48.819714+0000","last_undegraded":"2024-03-12T03:34:48.819714+0000","last_fullsized":"2024-03-12T03:34:48.819714+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:32.746027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:32.746027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:33.564397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"90'6","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147341+0000","last_change":"2024-03-12T03:34:34.274478+0000","last_active":"2024-03-12T03:34:48.147341+0000","last_peered":"2024-03-12T03:34:48.147341+0000","last_clean":"2024-03-12T03:34:48.147341+0000","last_became_active":"2024-03-12T03:33:23.161801+0000","last_became_peered":"2024-03-12T03:33:23.161801+0000","last_unstale":"2024-03-12T03:34:48.147341+0000","last_undegraded":"2024-03-12T03:34:48.147341+0000","last_fullsized":"2024-03-12T03:34:48.147341+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.274353+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.042399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147514+0000","last_change":"2024-03-12T03:34:40.191618+0000","last_active":"2024-03-12T03:34:48.147514+0000","last_peered":"2024-03-12T03:34:48.147514+0000","last_clean":"2024-03-12T03:34:48.147514+0000","last_became_active":"2024-03-12T03:33:27.381110+0000","last_became_peered":"2024-03-12T03:33:27.381110+0000","last_unstale":"2024-03-12T03:34:48.147514+0000","last_undegraded":"2024-03-12T03:34:48.147514+0000","last_fullsized":"2024-03-12T03:34:48.147514+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.191538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.480916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:53.878562+0000","last_change":"2024-03-12T03:34:53.878589+0000","last_active":"2024-03-12T03:34:53.878562+0000","last_peered":"2024-03-12T03:34:53.878562+0000","last_clean":"2024-03-12T03:34:53.878562+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:34:53.878562+0000","last_undegraded":"2024-03-12T03:34:53.878562+0000","last_fullsized":"2024-03-12T03:34:53.878562+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:53.878484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:53.878484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.602882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147736+0000","last_change":"2024-03-12T03:34:25.171761+0000","last_active":"2024-03-12T03:34:48.147736+0000","last_peered":"2024-03-12T03:34:48.147736+0000","last_clean":"2024-03-12T03:34:48.147736+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:34:48.147736+0000","last_undegraded":"2024-03-12T03:34:48.147736+0000","last_fullsized":"2024-03-12T03:34:48.147736+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:25.171622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.171622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:37.280670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'18","reported_seq":93,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147525+0000","last_change":"2024-03-12T03:33:23.552489+0000","last_active":"2024-03-12T03:34:48.147525+0000","last_peered":"2024-03-12T03:34:48.147525+0000","last_clean":"2024-03-12T03:34:48.147525+0000","last_became_active":"2024-03-12T03:33:23.552210+0000","last_became_peered":"2024-03-12T03:33:23.552210+0000","last_unstale":"2024-03-12T03:34:48.147525+0000","last_undegraded":"2024-03-12T03:34:48.147525+0000","last_fullsized":"2024-03-12T03:34:48.147525+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818700+0000","last_change":"2024-03-12T03:33:27.193134+0000","last_active":"2024-03-12T03:34:48.818700+0000","last_peered":"2024-03-12T03:34:48.818700+0000","last_clean":"2024-03-12T03:34:48.818700+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:34:48.818700+0000","last_undegraded":"2024-03-12T03:34:48.818700+0000","last_fullsized":"2024-03-12T03:34:48.818700+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147678+0000","last_change":"2024-03-12T03:34:45.089928+0000","last_active":"2024-03-12T03:34:48.147678+0000","last_peered":"2024-03-12T03:34:48.147678+0000","last_clean":"2024-03-12T03:34:48.147678+0000","last_became_active":"2024-03-12T03:33:25.183725+0000","last_became_peered":"2024-03-12T03:33:25.183725+0000","last_unstale":"2024-03-12T03:34:48.147678+0000","last_undegraded":"2024-03-12T03:34:48.147678+0000","last_fullsized":"2024-03-12T03:34:48.147678+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:45.089762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:45.089762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.500637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819348+0000","last_change":"2024-03-12T03:34:41.698397+0000","last_active":"2024-03-12T03:34:48.819348+0000","last_peered":"2024-03-12T03:34:48.819348+0000","last_clean":"2024-03-12T03:34:48.819348+0000","last_became_active":"2024-03-12T03:33:21.107037+0000","last_became_peered":"2024-03-12T03:33:21.107037+0000","last_unstale":"2024-03-12T03:34:48.819348+0000","last_undegraded":"2024-03-12T03:34:48.819348+0000","last_fullsized":"2024-03-12T03:34:48.819348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.698268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.698268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:05.134034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820148+0000","last_change":"2024-03-12T03:33:23.161686+0000","last_active":"2024-03-12T03:34:48.820148+0000","last_peered":"2024-03-12T03:34:48.820148+0000","last_clean":"2024-03-12T03:34:48.820148+0000","last_became_active":"2024-03-12T03:33:23.160818+0000","last_became_peered":"2024-03-12T03:33:23.160818+0000","last_unstale":"2024-03-12T03:34:48.820148+0000","last_undegraded":"2024-03-12T03:34:48.820148+0000","last_fullsized":"2024-03-12T03:34:48.820148+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.366562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147313+0000","last_change":"2024-03-12T03:33:27.192215+0000","last_active":"2024-03-12T03:34:48.147313+0000","last_peered":"2024-03-12T03:34:48.147313+0000","last_clean":"2024-03-12T03:34:48.147313+0000","last_became_active":"2024-03-12T03:33:27.191710+0000","last_became_peered":"2024-03-12T03:33:27.191710+0000","last_unstale":"2024-03-12T03:34:48.147313+0000","last_undegraded":"2024-03-12T03:34:48.147313+0000","last_fullsized":"2024-03-12T03:34:48.147313+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819961+0000","last_change":"2024-03-12T03:34:33.258733+0000","last_active":"2024-03-12T03:34:48.819961+0000","last_peered":"2024-03-12T03:34:48.819961+0000","last_clean":"2024-03-12T03:34:48.819961+0000","last_became_active":"2024-03-12T03:33:25.182004+0000","last_became_peered":"2024-03-12T03:33:25.182004+0000","last_unstale":"2024-03-12T03:34:48.819961+0000","last_undegraded":"2024-03-12T03:34:48.819961+0000","last_fullsized":"2024-03-12T03:34:48.819961+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:33.258626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:33.258626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.420748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:50.076875+0000","last_change":"2024-03-12T03:34:50.076895+0000","last_active":"2024-03-12T03:34:50.076875+0000","last_peered":"2024-03-12T03:34:50.076875+0000","last_clean":"2024-03-12T03:34:50.076875+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:34:50.076875+0000","last_undegraded":"2024-03-12T03:34:50.076875+0000","last_fullsized":"2024-03-12T03:34:50.076875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:50.076814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:50.076814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:58.283856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144708+0000","last_change":"2024-03-12T03:34:34.953658+0000","last_active":"2024-03-12T03:34:48.144708+0000","last_peered":"2024-03-12T03:34:48.144708+0000","last_clean":"2024-03-12T03:34:48.144708+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:34:48.144708+0000","last_undegraded":"2024-03-12T03:34:48.144708+0000","last_fullsized":"2024-03-12T03:34:48.144708+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:34:34.953565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.953565+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.161677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819583+0000","last_change":"2024-03-12T03:34:34.718591+0000","last_active":"2024-03-12T03:34:48.819583+0000","last_peered":"2024-03-12T03:34:48.819583+0000","last_clean":"2024-03-12T03:34:48.819583+0000","last_became_active":"2024-03-12T03:33:27.194094+0000","last_became_peered":"2024-03-12T03:33:27.194094+0000","last_unstale":"2024-03-12T03:34:48.819583+0000","last_undegraded":"2024-03-12T03:34:48.819583+0000","last_fullsized":"2024-03-12T03:34:48.819583+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:34.718509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.718509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.921678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147766+0000","last_change":"2024-03-12T03:34:38.139709+0000","last_active":"2024-03-12T03:34:48.147766+0000","last_peered":"2024-03-12T03:34:48.147766+0000","last_clean":"2024-03-12T03:34:48.147766+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:34:48.147766+0000","last_undegraded":"2024-03-12T03:34:48.147766+0000","last_fullsized":"2024-03-12T03:34:48.147766+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:38.139582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.139582+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:39.751323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819095+0000","last_change":"2024-03-12T03:34:38.663634+0000","last_active":"2024-03-12T03:34:48.819095+0000","last_peered":"2024-03-12T03:34:48.819095+0000","last_clean":"2024-03-12T03:34:48.819095+0000","last_became_active":"2024-03-12T03:33:21.573244+0000","last_became_peered":"2024-03-12T03:33:21.573244+0000","last_unstale":"2024-03-12T03:34:48.819095+0000","last_undegraded":"2024-03-12T03:34:48.819095+0000","last_fullsized":"2024-03-12T03:34:48.819095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:34:38.663542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.663542+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:59.053797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":100,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147785+0000","last_change":"2024-03-12T03:33:23.552006+0000","last_active":"2024-03-12T03:34:48.147785+0000","last_peered":"2024-03-12T03:34:48.147785+0000","last_clean":"2024-03-12T03:34:48.147785+0000","last_became_active":"2024-03-12T03:33:23.551706+0000","last_became_peered":"2024-03-12T03:33:23.551706+0000","last_unstale":"2024-03-12T03:34:48.147785+0000","last_undegraded":"2024-03-12T03:34:48.147785+0000","last_fullsized":"2024-03-12T03:34:48.147785+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145312+0000","last_change":"2024-03-12T03:33:27.185209+0000","last_active":"2024-03-12T03:34:48.145312+0000","last_peered":"2024-03-12T03:34:48.145312+0000","last_clean":"2024-03-12T03:34:48.145312+0000","last_became_active":"2024-03-12T03:33:27.184916+0000","last_became_peered":"2024-03-12T03:33:27.184916+0000","last_unstale":"2024-03-12T03:34:48.145312+0000","last_undegraded":"2024-03-12T03:34:48.145312+0000","last_fullsized":"2024-03-12T03:34:48.145312+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148098+0000","last_change":"2024-03-12T03:33:25.179522+0000","last_active":"2024-03-12T03:34:48.148098+0000","last_peered":"2024-03-12T03:34:48.148098+0000","last_clean":"2024-03-12T03:34:48.148098+0000","last_became_active":"2024-03-12T03:33:25.179158+0000","last_became_peered":"2024-03-12T03:33:25.179158+0000","last_unstale":"2024-03-12T03:34:48.148098+0000","last_undegraded":"2024-03-12T03:34:48.148098+0000","last_fullsized":"2024-03-12T03:34:48.148098+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819199+0000","last_change":"2024-03-12T03:34:40.753233+0000","last_active":"2024-03-12T03:34:48.819199+0000","last_peered":"2024-03-12T03:34:48.819199+0000","last_clean":"2024-03-12T03:34:48.819199+0000","last_became_active":"2024-03-12T03:33:21.101819+0000","last_became_peered":"2024-03-12T03:33:21.101819+0000","last_unstale":"2024-03-12T03:34:48.819199+0000","last_undegraded":"2024-03-12T03:34:48.819199+0000","last_fullsized":"2024-03-12T03:34:48.819199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.753153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.816911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":97,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147821+0000","last_change":"2024-03-12T03:34:29.083183+0000","last_active":"2024-03-12T03:34:48.147821+0000","last_peered":"2024-03-12T03:34:48.147821+0000","last_clean":"2024-03-12T03:34:48.147821+0000","last_became_active":"2024-03-12T03:33:23.553480+0000","last_became_peered":"2024-03-12T03:33:23.553480+0000","last_unstale":"2024-03-12T03:34:48.147821+0000","last_undegraded":"2024-03-12T03:34:48.147821+0000","last_fullsized":"2024-03-12T03:34:48.147821+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:34:29.083063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:29.083063+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:45.174021+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819151+0000","last_change":"2024-03-12T03:34:43.039900+0000","last_active":"2024-03-12T03:34:48.819151+0000","last_peered":"2024-03-12T03:34:48.819151+0000","last_clean":"2024-03-12T03:34:48.819151+0000","last_became_active":"2024-03-12T03:33:27.187657+0000","last_became_peered":"2024-03-12T03:33:27.187657+0000","last_unstale":"2024-03-12T03:34:48.819151+0000","last_undegraded":"2024-03-12T03:34:48.819151+0000","last_fullsized":"2024-03-12T03:34:48.819151+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:43.039799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.039799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:01.927684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148407+0000","last_change":"2024-03-12T03:33:25.175313+0000","last_active":"2024-03-12T03:34:48.148407+0000","last_peered":"2024-03-12T03:34:48.148407+0000","last_clean":"2024-03-12T03:34:48.148407+0000","last_became_active":"2024-03-12T03:33:25.174953+0000","last_became_peered":"2024-03-12T03:33:25.174953+0000","last_unstale":"2024-03-12T03:34:48.148407+0000","last_undegraded":"2024-03-12T03:34:48.148407+0000","last_fullsized":"2024-03-12T03:34:48.148407+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:52.848443+0000","last_change":"2024-03-12T03:34:52.848468+0000","last_active":"2024-03-12T03:34:52.848443+0000","last_peered":"2024-03-12T03:34:52.848443+0000","last_clean":"2024-03-12T03:34:52.848443+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:34:52.848443+0000","last_undegraded":"2024-03-12T03:34:52.848443+0000","last_fullsized":"2024-03-12T03:34:52.848443+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:52.848366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:52.848366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.609011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":101,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819952+0000","last_change":"2024-03-12T03:33:23.552425+0000","last_active":"2024-03-12T03:34:48.819952+0000","last_peered":"2024-03-12T03:34:48.819952+0000","last_clean":"2024-03-12T03:34:48.819952+0000","last_became_active":"2024-03-12T03:33:23.552164+0000","last_became_peered":"2024-03-12T03:33:23.552164+0000","last_unstale":"2024-03-12T03:34:48.819952+0000","last_undegraded":"2024-03-12T03:34:48.819952+0000","last_fullsized":"2024-03-12T03:34:48.819952+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.480291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819188+0000","last_change":"2024-03-12T03:34:46.767546+0000","last_active":"2024-03-12T03:34:48.819188+0000","last_peered":"2024-03-12T03:34:48.819188+0000","last_clean":"2024-03-12T03:34:48.819188+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:34:48.819188+0000","last_undegraded":"2024-03-12T03:34:48.819188+0000","last_fullsized":"2024-03-12T03:34:48.819188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:46.767453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:46.767453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:56.661402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147746+0000","last_change":"2024-03-12T03:33:25.181046+0000","last_active":"2024-03-12T03:34:48.147746+0000","last_peered":"2024-03-12T03:34:48.147746+0000","last_clean":"2024-03-12T03:34:48.147746+0000","last_became_active":"2024-03-12T03:33:25.180752+0000","last_became_peered":"2024-03-12T03:33:25.180752+0000","last_unstale":"2024-03-12T03:34:48.147746+0000","last_undegraded":"2024-03-12T03:34:48.147746+0000","last_fullsized":"2024-03-12T03:34:48.147746+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:52.874476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.158563+0000","last_change":"2024-03-12T03:34:26.565344+0000","last_active":"2024-03-12T03:34:48.158563+0000","last_peered":"2024-03-12T03:34:48.158563+0000","last_clean":"2024-03-12T03:34:48.158563+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:34:48.158563+0000","last_undegraded":"2024-03-12T03:34:48.158563+0000","last_fullsized":"2024-03-12T03:34:48.158563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:26.565245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:26.565245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:28.167100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148494+0000","last_change":"2024-03-12T03:33:23.552149+0000","last_active":"2024-03-12T03:34:48.148494+0000","last_peered":"2024-03-12T03:34:48.148494+0000","last_clean":"2024-03-12T03:34:48.148494+0000","last_became_active":"2024-03-12T03:33:23.551773+0000","last_became_peered":"2024-03-12T03:33:23.551773+0000","last_unstale":"2024-03-12T03:34:48.148494+0000","last_undegraded":"2024-03-12T03:34:48.148494+0000","last_fullsized":"2024-03-12T03:34:48.148494+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147234+0000","last_change":"2024-03-12T03:33:27.382928+0000","last_active":"2024-03-12T03:34:48.147234+0000","last_peered":"2024-03-12T03:34:48.147234+0000","last_clean":"2024-03-12T03:34:48.147234+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:34:48.147234+0000","last_undegraded":"2024-03-12T03:34:48.147234+0000","last_fullsized":"2024-03-12T03:34:48.147234+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.104295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819589+0000","last_change":"2024-03-12T03:33:21.107807+0000","last_active":"2024-03-12T03:34:48.819589+0000","last_peered":"2024-03-12T03:34:48.819589+0000","last_clean":"2024-03-12T03:34:48.819589+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:34:48.819589+0000","last_undegraded":"2024-03-12T03:34:48.819589+0000","last_fullsized":"2024-03-12T03:34:48.819589+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"90'36","reported_seq":127,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147179+0000","last_change":"2024-03-12T03:34:39.184089+0000","last_active":"2024-03-12T03:34:48.147179+0000","last_peered":"2024-03-12T03:34:48.147179+0000","last_clean":"2024-03-12T03:34:48.147179+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:34:48.147179+0000","last_undegraded":"2024-03-12T03:34:48.147179+0000","last_fullsized":"2024-03-12T03:34:48.147179+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.183984+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.564093+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818994+0000","last_change":"2024-03-12T03:34:39.659403+0000","last_active":"2024-03-12T03:34:48.818994+0000","last_peered":"2024-03-12T03:34:48.818994+0000","last_clean":"2024-03-12T03:34:48.818994+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:34:48.818994+0000","last_undegraded":"2024-03-12T03:34:48.818994+0000","last_fullsized":"2024-03-12T03:34:48.818994+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:39.659277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.659277+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:08.903088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147582+0000","last_change":"2024-03-12T03:33:27.191091+0000","last_active":"2024-03-12T03:34:48.147582+0000","last_peered":"2024-03-12T03:34:48.147582+0000","last_clean":"2024-03-12T03:34:48.147582+0000","last_became_active":"2024-03-12T03:33:27.190753+0000","last_became_peered":"2024-03-12T03:33:27.190753+0000","last_unstale":"2024-03-12T03:34:48.147582+0000","last_undegraded":"2024-03-12T03:34:48.147582+0000","last_fullsized":"2024-03-12T03:34:48.147582+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.061395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819447+0000","last_change":"2024-03-12T03:34:23.842758+0000","last_active":"2024-03-12T03:34:48.819447+0000","last_peered":"2024-03-12T03:34:48.819447+0000","last_clean":"2024-03-12T03:34:48.819447+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:34:48.819447+0000","last_undegraded":"2024-03-12T03:34:48.819447+0000","last_fullsized":"2024-03-12T03:34:48.819447+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:23.842628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.842628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.823849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":108,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145079+0000","last_change":"2024-03-12T03:33:23.163665+0000","last_active":"2024-03-12T03:34:48.145079+0000","last_peered":"2024-03-12T03:34:48.145079+0000","last_clean":"2024-03-12T03:34:48.145079+0000","last_became_active":"2024-03-12T03:33:23.163351+0000","last_became_peered":"2024-03-12T03:33:23.163351+0000","last_unstale":"2024-03-12T03:34:48.145079+0000","last_undegraded":"2024-03-12T03:34:48.145079+0000","last_fullsized":"2024-03-12T03:34:48.145079+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:51.958851+0000","last_change":"2024-03-12T03:34:51.958873+0000","last_active":"2024-03-12T03:34:51.958851+0000","last_peered":"2024-03-12T03:34:51.958851+0000","last_clean":"2024-03-12T03:34:51.958851+0000","last_became_active":"2024-03-12T03:33:25.185596+0000","last_became_peered":"2024-03-12T03:33:25.185596+0000","last_unstale":"2024-03-12T03:34:51.958851+0000","last_undegraded":"2024-03-12T03:34:51.958851+0000","last_fullsized":"2024-03-12T03:34:51.958851+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:51.958777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:51.958777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:20.123226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818706+0000","last_change":"2024-03-12T03:34:43.716123+0000","last_active":"2024-03-12T03:34:48.818706+0000","last_peered":"2024-03-12T03:34:48.818706+0000","last_clean":"2024-03-12T03:34:48.818706+0000","last_became_active":"2024-03-12T03:33:27.187955+0000","last_became_peered":"2024-03-12T03:33:27.187955+0000","last_unstale":"2024-03-12T03:34:48.818706+0000","last_undegraded":"2024-03-12T03:34:48.818706+0000","last_fullsized":"2024-03-12T03:34:48.818706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:43.716021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.716021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:01.645227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818709+0000","last_change":"2024-03-12T03:33:25.180331+0000","last_active":"2024-03-12T03:34:48.818709+0000","last_peered":"2024-03-12T03:34:48.818709+0000","last_clean":"2024-03-12T03:34:48.818709+0000","last_became_active":"2024-03-12T03:33:25.179593+0000","last_became_peered":"2024-03-12T03:33:25.179593+0000","last_unstale":"2024-03-12T03:34:48.818709+0000","last_undegraded":"2024-03-12T03:34:48.818709+0000","last_fullsized":"2024-03-12T03:34:48.818709+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.050267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819470+0000","last_change":"2024-03-12T03:34:30.792781+0000","last_active":"2024-03-12T03:34:48.819470+0000","last_peered":"2024-03-12T03:34:48.819470+0000","last_clean":"2024-03-12T03:34:48.819470+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:34:48.819470+0000","last_undegraded":"2024-03-12T03:34:48.819470+0000","last_fullsized":"2024-03-12T03:34:48.819470+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:30.792702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.792702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:50.132706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":112,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.158771+0000","last_change":"2024-03-12T03:34:46.722408+0000","last_active":"2024-03-12T03:34:48.158771+0000","last_peered":"2024-03-12T03:34:48.158771+0000","last_clean":"2024-03-12T03:34:48.158771+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:34:48.158771+0000","last_undegraded":"2024-03-12T03:34:48.158771+0000","last_fullsized":"2024-03-12T03:34:48.158771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:34:46.722278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:46.722278+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.145172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818954+0000","last_change":"2024-03-12T03:34:37.737471+0000","last_active":"2024-03-12T03:34:48.818954+0000","last_peered":"2024-03-12T03:34:48.818954+0000","last_clean":"2024-03-12T03:34:48.818954+0000","last_became_active":"2024-03-12T03:33:27.189948+0000","last_became_peered":"2024-03-12T03:33:27.189948+0000","last_unstale":"2024-03-12T03:34:48.818954+0000","last_undegraded":"2024-03-12T03:34:48.818954+0000","last_fullsized":"2024-03-12T03:34:48.818954+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:37.737386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:37.737386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.387230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818751+0000","last_change":"2024-03-12T03:33:25.184715+0000","last_active":"2024-03-12T03:34:48.818751+0000","last_peered":"2024-03-12T03:34:48.818751+0000","last_clean":"2024-03-12T03:34:48.818751+0000","last_became_active":"2024-03-12T03:33:25.184289+0000","last_became_peered":"2024-03-12T03:33:25.184289+0000","last_unstale":"2024-03-12T03:34:48.818751+0000","last_undegraded":"2024-03-12T03:34:48.818751+0000","last_fullsized":"2024-03-12T03:34:48.818751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.160445+0000","last_change":"2024-03-12T03:34:48.160481+0000","last_active":"2024-03-12T03:34:48.160445+0000","last_peered":"2024-03-12T03:34:48.160445+0000","last_clean":"2024-03-12T03:34:48.160445+0000","last_became_active":"2024-03-12T03:33:21.102410+0000","last_became_peered":"2024-03-12T03:33:21.102410+0000","last_unstale":"2024-03-12T03:34:48.160445+0000","last_undegraded":"2024-03-12T03:34:48.160445+0000","last_fullsized":"2024-03-12T03:34:48.160445+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:34:48.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:17.839681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819359+0000","last_change":"2024-03-12T03:33:23.164442+0000","last_active":"2024-03-12T03:34:48.819359+0000","last_peered":"2024-03-12T03:34:48.819359+0000","last_clean":"2024-03-12T03:34:48.819359+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:34:48.819359+0000","last_undegraded":"2024-03-12T03:34:48.819359+0000","last_fullsized":"2024-03-12T03:34:48.819359+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:50.796888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:49.088162+0000","last_change":"2024-03-12T03:34:49.088183+0000","last_active":"2024-03-12T03:34:49.088162+0000","last_peered":"2024-03-12T03:34:49.088162+0000","last_clean":"2024-03-12T03:34:49.088162+0000","last_became_active":"2024-03-12T03:33:27.185041+0000","last_became_peered":"2024-03-12T03:33:27.185041+0000","last_unstale":"2024-03-12T03:34:49.088162+0000","last_undegraded":"2024-03-12T03:34:49.088162+0000","last_fullsized":"2024-03-12T03:34:49.088162+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.088098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.088098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.622130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147376+0000","last_change":"2024-03-12T03:34:22.168794+0000","last_active":"2024-03-12T03:34:48.147376+0000","last_peered":"2024-03-12T03:34:48.147376+0000","last_clean":"2024-03-12T03:34:48.147376+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:34:48.147376+0000","last_undegraded":"2024-03-12T03:34:48.147376+0000","last_fullsized":"2024-03-12T03:34:48.147376+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:34:22.168654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:22.168654+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:48.338217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144560+0000","last_change":"2024-03-12T03:33:21.107222+0000","last_active":"2024-03-12T03:34:48.144560+0000","last_peered":"2024-03-12T03:34:48.144560+0000","last_clean":"2024-03-12T03:34:48.144560+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:34:48.144560+0000","last_undegraded":"2024-03-12T03:34:48.144560+0000","last_fullsized":"2024-03-12T03:34:48.144560+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'35","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:54.301800+0000","last_change":"2024-03-12T03:34:54.301841+0000","last_active":"2024-03-12T03:34:54.301800+0000","last_peered":"2024-03-12T03:34:54.301800+0000","last_clean":"2024-03-12T03:34:54.301800+0000","last_became_active":"2024-03-12T03:33:50.151664+0000","last_became_peered":"2024-03-12T03:33:50.151664+0000","last_unstale":"2024-03-12T03:34:54.301800+0000","last_undegraded":"2024-03-12T03:34:54.301800+0000","last_fullsized":"2024-03-12T03:34:54.301800+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'35","last_scrub_stamp":"2024-03-12T03:34:54.301725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:54.301725+0000","objects_scrubbed":11,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:23.962972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819470+0000","last_change":"2024-03-12T03:34:38.698005+0000","last_active":"2024-03-12T03:34:48.819470+0000","last_peered":"2024-03-12T03:34:48.819470+0000","last_clean":"2024-03-12T03:34:48.819470+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:34:48.819470+0000","last_undegraded":"2024-03-12T03:34:48.819470+0000","last_fullsized":"2024-03-12T03:34:48.819470+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:38.697923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.697923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:07.086680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819534+0000","last_change":"2024-03-12T03:34:40.663082+0000","last_active":"2024-03-12T03:34:48.819534+0000","last_peered":"2024-03-12T03:34:48.819534+0000","last_clean":"2024-03-12T03:34:48.819534+0000","last_became_active":"2024-03-12T03:33:27.380389+0000","last_became_peered":"2024-03-12T03:33:27.380389+0000","last_unstale":"2024-03-12T03:34:48.819534+0000","last_undegraded":"2024-03-12T03:34:48.819534+0000","last_fullsized":"2024-03-12T03:34:48.819534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.663002+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.064026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:51.444266+0000","last_change":"2024-03-12T03:34:51.444303+0000","last_active":"2024-03-12T03:34:51.444266+0000","last_peered":"2024-03-12T03:34:51.444266+0000","last_clean":"2024-03-12T03:34:51.444266+0000","last_became_active":"2024-03-12T03:33:21.572911+0000","last_became_peered":"2024-03-12T03:33:21.572911+0000","last_unstale":"2024-03-12T03:34:51.444266+0000","last_undegraded":"2024-03-12T03:34:51.444266+0000","last_fullsized":"2024-03-12T03:34:51.444266+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:51.444181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:51.444181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.657671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:49.918877+0000","last_change":"2024-03-12T03:34:49.918898+0000","last_active":"2024-03-12T03:34:49.918877+0000","last_peered":"2024-03-12T03:34:49.918877+0000","last_clean":"2024-03-12T03:34:49.918877+0000","last_became_active":"2024-03-12T03:33:20.093064+0000","last_became_peered":"2024-03-12T03:33:20.093064+0000","last_unstale":"2024-03-12T03:34:49.918877+0000","last_undegraded":"2024-03-12T03:34:49.918877+0000","last_fullsized":"2024-03-12T03:34:49.918877+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.918802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.918802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.157514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000516994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":108,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820127+0000","last_change":"2024-03-12T03:34:44.301692+0000","last_active":"2024-03-12T03:34:48.820127+0000","last_peered":"2024-03-12T03:34:48.820127+0000","last_clean":"2024-03-12T03:34:48.820127+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:34:48.820127+0000","last_undegraded":"2024-03-12T03:34:48.820127+0000","last_fullsized":"2024-03-12T03:34:48.820127+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-12T03:34:44.301556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.301556+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.466292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819079+0000","last_change":"2024-03-12T03:34:42.086021+0000","last_active":"2024-03-12T03:34:48.819079+0000","last_peered":"2024-03-12T03:34:48.819079+0000","last_clean":"2024-03-12T03:34:48.819079+0000","last_became_active":"2024-03-12T03:33:25.179022+0000","last_became_peered":"2024-03-12T03:33:25.179022+0000","last_unstale":"2024-03-12T03:34:48.819079+0000","last_undegraded":"2024-03-12T03:34:48.819079+0000","last_fullsized":"2024-03-12T03:34:48.819079+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:42.085936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:42.085936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.119017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147829+0000","last_change":"2024-03-12T03:33:27.188886+0000","last_active":"2024-03-12T03:34:48.147829+0000","last_peered":"2024-03-12T03:34:48.147829+0000","last_clean":"2024-03-12T03:34:48.147829+0000","last_became_active":"2024-03-12T03:33:27.188551+0000","last_became_peered":"2024-03-12T03:33:27.188551+0000","last_unstale":"2024-03-12T03:34:48.147829+0000","last_undegraded":"2024-03-12T03:34:48.147829+0000","last_fullsized":"2024-03-12T03:34:48.147829+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:50.069549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":320,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818814+0000","last_change":"2024-03-12T03:33:29.565156+0000","last_active":"2024-03-12T03:34:48.818814+0000","last_peered":"2024-03-12T03:34:48.818814+0000","last_clean":"2024-03-12T03:34:48.818814+0000","last_became_active":"2024-03-12T03:33:11.319888+0000","last_became_peered":"2024-03-12T03:33:11.319888+0000","last_unstale":"2024-03-12T03:34:48.818814+0000","last_undegraded":"2024-03-12T03:34:48.818814+0000","last_fullsized":"2024-03-12T03:34:48.818814+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-12T03:33:29.565069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:33:29.565069+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147698+0000","last_change":"2024-03-12T03:33:21.101239+0000","last_active":"2024-03-12T03:34:48.147698+0000","last_peered":"2024-03-12T03:34:48.147698+0000","last_clean":"2024-03-12T03:34:48.147698+0000","last_became_active":"2024-03-12T03:33:21.100900+0000","last_became_peered":"2024-03-12T03:33:21.100900+0000","last_unstale":"2024-03-12T03:34:48.147698+0000","last_undegraded":"2024-03-12T03:34:48.147698+0000","last_fullsized":"2024-03-12T03:34:48.147698+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":110,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.159037+0000","last_change":"2024-03-12T03:34:35.664996+0000","last_active":"2024-03-12T03:34:48.159037+0000","last_peered":"2024-03-12T03:34:48.159037+0000","last_clean":"2024-03-12T03:34:48.159037+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:34:48.159037+0000","last_undegraded":"2024-03-12T03:34:48.159037+0000","last_fullsized":"2024-03-12T03:34:48.159037+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:34:35.664876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:35.664876+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:54.461746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819213+0000","last_change":"2024-03-12T03:34:47.796670+0000","last_active":"2024-03-12T03:34:48.819213+0000","last_peered":"2024-03-12T03:34:48.819213+0000","last_clean":"2024-03-12T03:34:48.819213+0000","last_became_active":"2024-03-12T03:33:25.177762+0000","last_became_peered":"2024-03-12T03:33:25.177762+0000","last_unstale":"2024-03-12T03:34:48.819213+0000","last_undegraded":"2024-03-12T03:34:48.819213+0000","last_fullsized":"2024-03-12T03:34:48.819213+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.796561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.796561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.258919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820246+0000","last_change":"2024-03-12T03:34:47.352525+0000","last_active":"2024-03-12T03:34:48.820246+0000","last_peered":"2024-03-12T03:34:48.820246+0000","last_clean":"2024-03-12T03:34:48.820246+0000","last_became_active":"2024-03-12T03:33:27.191878+0000","last_became_peered":"2024-03-12T03:33:27.191878+0000","last_unstale":"2024-03-12T03:34:48.820246+0000","last_undegraded":"2024-03-12T03:34:48.820246+0000","last_fullsized":"2024-03-12T03:34:48.820246+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.352450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.352450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:01.666869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819126+0000","last_change":"2024-03-12T03:33:21.097466+0000","last_active":"2024-03-12T03:34:48.819126+0000","last_peered":"2024-03-12T03:34:48.819126+0000","last_clean":"2024-03-12T03:34:48.819126+0000","last_became_active":"2024-03-12T03:33:21.097086+0000","last_became_peered":"2024-03-12T03:33:21.097086+0000","last_unstale":"2024-03-12T03:34:48.819126+0000","last_undegraded":"2024-03-12T03:34:48.819126+0000","last_fullsized":"2024-03-12T03:34:48.819126+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.630139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":174,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:54.259942+0000","last_change":"2024-03-12T03:34:21.799658+0000","last_active":"2024-03-12T03:34:54.259942+0000","last_peered":"2024-03-12T03:34:54.259942+0000","last_clean":"2024-03-12T03:34:54.259942+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:34:54.259942+0000","last_undegraded":"2024-03-12T03:34:54.259942+0000","last_fullsized":"2024-03-12T03:34:54.259942+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:34:21.799534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:21.799534+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.973279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819685+0000","last_change":"2024-03-12T03:34:39.707369+0000","last_active":"2024-03-12T03:34:48.819685+0000","last_peered":"2024-03-12T03:34:48.819685+0000","last_clean":"2024-03-12T03:34:48.819685+0000","last_became_active":"2024-03-12T03:33:23.168750+0000","last_became_peered":"2024-03-12T03:33:23.168750+0000","last_unstale":"2024-03-12T03:34:48.819685+0000","last_undegraded":"2024-03-12T03:34:48.819685+0000","last_fullsized":"2024-03-12T03:34:48.819685+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:34:39.707264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.707264+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:43.684441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148076+0000","last_change":"2024-03-12T03:33:25.177321+0000","last_active":"2024-03-12T03:34:48.148076+0000","last_peered":"2024-03-12T03:34:48.148076+0000","last_clean":"2024-03-12T03:34:48.148076+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:34:48.148076+0000","last_undegraded":"2024-03-12T03:34:48.148076+0000","last_fullsized":"2024-03-12T03:34:48.148076+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:51.569538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819556+0000","last_change":"2024-03-12T03:34:45.344064+0000","last_active":"2024-03-12T03:34:48.819556+0000","last_peered":"2024-03-12T03:34:48.819556+0000","last_clean":"2024-03-12T03:34:48.819556+0000","last_became_active":"2024-03-12T03:33:27.380458+0000","last_became_peered":"2024-03-12T03:33:27.380458+0000","last_unstale":"2024-03-12T03:34:48.819556+0000","last_undegraded":"2024-03-12T03:34:48.819556+0000","last_fullsized":"2024-03-12T03:34:48.819556+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:45.343945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:45.343945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.292957+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820380+0000","last_change":"2024-03-12T03:34:34.272895+0000","last_active":"2024-03-12T03:34:48.820380+0000","last_peered":"2024-03-12T03:34:48.820380+0000","last_clean":"2024-03-12T03:34:48.820380+0000","last_became_active":"2024-03-12T03:33:21.573003+0000","last_became_peered":"2024-03-12T03:33:21.573003+0000","last_unstale":"2024-03-12T03:34:48.820380+0000","last_undegraded":"2024-03-12T03:34:48.820380+0000","last_fullsized":"2024-03-12T03:34:48.820380+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:34.272759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.272759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:47.242996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":109,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819841+0000","last_change":"2024-03-12T03:33:23.551923+0000","last_active":"2024-03-12T03:34:48.819841+0000","last_peered":"2024-03-12T03:34:48.819841+0000","last_clean":"2024-03-12T03:34:48.819841+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:34:48.819841+0000","last_undegraded":"2024-03-12T03:34:48.819841+0000","last_fullsized":"2024-03-12T03:34:48.819841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:50.306884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.159120+0000","last_change":"2024-03-12T03:34:28.572496+0000","last_active":"2024-03-12T03:34:48.159120+0000","last_peered":"2024-03-12T03:34:48.159120+0000","last_clean":"2024-03-12T03:34:48.159120+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:34:48.159120+0000","last_undegraded":"2024-03-12T03:34:48.159120+0000","last_fullsized":"2024-03-12T03:34:48.159120+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:28.572379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:28.572379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:58.234927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:51.734803+0000","last_change":"2024-03-12T03:34:51.734825+0000","last_active":"2024-03-12T03:34:51.734803+0000","last_peered":"2024-03-12T03:34:51.734803+0000","last_clean":"2024-03-12T03:34:51.734803+0000","last_became_active":"2024-03-12T03:33:27.194266+0000","last_became_peered":"2024-03-12T03:33:27.194266+0000","last_unstale":"2024-03-12T03:34:51.734803+0000","last_undegraded":"2024-03-12T03:34:51.734803+0000","last_fullsized":"2024-03-12T03:34:51.734803+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:51.734742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:51.734742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:12.279921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147637+0000","last_change":"2024-03-12T03:34:41.152579+0000","last_active":"2024-03-12T03:34:48.147637+0000","last_peered":"2024-03-12T03:34:48.147637+0000","last_clean":"2024-03-12T03:34:48.147637+0000","last_became_active":"2024-03-12T03:33:21.573251+0000","last_became_peered":"2024-03-12T03:33:21.573251+0000","last_unstale":"2024-03-12T03:34:48.147637+0000","last_undegraded":"2024-03-12T03:34:48.147637+0000","last_fullsized":"2024-03-12T03:34:48.147637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.152440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.152440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.371887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":96,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820053+0000","last_change":"2024-03-12T03:34:35.282309+0000","last_active":"2024-03-12T03:34:48.820053+0000","last_peered":"2024-03-12T03:34:48.820053+0000","last_clean":"2024-03-12T03:34:48.820053+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:34:48.820053+0000","last_undegraded":"2024-03-12T03:34:48.820053+0000","last_fullsized":"2024-03-12T03:34:48.820053+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:34:35.282203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:35.282203+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.121154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819251+0000","last_change":"2024-03-12T03:34:30.767076+0000","last_active":"2024-03-12T03:34:48.819251+0000","last_peered":"2024-03-12T03:34:48.819251+0000","last_clean":"2024-03-12T03:34:48.819251+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:34:48.819251+0000","last_undegraded":"2024-03-12T03:34:48.819251+0000","last_fullsized":"2024-03-12T03:34:48.819251+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:30.766975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.766975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:45.275798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819077+0000","last_change":"2024-03-12T03:34:44.750269+0000","last_active":"2024-03-12T03:34:48.819077+0000","last_peered":"2024-03-12T03:34:48.819077+0000","last_clean":"2024-03-12T03:34:48.819077+0000","last_became_active":"2024-03-12T03:33:27.383249+0000","last_became_peered":"2024-03-12T03:33:27.383249+0000","last_unstale":"2024-03-12T03:34:48.819077+0000","last_undegraded":"2024-03-12T03:34:48.819077+0000","last_fullsized":"2024-03-12T03:34:48.819077+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:44.750135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.750135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.084645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":66,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819787+0000","last_change":"2024-03-12T03:33:21.098280+0000","last_active":"2024-03-12T03:34:48.819787+0000","last_peered":"2024-03-12T03:34:48.819787+0000","last_clean":"2024-03-12T03:34:48.819787+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:34:48.819787+0000","last_undegraded":"2024-03-12T03:34:48.819787+0000","last_fullsized":"2024-03-12T03:34:48.819787+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":102,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144849+0000","last_change":"2024-03-12T03:34:47.043566+0000","last_active":"2024-03-12T03:34:48.144849+0000","last_peered":"2024-03-12T03:34:48.144849+0000","last_clean":"2024-03-12T03:34:48.144849+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:34:48.144849+0000","last_undegraded":"2024-03-12T03:34:48.144849+0000","last_fullsized":"2024-03-12T03:34:48.144849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-03-12T03:34:47.043433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.043433+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.321445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148144+0000","last_change":"2024-03-12T03:34:25.168128+0000","last_active":"2024-03-12T03:34:48.148144+0000","last_peered":"2024-03-12T03:34:48.148144+0000","last_clean":"2024-03-12T03:34:48.148144+0000","last_became_active":"2024-03-12T03:33:25.177311+0000","last_became_peered":"2024-03-12T03:33:25.177311+0000","last_unstale":"2024-03-12T03:34:48.148144+0000","last_undegraded":"2024-03-12T03:34:48.148144+0000","last_fullsized":"2024-03-12T03:34:48.148144+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:25.168002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.168002+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.036057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144500+0000","last_change":"2024-03-12T03:33:27.188212+0000","last_active":"2024-03-12T03:34:48.144500+0000","last_peered":"2024-03-12T03:34:48.144500+0000","last_clean":"2024-03-12T03:34:48.144500+0000","last_became_active":"2024-03-12T03:33:27.187864+0000","last_became_peered":"2024-03-12T03:33:27.187864+0000","last_unstale":"2024-03-12T03:34:48.144500+0000","last_undegraded":"2024-03-12T03:34:48.144500+0000","last_fullsized":"2024-03-12T03:34:48.144500+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148221+0000","last_change":"2024-03-12T03:34:47.155013+0000","last_active":"2024-03-12T03:34:48.148221+0000","last_peered":"2024-03-12T03:34:48.148221+0000","last_clean":"2024-03-12T03:34:48.148221+0000","last_became_active":"2024-03-12T03:33:21.104430+0000","last_became_peered":"2024-03-12T03:33:21.104430+0000","last_unstale":"2024-03-12T03:34:48.148221+0000","last_undegraded":"2024-03-12T03:34:48.148221+0000","last_fullsized":"2024-03-12T03:34:48.148221+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.154925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.154925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:48.895337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":117,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147818+0000","last_change":"2024-03-12T03:33:23.551957+0000","last_active":"2024-03-12T03:34:48.147818+0000","last_peered":"2024-03-12T03:34:48.147818+0000","last_clean":"2024-03-12T03:34:48.147818+0000","last_became_active":"2024-03-12T03:33:23.551623+0000","last_became_peered":"2024-03-12T03:33:23.551623+0000","last_unstale":"2024-03-12T03:34:48.147818+0000","last_undegraded":"2024-03-12T03:34:48.147818+0000","last_fullsized":"2024-03-12T03:34:48.147818+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145182+0000","last_change":"2024-03-12T03:33:25.182046+0000","last_active":"2024-03-12T03:34:48.145182+0000","last_peered":"2024-03-12T03:34:48.145182+0000","last_clean":"2024-03-12T03:34:48.145182+0000","last_became_active":"2024-03-12T03:33:25.181020+0000","last_became_peered":"2024-03-12T03:33:25.181020+0000","last_unstale":"2024-03-12T03:34:48.145182+0000","last_undegraded":"2024-03-12T03:34:48.145182+0000","last_fullsized":"2024-03-12T03:34:48.145182+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147561+0000","last_change":"2024-03-12T03:33:27.190398+0000","last_active":"2024-03-12T03:34:48.147561+0000","last_peered":"2024-03-12T03:34:48.147561+0000","last_clean":"2024-03-12T03:34:48.147561+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:34:48.147561+0000","last_undegraded":"2024-03-12T03:34:48.147561+0000","last_fullsized":"2024-03-12T03:34:48.147561+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:53.538609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145398+0000","last_change":"2024-03-12T03:33:25.176357+0000","last_active":"2024-03-12T03:34:48.145398+0000","last_peered":"2024-03-12T03:34:48.145398+0000","last_clean":"2024-03-12T03:34:48.145398+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:34:48.145398+0000","last_undegraded":"2024-03-12T03:34:48.145398+0000","last_fullsized":"2024-03-12T03:34:48.145398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147466+0000","last_change":"2024-03-12T03:33:21.104557+0000","last_active":"2024-03-12T03:34:48.147466+0000","last_peered":"2024-03-12T03:34:48.147466+0000","last_clean":"2024-03-12T03:34:48.147466+0000","last_became_active":"2024-03-12T03:33:21.104293+0000","last_became_peered":"2024-03-12T03:33:21.104293+0000","last_unstale":"2024-03-12T03:34:48.147466+0000","last_undegraded":"2024-03-12T03:34:48.147466+0000","last_fullsized":"2024-03-12T03:34:48.147466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819014+0000","last_change":"2024-03-12T03:33:50.166158+0000","last_active":"2024-03-12T03:34:48.819014+0000","last_peered":"2024-03-12T03:34:48.819014+0000","last_clean":"2024-03-12T03:34:48.819014+0000","last_became_active":"2024-03-12T03:33:50.153240+0000","last_became_peered":"2024-03-12T03:33:50.153240+0000","last_unstale":"2024-03-12T03:34:48.819014+0000","last_undegraded":"2024-03-12T03:34:48.819014+0000","last_fullsized":"2024-03-12T03:34:48.819014+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147547+0000","last_change":"2024-03-12T03:33:27.381253+0000","last_active":"2024-03-12T03:34:48.147547+0000","last_peered":"2024-03-12T03:34:48.147547+0000","last_clean":"2024-03-12T03:34:48.147547+0000","last_became_active":"2024-03-12T03:33:27.380896+0000","last_became_peered":"2024-03-12T03:33:27.380896+0000","last_unstale":"2024-03-12T03:34:48.147547+0000","last_undegraded":"2024-03-12T03:34:48.147547+0000","last_fullsized":"2024-03-12T03:34:48.147547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.148070+0000","last_change":"2024-03-12T03:34:36.138975+0000","last_active":"2024-03-12T03:34:48.148070+0000","last_peered":"2024-03-12T03:34:48.148070+0000","last_clean":"2024-03-12T03:34:48.148070+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:34:48.148070+0000","last_undegraded":"2024-03-12T03:34:48.148070+0000","last_fullsized":"2024-03-12T03:34:48.148070+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.138847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.138847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.950469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144623+0000","last_change":"2024-03-12T03:33:21.573917+0000","last_active":"2024-03-12T03:34:48.144623+0000","last_peered":"2024-03-12T03:34:48.144623+0000","last_clean":"2024-03-12T03:34:48.144623+0000","last_became_active":"2024-03-12T03:33:21.573257+0000","last_became_peered":"2024-03-12T03:33:21.573257+0000","last_unstale":"2024-03-12T03:34:48.144623+0000","last_undegraded":"2024-03-12T03:34:48.144623+0000","last_fullsized":"2024-03-12T03:34:48.144623+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:34:49.737276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145215+0000","last_change":"2024-03-12T03:33:23.168956+0000","last_active":"2024-03-12T03:34:48.145215+0000","last_peered":"2024-03-12T03:34:48.145215+0000","last_clean":"2024-03-12T03:34:48.145215+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:34:48.145215+0000","last_undegraded":"2024-03-12T03:34:48.145215+0000","last_fullsized":"2024-03-12T03:34:48.145215+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.819236+0000","last_change":"2024-03-12T03:34:31.142859+0000","last_active":"2024-03-12T03:34:48.819236+0000","last_peered":"2024-03-12T03:34:48.819236+0000","last_clean":"2024-03-12T03:34:48.819236+0000","last_became_active":"2024-03-12T03:33:27.187815+0000","last_became_peered":"2024-03-12T03:33:27.187815+0000","last_unstale":"2024-03-12T03:34:48.819236+0000","last_undegraded":"2024-03-12T03:34:48.819236+0000","last_fullsized":"2024-03-12T03:34:48.819236+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:31.142765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:31.142765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:54.545296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.820450+0000","last_change":"2024-03-12T03:34:30.280067+0000","last_active":"2024-03-12T03:34:48.820450+0000","last_peered":"2024-03-12T03:34:48.820450+0000","last_clean":"2024-03-12T03:34:48.820450+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:34:48.820450+0000","last_undegraded":"2024-03-12T03:34:48.820450+0000","last_fullsized":"2024-03-12T03:34:48.820450+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:30.279966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.279966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.784484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.158678+0000","last_change":"2024-03-12T03:34:23.646936+0000","last_active":"2024-03-12T03:34:48.158678+0000","last_peered":"2024-03-12T03:34:48.158678+0000","last_clean":"2024-03-12T03:34:48.158678+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:34:48.158678+0000","last_undegraded":"2024-03-12T03:34:48.158678+0000","last_fullsized":"2024-03-12T03:34:48.158678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.646819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:33.240250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145216+0000","last_change":"2024-03-12T03:33:23.164300+0000","last_active":"2024-03-12T03:34:48.145216+0000","last_peered":"2024-03-12T03:34:48.145216+0000","last_clean":"2024-03-12T03:34:48.145216+0000","last_became_active":"2024-03-12T03:33:23.163871+0000","last_became_peered":"2024-03-12T03:33:23.163871+0000","last_unstale":"2024-03-12T03:34:48.145216+0000","last_undegraded":"2024-03-12T03:34:48.145216+0000","last_fullsized":"2024-03-12T03:34:48.145216+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.145400+0000","last_change":"2024-03-12T03:33:27.184691+0000","last_active":"2024-03-12T03:34:48.145400+0000","last_peered":"2024-03-12T03:34:48.145400+0000","last_clean":"2024-03-12T03:34:48.145400+0000","last_became_active":"2024-03-12T03:33:27.184405+0000","last_became_peered":"2024-03-12T03:33:27.184405+0000","last_unstale":"2024-03-12T03:34:48.145400+0000","last_undegraded":"2024-03-12T03:34:48.145400+0000","last_fullsized":"2024-03-12T03:34:48.145400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.818785+0000","last_change":"2024-03-12T03:33:25.178413+0000","last_active":"2024-03-12T03:34:48.818785+0000","last_peered":"2024-03-12T03:34:48.818785+0000","last_clean":"2024-03-12T03:34:48.818785+0000","last_became_active":"2024-03-12T03:33:25.177613+0000","last_became_peered":"2024-03-12T03:33:25.177613+0000","last_unstale":"2024-03-12T03:34:48.818785+0000","last_undegraded":"2024-03-12T03:34:48.818785+0000","last_fullsized":"2024-03-12T03:34:48.818785+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:33:24.155889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.147771+0000","last_change":"2024-03-12T03:33:21.097634+0000","last_active":"2024-03-12T03:34:48.147771+0000","last_peered":"2024-03-12T03:34:48.147771+0000","last_clean":"2024-03-12T03:34:48.147771+0000","last_became_active":"2024-03-12T03:33:21.097304+0000","last_became_peered":"2024-03-12T03:33:21.097304+0000","last_unstale":"2024-03-12T03:34:48.147771+0000","last_undegraded":"2024-03-12T03:34:48.147771+0000","last_fullsized":"2024-03-12T03:34:48.147771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":76,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-12T03:34:48.144768+0000","last_change":"2024-03-12T03:33:23.163246+0000","last_active":"2024-03-12T03:34:48.144768+0000","last_peered":"2024-03-12T03:34:48.144768+0000","last_clean":"2024-03-12T03:34:48.144768+0000","last_became_active":"2024-03-12T03:33:23.162657+0000","last_became_peered":"2024-03-12T03:33:23.162657+0000","last_unstale":"2024-03-12T03:34:48.144768+0000","last_undegraded":"2024-03-12T03:34:48.144768+0000","last_fullsized":"2024-03-12T03:34:48.144768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":608,"ondisk_log_size":608,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070488,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28448,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716704,"statfs":{"total":95995035648,"available":95965904896,"internally_reserved":0,"allocated":1466368,"data_stored":974514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332125,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28424,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716728,"statfs":{"total":95995035648,"available":95965929472,"internally_reserved":0,"allocated":1441792,"data_stored":972551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":383813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32608,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712544,"statfs":{"total":95995035648,"available":95961645056,"internally_reserved":0,"allocated":1466368,"data_stored":973768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":851968,"data_stored":383416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117035,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":383510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953512,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":383686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659170,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":382854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:35:01.084 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-12T03:35:01.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd dump 2024-03-12T03:35:01.330 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:01 smithi173 ceph-mon[33544]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 0 op/s 2024-03-12T03:35:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:01 smithi173 ceph-mon[33544]: 5.1 scrub starts 2024-03-12T03:35:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:01 smithi173 ceph-mon[33544]: 5.1 scrub ok 2024-03-12T03:35:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:01 smithi173 ceph-mon[33544]: 4.17 scrub starts 2024-03-12T03:35:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:01 smithi173 ceph-mon[33544]: 4.17 scrub ok 2024-03-12T03:35:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:01 smithi173 ceph-mon[33544]: 6.a scrub starts 2024-03-12T03:35:01.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:01 smithi173 ceph-mon[33544]: 6.a scrub ok 2024-03-12T03:35:01.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:01 smithi173 ceph-mon[33544]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:35:01.747 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[29344]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 0 op/s 2024-03-12T03:35:01.747 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[29344]: 5.1 scrub starts 2024-03-12T03:35:01.747 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[29344]: 5.1 scrub ok 2024-03-12T03:35:01.747 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[29344]: 4.17 scrub starts 2024-03-12T03:35:01.747 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[29344]: 4.17 scrub ok 2024-03-12T03:35:01.747 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[29344]: 6.a scrub starts 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[29344]: 6.a scrub ok 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[29344]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[39949]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 0 op/s 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[39949]: 5.1 scrub starts 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[39949]: 5.1 scrub ok 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[39949]: 4.17 scrub starts 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[39949]: 4.17 scrub ok 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[39949]: 6.a scrub starts 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[39949]: 6.a scrub ok 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-mon[39949]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:35:02.117 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:35:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:35:02.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:35:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:35:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:02 smithi173 ceph-mon[33544]: 6.18 scrub starts 2024-03-12T03:35:02.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:02 smithi173 ceph-mon[33544]: 6.18 scrub ok 2024-03-12T03:35:02.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:02 smithi173 ceph-mon[33544]: 5.7 deep-scrub starts 2024-03-12T03:35:02.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:02 smithi173 ceph-mon[33544]: 5.7 deep-scrub ok 2024-03-12T03:35:02.807 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:02 smithi169 ceph-mon[29344]: 6.18 scrub starts 2024-03-12T03:35:02.807 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:02 smithi169 ceph-mon[29344]: 6.18 scrub ok 2024-03-12T03:35:02.807 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:02 smithi169 ceph-mon[29344]: 5.7 deep-scrub starts 2024-03-12T03:35:02.807 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:02 smithi169 ceph-mon[29344]: 5.7 deep-scrub ok 2024-03-12T03:35:02.807 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:02 smithi169 ceph-mon[39949]: 6.18 scrub starts 2024-03-12T03:35:02.807 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:02 smithi169 ceph-mon[39949]: 6.18 scrub ok 2024-03-12T03:35:02.807 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:02 smithi169 ceph-mon[39949]: 5.7 deep-scrub starts 2024-03-12T03:35:02.807 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:02 smithi169 ceph-mon[39949]: 5.7 deep-scrub ok 2024-03-12T03:35:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:epoch 94 2024-03-12T03:35:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:fsid 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:35:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:created 2024-03-12T03:27:53.319986+0000 2024-03-12T03:35:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:modified 2024-03-12T03:34:48.141400+0000 2024-03-12T03:35:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-12T03:35:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:crush_version 18 2024-03-12T03:35:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:full_ratio 0.95 2024-03-12T03:35:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:backfillfull_ratio 0.9 2024-03-12T03:35:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:nearfull_ratio 0.85 2024-03-12T03:35:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:require_min_compat_client luminous 2024-03-12T03:35:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:min_compat_client luminous 2024-03-12T03:35:03.370 INFO:teuthology.orchestra.run.smithi169.stdout:require_osd_release squid 2024-03-12T03:35:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:stretch_mode_enabled false 2024-03-12T03:35:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-12T03:35:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 70 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-12T03:35:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-12T03:35:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-12T03:35:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-12T03:35:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-12T03:35:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:max_osd 8 2024-03-12T03:35:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.169:6801/676779916 v2:172.21.15.169:6802/676779916 exists,up c44466b0-3112-44dc-b5f3-d6494b958761 2024-03-12T03:35:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.169:6805/130042595 v2:172.21.15.169:6806/130042595 exists,up 184c5c97-a992-4be9-bcd3-c08cb58a0e56 2024-03-12T03:35:03.371 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2 up in weight 1 up_from 18 up_thru 78 down_at 0 last_clean_interval [0,0) v2:172.21.15.169:6809/2751816702 v2:172.21.15.169:6810/2751816702 exists,up beb15e06-1ca4-46a7-b640-efd1991367e8 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3 up in weight 1 up_from 27 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.169:6813/195572266 v2:172.21.15.169:6814/195572266 exists,up a00b4c50-e064-4c9e-85dc-b0a75ea5bf37 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4 up in weight 1 up_from 32 up_thru 78 down_at 0 last_clean_interval [0,0) v2:172.21.15.173:6800/1397212305 v2:172.21.15.173:6801/1397212305 exists,up 3a6121ef-6309-41b7-a964-6772c010b3a0 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5 up in weight 1 up_from 41 up_thru 78 down_at 0 last_clean_interval [0,0) v2:172.21.15.173:6804/3525888417 v2:172.21.15.173:6805/3525888417 exists,up 3c2f6dd9-9784-4a07-8710-55ac3de82964 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6 up in weight 1 up_from 51 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.173:6808/2147192629 v2:172.21.15.173:6809/2147192629 exists,up d7759df7-1978-4d70-94a2-721540804f59 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7 up in weight 1 up_from 59 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.173:6812/4014976876 v2:172.21.15.173:6813/4014976876 exists,up 8843039e-2b17-4230-9cca-5788a2294d8c 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 4.4 [1,5] 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 4.f [1,2] 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 4.1b [1,2] 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/4041747161 expires 2024-03-13T03:33:59.734991+0000 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:6800/2779395456 expires 2024-03-13T03:33:59.734991+0000 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/555866474 expires 2024-03-13T03:33:59.734991+0000 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/342292643 expires 2024-03-13T03:33:59.734991+0000 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/197210445 expires 2024-03-13T03:33:59.734991+0000 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/1430250501 expires 2024-03-13T03:33:59.734991+0000 2024-03-12T03:35:03.372 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/3918553041 expires 2024-03-13T03:28:48.518732+0000 2024-03-12T03:35:03.373 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/1020514981 expires 2024-03-13T03:28:21.868923+0000 2024-03-12T03:35:03.373 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/284191239 expires 2024-03-13T03:28:21.868923+0000 2024-03-12T03:35:03.373 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/1212459137 expires 2024-03-13T03:28:48.518732+0000 2024-03-12T03:35:03.373 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:6800/204892396 expires 2024-03-13T03:28:21.868923+0000 2024-03-12T03:35:03.373 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/2751007072 expires 2024-03-13T03:28:48.518732+0000 2024-03-12T03:35:03.373 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/2349376510 expires 2024-03-13T03:28:21.868923+0000 2024-03-12T03:35:03.373 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:6800/108891420 expires 2024-03-13T03:28:48.518732+0000 2024-03-12T03:35:03.472 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 bash[64222]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-12T03:35:03.472 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 bash[64222]: Writing manifest to image destination 2024-03-12T03:35:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:03 smithi169 ceph-mon[29344]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:03 smithi169 ceph-mon[29344]: 4.11 scrub starts 2024-03-12T03:35:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:03 smithi169 ceph-mon[29344]: 4.11 scrub ok 2024-03-12T03:35:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:03 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:03 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1263926354' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-12T03:35:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:03 smithi169 ceph-mon[39949]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:03 smithi169 ceph-mon[39949]: 4.11 scrub starts 2024-03-12T03:35:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:03 smithi169 ceph-mon[39949]: 4.11 scrub ok 2024-03-12T03:35:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:03 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:03 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1263926354' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-12T03:35:03.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 podman[64222]: 2024-03-12 03:35:03.622482958 +0000 UTC m=+6.357382935 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-12T03:35:03.731 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-mon[33544]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:03.731 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-mon[33544]: 4.11 scrub starts 2024-03-12T03:35:03.731 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-mon[33544]: 4.11 scrub ok 2024-03-12T03:35:03.731 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:03.731 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1263926354' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-12T03:35:03.987 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 podman[64222]: 2024-03-12 03:35:03.729471063 +0000 UTC m=+6.464371082 container create e391b52dfd8cddcbd4382e3ecb2927d0a041a3f2853c962962b9eb7a68618870 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:35:03.988 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 podman[64222]: 2024-03-12 03:35:03.94056893 +0000 UTC m=+6.675468908 container init e391b52dfd8cddcbd4382e3ecb2927d0a041a3f2853c962962b9eb7a68618870 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:35:03.988 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 podman[64222]: 2024-03-12 03:35:03.94343892 +0000 UTC m=+6.678338898 container start e391b52dfd8cddcbd4382e3ecb2927d0a041a3f2853c962962b9eb7a68618870 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:35:03.988 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.947Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-12T03:35:03.988 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.947Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-12T03:35:03.988 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.948Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-12T03:35:03.989 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.948Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-12T03:35:03.989 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.948Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-12T03:35:03.989 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.948Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T03:35:03.989 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-12T03:35:03.989 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=arp 2024-03-12T03:35:03.989 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-12T03:35:03.990 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-12T03:35:03.990 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-12T03:35:03.990 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-12T03:35:03.990 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-12T03:35:03.990 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-12T03:35:03.990 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-12T03:35:03.991 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-12T03:35:03.991 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=edac 2024-03-12T03:35:03.991 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-12T03:35:03.991 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-12T03:35:03.991 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-12T03:35:03.991 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-12T03:35:03.992 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-12T03:35:03.992 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-12T03:35:03.992 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-12T03:35:03.992 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-12T03:35:03.992 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-12T03:35:03.992 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-12T03:35:03.993 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-12T03:35:03.993 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-12T03:35:03.993 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-12T03:35:03.993 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-12T03:35:03.993 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-12T03:35:03.993 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-12T03:35:03.994 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=os 2024-03-12T03:35:03.994 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-12T03:35:03.994 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-12T03:35:03.994 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-12T03:35:03.994 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-12T03:35:03.994 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-12T03:35:03.995 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-12T03:35:03.995 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-12T03:35:03.995 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=stat 2024-03-12T03:35:03.995 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-12T03:35:03.995 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-12T03:35:03.995 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-12T03:35:03.995 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=time 2024-03-12T03:35:03.996 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-12T03:35:03.996 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=uname 2024-03-12T03:35:03.996 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-12T03:35:03.996 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-12T03:35:03.996 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.949Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-12T03:35:03.996 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.950Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-12T03:35:03.996 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b[64278]: ts=2024-03-12T03:35:03.950Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-12T03:35:04.104 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.169:6801/676779916 v2:172.21.15.169:6802/676779916 exists,up c44466b0-3112-44dc-b5f3-d6494b958761', 'osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.169:6805/130042595 v2:172.21.15.169:6806/130042595 exists,up 184c5c97-a992-4be9-bcd3-c08cb58a0e56', 'osd.2 up in weight 1 up_from 18 up_thru 78 down_at 0 last_clean_interval [0,0) v2:172.21.15.169:6809/2751816702 v2:172.21.15.169:6810/2751816702 exists,up beb15e06-1ca4-46a7-b640-efd1991367e8', 'osd.3 up in weight 1 up_from 27 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.169:6813/195572266 v2:172.21.15.169:6814/195572266 exists,up a00b4c50-e064-4c9e-85dc-b0a75ea5bf37', 'osd.4 up in weight 1 up_from 32 up_thru 78 down_at 0 last_clean_interval [0,0) v2:172.21.15.173:6800/1397212305 v2:172.21.15.173:6801/1397212305 exists,up 3a6121ef-6309-41b7-a964-6772c010b3a0', 'osd.5 up in weight 1 up_from 41 up_thru 78 down_at 0 last_clean_interval [0,0) v2:172.21.15.173:6804/3525888417 v2:172.21.15.173:6805/3525888417 exists,up 3c2f6dd9-9784-4a07-8710-55ac3de82964', 'osd.6 up in weight 1 up_from 51 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.173:6808/2147192629 v2:172.21.15.173:6809/2147192629 exists,up d7759df7-1978-4d70-94a2-721540804f59', 'osd.7 up in weight 1 up_from 59 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.173:6812/4014976876 v2:172.21.15.173:6813/4014976876 exists,up 8843039e-2b17-4230-9cca-5788a2294d8c'] 2024-03-12T03:35:04.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon mon.a config show 2024-03-12T03:35:04.361 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:04.486 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:04 smithi173 bash[64222]: e391b52dfd8cddcbd4382e3ecb2927d0a041a3f2853c962962b9eb7a68618870 2024-03-12T03:35:04.486 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:35:04 smithi173 systemd[1]: Started Ceph node-exporter.b for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:35:05.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[29344]: 4.1e scrub starts 2024-03-12T03:35:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[29344]: 4.1e scrub ok 2024-03-12T03:35:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[29344]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[29344]: 5.17 deep-scrub starts 2024-03-12T03:35:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[29344]: 5.17 deep-scrub ok 2024-03-12T03:35:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:35:05.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[39949]: 4.1e scrub starts 2024-03-12T03:35:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[39949]: 4.1e scrub ok 2024-03-12T03:35:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[39949]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[39949]: 5.17 deep-scrub starts 2024-03-12T03:35:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[39949]: 5.17 deep-scrub ok 2024-03-12T03:35:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:35:05.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:05 smithi173 ceph-mon[33544]: 4.1e scrub starts 2024-03-12T03:35:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:05 smithi173 ceph-mon[33544]: 4.1e scrub ok 2024-03-12T03:35:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:05 smithi173 ceph-mon[33544]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:05 smithi173 ceph-mon[33544]: 5.17 deep-scrub starts 2024-03-12T03:35:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:05 smithi173 ceph-mon[33544]: 5.17 deep-scrub ok 2024-03-12T03:35:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:05 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:35:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:06.160 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:35:06.160 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mon.a", 2024-03-12T03:35:06.160 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster": "ceph", 2024-03-12T03:35:06.160 INFO:teuthology.orchestra.run.smithi169.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-12T03:35:06.160 INFO:teuthology.orchestra.run.smithi169.stdout: "admin_socket_mode": "", 2024-03-12T03:35:06.160 INFO:teuthology.orchestra.run.smithi169.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-12T03:35:06.160 INFO:teuthology.orchestra.run.smithi169.stdout: "auth_client_required": "cephx, none", 2024-03-12T03:35:06.160 INFO:teuthology.orchestra.run.smithi169.stdout: "auth_cluster_required": "cephx", 2024-03-12T03:35:06.160 INFO:teuthology.orchestra.run.smithi169.stdout: "auth_debug": "true", 2024-03-12T03:35:06.160 INFO:teuthology.orchestra.run.smithi169.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-12T03:35:06.160 INFO:teuthology.orchestra.run.smithi169.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-12T03:35:06.160 INFO:teuthology.orchestra.run.smithi169.stdout: "auth_service_required": "cephx", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "auth_supported": "", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_aio": "true", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_aio_poll_ms": "250", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_aio_reap_max": "16", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_async_discard": "false", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_block_size": "4096", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_debug_aio": "false", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_debug_inflight_ios": "false", 2024-03-12T03:35:06.161 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_enable_discard": "false", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_flock_retry": "3", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_inject_crash": "0", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_ioring": "false", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_ioring_hipri": "false", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bdev_type": "", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_alloc_size": "1048576", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_allocator": "hybrid", 2024-03-12T03:35:06.162 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_buffered_io": "true", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_check_for_zeros": "false", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_compact_log_sync": "false", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_max_log_runway": "4194304", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_max_prefetch": "1048576", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_min_flush_size": "524288", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_min_log_runway": "1048576", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_replay_recovery": "false", 2024-03-12T03:35:06.163 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_sync_write": "false", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_allocation_from_file": "true", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_allocator": "hybrid", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-12T03:35:06.164 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_block_create": "true", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_block_db_create": "false", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_block_db_path": "", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_block_db_size": "0", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_block_path": "", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_block_preallocate_file": "false", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_block_size": "107374182400", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_block_wal_create": "false", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_block_wal_path": "", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_block_wal_size": "100663296", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_bluefs": "true", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-12T03:35:06.165 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_autotune": "true", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_size": "0", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-12T03:35:06.166 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_type": "2q", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_clone_cow": "true", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_compression_mode": "none", 2024-03-12T03:35:06.167 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_csum_type": "crc32c", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_freelist": "false", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-12T03:35:06.168 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_small_allocations": "0", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_default_buffered_read": "true", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_default_buffered_write": "false", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-12T03:35:06.169 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fail_eio": "false", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_on_mount": "false", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_on_umount": "false", 2024-03-12T03:35:06.170 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_ignore_data_csum": "false", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_log_op_age": "5.000000", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_max_alloc_size": "0", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_max_blob_size": "0", 2024-03-12T03:35:06.171 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_max_deferred_txc": "32", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_min_alloc_size": "0", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_nid_prealloc": "1024", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_retry_disk_reads": "3", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_rocksdb_cf": "true", 2024-03-12T03:35:06.172 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-12T03:35:06.173 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-12T03:35:06.173 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-12T03:35:06.173 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-12T03:35:06.173 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-12T03:35:06.173 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-12T03:35:06.173 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_spdk_mem": "512", 2024-03-12T03:35:06.173 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-12T03:35:06.173 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_tracing": "false", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-12T03:35:06.174 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_zero_block_detection": "false", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-12T03:35:06.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "cephx_cluster_require_signatures": "false", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "cephx_cluster_require_version": "2", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "cephx_require_signatures": "false", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "cephx_require_version": "2", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "cephx_service_require_signatures": "false", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "cephx_service_require_version": "2", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "cephx_sign_messages": "true", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "chdir": "", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "client_acl_type": "", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "client_asio_thread_count": "2", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "client_cache_mid": "0.750000", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "client_cache_size": "16384", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "client_caps_release_delay": "5", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "client_check_pool_perm": "true", 2024-03-12T03:35:06.176 INFO:teuthology.orchestra.run.smithi169.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_debug_force_sync_read": "false", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_debug_getattr_caps": "false", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_debug_inject_tick_delay": "0", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_die_on_failed_remount": "false", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_dirsize_rbytes": "true", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_force_lazyio": "false", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_fs": "", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_inject_release_failure": "false", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_max_inline_size": "4096", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_mds_namespace": "", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_metadata": "", 2024-03-12T03:35:06.177 INFO:teuthology.orchestra.run.smithi169.stdout: "client_mount_gid": "-1", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_mount_timeout": "300", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_mount_uid": "-1", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_mountpoint": "/", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_notify_timeout": "10", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_oc": "true", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_oc_max_dirty": "104857600", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_oc_max_objects": "1000", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_oc_size": "209715200", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_oc_target_dirty": "8388608", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_permissions": "true", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_quota": "true", 2024-03-12T03:35:06.178 INFO:teuthology.orchestra.run.smithi169.stdout: "client_quota_df": "true", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "client_readahead_max_bytes": "0", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "client_readahead_max_periods": "4", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "client_readahead_min": "131072", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "client_reconnect_stale": "false", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "client_shutdown_timeout": "30", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "client_snapdir": ".snap", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "client_tick_interval": "1", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "client_trace": "", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "client_try_dentry_invalidate": "false", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "client_use_faked_inos": "false", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "client_use_random_mds": "false", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "clog_to_graylog": "false", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "clog_to_graylog_port": "12201", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "clog_to_monitors": "default=true", 2024-03-12T03:35:06.179 INFO:teuthology.orchestra.run.smithi169.stdout: "clog_to_syslog": "false", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "clog_to_syslog_level": "info", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "-", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_network": "", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_network_interface": "", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "compressor_zlib_isal": "false", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "compressor_zlib_level": "5", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "compressor_zlib_winsize": "-15", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "compressor_zstd_level": "1", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "crimson_alien_op_num_threads": "6", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-12T03:35:06.180 INFO:teuthology.orchestra.run.smithi169.stdout: "crimson_seastar_cpu_cores": "", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_location": "", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_location_hook": "", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_location_hook_timeout": "10", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "daemonize": "false", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "daos_pool": "tank", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_alienstore": "0/5", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_allow_any_pool_priority": "false", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_asok": "1/5", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_asok_assert_abort": "false", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_asserts_on_shutdown": "true", 2024-03-12T03:35:06.181 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_auth": "1/5", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_bdev": "1/3", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_bluefs": "1/5", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_bluestore": "1/5", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_buffer": "0/1", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_ceph_exporter": "1/5", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_cephfs_mirror": "0/5", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_cephsqlite": "0/5", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_client": "0/5", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_compressor": "1/5", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_context": "0/1", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_crimson_interrupt": "0/5", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_crush": "1/1", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_crypto": "1/5", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_cyanstore": "0/5", 2024-03-12T03:35:06.182 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_deliberately_leak_memory": "false", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_disable_randomized_ping": "false", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_dpdk": "1/5", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_eventtrace": "1/5", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_filer": "0/1", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_filestore": "1/3", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_finisher": "1/1", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_fuse": "1/5", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_heartbeat_testing_span": "0", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_heartbeatmap": "1/5", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_javaclient": "1/5", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_journal": "1/3", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_journaler": "0/5", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_kstore": "1/5", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_lockdep": "0/1", 2024-03-12T03:35:06.183 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_mclock": "1/5", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_mds": "1/5", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_mds_balancer": "1/5", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_mds_locker": "1/5", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_mds_log": "1/5", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_mds_log_expire": "1/5", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_mds_migrator": "1/5", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_memstore": "1/5", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_mgr": "2/5", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_mgrc": "1/5", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_mon": "20/20", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_monc": "0/10", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_ms": "1/1", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_none": "0/5", 2024-03-12T03:35:06.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_objclass": "0/5", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_objectcacher": "0/5", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_objecter": "0/1", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_optracker": "0/5", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_osd": "1/5", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_paxos": "20/20", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_perfcounter": "1/5", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_prioritycache": "1/5", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rados": "0/5", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rbd": "0/5", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rbd_mirror": "0/5", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rbd_pwl": "0/5", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rbd_replay": "0/5", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_refs": "0/0", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_reserver": "1/1", 2024-03-12T03:35:06.185 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rgw": "1/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rgw_access": "1/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rgw_datacache": "1/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rgw_dbstore": "1/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rgw_flight": "1/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rgw_sync": "1/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_rocksdb": "4/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore": "0/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_backref": "0/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_cache": "0/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_cleaner": "0/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_device": "0/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_epm": "0/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-12T03:35:06.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_journal": "0/5", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_lba": "0/5", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_odata": "0/5", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_omap": "0/5", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_onode": "0/5", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_t": "0/5", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_seastore_tm": "0/5", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_striper": "0/1", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_test": "0/5", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_throttle": "1/1", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_timer": "0/1", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_tp": "0/5", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "debug_trace": "1/5", 2024-03-12T03:35:06.187 INFO:teuthology.orchestra.run.smithi169.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "device_failure_prediction_mode": "none", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "err_to_graylog": "false", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "err_to_journald": "false", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "err_to_stderr": "true", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "err_to_syslog": "false", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "event_tracing": "false", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "exporter_addr": "0.0.0.0", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "exporter_http_port": "9926", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "exporter_prio_limit": "5", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "exporter_sort_metrics": "true", 2024-03-12T03:35:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: "exporter_stats_period": "5", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "fake_statfs_for_testing": "0", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "fatal_signal_handlers": "true", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filer_max_purge_ops": "10", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filer_max_truncate_ops": "128", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_apply_finisher_threads": "1", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_blackhole": "false", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_btrfs_clone_range": "true", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_btrfs_snap": "true", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_caller_concurrency": "10", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_collect_device_partition_information": "true", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_commit_timeout": "600.000000", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_debug_inject_read_err": "false", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_debug_omap_check": "false", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-12T03:35:06.189 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_debug_verify_split": "false", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_dump_file": "", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_fadvise": "true", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_fail_eio": "true", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_fd_cache_shards": "16", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_fd_cache_size": "128", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_fiemap": "false", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_fiemap_threshold": "4096", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_inject_stall": "0", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_journal_parallel": "false", 2024-03-12T03:35:06.190 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_journal_trailing": "false", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_journal_writeahead": "false", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_kill_at": "0", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_inline_xattrs": "0", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_xattr_value_size": "0", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-12T03:35:06.191 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_merge_threshold": "-10", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_odsync_write": "false", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_omap_backend": "rocksdb", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_omap_backend_path": "", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_op_thread_timeout": "60", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_op_threads": "2", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_punch_hole": "false", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-12T03:35:06.192 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_queue_max_ops": "50", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_seek_data_hole": "false", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_sloppy_crc": "false", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_splice": "false", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_split_multiple": "2", 2024-03-12T03:35:06.193 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_split_rand_factor": "20", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_update_to": "1000", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_enable": "true", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-12T03:35:06.194 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_xfs_extsize": "false", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "filestore_zfs_snap": "false", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fio_dir": "/tmp/fio", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fsid": "677d624a-e020-11ee-95c8-87774f69a715", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_allow_other": "true", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_atomic_o_trunc": "true", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_big_writes": "true", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_debug": "false", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_default_permissions": "false", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_disable_pagecache": "false", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_max_write": "0", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_multithreaded": "true", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_require_active_mds": "true", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_set_user_groups": "true", 2024-03-12T03:35:06.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_splice_move": "true", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_splice_read": "true", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_splice_write": "true", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "fuse_use_invalidate_cb": "true", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "gss_target_name": "ceph", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_file": "", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_inject_failure": "0", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_interval": "5", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "host": "smithi169", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-12T03:35:06.196 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "inject_early_sigterm": "false", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "jaeger_agent_port": "6799", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "jaeger_tracing_enable": "false", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_aio": "true", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_align_min_size": "65536", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_block_align": "true", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_block_size": "4096", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_dio": "true", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_discard": "false", 2024-03-12T03:35:06.197 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_force_aio": "false", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_ignore_corruption": "false", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_max_write_bytes": "10485760", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_max_write_entries": "100", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_replay_from": "0", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_write_header_frequency": "0", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journal_zero_on_create": "false", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journaler_prefetch_periods": "10", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journaler_prezero_periods": "5", 2024-03-12T03:35:06.198 INFO:teuthology.orchestra.run.smithi169.stdout: "journaler_write_head_interval": "15", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "key": "", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "keyfile": "", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "kstore_backend": "rocksdb", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "kstore_default_stripe_size": "65536", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "kstore_fsck_on_mount": "false", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "kstore_max_bytes": "67108864", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "kstore_max_ops": "512", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "kstore_nid_prealloc": "1024", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "kstore_onode_map_size": "1024", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "kstore_sync_submit_transaction": "false", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "kstore_sync_transaction": "false", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "librados_thread_count": "2", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "lockdep": "false", 2024-03-12T03:35:06.199 INFO:teuthology.orchestra.run.smithi169.stdout: "lockdep_force_backtrace": "false", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_coarse_timestamps": "true", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_flush_on_exit": "false", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_graylog_host": "127.0.0.1", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_graylog_port": "12201", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_max_new": "1000", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_max_recent": "10000", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_stderr_prefix": "", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_stop_at_utilization": "0.970000", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": "true", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_graylog": "false", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_journald": "false", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_stderr": "false", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_syslog": "false", 2024-03-12T03:35:06.200 INFO:teuthology.orchestra.run.smithi169.stdout: "max_rotating_auth_attempts": "10", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_action_on_write_error": "1", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_alternate_name_max": "8192", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_asio_thread_count": "2", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_export_pin": "true", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_fragment_dirs": "true", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_fragment_interval": "5", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_interval": "10", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_max": "-1", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_max_until": "-1", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_merge_size": "50", 2024-03-12T03:35:06.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_midchunk": "0.300000", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_min_start": "0.200000", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_minchunk": "0.001000", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_mode": "0", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_need_max": "1.200000", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_need_min": "0.800000", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_overload_epochs": "2", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_split_bits": "3", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_split_size": "10000", 2024-03-12T03:35:06.202 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_target_decay": "10.000000", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_beacon_grace": "15.000000", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_beacon_interval": "4.000000", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache_mid": "0.700000", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache_release_free_interval": "10", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache_reservation": "0.050000", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache_trim_interval": "1", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache_trim_threshold": "262144", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-12T03:35:06.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_client_prealloc_inos": "1000", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_connect_bootstrapping": "false", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_damage_table_max_entries": "10000", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_debug_auth_pins": "false", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_debug_frag": "false", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_debug_scatterstat": "false", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_debug_subtrees": "false", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_decay_halflife": "5.000000", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_default_dir_hash": "2", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_defer_session_stale": "true", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_deny_all_reconnect": "false", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_dir_keys_per_op": "16384", 2024-03-12T03:35:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_dir_max_commit_size": "10", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_dir_max_entries": "0", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_dir_prefetch": "true", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_dump_cache_on_map": "false", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_early_reply": "true", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_enable_op_tracker": "true", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_enforce_unique_name": "true", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_export_ephemeral_random": "true", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-12T03:35:06.205 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_health_summarize_threshold": "10", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_inject_health_dummy": "false", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_inject_migrator_session_race": "false", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-12T03:35:06.206 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_join_fs": "", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_journal_format": "1", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_kill_create_at": "0", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_kill_export_at": "0", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_kill_import_at": "0", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_kill_journal_at": "0", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_kill_journal_expire_at": "0", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_kill_journal_replay_at": "0", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_kill_link_at": "0", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_kill_mdstable_at": "0", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_kill_openc_at": "0", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_kill_rename_at": "0", 2024-03-12T03:35:06.207 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_kill_shutdown_at": "0", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_event_large_threshold": "524288", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_events_per_segment": "1024", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_max_events": "-1", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_max_segments": "128", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_pause": "false", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_segment_size": "0", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_trim_threshold": "128", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log_warn_factor": "2.000000", 2024-03-12T03:35:06.208 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_max_caps_per_client": "1048576", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_max_completed_flushes": "100000", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_max_completed_requests": "100000", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_max_export_size": "20971520", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_max_file_recover": "32", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_max_purge_files": "64", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_max_purge_ops": "8192", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_max_snaps_per_dir": "100", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_metrics_update_interval": "2", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_min_caps_per_client": "100", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_min_caps_working_set": "10000", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_numa_node": "-1", 2024-03-12T03:35:06.209 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_op_complaint_time": "30.000000", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_op_history_duration": "600", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_op_history_size": "20", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_op_history_slow_op_size": "20", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_op_log_threshold": "5", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_ping_grace": "15", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_ping_interval": "5", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_recall_max_caps": "30000", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-12T03:35:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_recall_warning_threshold": "262144", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_replay_interval": "1.000000", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_root_ino_gid": "0", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_root_ino_uid": "0", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-12T03:35:06.211 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_shutdown_check": "0", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_skip_ino": "0", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_snap_max_uid": "4294967294", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_snap_min_uid": "0", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_snap_rstat": "false", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_standby_replay_damaged": "false", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_symlink_recovery": "true", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_thrash_exports": "0", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_thrash_fragments": "0", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_tick_interval": "5.000000", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_valgrind_exit": "false", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_verify_backtrace": "1", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_verify_scatter": "false", 2024-03-12T03:35:06.212 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_wipe_sessions": "false", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool_debug": "false", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "memstore_device_bytes": "1073741824", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "memstore_page_set": "false", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "memstore_page_size": "65536", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_client_bytes": "134217728", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_client_messages": "512", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_disabled_modules": "", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_max_pg_num_change": "128", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_mds_bytes": "134217728", 2024-03-12T03:35:06.213 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_mds_messages": "128", 2024-03-12T03:35:06.215 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-12T03:35:06.215 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_mon_bytes": "134217728", 2024-03-12T03:35:06.215 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_mon_messages": "128", 2024-03-12T03:35:06.215 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_osd_bytes": "536870912", 2024-03-12T03:35:06.215 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_osd_messages": "8192", 2024-03-12T03:35:06.215 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_pool": "true", 2024-03-12T03:35:06.215 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-12T03:35:06.215 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_standby_modules": "true", 2024-03-12T03:35:06.215 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_stats_period": "5", 2024-03-12T03:35:06.215 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_stats_threshold": "5", 2024-03-12T03:35:06.215 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_tick_period": "2", 2024-03-12T03:35:06.215 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_allow_pool_delete": "true", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_allow_pool_size_one": "false", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_auth_validate_all_caps": "true", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_client_bytes": "104857600", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_client_directed_command_retry": "2", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_client_hunt_parallel": "3", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_client_log_interval": "1.000000", 2024-03-12T03:35:06.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_client_ping_interval": "10.000000", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_client_target_rank": "-1", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cluster_log_file_level": "debug", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cluster_log_to_file": "true", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cluster_log_to_journald": "true", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-12T03:35:06.217 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_compact_on_bootstrap": "false", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_compact_on_start": "false", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_compact_on_trim": "true", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_cpu_threads": "4", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_crush_min_required_version": "hammer", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_daemon_bytes": "419430400", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_data_avail_crit": "5", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_data_avail_warn": "5", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_data_size_warn": "16106127360", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-12T03:35:06.218 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_debug_dump_json": "false", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_debug_dump_transactions": "false", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_debug_extra_checks": "false", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_debug_no_require_reef": "false", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_debug_no_require_squid": "false", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_down_mkfs_grace": "60", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_election_default_strategy": "1", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_election_timeout": "5.000000", 2024-03-12T03:35:06.219 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_elector_ping_divisor": "2", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_enable_op_tracker": "true", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_fake_pool_delete": "false", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_force_quorum_join": "false", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_globalid_prealloc": "10000", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_health_detail_to_clog": "true", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_health_log_update_period": "5", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_health_max_detail": "50", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_health_to_clog": "true", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_health_to_clog_interval": "600", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_host": "[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]", 2024-03-12T03:35:06.220 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_host_override": "", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_initial_members": "", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_lease": "5.000000", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_log_full_interval": "50", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_log_max": "10000", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_log_max_summary": "50", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-12T03:35:06.221 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_max_log_epochs": "500", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_max_osd": "10000", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_max_pg_per_osd": "10000", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_max_pool_pg_num": "65536", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_mds_force_trim_to": "0", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_mds_skip_sanity": "false", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_memory_autotune": "true", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_memory_target": "2147483648", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_mgr_beacon_grace": "30", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_mgr_digest_period": "5", 2024-03-12T03:35:06.222 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_mgr_inactive_grace": "60", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_min_osdmap_epochs": "50", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_op_complaint_time": "30", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_op_history_duration": "600", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_op_history_size": "20", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_op_history_slow_op_size": "20", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_op_log_threshold": "5", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_auto_mark_in": "false", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-12T03:35:06.223 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_cache_size": "500", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_down_out_interval": "600", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_force_trim_to": "0", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-12T03:35:06.224 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_min_down_reporters": "2", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_report_timeout": "900", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-12T03:35:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_pg_stuck_threshold": "60", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_probe_timeout": "2.000000", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_reweight_max_change": "0.050000", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_reweight_max_osds": "4", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-12T03:35:06.226 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_scrub_interval": "86400", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_scrub_max_keys": "100", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_scrub_timeout": "300", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_session_timeout": "300", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_smart_report_timeout": "5", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_stat_smooth_intervals": "6", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_stretch_pool_min_size": "2", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_stretch_pool_size": "4", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_sync_debug": "false", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-12T03:35:06.227 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_sync_provider_kill_at": "0", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_sync_requester_kill_at": "0", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_sync_timeout": "60.000000", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_target_pg_per_osd": "100", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_tick_interval": "5", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_timecheck_interval": "300.000000", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_use_min_delay_socket": "false", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_older_version_delay": "604800", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-12T03:35:06.228 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_misplaced": "false", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_older_version": "true", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "monmap": "", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-12T03:35:06.229 INFO:teuthology.orchestra.run.smithi169.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "motr_tracing_enabled": "false", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_op_threads": "3", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_cm": "false", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_device_name": "", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_dscp": "96", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_local_gid": "", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-12T03:35:06.230 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_port_num": "1", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_sl": "3", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_support_srq": "true", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_rdma_type": "ib", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_async_reap_threshold": "5", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_bind_before_connect": "false", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_bind_ipv4": "true", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_bind_ipv6": "false", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_bind_msgr1": "false", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_bind_msgr2": "true", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_bind_port_max": "7568", 2024-03-12T03:35:06.231 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_bind_port_min": "6800", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_bind_retry_count": "3", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_bind_retry_delay": "5", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_blackhole_client": "false", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_blackhole_mds": "false", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_blackhole_mgr": "false", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_blackhole_mon": "false", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_blackhole_osd": "false", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_client_mode": "crc secure", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_cluster_mode": "crc secure", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_cluster_type": "", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_compress_secure": "false", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_connection_idle_timeout": "900", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_connection_ready_timeout": "10", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_crc_data": "true", 2024-03-12T03:35:06.232 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_crc_header": "true", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_die_on_bad_msg": "false", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_die_on_bug": "true", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_die_on_old_message": "true", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_die_on_skipped_message": "false", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_coremask": "0xF", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_enable_tso": "true", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_hugepages": "", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-12T03:35:06.233 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_lro": "true", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_memory_channel": "4", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_pmd": "", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_port_id": "0", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_dump_on_send": "false", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_initial_backoff": "0.200000", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_inject_delay_max": "1.000000", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_inject_delay_type": "", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_inject_network_congestion": "0", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_inject_socket_failures": "0", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_learn_addr_from_peer": "true", 2024-03-12T03:35:06.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_max_accept_failures": "4", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_max_backoff": "15.000000", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_mon_client_mode": "secure crc", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_mon_service_mode": "secure crc", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_osd_compress_min_size": "1024", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_osd_compress_mode": "none", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_pq_min_cost": "65536", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_public_type": "", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_service_mode": "crc secure", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_tcp_listen_backlog": "512", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_tcp_nodelay": "true", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_tcp_rcvbuf": "0", 2024-03-12T03:35:06.235 INFO:teuthology.orchestra.run.smithi169.stdout: "ms_type": "async+posix", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "no_config_file": "false", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter_completion_locks_per_session": "32", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter_inflight_ops": "1024", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter_tick_interval": "5.000000", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter_timeout": "10.000000", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "objectstore_blackhole": "false", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "openssl_engine_opts": "", 2024-03-12T03:35:06.236 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_agent_delay_time": "5.000000", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_agent_hist_halflife": "1000", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_agent_max_low_ops": "2", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_agent_max_ops": "4", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_agent_slop": "0.020000", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_asio_thread_count": "2", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_async_recovery_min_cost": "100", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_backfill_scan_max": "512", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_backfill_scan_min": "64", 2024-03-12T03:35:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_backoff_on_degraded": "false", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_backoff_on_peering": "false", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_backoff_on_unfound": "true", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_beacon_report_interval": "300", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bench_duration": "30", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bench_max_block_size": "67108864", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_blkin_trace_all": "false", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_check_for_log_corruption": "false", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-12T03:35:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_class_update_on_start": "true", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_client_message_cap": "256", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_client_message_size_cap": "524288000", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_client_op_priority": "63", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_client_watch_timeout": "30", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_command_max_records": "256", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_compact_on_start": "false", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_crush_update_on_start": "true", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_crush_update_weight_set": "true", 2024-03-12T03:35:06.239 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_feed_pullee": "-1", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_misdirected_ops": "false", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_no_acting_change": "false", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_no_purge_strays": "false", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_op_order": "false", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-12T03:35:06.240 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_shutdown": "false", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_trim_objects": "false", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_verify_snaps": "false", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_deep_scrub_keys": "1024", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-12T03:35:06.241 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_deep_scrub_stride": "524288", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_default_notify_timeout": "30", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_delete_sleep": "0.000000", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_discard_disconnected_ops": "true", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_enable_op_tracker": "true", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_fast_info": "true", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_fast_shutdown": "true", 2024-03-12T03:35:06.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_function_tracing": "false", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_heartbeat_grace": "20", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_heartbeat_interval": "6", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_heartbeat_min_peers": "10", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_heartbeat_min_size": "2000", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_heartbeat_stale": "600", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_hit_set_max_size": "100000", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_hit_set_min_size": "1000", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-12T03:35:06.243 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_journal_size": "5120", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_kill_backfill_at": "0", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_map_cache_size": "50", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_map_dedup": "true", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_map_message_max": "40", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_map_share_max_epochs": "40", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_attr_name_len": "100", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_attr_size": "0", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_backfills": "1", 2024-03-12T03:35:06.244 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_markdown_count": "5", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_markdown_period": "600", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_object_name_len": "2048", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_object_namespace_len": "256", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_object_size": "134217728", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_pg_blocked_by": "16", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_pg_log_entries": "10000", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_pgls": "1024", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_push_cost": "8388608", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_push_objects": "10", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_scrubs": "3", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_trimming_pgs": "2", 2024-03-12T03:35:06.245 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_write_op_reply_len": "64", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_max_write_size": "90", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_profile": "balanced", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-12T03:35:06.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_memory_base": "805306368", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_memory_cache_min": "134217728", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_memory_target": "4294967296", 2024-03-12T03:35:06.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_memory_target_autotune": "false", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_min_pg_log_entries": "250", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_min_recovery_priority": "0", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mon_report_interval": "5", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_num_cache_shards": "32", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_num_op_tracker_shard": "32", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_numa_auto_affinity": "true", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_numa_node": "-1", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_numa_prefer_iface": "true", 2024-03-12T03:35:06.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_objecter_finishers": "1", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_objectstore": "bluestore", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_objectstore_fuse": "false", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_objectstore_tracing": "false", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_complaint_time": "30.000000", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_history_duration": "600", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_history_size": "20", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_history_slow_op_size": "20", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_log_threshold": "5", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_num_shards": "0", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_num_shards_hdd": "5", 2024-03-12T03:35:06.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_num_shards_ssd": "8", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_pq_min_cost": "65536", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_queue_cut_off": "high", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_op_thread_timeout": "15", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_open_classes_on_start": "true", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_os_flags": "0", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_peering_op_priority": "255", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pg_delete_cost": "1048576", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pg_delete_priority": "5", 2024-03-12T03:35:06.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pg_log_trim_max": "10000", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pg_log_trim_min": "100", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-12T03:35:06.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_crimson": "false", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_flags": "0", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_min_size": "0", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_pg_num": "32", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_pgp_num": "0", 2024-03-12T03:35:06.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_read_ratio": "70", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_size": "3", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_default_type": "replicated", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_push_per_object_cost": "1000", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recover_clone_overlap": "true", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_cost": "20971520", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_max_active": "0", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-12T03:35:06.253 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_max_single_start": "1", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_op_priority": "3", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_priority": "5", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_sleep": "0.000000", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_repair_during_recovery": "false", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_requested_scrub_priority": "120", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-12T03:35:06.254 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_auto_repair": "false", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_begin_hour": "0", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_begin_week_day": "0", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_chunk_max": "25", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_chunk_min": "5", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_cost": "52428800", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_during_recovery": "false", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_end_hour": "0", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_end_week_day": "0", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_event_cost": "4096", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-12T03:35:06.255 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_invalid_stats": "true", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_max_preemptions": "5", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_priority": "5", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_sleep": "0.000000", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_skip_data_digest": "false", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_smart_report_timeout": "5", 2024-03-12T03:35:06.256 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_snap_trim_cost": "1048576", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_snap_trim_priority": "5", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_target_transaction_size": "30", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-12T03:35:06.257 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_tracing": "false", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_use_stale_snap": "false", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "osdc_blkin_trace_all": "false", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "paxos_kill_at": "0", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "paxos_max_join_drift": "10", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "paxos_min": "500", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "paxos_min_wait": "0.050000", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "paxos_propose_interval": "1.000000", 2024-03-12T03:35:06.258 INFO:teuthology.orchestra.run.smithi169.stdout: "paxos_service_trim_max": "500", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "paxos_service_trim_min": "10", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "paxos_stash_full_interval": "25", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "paxos_trim_max": "500", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "paxos_trim_min": "250", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "perf": "true", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "pid_file": "", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "-", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrv": "", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "public_bind_addr": "-", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "public_network": "", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "public_network_interface": "", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "qat_compressor_enabled": "false", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "qat_compressor_session_max_number": "256", 2024-03-12T03:35:06.259 INFO:teuthology.orchestra.run.smithi169.stdout: "rados_mon_op_timeout": "0", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rados_osd_op_timeout": "0", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rados_tracing": "false", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_atime_update_interval": "60", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_balance_parent_reads": "false", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_balance_snap_reads": "false", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_blkin_trace_all": "false", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_cache": "true", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-12T03:35:06.260 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_cache_policy": "writearound", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_cache_size": "33554432", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_clone_copy_on_read": "false", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_compression_hint": "none", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_concurrent_management_ops": "10", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_default_clone_format": "auto", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_default_data_pool": "", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_default_features": "61", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_default_format": "2", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_default_map_options": "", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_default_order": "22", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_default_pool": "rbd", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-12T03:35:06.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_default_stripe_count": "0", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_default_stripe_unit": "0", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_enable_alloc_hint": "true", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_io_scheduler": "simple", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-12T03:35:06.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_journal_order": "24", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_journal_pool": "", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_journal_splay_width": "4", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_localize_parent_reads": "false", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_localize_snap_reads": "false", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-12T03:35:06.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-12T03:35:06.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_mtime_update_interval": "60", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_non_blocking_aio": "true", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_op_thread_timeout": "60", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_op_threads": "1", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_parent_cache_enabled": "false", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_plugins": "", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_bps_burst": "0", 2024-03-12T03:35:06.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_bps_limit": "0", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_exclude_ops": "0", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_iops_burst": "0", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_iops_limit": "0", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-12T03:35:06.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_read_from_replica_policy": "default", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_skip_partial_discard": "true", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_tracing": "false", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_validate_names": "true", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_validate_pool": "true", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "restapi_base_url": "", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "restapi_log_level": "", 2024-03-12T03:35:06.267 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_acl_grants_max_num": "100", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_admin_entry": "admin", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_backend_store": "rados", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_barbican_url": "", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_beast_enable_async": "true", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_bucket_counters_cache": "false", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-12T03:35:06.268 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_cache_enabled": "true", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_cache_expiry_interval": "900", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_cache_lru_size": "10000", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_config_store": "rados", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_content_length_compat": "false", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_copy_obj_progress": "true", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_cors_rules_max_num": "100", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_cross_domain_policy": "", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_kmip_addr": "", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_kmip_password": "", 2024-03-12T03:35:06.269 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_kmip_username": "", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_require_ssl": "true", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-12T03:35:06.270 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_vault_addr": "", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_vault_auth": "token", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_vault_namespace": "", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_vault_prefix": "", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_vault_token_file": "", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_curl_buffersize": "524288", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_curl_low_speed_time": "300", 2024-03-12T03:35:06.271 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_d4n_port": "6379", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_data_log_changes_size": "1000", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_data_log_num_shards": "128", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-12T03:35:06.272 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_data_log_window": "30", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-12T03:35:06.273 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dns_name": "", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dns_s3website_name": "", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_dynamic_resharding": "true", 2024-03-12T03:35:06.274 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_enable_gc_threads": "true", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_enable_lc_threads": "true", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_enable_ops_log": "false", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_enable_quota_threads": "true", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_enable_static_website": "false", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_enable_usage_log": "false", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_enforce_swift_acls": "true", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_exit_timeout_secs": "120", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_expose_bucket": "false", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_extended_http_attrs": "", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_filter": "none", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_frontends": "beast port=7480", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_gc_max_deferred": "50", 2024-03-12T03:35:06.275 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_gc_max_objs": "32", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_gc_processor_period": "3600", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_init_timeout": "300", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_kafka_connection_idle": "30", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-12T03:35:06.276 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_admin_domain": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_admin_password": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_admin_password_path": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_admin_project": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_admin_tenant": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_admin_token": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_admin_token_path": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_admin_user": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_api_version": "2", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_barbican_domain": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_barbican_password": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_barbican_project": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_barbican_user": "", 2024-03-12T03:35:06.277 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_url": "", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_lc_debug_interval": "-1", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_lc_lock_max_time": "90", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_lc_max_objs": "32", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_lc_max_rules": "1000", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_lc_max_worker": "3", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_lc_thread_delay": "0", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-12T03:35:06.278 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_ldap_dnattr": "uid", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_ldap_searchfilter": "", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_log_http_headers": "", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_log_object_name_utc": "false", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_attr_name_len": "0", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_attr_size": "0", 2024-03-12T03:35:06.279 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_chunk_size": "4194304", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_listing_results": "1000", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_notify_retries": "10", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_put_param_size": "1048576", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_put_size": "5368709120", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_max_slo_entries": "1000", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_md_log_max_shards": "64", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-12T03:35:06.280 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_mp_lock_max_time": "600", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_max_gc": "300", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-12T03:35:06.281 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_num_async_rados_threads": "32", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_num_control_oids": "8", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_numa_node": "-1", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_objexp_chunk_size": "100", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_objexp_gc_interval": "600", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_op_thread_timeout": "600", 2024-03-12T03:35:06.282 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_op_tracing": "false", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_opa_token": "", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_opa_url": "", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_opa_verify_ssl": "true", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_ops_log_rados": "false", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_ops_log_socket_path": "", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_period_push_interval": "2.000000", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-12T03:35:06.283 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_posix_cache_lanes": "3", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_posix_cache_partitions": "3", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_print_continue": "true", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_rados_tracing": "false", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_realm": "", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_realm_id": "", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-12T03:35:06.284 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_region": "", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_request_uri": "", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_reshard_batch_size": "64", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_reshard_max_aio": "128", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_reshard_num_logs": "16", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_reshard_thread_interval": "600", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_resolve_cname": "false", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_run_sync_thread": "true", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-12T03:35:06.285 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_scheduler_type": "throttler", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_script_uri": "", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_service_provider_name": "", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sts_client_id": "", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sts_client_secret": "", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sts_entry": "sts", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sts_key": "sts", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sts_min_session_duration": "900", 2024-03-12T03:35:06.286 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sts_token_introspection_url": "", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_swift_account_in_url": "false", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_swift_auth_entry": "auth", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_swift_auth_url": "", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_swift_custom_header": "", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_swift_need_stats": "true", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_swift_tenant_name": "", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_swift_token_expiration": "86400", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_swift_url": "", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_swift_url_prefix": "swift", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sync_lease_period": "120", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-12T03:35:06.287 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_thread_pool_size": "512", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_torrent_comment": "", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_torrent_createby": "", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_torrent_encoding": "", 2024-03-12T03:35:06.288 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_torrent_flag": "false", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_torrent_origin": "", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_torrent_tracker": "", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_trust_forwarded_https": "false", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_usage_max_shards": "32", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_usage_max_user_shards": "1", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_use_opa_authz": "false", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_user_counters_cache": "false", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_user_max_buckets": "1000", 2024-03-12T03:35:06.289 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_user_policies_max_num": "100", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_user_unique_email": "true", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_verify_ssl": "true", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_zone": "", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_zone_id": "", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_zonegroup": "", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_zonegroup_id": "", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgwlc_auto_session_clear": "true", 2024-03-12T03:35:06.290 INFO:teuthology.orchestra.run.smithi169.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_block_size": "4096", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_cache_size": "536870912", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_index_type": "binary_search", 2024-03-12T03:35:06.291 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_partition_filters": "false", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_perf": "false", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "run_dir": "/var/run/ceph", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_block_create": "true", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_cache_lru_size": "67108864", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_default_max_object_size": "16777216", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_device_size": "53687091200", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_journal_batch_capacity": "16", 2024-03-12T03:35:06.292 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_main_device_type": "SSD", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "seastore_segment_size": "67108864", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "set_keepcaps": "false", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "setgroup": "ceph", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "setuser": "ceph", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "setuser_match_path": "", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "thp": "false", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "threadpool_default_timeout": "60", 2024-03-12T03:35:06.293 INFO:teuthology.orchestra.run.smithi169.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-12T03:35:06.294 INFO:teuthology.orchestra.run.smithi169.stdout: "throttler_perf_counter": "true" 2024-03-12T03:35:06.294 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:35:06.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:06 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:35:06.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:06 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:35:06.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:06 smithi173 ceph-mon[33544]: 5.a scrub starts 2024-03-12T03:35:06.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:06 smithi173 ceph-mon[33544]: 5.a scrub ok 2024-03-12T03:35:06.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:06 smithi173 ceph-mon[33544]: 4.19 deep-scrub starts 2024-03-12T03:35:06.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:06 smithi173 ceph-mon[33544]: 4.19 deep-scrub ok 2024-03-12T03:35:06.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:06 smithi173 ceph-mon[33544]: 6.19 scrub starts 2024-03-12T03:35:06.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:06 smithi173 ceph-mon[33544]: 6.19 scrub ok 2024-03-12T03:35:06.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:06 smithi173 ceph-mon[33544]: osdmap e95: 8 total, 8 up, 8 in 2024-03-12T03:35:06.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:35:06.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:35:06.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[29344]: 5.a scrub starts 2024-03-12T03:35:06.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[29344]: 5.a scrub ok 2024-03-12T03:35:06.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[29344]: 4.19 deep-scrub starts 2024-03-12T03:35:06.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[29344]: 4.19 deep-scrub ok 2024-03-12T03:35:06.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[29344]: 6.19 scrub starts 2024-03-12T03:35:06.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[29344]: 6.19 scrub ok 2024-03-12T03:35:06.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[29344]: osdmap e95: 8 total, 8 up, 8 in 2024-03-12T03:35:06.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:35:06.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:35:06.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[39949]: 5.a scrub starts 2024-03-12T03:35:06.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[39949]: 5.a scrub ok 2024-03-12T03:35:06.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[39949]: 4.19 deep-scrub starts 2024-03-12T03:35:06.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[39949]: 4.19 deep-scrub ok 2024-03-12T03:35:06.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[39949]: 6.19 scrub starts 2024-03-12T03:35:06.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[39949]: 6.19 scrub ok 2024-03-12T03:35:06.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:06 smithi169 ceph-mon[39949]: osdmap e95: 8 total, 8 up, 8 in 2024-03-12T03:35:07.407 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:07 smithi173 ceph-mon[33544]: 3.1 scrub starts 2024-03-12T03:35:07.407 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:07 smithi173 ceph-mon[33544]: 3.1 scrub ok 2024-03-12T03:35:07.407 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:07 smithi173 ceph-mon[33544]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:07.407 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:07 smithi173 ceph-mon[33544]: 4.e deep-scrub starts 2024-03-12T03:35:07.407 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:07 smithi173 ceph-mon[33544]: 4.e deep-scrub ok 2024-03-12T03:35:07.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:07 smithi169 ceph-mon[29344]: 3.1 scrub starts 2024-03-12T03:35:07.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:07 smithi169 ceph-mon[29344]: 3.1 scrub ok 2024-03-12T03:35:07.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:07 smithi169 ceph-mon[29344]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:07.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:07 smithi169 ceph-mon[29344]: 4.e deep-scrub starts 2024-03-12T03:35:07.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:07 smithi169 ceph-mon[29344]: 4.e deep-scrub ok 2024-03-12T03:35:07.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:07 smithi169 ceph-mon[39949]: 3.1 scrub starts 2024-03-12T03:35:07.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:07 smithi169 ceph-mon[39949]: 3.1 scrub ok 2024-03-12T03:35:07.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:07 smithi169 ceph-mon[39949]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:07.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:07 smithi169 ceph-mon[39949]: 4.e deep-scrub starts 2024-03-12T03:35:07.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:07 smithi169 ceph-mon[39949]: 4.e deep-scrub ok 2024-03-12T03:35:07.900 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-12T03:35:08.175 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:08.402 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:08 smithi169 ceph-mon[29344]: 4.3 scrub starts 2024-03-12T03:35:08.402 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:08 smithi169 ceph-mon[29344]: 4.3 scrub ok 2024-03-12T03:35:08.403 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:08 smithi169 ceph-mon[29344]: 4.f scrub starts 2024-03-12T03:35:08.403 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:08 smithi169 ceph-mon[29344]: 4.f scrub ok 2024-03-12T03:35:08.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:08 smithi169 ceph-mon[39949]: 4.3 scrub starts 2024-03-12T03:35:08.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:08 smithi169 ceph-mon[39949]: 4.3 scrub ok 2024-03-12T03:35:08.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:08 smithi169 ceph-mon[39949]: 4.f scrub starts 2024-03-12T03:35:08.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:08 smithi169 ceph-mon[39949]: 4.f scrub ok 2024-03-12T03:35:08.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:08 smithi173 ceph-mon[33544]: 4.3 scrub starts 2024-03-12T03:35:08.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:08 smithi173 ceph-mon[33544]: 4.3 scrub ok 2024-03-12T03:35:08.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:08 smithi173 ceph-mon[33544]: 4.f scrub starts 2024-03-12T03:35:08.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:08 smithi173 ceph-mon[33544]: 4.f scrub ok 2024-03-12T03:35:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[39949]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[39949]: 4.b scrub starts 2024-03-12T03:35:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[39949]: 4.b scrub ok 2024-03-12T03:35:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[39949]: 5.14 scrub starts 2024-03-12T03:35:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[39949]: 5.14 scrub ok 2024-03-12T03:35:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[39949]: 5.16 scrub starts 2024-03-12T03:35:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[39949]: 5.16 scrub ok 2024-03-12T03:35:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[29344]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[29344]: 4.b scrub starts 2024-03-12T03:35:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[29344]: 4.b scrub ok 2024-03-12T03:35:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[29344]: 5.14 scrub starts 2024-03-12T03:35:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[29344]: 5.14 scrub ok 2024-03-12T03:35:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[29344]: 5.16 scrub starts 2024-03-12T03:35:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[29344]: 5.16 scrub ok 2024-03-12T03:35:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:09 smithi173 ceph-mon[33544]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:09 smithi173 ceph-mon[33544]: 4.b scrub starts 2024-03-12T03:35:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:09 smithi173 ceph-mon[33544]: 4.b scrub ok 2024-03-12T03:35:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:09 smithi173 ceph-mon[33544]: 5.14 scrub starts 2024-03-12T03:35:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:09 smithi173 ceph-mon[33544]: 5.14 scrub ok 2024-03-12T03:35:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:09 smithi173 ceph-mon[33544]: 5.16 scrub starts 2024-03-12T03:35:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:09 smithi173 ceph-mon[33544]: 5.16 scrub ok 2024-03-12T03:35:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:10.498 INFO:teuthology.orchestra.run.smithi169.stdout:mon.a: {} 2024-03-12T03:35:10.511 INFO:teuthology.orchestra.run.smithi169.stdout:mon.b: {} 2024-03-12T03:35:10.523 INFO:teuthology.orchestra.run.smithi169.stdout:mon.c: {} 2024-03-12T03:35:10.524 INFO:teuthology.orchestra.run.smithi169.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-12T03:35:10.524 INFO:teuthology.orchestra.run.smithi169.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-12T03:35:10.524 INFO:teuthology.orchestra.run.smithi169.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-12T03:35:10.534 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:10 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1779201390' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:35:10.534 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:10 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1779201390' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:35:10.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:10 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1779201390' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:35:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:11 smithi169 ceph-mon[29344]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:11 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:11.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:11 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:11 smithi169 ceph-mon[39949]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:11 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:11 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:11.876 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-12T03:35:11.886 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-12T03:35:11.922 INFO:tasks.radosbench:Beginning radosbench... 2024-03-12T03:35:11.923 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-12T03:35:11.923 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-12T03:35:11.923 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-12T03:35:11.923 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:35:11.924 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:35:11.924 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-12T03:35:11.924 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-12T03:35:11.924 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-12T03:35:11.924 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-03-12T03:35:11.925 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-12T03:35:11.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:11.925 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-12T03:35:11.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:35:11.926 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-12T03:35:11.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-12T03:35:11.927 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-12T03:35:11.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:35:11.959 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:11.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:11 smithi173 ceph-mon[33544]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:11 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:11 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:12.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:12.091 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:12.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:12.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:35:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:35:12.192 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:12.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:12.230 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:12.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:12.235 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:12.264 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:12.285 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:12.331 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:12.358 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:12.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:12.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:12.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:12.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:12.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:35:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:35:12.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:12.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:12.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:12.656 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:12 smithi169 ceph-mon[29344]: 5.b scrub starts 2024-03-12T03:35:12.656 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:12 smithi169 ceph-mon[29344]: 5.b scrub ok 2024-03-12T03:35:12.656 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:12 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:35:12.656 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:12 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:35:12.656 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:12 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:12.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:12 smithi169 ceph-mon[39949]: 5.b scrub starts 2024-03-12T03:35:12.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:12 smithi169 ceph-mon[39949]: 5.b scrub ok 2024-03-12T03:35:12.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:12 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:35:12.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:12 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:35:12.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:12 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:12.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:12.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:12.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:12.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:12.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:12.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:12.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:12.941 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:12.942 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:12.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:12 smithi173 ceph-mon[33544]: 5.b scrub starts 2024-03-12T03:35:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:12 smithi173 ceph-mon[33544]: 5.b scrub ok 2024-03-12T03:35:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:12 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:35:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:12 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:35:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:12 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:12.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:12.996 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:12.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:13.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:13.124 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:13.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:13.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:13.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:13.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:13.251 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:13.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:13.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:13.375 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:13.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:13.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:13.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:13.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:13.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:13.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:13.631 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:13.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[29344]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[29344]: 4.16 scrub starts 2024-03-12T03:35:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[29344]: 4.16 scrub ok 2024-03-12T03:35:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[29344]: 5.1a scrub starts 2024-03-12T03:35:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[29344]: 5.1a scrub ok 2024-03-12T03:35:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[29344]: 6.17 scrub starts 2024-03-12T03:35:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[29344]: 6.17 scrub ok 2024-03-12T03:35:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[29344]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-12T03:35:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[29344]: Reconfiguring daemon alertmanager.a on smithi169 2024-03-12T03:35:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[39949]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[39949]: 4.16 scrub starts 2024-03-12T03:35:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[39949]: 4.16 scrub ok 2024-03-12T03:35:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[39949]: 5.1a scrub starts 2024-03-12T03:35:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[39949]: 5.1a scrub ok 2024-03-12T03:35:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[39949]: 6.17 scrub starts 2024-03-12T03:35:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[39949]: 6.17 scrub ok 2024-03-12T03:35:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[39949]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-12T03:35:13.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[39949]: Reconfiguring daemon alertmanager.a on smithi169 2024-03-12T03:35:13.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:13 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:13.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:13.755 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:13.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:13.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:13.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:13.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:13.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:13.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:13 smithi173 ceph-mon[33544]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:13 smithi173 ceph-mon[33544]: 4.16 scrub starts 2024-03-12T03:35:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:13 smithi173 ceph-mon[33544]: 4.16 scrub ok 2024-03-12T03:35:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:13 smithi173 ceph-mon[33544]: 5.1a scrub starts 2024-03-12T03:35:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:13 smithi173 ceph-mon[33544]: 5.1a scrub ok 2024-03-12T03:35:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:13 smithi173 ceph-mon[33544]: 6.17 scrub starts 2024-03-12T03:35:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:13 smithi173 ceph-mon[33544]: 6.17 scrub ok 2024-03-12T03:35:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:13 smithi173 ceph-mon[33544]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-12T03:35:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:13 smithi173 ceph-mon[33544]: Reconfiguring daemon alertmanager.a on smithi169 2024-03-12T03:35:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:13 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:13.987 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:14.011 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:14.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:14.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:14.136 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:14.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:14.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:14.261 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:14.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:14.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:14.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:14.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:14.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:14.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:14.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:14.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:14.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:14.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:14.646 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:14.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:14.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:14.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:14.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:14.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:14.901 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:14.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:14 smithi169 ceph-mon[29344]: 3.f scrub starts 2024-03-12T03:35:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:14 smithi169 ceph-mon[29344]: 3.f scrub ok 2024-03-12T03:35:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:14 smithi169 ceph-mon[29344]: 3.8 scrub starts 2024-03-12T03:35:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:14 smithi169 ceph-mon[29344]: 3.8 scrub ok 2024-03-12T03:35:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:14 smithi169 ceph-mon[39949]: 3.f scrub starts 2024-03-12T03:35:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:14 smithi169 ceph-mon[39949]: 3.f scrub ok 2024-03-12T03:35:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:14 smithi169 ceph-mon[39949]: 3.8 scrub starts 2024-03-12T03:35:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:14 smithi169 ceph-mon[39949]: 3.8 scrub ok 2024-03-12T03:35:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:14 smithi173 ceph-mon[33544]: 3.f scrub starts 2024-03-12T03:35:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:14 smithi173 ceph-mon[33544]: 3.f scrub ok 2024-03-12T03:35:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:14 smithi173 ceph-mon[33544]: 3.8 scrub starts 2024-03-12T03:35:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:14 smithi173 ceph-mon[33544]: 3.8 scrub ok 2024-03-12T03:35:15.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:15.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:15.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:15.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:15.132 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:15.157 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:15.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:15.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:15.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:15.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:15.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:15.414 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:15.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:15.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:15.539 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:15.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:15.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:15.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:15.665 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:15.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:15.668 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:15 smithi169 ceph-mon[29344]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:15.668 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:15.669 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:15 smithi169 ceph-mon[39949]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:15.669 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:15.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:15.794 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:15.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:15.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:15.920 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:15.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:15.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:15 smithi173 ceph-mon[33544]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:16.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:16.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:16.046 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:16.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:16.147 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:16.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:16.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:16.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:16.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:16.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:16.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:16.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:16.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:16.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:16.553 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:16.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:16.640 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:16 smithi169 ceph-mon[29344]: 3.2 scrub starts 2024-03-12T03:35:16.640 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:16 smithi169 ceph-mon[29344]: 3.2 scrub ok 2024-03-12T03:35:16.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:16 smithi169 ceph-mon[39949]: 3.2 scrub starts 2024-03-12T03:35:16.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:16 smithi169 ceph-mon[39949]: 3.2 scrub ok 2024-03-12T03:35:16.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:16.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:16.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:16.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:16.785 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:16.809 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:16.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:16.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:16.904 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:35:16.904 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:35:16.904 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-03-12T03:35:16.904 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:35:16.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:16.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:16.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:16.950 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:35:16.950 INFO:teuthology.orchestra.run.smithi169.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-12T03:35:16.950 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:35:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:16 smithi173 ceph-mon[33544]: 3.2 scrub starts 2024-03-12T03:35:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:16 smithi173 ceph-mon[33544]: 3.2 scrub ok 2024-03-12T03:35:17.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:17.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:17.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:17.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:17.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:17.194 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:17.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:17.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:17.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:17.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:17.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:17.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:17.452 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:17.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:17.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:17.580 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:17.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:17.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:17 smithi169 ceph-mon[29344]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:17.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:17 smithi169 ceph-mon[29344]: 6.12 scrub starts 2024-03-12T03:35:17.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:17 smithi169 ceph-mon[29344]: 6.12 scrub ok 2024-03-12T03:35:17.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:17 smithi169 ceph-mon[29344]: 3.b scrub starts 2024-03-12T03:35:17.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:17 smithi169 ceph-mon[29344]: 3.b scrub ok 2024-03-12T03:35:17.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:17 smithi169 ceph-mon[39949]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:17.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:17 smithi169 ceph-mon[39949]: 6.12 scrub starts 2024-03-12T03:35:17.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:17 smithi169 ceph-mon[39949]: 6.12 scrub ok 2024-03-12T03:35:17.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:17 smithi169 ceph-mon[39949]: 3.b scrub starts 2024-03-12T03:35:17.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:17 smithi169 ceph-mon[39949]: 3.b scrub ok 2024-03-12T03:35:17.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:17.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:17.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:17.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:17.833 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:17.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:17.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:17.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:17.959 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:17.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:17 smithi173 ceph-mon[33544]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:17 smithi173 ceph-mon[33544]: 6.12 scrub starts 2024-03-12T03:35:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:17 smithi173 ceph-mon[33544]: 6.12 scrub ok 2024-03-12T03:35:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:17 smithi173 ceph-mon[33544]: 3.b scrub starts 2024-03-12T03:35:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:17 smithi173 ceph-mon[33544]: 3.b scrub ok 2024-03-12T03:35:18.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:18.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:18.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:18.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:18.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:18.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:18.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:18.318 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:18.342 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:18.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:18.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:18.468 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:18.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:18.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:18.570 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:18.593 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:18.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:18.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:18.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:18.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:18.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:18.825 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:18.848 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:18.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:18.913 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:18 smithi169 ceph-mon[39949]: 5.18 scrub starts 2024-03-12T03:35:18.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:18 smithi169 ceph-mon[39949]: 5.18 scrub ok 2024-03-12T03:35:18.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:18 smithi169 ceph-mon[39949]: 6.d scrub starts 2024-03-12T03:35:18.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:18 smithi169 ceph-mon[39949]: 6.d scrub ok 2024-03-12T03:35:18.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:18 smithi169 ceph-mon[29344]: 5.18 scrub starts 2024-03-12T03:35:18.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:18 smithi169 ceph-mon[29344]: 5.18 scrub ok 2024-03-12T03:35:18.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:18 smithi169 ceph-mon[29344]: 6.d scrub starts 2024-03-12T03:35:18.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:18 smithi169 ceph-mon[29344]: 6.d scrub ok 2024-03-12T03:35:18.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:18.974 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:18.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:18.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:18 smithi173 ceph-mon[33544]: 5.18 scrub starts 2024-03-12T03:35:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:18 smithi173 ceph-mon[33544]: 5.18 scrub ok 2024-03-12T03:35:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:18 smithi173 ceph-mon[33544]: 6.d scrub starts 2024-03-12T03:35:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:18 smithi173 ceph-mon[33544]: 6.d scrub ok 2024-03-12T03:35:19.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:19.100 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:19.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:19.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:19.225 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:19.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:19.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:19.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:19.351 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:19.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:19.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:19.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:19.480 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:19.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:19.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:19.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:19.607 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:19.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:19.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:19.731 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:19.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:19.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:19.814 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:19 smithi169 ceph-mon[29344]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-03-12T03:35:19.814 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:19 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:35:19.814 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:19 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:35:19.814 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:19 smithi169 ceph-mon[39949]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-03-12T03:35:19.814 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:19 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:35:19.814 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:19 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:35:19.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:19.857 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:19.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:19.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:19.981 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:19.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:19 smithi173 ceph-mon[33544]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-03-12T03:35:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:19 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:35:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:19 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:35:20.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:20.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:20.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:20.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:20.235 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:20.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:20.337 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:20.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:20.361 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:20.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:20.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:20.491 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:20.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:20.590 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:20 smithi169 ceph-mon[29344]: 4.1a scrub starts 2024-03-12T03:35:20.590 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:20 smithi169 ceph-mon[29344]: 4.1a scrub ok 2024-03-12T03:35:20.590 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:20 smithi169 ceph-mon[29344]: osdmap e96: 8 total, 8 up, 8 in 2024-03-12T03:35:20.591 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:20 smithi169 ceph-mon[39949]: 4.1a scrub starts 2024-03-12T03:35:20.591 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:20 smithi169 ceph-mon[39949]: 4.1a scrub ok 2024-03-12T03:35:20.591 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:20 smithi169 ceph-mon[39949]: osdmap e96: 8 total, 8 up, 8 in 2024-03-12T03:35:20.592 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:20.617 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:20.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:20.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-12T03:35:20.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:20.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:20.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:20.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:20.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:20.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:20.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:20.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:20 smithi173 ceph-mon[33544]: 4.1a scrub starts 2024-03-12T03:35:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:20 smithi173 ceph-mon[33544]: 4.1a scrub ok 2024-03-12T03:35:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:20 smithi173 ceph-mon[33544]: osdmap e96: 8 total, 8 up, 8 in 2024-03-12T03:35:21.004 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:21.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:21.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:21.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:21.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:21.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:21.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:21.360 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:21.385 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:21.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:21.486 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:21.510 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:21.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:21.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:21.636 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:21.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:21.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:21.642 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:35:21.662 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:35:21.681 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:35:21.701 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:35:21.721 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:35:21.738 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:21.743 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:35:21.760 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:21.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:21.782 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:35:21.782 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:21 smithi169 ceph-mon[29344]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:21.782 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:21 smithi169 ceph-mon[29344]: 5.e scrub starts 2024-03-12T03:35:21.782 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:21 smithi169 ceph-mon[29344]: 5.e scrub ok 2024-03-12T03:35:21.782 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:21 smithi169 ceph-mon[39949]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:21.783 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:21 smithi169 ceph-mon[39949]: 5.e scrub starts 2024-03-12T03:35:21.783 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:21 smithi169 ceph-mon[39949]: 5.e scrub ok 2024-03-12T03:35:21.818 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:35:21.820 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:35:21.820 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:35:21.820 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:35:21.820 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:35:21.820 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:35:21.821 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:35:21.821 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:35:21.821 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:35:21.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:21.887 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:21.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:21 smithi173 ceph-mon[33544]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:21 smithi173 ceph-mon[33544]: 5.e scrub starts 2024-03-12T03:35:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:21 smithi173 ceph-mon[33544]: 5.e scrub ok 2024-03-12T03:35:21.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:22.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:22.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:22.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:22.118 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:22.141 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:22.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:22.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:35:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:35:22.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:22.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:22.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:22.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:22.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:22.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:22.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:22.528 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:22.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:22.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:22.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:22.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:22.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:35:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:35:22.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:22.776 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:35:22.778 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'unique_pool_0' created 2024-03-12T03:35:22.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:22.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:22.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:22.916 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:22.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:23.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:23.042 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:23.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:23.144 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:23.146 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:22 smithi169 ceph-mon[29344]: 6.1a scrub starts 2024-03-12T03:35:23.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:22 smithi169 ceph-mon[29344]: 6.1a scrub ok 2024-03-12T03:35:23.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:22 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/209612913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:35:23.147 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:22 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3965812326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:35:23.147 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:22 smithi169 ceph-mon[39949]: 6.1a scrub starts 2024-03-12T03:35:23.147 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:22 smithi169 ceph-mon[39949]: 6.1a scrub ok 2024-03-12T03:35:23.147 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:22 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/209612913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:35:23.147 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:22 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3965812326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:35:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:23.168 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:23.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:23.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:22 smithi173 ceph-mon[33544]: 6.1a scrub starts 2024-03-12T03:35:23.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:22 smithi173 ceph-mon[33544]: 6.1a scrub ok 2024-03-12T03:35:23.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:22 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/209612913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:35:23.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:22 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3965812326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:35:23.269 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:23.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:23.293 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:23.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:23.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:23.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:23.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:23.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:23.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:23.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:23.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:23.677 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:23.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:23.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:23.806 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:23.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:23.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:23.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[29344]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:23.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:23.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/436819608' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-12T03:35:23.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-12T03:35:23.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:35:23.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3965812326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:35:23.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-12T03:35:23.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[29344]: osdmap e97: 8 total, 8 up, 8 in 2024-03-12T03:35:23.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[39949]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:23.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:23.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/436819608' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-12T03:35:23.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-12T03:35:23.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:35:23.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3965812326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:35:23.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-12T03:35:23.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:23 smithi169 ceph-mon[39949]: osdmap e97: 8 total, 8 up, 8 in 2024-03-12T03:35:23.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:23.934 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-12T03:35:23.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:23.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:23.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:24.037 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:24.061 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:24.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:24.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:24.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:24.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:24.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:23 smithi173 ceph-mon[33544]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:35:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:23 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:23 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/436819608' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-12T03:35:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:23 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-12T03:35:24.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:23 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:35:24.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:23 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3965812326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:35:24.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:23 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-12T03:35:24.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:23 smithi173 ceph-mon[33544]: osdmap e97: 8 total, 8 up, 8 in 2024-03-12T03:35:24.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:24.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:24.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:24.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:24.448 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:24.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:24.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:24.574 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:24.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:24.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:24.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:24.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:24.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:24.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:24.805 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:24.833 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:24.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:24.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:24.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:24.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:25.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:25.084 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:24 smithi169 ceph-mon[29344]: osdmap e98: 8 total, 8 up, 8 in 2024-03-12T03:35:25.084 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:24 smithi169 ceph-mon[29344]: pgmap v60: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:25.084 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:24 smithi169 ceph-mon[39949]: osdmap e98: 8 total, 8 up, 8 in 2024-03-12T03:35:25.084 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:24 smithi169 ceph-mon[39949]: pgmap v60: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:25.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:25.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:25.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:25.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:25.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:25.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:24 smithi173 ceph-mon[33544]: osdmap e98: 8 total, 8 up, 8 in 2024-03-12T03:35:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:24 smithi173 ceph-mon[33544]: pgmap v60: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:25.291 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:25.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:25.329 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:25.346 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:25.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:25.448 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:25.472 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:25.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:25.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:25.598 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:25.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:25.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:25.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:25.724 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:25.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:25.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:25.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:25.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:25.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:25.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:25.979 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:25.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:25.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:26.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:26.108 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:26.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:26.124 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:25 smithi169 ceph-mon[29344]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:35:26.124 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:25 smithi169 ceph-mon[29344]: osdmap e99: 8 total, 8 up, 8 in 2024-03-12T03:35:26.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:25 smithi169 ceph-mon[39949]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:35:26.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:25 smithi169 ceph-mon[39949]: osdmap e99: 8 total, 8 up, 8 in 2024-03-12T03:35:26.167 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:35:26.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:26.233 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:26.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:26.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:26.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:25 smithi173 ceph-mon[33544]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:35:26.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:25 smithi173 ceph-mon[33544]: osdmap e99: 8 total, 8 up, 8 in 2024-03-12T03:35:26.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:26.359 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:26.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:26.417 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:35:26.417 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:35:26.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:26.484 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:26.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:26.585 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:26.609 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:26.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:26.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:26.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:26.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:26.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:35:26.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:26.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:26.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:26.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:26.995 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:26.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:27.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:27.098 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:26 smithi169 ceph-mon[29344]: pgmap v62: 148 pgs: 9 unknown, 139 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:27.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:26 smithi169 ceph-mon[39949]: pgmap v62: 148 pgs: 9 unknown, 139 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:27.099 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:26 smithi169 systemd[1]: Stopping Ceph alertmanager.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:35:27.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:27.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:27.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:27.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:26 smithi173 ceph-mon[33544]: pgmap v62: 148 pgs: 9 unknown, 139 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:27.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:27.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:27.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:27.354 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:27.378 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:27.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:27.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:27.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:27.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:27.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:27.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:27.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:27.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:27.739 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:27.764 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:27.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:27.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:27.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:27.897 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:27.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:27.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:28.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:28.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:28.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:28.150 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:28.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:28.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:28.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:28.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:28.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:28.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:28.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:28.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:28.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:28.536 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:28.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:28.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:28.557 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:28.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:28.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:28.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:28.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:28.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:28.793 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:28.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:28.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:28.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:28.919 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:28.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:29.000 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:35:29.000 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:35:29.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:29.044 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:29.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:29.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:29.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:28 smithi169 ceph-mon[29344]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:29.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:28 smithi169 ceph-mon[39949]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:29.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:29.170 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:29.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:29.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:28 smithi173 ceph-mon[33544]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:35:29.271 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:29.295 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:29.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:29.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:29.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:29.423 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:29.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:29.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:29.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:29.549 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:29.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:29.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:29.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:29.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:29.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:29.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:29.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:29.832 INFO:teuthology.orchestra.run.smithi169.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-12T03:35:29.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:29.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:29.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:29.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-12T03:35:30.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:30.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:30.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:30.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:30.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:29 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:35:30.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:29 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:35:30.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:29 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2226810484' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:35:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:29 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:35:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:29 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:35:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:29 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2226810484' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:35:30.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:30.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:29 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:35:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:29 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:35:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:29 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2226810484' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:35:30.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:30.303 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:30.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:30.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:30.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:30.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:30.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:30.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:30.556 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:30.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:30.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:30.658 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:30.682 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:30.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:30.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:30.807 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:30.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:30.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:30.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:30.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:30.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:31.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:31.059 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:31.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:31.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:31.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:31.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:30 smithi169 ceph-mon[39949]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-03-12T03:35:31.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:30 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2226810484' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:35:31.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:30 smithi169 ceph-mon[39949]: osdmap e100: 8 total, 8 up, 8 in 2024-03-12T03:35:31.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:30 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:31.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:30 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:31.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:30 smithi169 ceph-mon[29344]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-03-12T03:35:31.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:30 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2226810484' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:35:31.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:30 smithi169 ceph-mon[29344]: osdmap e100: 8 total, 8 up, 8 in 2024-03-12T03:35:31.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:30 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:31.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:30 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:31.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:31.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:31.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:30 smithi173 ceph-mon[33544]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-03-12T03:35:31.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:30 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2226810484' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:35:31.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:30 smithi173 ceph-mon[33544]: osdmap e100: 8 total, 8 up, 8 in 2024-03-12T03:35:31.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:30 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:31.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:30 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:31.281 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:31.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:31.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:31.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:31.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:31.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:31.439 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:31.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:31.482 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-12T03:35:31.483 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-12T03:35:31.483 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-12T03:35:31.483 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:35:31.486 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-12T03:35:31.486 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-12T03:35:31.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:31.558 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:hints = 1 2024-03-12T03:35:31.559 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-12T03:35:31.559 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Object prefix: benchmark_data_smithi169_79822 2024-03-12T03:35:31.559 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:35:31.559 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 0 0 0 0 0 0 - 0 2024-03-12T03:35:31.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:31.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:31.654 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[74234]: ts=2024-03-12T03:35:31.263Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:35:31.654 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:31 smithi169 podman[79287]: 2024-03-12 03:35:31.284245615 +0000 UTC m=+3.412122566 container died f1777cd585edeb9b5af26e3b828ac507f32e733e80dbf71901ad9e47001d40a0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:31.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:31.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:31.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:31.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:31.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:31.848 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:31.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:31.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:31.922 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:31 smithi169 ceph-mon[29344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:35:31.922 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:31 smithi169 ceph-mon[39949]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:35:31.922 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:35:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:35:31.923 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:31 smithi169 podman[79287]: 2024-03-12 03:35:31.735895201 +0000 UTC m=+3.863772148 container cleanup f1777cd585edeb9b5af26e3b828ac507f32e733e80dbf71901ad9e47001d40a0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:31.923 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:31 smithi169 bash[79287]: ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a 2024-03-12T03:35:31.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:31.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:31.990 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:31.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:32.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:32.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:32.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:32.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:32.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:31 smithi173 ceph-mon[33544]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:35:32.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:32.255 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:32.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:32.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:32.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:32.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:32.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:32.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:32.534 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:32.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:32.559 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 1 16 3258 3242 202.614 202.625 0.00582424 0.00491995 2024-03-12T03:35:32.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:32.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:32.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:32.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:35:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:35:32.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:32.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:32.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:32.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:32.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:32.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:32.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:32.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:33.037 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:33.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:33.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:33.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:33.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:32 smithi169 ceph-mon[29344]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:33.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:32 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:33.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:32 smithi169 ceph-mon[39949]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:33.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:32 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:33.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:33.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:32 smithi173 ceph-mon[33544]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:35:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:32 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:33.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:33.344 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:33.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:33.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:33.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:33.492 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:33.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:33.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:33.559 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 2 16 6319 6303 196.917 191.312 0.00586228 0.00506636 2024-03-12T03:35:33.595 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:33.636 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:33.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:33.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:33.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:33.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:33.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:33.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:33.913 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:33.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:34.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:34.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:34.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:34.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:34.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:34.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:34.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:34.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:34.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:34.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:34.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:34.448 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:34.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:34.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:34.559 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 3 16 9316 9300 193.692 187.312 0.00311944 0.00515213 2024-03-12T03:35:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:34.587 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:34.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:34.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:34.726 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:34.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:34.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:34.860 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:34.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:34.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:34.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:34.989 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:34.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:35.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:35.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:35.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:35.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:35.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:35.253 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:35.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:35.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:35.315 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:35 smithi169 ceph-mon[29344]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:35:35.315 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:35 smithi169 ceph-mon[39949]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:35:35.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:35.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:35.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:35.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:35 smithi173 ceph-mon[33544]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:35:35.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:35.534 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:35.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:35.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:35.560 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 4 16 12028 12012 187.641 169.5 0.00572209 0.00532064 2024-03-12T03:35:35.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:35.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:35.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:35.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:35.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:35.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:35.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:35.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:35.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:35.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:36.044 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:36.094 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:36.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:36.183 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:35:36.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:36.244 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:36.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:36.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:36.283 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:36 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:35:36.283 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:36 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:35:36.283 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:36 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3630799624' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:35:36.283 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:36 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:35:36.284 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:36 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:35:36.284 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:36 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:35:36.284 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:36 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3630799624' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:35:36.284 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:36 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:35:36.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:36.382 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:36.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:36.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:36.548 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:36 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:35:36.548 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:36 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:35:36.548 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:36 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3630799624' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:35:36.548 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:36 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:35:36.560 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 5 16 14601 14585 182.274 160.812 0.00787011 0.00547822 2024-03-12T03:35:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:36.580 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:36.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:36.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:36.713 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:36.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:36.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:36.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:36.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:36.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:36.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:36.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:36.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:37.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:37.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:37.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:37.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:37.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:37.248 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:37.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:37.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:37.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:37.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:37.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:37.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:37 smithi169 ceph-mon[39949]: pgmap v68: 148 pgs: 148 active+clean; 345 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 34 MiB/s wr, 552 op/s 2024-03-12T03:35:37.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:37 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:35:37.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:37 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:35:37.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:37 smithi169 ceph-mon[39949]: osdmap e101: 8 total, 8 up, 8 in 2024-03-12T03:35:37.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:37 smithi169 ceph-mon[29344]: pgmap v68: 148 pgs: 148 active+clean; 345 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 34 MiB/s wr, 552 op/s 2024-03-12T03:35:37.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:37 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:35:37.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:37 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:35:37.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:37 smithi169 ceph-mon[29344]: osdmap e101: 8 total, 8 up, 8 in 2024-03-12T03:35:37.424 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:37 smithi169 podman[79773]: 2024-03-12 03:35:37.134153559 +0000 UTC m=+5.849834272 container remove f1777cd585edeb9b5af26e3b828ac507f32e733e80dbf71901ad9e47001d40a0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:37.425 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:37 smithi169 podman[79773]: 2024-03-12 03:35:37.252582879 +0000 UTC m=+5.968263629 volume remove 382f4ae691a9068c46eb58894cb2bfbf8092adf952abe51e3404d0640c963b20 2024-03-12T03:35:37.425 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:37 smithi169 bash[80223]: Error: no container with ID f1777cd585edeb9b5af26e3b828ac507f32e733e80dbf71901ad9e47001d40a0 found in database: no such container 2024-03-12T03:35:37.425 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:37 smithi169 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-12T03:35:37.425 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:37 smithi169 systemd[1]: Stopped Ceph alertmanager.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:35:37.425 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:37 smithi169 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.045s CPU time. 2024-03-12T03:35:37.425 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:37 smithi169 systemd[1]: Starting Ceph alertmanager.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:35:37.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:37.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:37 smithi173 ceph-mon[33544]: pgmap v68: 148 pgs: 148 active+clean; 345 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 34 MiB/s wr, 552 op/s 2024-03-12T03:35:37.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:37 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:35:37.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:37 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:35:37.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:37 smithi173 ceph-mon[33544]: osdmap e101: 8 total, 8 up, 8 in 2024-03-12T03:35:37.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:37.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:37.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:37.538 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:37.560 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 6 16 17209 17193 179.06 163 0.0158589 0.00557598 2024-03-12T03:35:37.579 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:35:37.579 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:35:37.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:37.644 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:37.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:37.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:37.795 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:37.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:37.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:37.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:37.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:37.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:38.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:38.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:38.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:38.163 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:38.205 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:38.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:38.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:38.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:38.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:38.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-12T03:35:38.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:38.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:38.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:38.560 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 7 15 19656 19641 175.332 153 0.0057309 0.005695 2024-03-12T03:35:38.569 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:38.613 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:38.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:38.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:38.744 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:38.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:38.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:38.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:38.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:38.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:39.019 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:39.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:39.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:39.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:39.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:39.209 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:39 smithi169 ceph-mon[39949]: pgmap v70: 148 pgs: 148 active+clean; 787 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-12T03:35:39.210 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:39 smithi169 ceph-mon[29344]: pgmap v70: 148 pgs: 148 active+clean; 787 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-12T03:35:39.251 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:39.300 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:39.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:39.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:39.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:39.446 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:39.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:39 smithi173 ceph-mon[33544]: pgmap v70: 148 pgs: 148 active+clean; 787 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-12T03:35:39.544 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:35:39.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:39.560 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 8 16 21900 21884 170.938 140.188 0.00428011 0.00584248 2024-03-12T03:35:39.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:39.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:39.593 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:35:39.636 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:35:39.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:39.682 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:35:39.719 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:35:39.719 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:39.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:39.759 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:35:39.796 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:35:39.821 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:39.834 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:35:39.838 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:35:39.839 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:35:39.839 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:35:39.839 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:35:39.839 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:35:39.839 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:35:39.839 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:35:39.839 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:35:39.861 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:39.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:39.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:39.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:40.002 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:40.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:40.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:40.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:40.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:40.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:40.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:40.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:40.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:40.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:40.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:40.407 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:40.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:40 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1547787099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:35:40.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:40.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:40.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:40.560 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 9 15 23832 23817 165.367 120.812 0.00455219 0.0060414 2024-03-12T03:35:40.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:40.671 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:40.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:40.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:40 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1547787099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:35:40.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:40 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1547787099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:35:40.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:40.797 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:40.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:40.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:40.935 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:40.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:40.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:41.003 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:35:41.037 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:41.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:41.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:41.166 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:41.211 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:41.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:41.312 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:41.353 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:41.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:41.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:41 smithi169 ceph-mon[29344]: pgmap v71: 148 pgs: 148 active+clean; 787 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-12T03:35:41.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:41 smithi169 ceph-mon[39949]: pgmap v71: 148 pgs: 148 active+clean; 787 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-12T03:35:41.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:41 smithi173 ceph-mon[33544]: pgmap v71: 148 pgs: 148 active+clean; 787 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-12T03:35:41.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:41.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:41.560 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 10 16 25711 25695 160.568 117.375 0.00292654 0.00622201 2024-03-12T03:35:41.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:41.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:41.628 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:41.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:41.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:41.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:41.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:41.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:41.914 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:41.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:41.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:42.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:42.047 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:42.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:42.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:42.172 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:35:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:35:42.188 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:42.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:42.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:42.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:42.332 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:42.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:42.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:42.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:42.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:42.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:42.561 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 11 16 27418 27402 155.668 106.688 0.00176228 0.00638503 2024-03-12T03:35:42.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:42.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:42.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:42.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:42.735 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:42.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:42.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:42.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:35:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:35:42.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:42.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:42.878 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:42.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:42.979 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:43.019 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:43.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:43.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:43.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:43.154 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:43.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:43.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:43.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:43.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:43.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:43.429 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:43.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:43.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:43 smithi173 ceph-mon[33544]: pgmap v72: 148 pgs: 148 active+clean; 1.5 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-12T03:35:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:43 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:43.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:43.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:43.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:43.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:43.560 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 12 16 29245 29229 152.211 114.188 0.00351634 0.00654381 2024-03-12T03:35:43.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:43.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:43 smithi169 ceph-mon[29344]: pgmap v72: 148 pgs: 148 active+clean; 1.5 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-12T03:35:43.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:43 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:43.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:43 smithi169 ceph-mon[39949]: pgmap v72: 148 pgs: 148 active+clean; 1.5 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-12T03:35:43.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:43 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:43.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:43.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:43.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:43.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:43.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:43.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:43.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:43.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:43.942 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:43.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:43.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:44.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:44.072 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:44.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:44.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:44.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:44.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:44.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:44.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:44.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:44.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:44.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:44.433 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:44.477 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:44.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:44.561 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 13 15 30645 30630 147.237 87.5625 0.0198328 0.00678677 2024-03-12T03:35:44.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:44.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:44.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:44.709 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:44.749 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:44.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:44.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:44.897 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:44.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:44.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:44.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:45.048 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:45.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:45.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:45.173 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:44 smithi169 podman[80845]: 2024-03-12 03:35:44.594421721 +0000 UTC m=+2.658117544 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-12T03:35:45.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:45.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:45.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:45.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:45.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:45.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:45.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:45.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:45 smithi173 ceph-mon[33544]: pgmap v73: 148 pgs: 148 active+clean; 1.5 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-12T03:35:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:45 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:45 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:45.521 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-mon[29344]: pgmap v73: 148 pgs: 148 active+clean; 1.5 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-12T03:35:45.521 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:45.521 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:45.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-mon[39949]: pgmap v73: 148 pgs: 148 active+clean; 1.5 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-12T03:35:45.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:45.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:35:45.521 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 podman[80845]: 2024-03-12 03:35:45.400976284 +0000 UTC m=+3.464672084 volume create 45aa4dd8f43e7447f7fb95b648003a1bac6448b7acc22a0ff0707591098db799 2024-03-12T03:35:45.522 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 podman[80845]: 2024-03-12 03:35:45.519564107 +0000 UTC m=+3.583259912 container create 7561c849829672b755d52fb85640fb77a5c1387d0d1187b8776661531e87bf78 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:45.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:45.561 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 14 16 32459 32443 144.809 113.312 0.00309622 0.00690018 2024-03-12T03:35:45.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:45.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:45.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:45.712 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:45.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:45.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:45.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:45.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:45.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:45.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:45.947 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:45.988 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:45.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:46.080 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:46.088 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:46.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:46.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:46.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:46.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:46.169 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 podman[80845]: 2024-03-12 03:35:45.897696729 +0000 UTC m=+3.961392554 container init 7561c849829672b755d52fb85640fb77a5c1387d0d1187b8776661531e87bf78 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:46.169 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 podman[80845]: 2024-03-12 03:35:45.905741602 +0000 UTC m=+3.969437405 container start 7561c849829672b755d52fb85640fb77a5c1387d0d1187b8776661531e87bf78 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:46.169 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[81118]: ts=2024-03-12T03:35:45.927Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-12T03:35:46.169 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[81118]: ts=2024-03-12T03:35:45.928Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-12T03:35:46.169 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[81118]: ts=2024-03-12T03:35:45.930Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.169 port=9094 2024-03-12T03:35:46.169 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[81118]: ts=2024-03-12T03:35:45.933Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-12T03:35:46.170 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[81118]: ts=2024-03-12T03:35:45.982Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:35:46.170 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[81118]: ts=2024-03-12T03:35:45.982Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:35:46.170 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[81118]: ts=2024-03-12T03:35:45.987Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-12T03:35:46.170 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[81118]: ts=2024-03-12T03:35:45.987Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-12T03:35:46.170 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:45 smithi169 bash[80845]: 7561c849829672b755d52fb85640fb77a5c1387d0d1187b8776661531e87bf78 2024-03-12T03:35:46.170 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:46 smithi169 systemd[1]: Started Ceph alertmanager.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:35:46.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:46.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:46.264 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:46.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:46.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:46.427 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:46.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:46.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:46.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:46.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:46.561 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 15 16 34165 34149 142.263 106.625 0.051222 0.00701683 2024-03-12T03:35:46.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:46.689 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:46.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:46.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:46.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:46.834 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:46.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:46.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:46.972 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:46.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:46.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:47.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:47.111 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:47.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:47.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:47 smithi169 ceph-mon[39949]: pgmap v74: 148 pgs: 148 active+clean; 1.8 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 147 MiB/s wr, 2.36k op/s 2024-03-12T03:35:47.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:47 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:47.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:47 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:47.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:47 smithi169 ceph-mon[39949]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:35:47.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:47 smithi169 ceph-mon[39949]: Reconfiguring daemon prometheus.a on smithi173 2024-03-12T03:35:47.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:47 smithi169 ceph-mon[29344]: pgmap v74: 148 pgs: 148 active+clean; 1.8 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 147 MiB/s wr, 2.36k op/s 2024-03-12T03:35:47.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:47 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:47.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:47 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:47.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:47 smithi169 ceph-mon[29344]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:35:47.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:47 smithi169 ceph-mon[29344]: Reconfiguring daemon prometheus.a on smithi173 2024-03-12T03:35:47.214 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:47.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:47.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:35:47.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:47.424 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:47.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:47.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:47 smithi173 ceph-mon[33544]: pgmap v74: 148 pgs: 148 active+clean; 1.8 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 147 MiB/s wr, 2.36k op/s 2024-03-12T03:35:47.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:47 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:47.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:47 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:47.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:47 smithi173 ceph-mon[33544]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:35:47.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:47 smithi173 ceph-mon[33544]: Reconfiguring daemon prometheus.a on smithi173 2024-03-12T03:35:47.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:47.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:47.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:47.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:47.564 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 16 16 36012 35996 140.584 115.438 0.0069206 0.00710771 2024-03-12T03:35:47.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:47.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:47.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:47.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:47.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:47.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:47.815 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:47.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:47.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:47.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:47.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:47.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:48.047 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:48.088 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:48.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:48.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:48.231 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:48.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:48.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:48.372 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:48.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:48.378 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:47 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[81118]: ts=2024-03-12T03:35:47.933Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000147461s 2024-03-12T03:35:48.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:48.519 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:48.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:48.562 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 17 16 37479 37463 137.704 91.6875 0.00748316 0.00725682 2024-03-12T03:35:48.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:48.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:48.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:48.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:48.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:48.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:48.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:48.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:48.880 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:48.920 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:48.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:49.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:49.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:49.064 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:49.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:49.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:49.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:49 smithi169 ceph-mon[29344]: pgmap v75: 148 pgs: 148 active+clean; 2.0 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 882 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-03-12T03:35:49.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:49 smithi169 ceph-mon[39949]: pgmap v75: 148 pgs: 148 active+clean; 2.0 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 882 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-03-12T03:35:49.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:49.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:49.249 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:35:49.254 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:35:49.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:49.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:49.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:49.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:49.441 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:49 smithi173 ceph-mon[33544]: pgmap v75: 148 pgs: 148 active+clean; 2.0 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 882 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-03-12T03:35:49.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:49.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:49.562 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 18 16 39112 39096 135.724 102.062 0.00709186 0.0073623 2024-03-12T03:35:49.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:49.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:49.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:49.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:49.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:49.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:49.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:49.892 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:49.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:49.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:49.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:50.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:50.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:50.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:50.126 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:50.172 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:50.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:50.217 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:35:50.217 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:35:50.217 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.217 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:40215 7.0 e101/97 7:0b3e94e0:::benchmark_data_smithi169_79822_object40214:head v 101'2513, mlcod=101'2509)", 2024-03-12T03:35:50.217 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:50.211843+0000", 2024-03-12T03:35:50.217 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0027571710000000001, 2024-03-12T03:35:50.217 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027968749999999999, 2024-03-12T03:35:50.217 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:35:50.217 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:35:50.217 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:35:50.217 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.218 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:35:50.218 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.211843+0000", 2024-03-12T03:35:50.218 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:35:50.218 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.218 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.218 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:35:50.218 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.211843+0000", 2024-03-12T03:35:50.218 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:35:50.218 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.218 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.218 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:35:50.218 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.211844+0000", 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1500000000000004e-07 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.211876+0000", 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2842999999999997e-05 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.211891+0000", 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4436e-05 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.211898+0000", 2024-03-12T03:35:50.219 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.233e-06 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.214505+0000", 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026064819999999998 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.214523+0000", 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8770000000000002e-05 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:35:50.220 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:40220 7.d e101/97 7:bbd62175:::benchmark_data_smithi169_79822_object40219:head v 101'2477, mlcod=101'2475)", 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:50.213713+0000", 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.00088679999999999998, 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00099967000000000003, 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.213713+0000", 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.221 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.213713+0000", 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.213714+0000", 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8100000000000001e-07 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.213775+0000", 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1105999999999994e-05 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.222 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.213791+0000", 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5475e-05 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.213798+0000", 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2869999999999999e-06 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.213876+0000", 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7489000000000003e-05 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:35:50.223 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:35:50.224 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:35:50.224 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:50.213895+0000", 2024-03-12T03:35:50.224 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.916e-05 2024-03-12T03:35:50.224 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:35:50.224 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:35:50.224 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:35:50.224 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:35:50.224 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:35:50.224 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 2 2024-03-12T03:35:50.224 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:35:50.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:50.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:50.308 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:50.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:50.409 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:50 smithi169 ceph-mon[29344]: from='client.24652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:35:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:50 smithi169 ceph-mon[39949]: from='client.24652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:35:50.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:50.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:50.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:50.454 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:50 smithi173 ceph-mon[33544]: from='client.24652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:35:50.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:50.562 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 19 15 40836 40821 134.255 107.812 0.0110985 0.00744321 2024-03-12T03:35:50.575 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-03-12T03:35:47.787526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2157386152,"num_objects":33140,"num_object_clones":0,"num_object_copies":99420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33140,"num_whiteouts":0,"num_read":1361,"num_read_kb":1175,"num_write":33717,"num_write_kb":2108701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33743,"ondisk_log_size":33743,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6371772,"kb_used_data":6073724,"kb_used_omap":12,"kb_used_meta":297907,"kb_avail":743589444,"statfs":{"total":767960285184,"available":761435590656,"internally_reserved":0,"allocated":6219493376,"data_stored":6216503456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":305057328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,9,10],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1331953664,"num_objects":20324,"num_object_clones":0,"num_object_copies":60972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20324,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":20324,"num_write_kb":1300736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.609126"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.175745+0000","last_change":"2024-03-12T03:35:21.106867+0000","last_active":"2024-03-12T03:35:36.175745+0000","last_peered":"2024-03-12T03:35:36.175745+0000","last_clean":"2024-03-12T03:35:36.175745+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:35:36.175745+0000","last_undegraded":"2024-03-12T03:35:36.175745+0000","last_fullsized":"2024-03-12T03:35:36.175745+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:21.106755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:21.106755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:24.765060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":79,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183208+0000","last_change":"2024-03-12T03:34:29.244162+0000","last_active":"2024-03-12T03:35:36.183208+0000","last_peered":"2024-03-12T03:35:36.183208+0000","last_clean":"2024-03-12T03:35:36.183208+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:35:36.183208+0000","last_undegraded":"2024-03-12T03:35:36.183208+0000","last_fullsized":"2024-03-12T03:35:36.183208+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:29.244078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:29.244078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.593696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.181858+0000","last_change":"2024-03-12T03:34:48.829003+0000","last_active":"2024-03-12T03:35:36.181858+0000","last_peered":"2024-03-12T03:35:36.181858+0000","last_clean":"2024-03-12T03:35:36.181858+0000","last_became_active":"2024-03-12T03:33:21.573080+0000","last_became_peered":"2024-03-12T03:33:21.573080+0000","last_unstale":"2024-03-12T03:35:36.181858+0000","last_undegraded":"2024-03-12T03:35:36.181858+0000","last_fullsized":"2024-03-12T03:35:36.181858+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:48.828926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:48.828926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:15.319549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":103,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.175672+0000","last_change":"2024-03-12T03:33:23.164433+0000","last_active":"2024-03-12T03:35:36.175672+0000","last_peered":"2024-03-12T03:35:36.175672+0000","last_clean":"2024-03-12T03:35:36.175672+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:35:36.175672+0000","last_undegraded":"2024-03-12T03:35:36.175672+0000","last_fullsized":"2024-03-12T03:35:36.175672+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188763+0000","last_change":"2024-03-12T03:33:27.382583+0000","last_active":"2024-03-12T03:35:36.188763+0000","last_peered":"2024-03-12T03:35:36.188763+0000","last_clean":"2024-03-12T03:35:36.188763+0000","last_became_active":"2024-03-12T03:33:27.382050+0000","last_became_peered":"2024-03-12T03:33:27.382050+0000","last_unstale":"2024-03-12T03:35:36.188763+0000","last_undegraded":"2024-03-12T03:35:36.188763+0000","last_fullsized":"2024-03-12T03:35:36.188763+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.001592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":79,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.175876+0000","last_change":"2024-03-12T03:35:17.100474+0000","last_active":"2024-03-12T03:35:36.175876+0000","last_peered":"2024-03-12T03:35:36.175876+0000","last_clean":"2024-03-12T03:35:36.175876+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:35:36.175876+0000","last_undegraded":"2024-03-12T03:35:36.175876+0000","last_fullsized":"2024-03-12T03:35:36.175876+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:17.100375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:17.100375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:39.744648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188189+0000","last_change":"2024-03-12T03:34:41.143075+0000","last_active":"2024-03-12T03:35:36.188189+0000","last_peered":"2024-03-12T03:35:36.188189+0000","last_clean":"2024-03-12T03:35:36.188189+0000","last_became_active":"2024-03-12T03:33:21.572876+0000","last_became_peered":"2024-03-12T03:33:21.572876+0000","last_unstale":"2024-03-12T03:35:36.188189+0000","last_undegraded":"2024-03-12T03:35:36.188189+0000","last_fullsized":"2024-03-12T03:35:36.188189+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.142994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.165643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"94'50","reported_seq":167,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.187743+0000","last_change":"2024-03-12T03:35:05.042142+0000","last_active":"2024-03-12T03:35:36.187743+0000","last_peered":"2024-03-12T03:35:36.187743+0000","last_clean":"2024-03-12T03:35:36.187743+0000","last_became_active":"2024-03-12T03:33:23.161549+0000","last_became_peered":"2024-03-12T03:33:23.161549+0000","last_unstale":"2024-03-12T03:35:36.187743+0000","last_undegraded":"2024-03-12T03:35:36.187743+0000","last_fullsized":"2024-03-12T03:35:36.187743+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.042030+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:12.264888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.181612+0000","last_change":"2024-03-12T03:35:00.731143+0000","last_active":"2024-03-12T03:35:36.181612+0000","last_peered":"2024-03-12T03:35:36.181612+0000","last_clean":"2024-03-12T03:35:36.181612+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:35:36.181612+0000","last_undegraded":"2024-03-12T03:35:36.181612+0000","last_fullsized":"2024-03-12T03:35:36.181612+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:00.731066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.731066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:14.276977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":79,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183353+0000","last_change":"2024-03-12T03:34:50.189337+0000","last_active":"2024-03-12T03:35:36.183353+0000","last_peered":"2024-03-12T03:35:36.183353+0000","last_clean":"2024-03-12T03:35:36.183353+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:35:36.183353+0000","last_undegraded":"2024-03-12T03:35:36.183353+0000","last_fullsized":"2024-03-12T03:35:36.183353+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:50.189256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:50.189256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:52.054548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":87,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183541+0000","last_change":"2024-03-12T03:34:51.188063+0000","last_active":"2024-03-12T03:35:36.183541+0000","last_peered":"2024-03-12T03:35:36.183541+0000","last_clean":"2024-03-12T03:35:36.183541+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:35:36.183541+0000","last_undegraded":"2024-03-12T03:35:36.183541+0000","last_fullsized":"2024-03-12T03:35:36.183541+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:34:51.187976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:08.066983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'21","reported_seq":123,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.175914+0000","last_change":"2024-03-12T03:35:19.084309+0000","last_active":"2024-03-12T03:35:36.175914+0000","last_peered":"2024-03-12T03:35:36.175914+0000","last_clean":"2024-03-12T03:35:36.175914+0000","last_became_active":"2024-03-12T03:33:23.162749+0000","last_became_peered":"2024-03-12T03:33:23.162749+0000","last_unstale":"2024-03-12T03:35:36.175914+0000","last_undegraded":"2024-03-12T03:35:36.175914+0000","last_fullsized":"2024-03-12T03:35:36.175914+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:35:19.084191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:19.084191+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:29.257405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183060+0000","last_change":"2024-03-12T03:35:05.181569+0000","last_active":"2024-03-12T03:35:36.183060+0000","last_peered":"2024-03-12T03:35:36.183060+0000","last_clean":"2024-03-12T03:35:36.183060+0000","last_became_active":"2024-03-12T03:33:27.187943+0000","last_became_peered":"2024-03-12T03:33:27.187943+0000","last_unstale":"2024-03-12T03:35:36.183060+0000","last_undegraded":"2024-03-12T03:35:36.183060+0000","last_fullsized":"2024-03-12T03:35:36.183060+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:05.181454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.181454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:33.321570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":79,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.186986+0000","last_change":"2024-03-12T03:35:11.909632+0000","last_active":"2024-03-12T03:35:36.186986+0000","last_peered":"2024-03-12T03:35:36.186986+0000","last_clean":"2024-03-12T03:35:36.186986+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:35:36.186986+0000","last_undegraded":"2024-03-12T03:35:36.186986+0000","last_fullsized":"2024-03-12T03:35:36.186986+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:11.909537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.909537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:26.882852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":87,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183606+0000","last_change":"2024-03-12T03:34:49.175167+0000","last_active":"2024-03-12T03:35:36.183606+0000","last_peered":"2024-03-12T03:35:36.183606+0000","last_clean":"2024-03-12T03:35:36.183606+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:35:36.183606+0000","last_undegraded":"2024-03-12T03:35:36.183606+0000","last_fullsized":"2024-03-12T03:35:36.183606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.175091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.175091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:59.268389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"86'14","reported_seq":122,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.176057+0000","last_change":"2024-03-12T03:34:23.795189+0000","last_active":"2024-03-12T03:35:36.176057+0000","last_peered":"2024-03-12T03:35:36.176057+0000","last_clean":"2024-03-12T03:35:36.176057+0000","last_became_active":"2024-03-12T03:33:50.155908+0000","last_became_peered":"2024-03-12T03:33:50.155908+0000","last_unstale":"2024-03-12T03:35:36.176057+0000","last_undegraded":"2024-03-12T03:35:36.176057+0000","last_fullsized":"2024-03-12T03:35:36.176057+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.795040+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":68,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.176540+0000","last_change":"2024-03-12T03:33:27.380906+0000","last_active":"2024-03-12T03:35:36.176540+0000","last_peered":"2024-03-12T03:35:36.176540+0000","last_clean":"2024-03-12T03:35:36.176540+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:35:36.176540+0000","last_undegraded":"2024-03-12T03:35:36.176540+0000","last_fullsized":"2024-03-12T03:35:36.176540+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":79,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.180840+0000","last_change":"2024-03-12T03:34:26.233162+0000","last_active":"2024-03-12T03:35:36.180840+0000","last_peered":"2024-03-12T03:35:36.180840+0000","last_clean":"2024-03-12T03:35:36.180840+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:35:36.180840+0000","last_undegraded":"2024-03-12T03:35:36.180840+0000","last_fullsized":"2024-03-12T03:35:36.180840+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:26.233072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:26.233072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.096475+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":87,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183130+0000","last_change":"2024-03-12T03:34:50.784072+0000","last_active":"2024-03-12T03:35:36.183130+0000","last_peered":"2024-03-12T03:35:36.183130+0000","last_clean":"2024-03-12T03:35:36.183130+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:35:36.183130+0000","last_undegraded":"2024-03-12T03:35:36.183130+0000","last_fullsized":"2024-03-12T03:35:36.183130+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:50.783950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:50.783950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.176860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":108,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188075+0000","last_change":"2024-03-12T03:34:58.889062+0000","last_active":"2024-03-12T03:35:36.188075+0000","last_peered":"2024-03-12T03:35:36.188075+0000","last_clean":"2024-03-12T03:35:36.188075+0000","last_became_active":"2024-03-12T03:33:23.160702+0000","last_became_peered":"2024-03-12T03:33:23.160702+0000","last_unstale":"2024-03-12T03:35:36.188075+0000","last_undegraded":"2024-03-12T03:35:36.188075+0000","last_fullsized":"2024-03-12T03:35:36.188075+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:34:58.888971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.888971+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:28.745266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":68,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.191904+0000","last_change":"2024-03-12T03:33:27.381440+0000","last_active":"2024-03-12T03:35:36.191904+0000","last_peered":"2024-03-12T03:35:36.191904+0000","last_clean":"2024-03-12T03:35:36.191904+0000","last_became_active":"2024-03-12T03:33:27.380880+0000","last_became_peered":"2024-03-12T03:33:27.380880+0000","last_unstale":"2024-03-12T03:35:36.191904+0000","last_undegraded":"2024-03-12T03:35:36.191904+0000","last_fullsized":"2024-03-12T03:35:36.191904+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.001693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":79,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.176398+0000","last_change":"2024-03-12T03:34:36.949207+0000","last_active":"2024-03-12T03:35:36.176398+0000","last_peered":"2024-03-12T03:35:36.176398+0000","last_clean":"2024-03-12T03:35:36.176398+0000","last_became_active":"2024-03-12T03:33:25.181087+0000","last_became_peered":"2024-03-12T03:33:25.181087+0000","last_unstale":"2024-03-12T03:35:36.176398+0000","last_undegraded":"2024-03-12T03:35:36.176398+0000","last_fullsized":"2024-03-12T03:35:36.176398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.949124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.949124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:57.461161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":87,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.175909+0000","last_change":"2024-03-12T03:34:44.029203+0000","last_active":"2024-03-12T03:35:36.175909+0000","last_peered":"2024-03-12T03:35:36.175909+0000","last_clean":"2024-03-12T03:35:36.175909+0000","last_became_active":"2024-03-12T03:33:21.573218+0000","last_became_peered":"2024-03-12T03:33:21.573218+0000","last_unstale":"2024-03-12T03:35:36.175909+0000","last_undegraded":"2024-03-12T03:35:36.175909+0000","last_fullsized":"2024-03-12T03:35:36.175909+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:44.029060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.029060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.635332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":110,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188335+0000","last_change":"2024-03-12T03:33:23.162704+0000","last_active":"2024-03-12T03:35:36.188335+0000","last_peered":"2024-03-12T03:35:36.188335+0000","last_clean":"2024-03-12T03:35:36.188335+0000","last_became_active":"2024-03-12T03:33:23.162427+0000","last_became_peered":"2024-03-12T03:33:23.162427+0000","last_unstale":"2024-03-12T03:35:36.188335+0000","last_undegraded":"2024-03-12T03:35:36.188335+0000","last_fullsized":"2024-03-12T03:35:36.188335+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.001663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.187312+0000","last_change":"2024-03-12T03:34:37.658704+0000","last_active":"2024-03-12T03:35:36.187312+0000","last_peered":"2024-03-12T03:35:36.187312+0000","last_clean":"2024-03-12T03:35:36.187312+0000","last_became_active":"2024-03-12T03:33:27.189152+0000","last_became_peered":"2024-03-12T03:33:27.189152+0000","last_unstale":"2024-03-12T03:35:36.187312+0000","last_undegraded":"2024-03-12T03:35:36.187312+0000","last_fullsized":"2024-03-12T03:35:36.187312+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:34:37.658602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:37.658602+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:38.697166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":120,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.647567+0000","last_change":"2024-03-12T03:34:33.696207+0000","last_active":"2024-03-12T03:35:36.647567+0000","last_peered":"2024-03-12T03:35:36.647567+0000","last_clean":"2024-03-12T03:35:36.647567+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:35:36.647567+0000","last_undegraded":"2024-03-12T03:35:36.647567+0000","last_fullsized":"2024-03-12T03:35:36.647567+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:33.696102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:33.696102+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183283+0000","last_change":"2024-03-12T03:34:43.324075+0000","last_active":"2024-03-12T03:35:36.183283+0000","last_peered":"2024-03-12T03:35:36.183283+0000","last_clean":"2024-03-12T03:35:36.183283+0000","last_became_active":"2024-03-12T03:33:21.097218+0000","last_became_peered":"2024-03-12T03:33:21.097218+0000","last_unstale":"2024-03-12T03:35:36.183283+0000","last_undegraded":"2024-03-12T03:35:36.183283+0000","last_fullsized":"2024-03-12T03:35:36.183283+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:34:43.323991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.323991+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:05.428737+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'17","reported_seq":112,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.182822+0000","last_change":"2024-03-12T03:35:03.718267+0000","last_active":"2024-03-12T03:35:36.182822+0000","last_peered":"2024-03-12T03:35:36.182822+0000","last_clean":"2024-03-12T03:35:36.182822+0000","last_became_active":"2024-03-12T03:33:23.552812+0000","last_became_peered":"2024-03-12T03:33:23.552812+0000","last_unstale":"2024-03-12T03:35:36.182822+0000","last_undegraded":"2024-03-12T03:35:36.182822+0000","last_fullsized":"2024-03-12T03:35:36.182822+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:35:03.718123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:03.718123+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.467204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.181057+0000","last_change":"2024-03-12T03:34:59.373439+0000","last_active":"2024-03-12T03:35:36.181057+0000","last_peered":"2024-03-12T03:35:36.181057+0000","last_clean":"2024-03-12T03:35:36.181057+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:35:36.181057+0000","last_undegraded":"2024-03-12T03:35:36.181057+0000","last_fullsized":"2024-03-12T03:35:36.181057+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:59.373323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:59.373323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:20.913803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.182553+0000","last_change":"2024-03-12T03:34:47.832795+0000","last_active":"2024-03-12T03:35:36.182553+0000","last_peered":"2024-03-12T03:35:36.182553+0000","last_clean":"2024-03-12T03:35:36.182553+0000","last_became_active":"2024-03-12T03:33:25.178346+0000","last_became_peered":"2024-03-12T03:33:25.178346+0000","last_unstale":"2024-03-12T03:35:36.182553+0000","last_undegraded":"2024-03-12T03:35:36.182553+0000","last_fullsized":"2024-03-12T03:35:36.182553+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.832716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.832716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.392333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":85,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.189614+0000","last_change":"2024-03-12T03:33:21.095860+0000","last_active":"2024-03-12T03:35:36.189614+0000","last_peered":"2024-03-12T03:35:36.189614+0000","last_clean":"2024-03-12T03:35:36.189614+0000","last_became_active":"2024-03-12T03:33:21.095556+0000","last_became_peered":"2024-03-12T03:33:21.095556+0000","last_unstale":"2024-03-12T03:35:36.189614+0000","last_undegraded":"2024-03-12T03:35:36.189614+0000","last_fullsized":"2024-03-12T03:35:36.189614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.001627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"94'23","reported_seq":121,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.177136+0000","last_change":"2024-03-12T03:34:52.730098+0000","last_active":"2024-03-12T03:35:36.177136+0000","last_peered":"2024-03-12T03:35:36.177136+0000","last_clean":"2024-03-12T03:35:36.177136+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:35:36.177136+0000","last_undegraded":"2024-03-12T03:35:36.177136+0000","last_fullsized":"2024-03-12T03:35:36.177136+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'23","last_scrub_stamp":"2024-03-12T03:34:52.729983+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:34:52.729983+0000","last_clean_scrub_stamp":"2024-03-12T03:34:52.729983+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.107241+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.186928+0000","last_change":"2024-03-12T03:35:15.855752+0000","last_active":"2024-03-12T03:35:36.186928+0000","last_peered":"2024-03-12T03:35:36.186928+0000","last_clean":"2024-03-12T03:35:36.186928+0000","last_became_active":"2024-03-12T03:33:27.189696+0000","last_became_peered":"2024-03-12T03:33:27.189696+0000","last_unstale":"2024-03-12T03:35:36.186928+0000","last_undegraded":"2024-03-12T03:35:36.186928+0000","last_fullsized":"2024-03-12T03:35:36.186928+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.855641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.855641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:31.748344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.174492+0000","last_change":"2024-03-12T03:34:36.701692+0000","last_active":"2024-03-12T03:35:36.174492+0000","last_peered":"2024-03-12T03:35:36.174492+0000","last_clean":"2024-03-12T03:35:36.174492+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:35:36.174492+0000","last_undegraded":"2024-03-12T03:35:36.174492+0000","last_fullsized":"2024-03-12T03:35:36.174492+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.701611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.701611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:53.280217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":87,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.182051+0000","last_change":"2024-03-12T03:34:32.746113+0000","last_active":"2024-03-12T03:35:36.182051+0000","last_peered":"2024-03-12T03:35:36.182051+0000","last_clean":"2024-03-12T03:35:36.182051+0000","last_became_active":"2024-03-12T03:33:21.096560+0000","last_became_peered":"2024-03-12T03:33:21.096560+0000","last_unstale":"2024-03-12T03:35:36.182051+0000","last_undegraded":"2024-03-12T03:35:36.182051+0000","last_fullsized":"2024-03-12T03:35:36.182051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:32.746027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:32.746027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:38.824190+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"90'6","reported_seq":92,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.189739+0000","last_change":"2024-03-12T03:34:34.274478+0000","last_active":"2024-03-12T03:35:36.189739+0000","last_peered":"2024-03-12T03:35:36.189739+0000","last_clean":"2024-03-12T03:35:36.189739+0000","last_became_active":"2024-03-12T03:33:23.161801+0000","last_became_peered":"2024-03-12T03:33:23.161801+0000","last_unstale":"2024-03-12T03:35:36.189739+0000","last_undegraded":"2024-03-12T03:35:36.189739+0000","last_fullsized":"2024-03-12T03:35:36.189739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.274353+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.042399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.189490+0000","last_change":"2024-03-12T03:34:40.191618+0000","last_active":"2024-03-12T03:35:36.189490+0000","last_peered":"2024-03-12T03:35:36.189490+0000","last_clean":"2024-03-12T03:35:36.189490+0000","last_became_active":"2024-03-12T03:33:27.381110+0000","last_became_peered":"2024-03-12T03:33:27.381110+0000","last_unstale":"2024-03-12T03:35:36.189490+0000","last_undegraded":"2024-03-12T03:35:36.189490+0000","last_fullsized":"2024-03-12T03:35:36.189490+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.191538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.480916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.186677+0000","last_change":"2024-03-12T03:34:53.878589+0000","last_active":"2024-03-12T03:35:36.186677+0000","last_peered":"2024-03-12T03:35:36.186677+0000","last_clean":"2024-03-12T03:35:36.186677+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:35:36.186677+0000","last_undegraded":"2024-03-12T03:35:36.186677+0000","last_fullsized":"2024-03-12T03:35:36.186677+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:53.878484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:53.878484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.602882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183973+0000","last_change":"2024-03-12T03:34:25.171761+0000","last_active":"2024-03-12T03:35:36.183973+0000","last_peered":"2024-03-12T03:35:36.183973+0000","last_clean":"2024-03-12T03:35:36.183973+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:35:36.183973+0000","last_undegraded":"2024-03-12T03:35:36.183973+0000","last_fullsized":"2024-03-12T03:35:36.183973+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:25.171622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.171622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:50.231250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'18","reported_seq":116,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.187750+0000","last_change":"2024-03-12T03:35:02.079317+0000","last_active":"2024-03-12T03:35:36.187750+0000","last_peered":"2024-03-12T03:35:36.187750+0000","last_clean":"2024-03-12T03:35:36.187750+0000","last_became_active":"2024-03-12T03:33:23.552210+0000","last_became_peered":"2024-03-12T03:33:23.552210+0000","last_unstale":"2024-03-12T03:35:36.187750+0000","last_undegraded":"2024-03-12T03:35:36.187750+0000","last_fullsized":"2024-03-12T03:35:36.187750+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:35:02.079224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:02.079224+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:19.194009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.181392+0000","last_change":"2024-03-12T03:34:57.781185+0000","last_active":"2024-03-12T03:35:36.181392+0000","last_peered":"2024-03-12T03:35:36.181392+0000","last_clean":"2024-03-12T03:35:36.181392+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:35:36.181392+0000","last_undegraded":"2024-03-12T03:35:36.181392+0000","last_fullsized":"2024-03-12T03:35:36.181392+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:57.781083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:57.781083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.508624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188149+0000","last_change":"2024-03-12T03:34:45.089928+0000","last_active":"2024-03-12T03:35:36.188149+0000","last_peered":"2024-03-12T03:35:36.188149+0000","last_clean":"2024-03-12T03:35:36.188149+0000","last_became_active":"2024-03-12T03:33:25.183725+0000","last_became_peered":"2024-03-12T03:33:25.183725+0000","last_unstale":"2024-03-12T03:35:36.188149+0000","last_undegraded":"2024-03-12T03:35:36.188149+0000","last_fullsized":"2024-03-12T03:35:36.188149+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:45.089762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:45.089762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.500637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.187850+0000","last_change":"2024-03-12T03:34:41.698397+0000","last_active":"2024-03-12T03:35:36.187850+0000","last_peered":"2024-03-12T03:35:36.187850+0000","last_clean":"2024-03-12T03:35:36.187850+0000","last_became_active":"2024-03-12T03:33:21.107037+0000","last_became_peered":"2024-03-12T03:33:21.107037+0000","last_unstale":"2024-03-12T03:35:36.187850+0000","last_undegraded":"2024-03-12T03:35:36.187850+0000","last_fullsized":"2024-03-12T03:35:36.187850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.698268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.698268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:05.134034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":100,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.181125+0000","last_change":"2024-03-12T03:34:58.413300+0000","last_active":"2024-03-12T03:35:36.181125+0000","last_peered":"2024-03-12T03:35:36.181125+0000","last_clean":"2024-03-12T03:35:36.181125+0000","last_became_active":"2024-03-12T03:33:23.160818+0000","last_became_peered":"2024-03-12T03:33:23.160818+0000","last_unstale":"2024-03-12T03:35:36.181125+0000","last_undegraded":"2024-03-12T03:35:36.181125+0000","last_fullsized":"2024-03-12T03:35:36.181125+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-12T03:34:58.413172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.413172+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:59.604035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.187534+0000","last_change":"2024-03-12T03:33:27.192215+0000","last_active":"2024-03-12T03:35:36.187534+0000","last_peered":"2024-03-12T03:35:36.187534+0000","last_clean":"2024-03-12T03:35:36.187534+0000","last_became_active":"2024-03-12T03:33:27.191710+0000","last_became_peered":"2024-03-12T03:33:27.191710+0000","last_unstale":"2024-03-12T03:35:36.187534+0000","last_undegraded":"2024-03-12T03:35:36.187534+0000","last_fullsized":"2024-03-12T03:35:36.187534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.001493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":79,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.180801+0000","last_change":"2024-03-12T03:34:33.258733+0000","last_active":"2024-03-12T03:35:36.180801+0000","last_peered":"2024-03-12T03:35:36.180801+0000","last_clean":"2024-03-12T03:35:36.180801+0000","last_became_active":"2024-03-12T03:33:25.182004+0000","last_became_peered":"2024-03-12T03:33:25.182004+0000","last_unstale":"2024-03-12T03:35:36.180801+0000","last_undegraded":"2024-03-12T03:35:36.180801+0000","last_fullsized":"2024-03-12T03:35:36.180801+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:33.258626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:33.258626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.420748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.176340+0000","last_change":"2024-03-12T03:34:50.076895+0000","last_active":"2024-03-12T03:35:36.176340+0000","last_peered":"2024-03-12T03:35:36.176340+0000","last_clean":"2024-03-12T03:35:36.176340+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:35:36.176340+0000","last_undegraded":"2024-03-12T03:35:36.176340+0000","last_fullsized":"2024-03-12T03:35:36.176340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:50.076814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:50.076814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:58.283856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":102,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.176158+0000","last_change":"2024-03-12T03:34:34.953658+0000","last_active":"2024-03-12T03:35:36.176158+0000","last_peered":"2024-03-12T03:35:36.176158+0000","last_clean":"2024-03-12T03:35:36.176158+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:35:36.176158+0000","last_undegraded":"2024-03-12T03:35:36.176158+0000","last_fullsized":"2024-03-12T03:35:36.176158+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:34:34.953565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.953565+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.161677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.181589+0000","last_change":"2024-03-12T03:34:34.718591+0000","last_active":"2024-03-12T03:35:36.181589+0000","last_peered":"2024-03-12T03:35:36.181589+0000","last_clean":"2024-03-12T03:35:36.181589+0000","last_became_active":"2024-03-12T03:33:27.194094+0000","last_became_peered":"2024-03-12T03:33:27.194094+0000","last_unstale":"2024-03-12T03:35:36.181589+0000","last_undegraded":"2024-03-12T03:35:36.181589+0000","last_fullsized":"2024-03-12T03:35:36.181589+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:34.718509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.718509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.921678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":116,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.647480+0000","last_change":"2024-03-12T03:34:38.139709+0000","last_active":"2024-03-12T03:35:36.647480+0000","last_peered":"2024-03-12T03:35:36.647480+0000","last_clean":"2024-03-12T03:35:36.647480+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:35:36.647480+0000","last_undegraded":"2024-03-12T03:35:36.647480+0000","last_fullsized":"2024-03-12T03:35:36.647480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:38.139582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.139582+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":94,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.187153+0000","last_change":"2024-03-12T03:34:38.663634+0000","last_active":"2024-03-12T03:35:36.187153+0000","last_peered":"2024-03-12T03:35:36.187153+0000","last_clean":"2024-03-12T03:35:36.187153+0000","last_became_active":"2024-03-12T03:33:21.573244+0000","last_became_peered":"2024-03-12T03:33:21.573244+0000","last_unstale":"2024-03-12T03:35:36.187153+0000","last_undegraded":"2024-03-12T03:35:36.187153+0000","last_fullsized":"2024-03-12T03:35:36.187153+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:34:38.663542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.663542+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:59.053797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":115,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.190711+0000","last_change":"2024-03-12T03:33:23.552006+0000","last_active":"2024-03-12T03:35:36.190711+0000","last_peered":"2024-03-12T03:35:36.190711+0000","last_clean":"2024-03-12T03:35:36.190711+0000","last_became_active":"2024-03-12T03:33:23.551706+0000","last_became_peered":"2024-03-12T03:33:23.551706+0000","last_unstale":"2024-03-12T03:35:36.190711+0000","last_undegraded":"2024-03-12T03:35:36.190711+0000","last_fullsized":"2024-03-12T03:35:36.190711+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:47.010997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.177047+0000","last_change":"2024-03-12T03:35:12.124255+0000","last_active":"2024-03-12T03:35:36.177047+0000","last_peered":"2024-03-12T03:35:36.177047+0000","last_clean":"2024-03-12T03:35:36.177047+0000","last_became_active":"2024-03-12T03:33:27.184916+0000","last_became_peered":"2024-03-12T03:33:27.184916+0000","last_unstale":"2024-03-12T03:35:36.177047+0000","last_undegraded":"2024-03-12T03:35:36.177047+0000","last_fullsized":"2024-03-12T03:35:36.177047+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:12.124167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:12.124167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:21.269917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":120,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:41.679362+0000","last_change":"2024-03-12T03:35:08.006530+0000","last_active":"2024-03-12T03:35:41.679362+0000","last_peered":"2024-03-12T03:35:41.679362+0000","last_clean":"2024-03-12T03:35:41.679362+0000","last_became_active":"2024-03-12T03:33:25.179158+0000","last_became_peered":"2024-03-12T03:33:25.179158+0000","last_unstale":"2024-03-12T03:35:41.679362+0000","last_undegraded":"2024-03-12T03:35:41.679362+0000","last_fullsized":"2024-03-12T03:35:41.679362+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:35:08.006375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:08.006375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:29.223748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.182306+0000","last_change":"2024-03-12T03:34:40.753233+0000","last_active":"2024-03-12T03:35:36.182306+0000","last_peered":"2024-03-12T03:35:36.182306+0000","last_clean":"2024-03-12T03:35:36.182306+0000","last_became_active":"2024-03-12T03:33:21.101819+0000","last_became_peered":"2024-03-12T03:33:21.101819+0000","last_unstale":"2024-03-12T03:35:36.182306+0000","last_undegraded":"2024-03-12T03:35:36.182306+0000","last_fullsized":"2024-03-12T03:35:36.182306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.753153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.816911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":111,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183732+0000","last_change":"2024-03-12T03:34:29.083183+0000","last_active":"2024-03-12T03:35:36.183732+0000","last_peered":"2024-03-12T03:35:36.183732+0000","last_clean":"2024-03-12T03:35:36.183732+0000","last_became_active":"2024-03-12T03:33:23.553480+0000","last_became_peered":"2024-03-12T03:33:23.553480+0000","last_unstale":"2024-03-12T03:35:36.183732+0000","last_undegraded":"2024-03-12T03:35:36.183732+0000","last_fullsized":"2024-03-12T03:35:36.183732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:34:29.083063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:29.083063+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:45.174021+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188352+0000","last_change":"2024-03-12T03:34:43.039900+0000","last_active":"2024-03-12T03:35:36.188352+0000","last_peered":"2024-03-12T03:35:36.188352+0000","last_clean":"2024-03-12T03:35:36.188352+0000","last_became_active":"2024-03-12T03:33:27.187657+0000","last_became_peered":"2024-03-12T03:33:27.187657+0000","last_unstale":"2024-03-12T03:35:36.188352+0000","last_undegraded":"2024-03-12T03:35:36.188352+0000","last_fullsized":"2024-03-12T03:35:36.188352+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:43.039799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.039799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:01.927684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":79,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.191809+0000","last_change":"2024-03-12T03:35:04.007338+0000","last_active":"2024-03-12T03:35:36.191809+0000","last_peered":"2024-03-12T03:35:36.191809+0000","last_clean":"2024-03-12T03:35:36.191809+0000","last_became_active":"2024-03-12T03:33:25.174953+0000","last_became_peered":"2024-03-12T03:33:25.174953+0000","last_unstale":"2024-03-12T03:35:36.191809+0000","last_undegraded":"2024-03-12T03:35:36.191809+0000","last_fullsized":"2024-03-12T03:35:36.191809+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:35:04.007252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:31.186652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.186848+0000","last_change":"2024-03-12T03:34:52.848468+0000","last_active":"2024-03-12T03:35:36.186848+0000","last_peered":"2024-03-12T03:35:36.186848+0000","last_clean":"2024-03-12T03:35:36.186848+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:35:36.186848+0000","last_undegraded":"2024-03-12T03:35:36.186848+0000","last_fullsized":"2024-03-12T03:35:36.186848+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:52.848366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:52.848366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.609011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":123,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.182383+0000","last_change":"2024-03-12T03:35:11.786139+0000","last_active":"2024-03-12T03:35:36.182383+0000","last_peered":"2024-03-12T03:35:36.182383+0000","last_clean":"2024-03-12T03:35:36.182383+0000","last_became_active":"2024-03-12T03:33:23.552164+0000","last_became_peered":"2024-03-12T03:33:23.552164+0000","last_unstale":"2024-03-12T03:35:36.182383+0000","last_undegraded":"2024-03-12T03:35:36.182383+0000","last_fullsized":"2024-03-12T03:35:36.182383+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:35:11.786040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.786040+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:24.463019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.187747+0000","last_change":"2024-03-12T03:34:46.767546+0000","last_active":"2024-03-12T03:35:36.187747+0000","last_peered":"2024-03-12T03:35:36.187747+0000","last_clean":"2024-03-12T03:35:36.187747+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:35:36.187747+0000","last_undegraded":"2024-03-12T03:35:36.187747+0000","last_fullsized":"2024-03-12T03:35:36.187747+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:46.767453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:46.767453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:56.661402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":79,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183954+0000","last_change":"2024-03-12T03:35:08.194100+0000","last_active":"2024-03-12T03:35:36.183954+0000","last_peered":"2024-03-12T03:35:36.183954+0000","last_clean":"2024-03-12T03:35:36.183954+0000","last_became_active":"2024-03-12T03:33:25.180752+0000","last_became_peered":"2024-03-12T03:33:25.180752+0000","last_unstale":"2024-03-12T03:35:36.183954+0000","last_undegraded":"2024-03-12T03:35:36.183954+0000","last_fullsized":"2024-03-12T03:35:36.183954+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:08.193996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:08.193996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:20.553560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.174399+0000","last_change":"2024-03-12T03:34:26.565344+0000","last_active":"2024-03-12T03:35:36.174399+0000","last_peered":"2024-03-12T03:35:36.174399+0000","last_clean":"2024-03-12T03:35:36.174399+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:35:36.174399+0000","last_undegraded":"2024-03-12T03:35:36.174399+0000","last_fullsized":"2024-03-12T03:35:36.174399+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:26.565245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:26.565245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:42.245276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":104,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.191935+0000","last_change":"2024-03-12T03:35:00.079249+0000","last_active":"2024-03-12T03:35:36.191935+0000","last_peered":"2024-03-12T03:35:36.191935+0000","last_clean":"2024-03-12T03:35:36.191935+0000","last_became_active":"2024-03-12T03:33:23.551773+0000","last_became_peered":"2024-03-12T03:33:23.551773+0000","last_unstale":"2024-03-12T03:35:36.191935+0000","last_undegraded":"2024-03-12T03:35:36.191935+0000","last_fullsized":"2024-03-12T03:35:36.191935+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:35:00.079152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.079152+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:21.325882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"101'1950","reported_seq":1968,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.800510+0000","last_change":"2024-03-12T03:35:23.782770+0000","last_active":"2024-03-12T03:35:45.800510+0000","last_peered":"2024-03-12T03:35:45.800510+0000","last_clean":"2024-03-12T03:35:45.800510+0000","last_became_active":"2024-03-12T03:35:23.782112+0000","last_became_peered":"2024-03-12T03:35:23.782112+0000","last_unstale":"2024-03-12T03:35:45.800510+0000","last_undegraded":"2024-03-12T03:35:45.800510+0000","last_fullsized":"2024-03-12T03:35:45.800510+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":1950,"log_dups_size":0,"ondisk_log_size":1950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:41.457014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127795200,"num_objects":1950,"num_object_clones":0,"num_object_copies":5850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1950,"num_write_kb":124800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183232+0000","last_change":"2024-03-12T03:35:00.250874+0000","last_active":"2024-03-12T03:35:36.183232+0000","last_peered":"2024-03-12T03:35:36.183232+0000","last_clean":"2024-03-12T03:35:36.183232+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:35:36.183232+0000","last_undegraded":"2024-03-12T03:35:36.183232+0000","last_fullsized":"2024-03-12T03:35:36.183232+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:00.250780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.250780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:05.004387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":100,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.187673+0000","last_change":"2024-03-12T03:35:12.907083+0000","last_active":"2024-03-12T03:35:36.187673+0000","last_peered":"2024-03-12T03:35:36.187673+0000","last_clean":"2024-03-12T03:35:36.187673+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:35:36.187673+0000","last_undegraded":"2024-03-12T03:35:36.187673+0000","last_fullsized":"2024-03-12T03:35:36.187673+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:35:12.906988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:12.906988+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:35.363926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"90'36","reported_seq":142,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183113+0000","last_change":"2024-03-12T03:34:39.184089+0000","last_active":"2024-03-12T03:35:36.183113+0000","last_peered":"2024-03-12T03:35:36.183113+0000","last_clean":"2024-03-12T03:35:36.183113+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:35:36.183113+0000","last_undegraded":"2024-03-12T03:35:36.183113+0000","last_fullsized":"2024-03-12T03:35:36.183113+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.183984+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.564093+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":117,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:40.781902+0000","last_change":"2024-03-12T03:34:39.659403+0000","last_active":"2024-03-12T03:35:40.781902+0000","last_peered":"2024-03-12T03:35:40.781902+0000","last_clean":"2024-03-12T03:35:40.781902+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:35:40.781902+0000","last_undegraded":"2024-03-12T03:35:40.781902+0000","last_fullsized":"2024-03-12T03:35:40.781902+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:39.659277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.659277+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:08.903088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"101'2058","reported_seq":2076,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.789133+0000","last_change":"2024-03-12T03:35:23.777596+0000","last_active":"2024-03-12T03:35:45.789133+0000","last_peered":"2024-03-12T03:35:45.789133+0000","last_clean":"2024-03-12T03:35:45.789133+0000","last_became_active":"2024-03-12T03:35:23.777272+0000","last_became_peered":"2024-03-12T03:35:23.777272+0000","last_unstale":"2024-03-12T03:35:45.789133+0000","last_undegraded":"2024-03-12T03:35:45.789133+0000","last_fullsized":"2024-03-12T03:35:45.789133+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:26.791230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134873088,"num_objects":2058,"num_object_clones":0,"num_object_copies":6174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2058,"num_write_kb":131712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":68,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.189039+0000","last_change":"2024-03-12T03:33:27.191091+0000","last_active":"2024-03-12T03:35:36.189039+0000","last_peered":"2024-03-12T03:35:36.189039+0000","last_clean":"2024-03-12T03:35:36.189039+0000","last_became_active":"2024-03-12T03:33:27.190753+0000","last_became_peered":"2024-03-12T03:33:27.190753+0000","last_unstale":"2024-03-12T03:35:36.189039+0000","last_undegraded":"2024-03-12T03:35:36.189039+0000","last_fullsized":"2024-03-12T03:35:36.189039+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.187549+0000","last_change":"2024-03-12T03:34:23.842758+0000","last_active":"2024-03-12T03:35:36.187549+0000","last_peered":"2024-03-12T03:35:36.187549+0000","last_clean":"2024-03-12T03:35:36.187549+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:35:36.187549+0000","last_undegraded":"2024-03-12T03:35:36.187549+0000","last_fullsized":"2024-03-12T03:35:36.187549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:23.842628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.842628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.823849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":123,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.177289+0000","last_change":"2024-03-12T03:33:23.163665+0000","last_active":"2024-03-12T03:35:36.177289+0000","last_peered":"2024-03-12T03:35:36.177289+0000","last_clean":"2024-03-12T03:35:36.177289+0000","last_became_active":"2024-03-12T03:33:23.163351+0000","last_became_peered":"2024-03-12T03:33:23.163351+0000","last_unstale":"2024-03-12T03:35:36.177289+0000","last_undegraded":"2024-03-12T03:35:36.177289+0000","last_fullsized":"2024-03-12T03:35:36.177289+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188097+0000","last_change":"2024-03-12T03:34:51.958873+0000","last_active":"2024-03-12T03:35:36.188097+0000","last_peered":"2024-03-12T03:35:36.188097+0000","last_clean":"2024-03-12T03:35:36.188097+0000","last_became_active":"2024-03-12T03:33:25.185596+0000","last_became_peered":"2024-03-12T03:33:25.185596+0000","last_unstale":"2024-03-12T03:35:36.188097+0000","last_undegraded":"2024-03-12T03:35:36.188097+0000","last_fullsized":"2024-03-12T03:35:36.188097+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:51.958777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:51.958777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:20.123226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"101'2029","reported_seq":2047,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.749379+0000","last_change":"2024-03-12T03:35:23.778943+0000","last_active":"2024-03-12T03:35:45.749379+0000","last_peered":"2024-03-12T03:35:45.749379+0000","last_clean":"2024-03-12T03:35:45.749379+0000","last_became_active":"2024-03-12T03:35:23.778606+0000","last_became_peered":"2024-03-12T03:35:23.778606+0000","last_unstale":"2024-03-12T03:35:45.749379+0000","last_undegraded":"2024-03-12T03:35:45.749379+0000","last_fullsized":"2024-03-12T03:35:45.749379+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:50.273914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132972544,"num_objects":2029,"num_object_clones":0,"num_object_copies":6087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2029,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.186361+0000","last_change":"2024-03-12T03:34:43.716123+0000","last_active":"2024-03-12T03:35:36.186361+0000","last_peered":"2024-03-12T03:35:36.186361+0000","last_clean":"2024-03-12T03:35:36.186361+0000","last_became_active":"2024-03-12T03:33:27.187955+0000","last_became_peered":"2024-03-12T03:33:27.187955+0000","last_unstale":"2024-03-12T03:35:36.186361+0000","last_undegraded":"2024-03-12T03:35:36.186361+0000","last_fullsized":"2024-03-12T03:35:36.186361+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:43.716021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.716021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:01.645227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188446+0000","last_change":"2024-03-12T03:35:11.095883+0000","last_active":"2024-03-12T03:35:36.188446+0000","last_peered":"2024-03-12T03:35:36.188446+0000","last_clean":"2024-03-12T03:35:36.188446+0000","last_became_active":"2024-03-12T03:33:25.179593+0000","last_became_peered":"2024-03-12T03:33:25.179593+0000","last_unstale":"2024-03-12T03:35:36.188446+0000","last_undegraded":"2024-03-12T03:35:36.188446+0000","last_fullsized":"2024-03-12T03:35:36.188446+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:11.095766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.095766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:12.302418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.187988+0000","last_change":"2024-03-12T03:34:30.792781+0000","last_active":"2024-03-12T03:35:36.187988+0000","last_peered":"2024-03-12T03:35:36.187988+0000","last_clean":"2024-03-12T03:35:36.187988+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:35:36.187988+0000","last_undegraded":"2024-03-12T03:35:36.187988+0000","last_fullsized":"2024-03-12T03:35:36.187988+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:30.792702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.792702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:50.132706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":127,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.174643+0000","last_change":"2024-03-12T03:34:46.722408+0000","last_active":"2024-03-12T03:35:36.174643+0000","last_peered":"2024-03-12T03:35:36.174643+0000","last_clean":"2024-03-12T03:35:36.174643+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:35:36.174643+0000","last_undegraded":"2024-03-12T03:35:36.174643+0000","last_fullsized":"2024-03-12T03:35:36.174643+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:34:46.722278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:46.722278+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.145172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"101'2061","reported_seq":2079,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.780990+0000","last_change":"2024-03-12T03:35:23.785560+0000","last_active":"2024-03-12T03:35:45.780990+0000","last_peered":"2024-03-12T03:35:45.780990+0000","last_clean":"2024-03-12T03:35:45.780990+0000","last_became_active":"2024-03-12T03:35:23.784701+0000","last_became_peered":"2024-03-12T03:35:23.784701+0000","last_unstale":"2024-03-12T03:35:45.780990+0000","last_undegraded":"2024-03-12T03:35:45.780990+0000","last_fullsized":"2024-03-12T03:35:45.780990+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2061,"log_dups_size":0,"ondisk_log_size":2061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:26.081129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135069696,"num_objects":2061,"num_object_clones":0,"num_object_copies":6183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2061,"num_write_kb":131904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.182571+0000","last_change":"2024-03-12T03:34:37.737471+0000","last_active":"2024-03-12T03:35:36.182571+0000","last_peered":"2024-03-12T03:35:36.182571+0000","last_clean":"2024-03-12T03:35:36.182571+0000","last_became_active":"2024-03-12T03:33:27.189948+0000","last_became_peered":"2024-03-12T03:33:27.189948+0000","last_unstale":"2024-03-12T03:35:36.182571+0000","last_undegraded":"2024-03-12T03:35:36.182571+0000","last_fullsized":"2024-03-12T03:35:36.182571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:37.737386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:37.737386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.387230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188723+0000","last_change":"2024-03-12T03:35:04.986995+0000","last_active":"2024-03-12T03:35:36.188723+0000","last_peered":"2024-03-12T03:35:36.188723+0000","last_clean":"2024-03-12T03:35:36.188723+0000","last_became_active":"2024-03-12T03:33:25.184289+0000","last_became_peered":"2024-03-12T03:33:25.184289+0000","last_unstale":"2024-03-12T03:35:36.188723+0000","last_undegraded":"2024-03-12T03:35:36.188723+0000","last_fullsized":"2024-03-12T03:35:36.188723+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:04.986909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:04.986909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:09.694715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183489+0000","last_change":"2024-03-12T03:34:48.160481+0000","last_active":"2024-03-12T03:35:36.183489+0000","last_peered":"2024-03-12T03:35:36.183489+0000","last_clean":"2024-03-12T03:35:36.183489+0000","last_became_active":"2024-03-12T03:33:21.102410+0000","last_became_peered":"2024-03-12T03:33:21.102410+0000","last_unstale":"2024-03-12T03:35:36.183489+0000","last_undegraded":"2024-03-12T03:35:36.183489+0000","last_fullsized":"2024-03-12T03:35:36.183489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:34:48.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:17.839681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":112,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.182950+0000","last_change":"2024-03-12T03:35:07.777251+0000","last_active":"2024-03-12T03:35:36.182950+0000","last_peered":"2024-03-12T03:35:36.182950+0000","last_clean":"2024-03-12T03:35:36.182950+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:35:36.182950+0000","last_undegraded":"2024-03-12T03:35:36.182950+0000","last_fullsized":"2024-03-12T03:35:36.182950+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:35:07.777146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:07.777146+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:17.920452+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"101'2044","reported_seq":2062,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.756838+0000","last_change":"2024-03-12T03:35:23.784948+0000","last_active":"2024-03-12T03:35:45.756838+0000","last_peered":"2024-03-12T03:35:45.756838+0000","last_clean":"2024-03-12T03:35:45.756838+0000","last_became_active":"2024-03-12T03:35:23.784628+0000","last_became_peered":"2024-03-12T03:35:23.784628+0000","last_unstale":"2024-03-12T03:35:45.756838+0000","last_undegraded":"2024-03-12T03:35:45.756838+0000","last_fullsized":"2024-03-12T03:35:45.756838+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:33.288176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133955584,"num_objects":2044,"num_object_clones":0,"num_object_copies":6132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2044,"num_write_kb":130816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.191360+0000","last_change":"2024-03-12T03:34:49.088183+0000","last_active":"2024-03-12T03:35:36.191360+0000","last_peered":"2024-03-12T03:35:36.191360+0000","last_clean":"2024-03-12T03:35:36.191360+0000","last_became_active":"2024-03-12T03:33:27.185041+0000","last_became_peered":"2024-03-12T03:33:27.185041+0000","last_unstale":"2024-03-12T03:35:36.191360+0000","last_undegraded":"2024-03-12T03:35:36.191360+0000","last_fullsized":"2024-03-12T03:35:36.191360+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.088098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.088098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.622130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":99,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183400+0000","last_change":"2024-03-12T03:34:22.168794+0000","last_active":"2024-03-12T03:35:36.183400+0000","last_peered":"2024-03-12T03:35:36.183400+0000","last_clean":"2024-03-12T03:35:36.183400+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:35:36.183400+0000","last_undegraded":"2024-03-12T03:35:36.183400+0000","last_fullsized":"2024-03-12T03:35:36.183400+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:34:22.168654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:22.168654+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:48.338217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.175958+0000","last_change":"2024-03-12T03:34:59.105754+0000","last_active":"2024-03-12T03:35:36.175958+0000","last_peered":"2024-03-12T03:35:36.175958+0000","last_clean":"2024-03-12T03:35:36.175958+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:35:36.175958+0000","last_undegraded":"2024-03-12T03:35:36.175958+0000","last_fullsized":"2024-03-12T03:35:36.175958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:59.105634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:59.105634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:02.625520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'35","reported_seq":111,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.647489+0000","last_change":"2024-03-12T03:34:54.301841+0000","last_active":"2024-03-12T03:35:36.647489+0000","last_peered":"2024-03-12T03:35:36.647489+0000","last_clean":"2024-03-12T03:35:36.647489+0000","last_became_active":"2024-03-12T03:33:50.151664+0000","last_became_peered":"2024-03-12T03:33:50.151664+0000","last_unstale":"2024-03-12T03:35:36.647489+0000","last_undegraded":"2024-03-12T03:35:36.647489+0000","last_fullsized":"2024-03-12T03:35:36.647489+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'35","last_scrub_stamp":"2024-03-12T03:34:54.301725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:54.301725+0000","objects_scrubbed":11,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:23.962972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183815+0000","last_change":"2024-03-12T03:34:38.698005+0000","last_active":"2024-03-12T03:35:36.183815+0000","last_peered":"2024-03-12T03:35:36.183815+0000","last_clean":"2024-03-12T03:35:36.183815+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:35:36.183815+0000","last_undegraded":"2024-03-12T03:35:36.183815+0000","last_fullsized":"2024-03-12T03:35:36.183815+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:38.697923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.697923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:07.086680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"101'2081","reported_seq":2099,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.761439+0000","last_change":"2024-03-12T03:35:23.776994+0000","last_active":"2024-03-12T03:35:45.761439+0000","last_peered":"2024-03-12T03:35:45.761439+0000","last_clean":"2024-03-12T03:35:45.761439+0000","last_became_active":"2024-03-12T03:35:23.776662+0000","last_became_peered":"2024-03-12T03:35:23.776662+0000","last_unstale":"2024-03-12T03:35:45.761439+0000","last_undegraded":"2024-03-12T03:35:45.761439+0000","last_fullsized":"2024-03-12T03:35:45.761439+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:37.352608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136380416,"num_objects":2081,"num_object_clones":0,"num_object_copies":6243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":133184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":78,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188135+0000","last_change":"2024-03-12T03:34:40.663082+0000","last_active":"2024-03-12T03:35:36.188135+0000","last_peered":"2024-03-12T03:35:36.188135+0000","last_clean":"2024-03-12T03:35:36.188135+0000","last_became_active":"2024-03-12T03:33:27.380389+0000","last_became_peered":"2024-03-12T03:33:27.380389+0000","last_unstale":"2024-03-12T03:35:36.188135+0000","last_undegraded":"2024-03-12T03:35:36.188135+0000","last_fullsized":"2024-03-12T03:35:36.188135+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.663002+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.064026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183876+0000","last_change":"2024-03-12T03:34:51.444303+0000","last_active":"2024-03-12T03:35:36.183876+0000","last_peered":"2024-03-12T03:35:36.183876+0000","last_clean":"2024-03-12T03:35:36.183876+0000","last_became_active":"2024-03-12T03:33:21.572911+0000","last_became_peered":"2024-03-12T03:33:21.572911+0000","last_unstale":"2024-03-12T03:35:36.183876+0000","last_undegraded":"2024-03-12T03:35:36.183876+0000","last_fullsized":"2024-03-12T03:35:36.183876+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:51.444181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:51.444181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.657671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188158+0000","last_change":"2024-03-12T03:34:49.918898+0000","last_active":"2024-03-12T03:35:36.188158+0000","last_peered":"2024-03-12T03:35:36.188158+0000","last_clean":"2024-03-12T03:35:36.188158+0000","last_became_active":"2024-03-12T03:33:20.093064+0000","last_became_peered":"2024-03-12T03:33:20.093064+0000","last_unstale":"2024-03-12T03:35:36.188158+0000","last_undegraded":"2024-03-12T03:35:36.188158+0000","last_fullsized":"2024-03-12T03:35:36.188158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.918802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.918802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.157514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000516994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":123,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.181172+0000","last_change":"2024-03-12T03:34:44.301692+0000","last_active":"2024-03-12T03:35:36.181172+0000","last_peered":"2024-03-12T03:35:36.181172+0000","last_clean":"2024-03-12T03:35:36.181172+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:35:36.181172+0000","last_undegraded":"2024-03-12T03:35:36.181172+0000","last_fullsized":"2024-03-12T03:35:36.181172+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-12T03:34:44.301556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.301556+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.466292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188361+0000","last_change":"2024-03-12T03:34:42.086021+0000","last_active":"2024-03-12T03:35:36.188361+0000","last_peered":"2024-03-12T03:35:36.188361+0000","last_clean":"2024-03-12T03:35:36.188361+0000","last_became_active":"2024-03-12T03:33:25.179022+0000","last_became_peered":"2024-03-12T03:33:25.179022+0000","last_unstale":"2024-03-12T03:35:36.188361+0000","last_undegraded":"2024-03-12T03:35:36.188361+0000","last_fullsized":"2024-03-12T03:35:36.188361+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:42.085936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:42.085936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.119017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"101'2039","reported_seq":2057,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.768576+0000","last_change":"2024-03-12T03:35:23.785163+0000","last_active":"2024-03-12T03:35:45.768576+0000","last_peered":"2024-03-12T03:35:45.768576+0000","last_clean":"2024-03-12T03:35:45.768576+0000","last_became_active":"2024-03-12T03:35:23.784880+0000","last_became_peered":"2024-03-12T03:35:23.784880+0000","last_unstale":"2024-03-12T03:35:45.768576+0000","last_undegraded":"2024-03-12T03:35:45.768576+0000","last_fullsized":"2024-03-12T03:35:45.768576+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2039,"log_dups_size":0,"ondisk_log_size":2039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:41.806437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133627904,"num_objects":2039,"num_object_clones":0,"num_object_copies":6117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":130496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183942+0000","last_change":"2024-03-12T03:34:58.277475+0000","last_active":"2024-03-12T03:35:36.183942+0000","last_peered":"2024-03-12T03:35:36.183942+0000","last_clean":"2024-03-12T03:35:36.183942+0000","last_became_active":"2024-03-12T03:33:27.188551+0000","last_became_peered":"2024-03-12T03:33:27.188551+0000","last_unstale":"2024-03-12T03:35:36.183942+0000","last_undegraded":"2024-03-12T03:35:36.183942+0000","last_fullsized":"2024-03-12T03:35:36.183942+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:58.277378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.277378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:16.176471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":343,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.186470+0000","last_change":"2024-03-12T03:34:58.903289+0000","last_active":"2024-03-12T03:35:36.186470+0000","last_peered":"2024-03-12T03:35:36.186470+0000","last_clean":"2024-03-12T03:35:36.186470+0000","last_became_active":"2024-03-12T03:33:11.319888+0000","last_became_peered":"2024-03-12T03:33:11.319888+0000","last_unstale":"2024-03-12T03:35:36.186470+0000","last_undegraded":"2024-03-12T03:35:36.186470+0000","last_fullsized":"2024-03-12T03:35:36.186470+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:34:58.903163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.903163+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:07.967565+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188085+0000","last_change":"2024-03-12T03:35:14.996451+0000","last_active":"2024-03-12T03:35:36.188085+0000","last_peered":"2024-03-12T03:35:36.188085+0000","last_clean":"2024-03-12T03:35:36.188085+0000","last_became_active":"2024-03-12T03:33:21.100900+0000","last_became_peered":"2024-03-12T03:33:21.100900+0000","last_unstale":"2024-03-12T03:35:36.188085+0000","last_undegraded":"2024-03-12T03:35:36.188085+0000","last_fullsized":"2024-03-12T03:35:36.188085+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:14.996357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:14.996357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:35.315899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":135,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.647495+0000","last_change":"2024-03-12T03:34:35.664996+0000","last_active":"2024-03-12T03:35:36.647495+0000","last_peered":"2024-03-12T03:35:36.647495+0000","last_clean":"2024-03-12T03:35:36.647495+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:35:36.647495+0000","last_undegraded":"2024-03-12T03:35:36.647495+0000","last_fullsized":"2024-03-12T03:35:36.647495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:34:35.664876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:35.664876+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:54.461746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.187315+0000","last_change":"2024-03-12T03:34:47.796670+0000","last_active":"2024-03-12T03:35:36.187315+0000","last_peered":"2024-03-12T03:35:36.187315+0000","last_clean":"2024-03-12T03:35:36.187315+0000","last_became_active":"2024-03-12T03:33:25.177762+0000","last_became_peered":"2024-03-12T03:33:25.177762+0000","last_unstale":"2024-03-12T03:35:36.187315+0000","last_undegraded":"2024-03-12T03:35:36.187315+0000","last_fullsized":"2024-03-12T03:35:36.187315+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.796561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.796561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.258919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"101'2030","reported_seq":2048,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.761989+0000","last_change":"2024-03-12T03:35:23.786371+0000","last_active":"2024-03-12T03:35:45.761989+0000","last_peered":"2024-03-12T03:35:45.761989+0000","last_clean":"2024-03-12T03:35:45.761989+0000","last_became_active":"2024-03-12T03:35:23.786059+0000","last_became_peered":"2024-03-12T03:35:23.786059+0000","last_unstale":"2024-03-12T03:35:45.761989+0000","last_undegraded":"2024-03-12T03:35:45.761989+0000","last_fullsized":"2024-03-12T03:35:45.761989+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2030,"log_dups_size":0,"ondisk_log_size":2030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:44.824542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133038080,"num_objects":2030,"num_object_clones":0,"num_object_copies":6090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2030,"num_write_kb":129920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.180890+0000","last_change":"2024-03-12T03:34:47.352525+0000","last_active":"2024-03-12T03:35:36.180890+0000","last_peered":"2024-03-12T03:35:36.180890+0000","last_clean":"2024-03-12T03:35:36.180890+0000","last_became_active":"2024-03-12T03:33:27.191878+0000","last_became_peered":"2024-03-12T03:33:27.191878+0000","last_unstale":"2024-03-12T03:35:36.180890+0000","last_undegraded":"2024-03-12T03:35:36.180890+0000","last_fullsized":"2024-03-12T03:35:36.180890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.352450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.352450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:01.666869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.182745+0000","last_change":"2024-03-12T03:35:05.715931+0000","last_active":"2024-03-12T03:35:36.182745+0000","last_peered":"2024-03-12T03:35:36.182745+0000","last_clean":"2024-03-12T03:35:36.182745+0000","last_became_active":"2024-03-12T03:33:21.097086+0000","last_became_peered":"2024-03-12T03:33:21.097086+0000","last_unstale":"2024-03-12T03:35:36.182745+0000","last_undegraded":"2024-03-12T03:35:36.182745+0000","last_fullsized":"2024-03-12T03:35:36.182745+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:05.715849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.715849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:19.094010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":239,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:44.328322+0000","last_change":"2024-03-12T03:34:21.799658+0000","last_active":"2024-03-12T03:35:44.328322+0000","last_peered":"2024-03-12T03:35:44.328322+0000","last_clean":"2024-03-12T03:35:44.328322+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:35:44.328322+0000","last_undegraded":"2024-03-12T03:35:44.328322+0000","last_fullsized":"2024-03-12T03:35:44.328322+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:34:21.799534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:21.799534+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:46.973279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":134,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":102,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.181702+0000","last_change":"2024-03-12T03:34:39.707369+0000","last_active":"2024-03-12T03:35:36.181702+0000","last_peered":"2024-03-12T03:35:36.181702+0000","last_clean":"2024-03-12T03:35:36.181702+0000","last_became_active":"2024-03-12T03:33:23.168750+0000","last_became_peered":"2024-03-12T03:33:23.168750+0000","last_unstale":"2024-03-12T03:35:36.181702+0000","last_undegraded":"2024-03-12T03:35:36.181702+0000","last_fullsized":"2024-03-12T03:35:36.181702+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:34:39.707264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.707264+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:43.684441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.184160+0000","last_change":"2024-03-12T03:35:01.257832+0000","last_active":"2024-03-12T03:35:36.184160+0000","last_peered":"2024-03-12T03:35:36.184160+0000","last_clean":"2024-03-12T03:35:36.184160+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:35:36.184160+0000","last_undegraded":"2024-03-12T03:35:36.184160+0000","last_fullsized":"2024-03-12T03:35:36.184160+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:35:01.257725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:12.838678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"101'2106","reported_seq":2124,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.773585+0000","last_change":"2024-03-12T03:35:23.778292+0000","last_active":"2024-03-12T03:35:45.773585+0000","last_peered":"2024-03-12T03:35:45.773585+0000","last_clean":"2024-03-12T03:35:45.773585+0000","last_became_active":"2024-03-12T03:35:23.777945+0000","last_became_peered":"2024-03-12T03:35:23.777945+0000","last_unstale":"2024-03-12T03:35:45.773585+0000","last_undegraded":"2024-03-12T03:35:45.773585+0000","last_fullsized":"2024-03-12T03:35:45.773585+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2106,"log_dups_size":0,"ondisk_log_size":2106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:44.975408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138018816,"num_objects":2106,"num_object_clones":0,"num_object_copies":6318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":134784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183123+0000","last_change":"2024-03-12T03:34:45.344064+0000","last_active":"2024-03-12T03:35:36.183123+0000","last_peered":"2024-03-12T03:35:36.183123+0000","last_clean":"2024-03-12T03:35:36.183123+0000","last_became_active":"2024-03-12T03:33:27.380458+0000","last_became_peered":"2024-03-12T03:33:27.380458+0000","last_unstale":"2024-03-12T03:35:36.183123+0000","last_undegraded":"2024-03-12T03:35:36.183123+0000","last_fullsized":"2024-03-12T03:35:36.183123+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:45.343945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:45.343945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.292957+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.181292+0000","last_change":"2024-03-12T03:34:34.272895+0000","last_active":"2024-03-12T03:35:36.181292+0000","last_peered":"2024-03-12T03:35:36.181292+0000","last_clean":"2024-03-12T03:35:36.181292+0000","last_became_active":"2024-03-12T03:33:21.573003+0000","last_became_peered":"2024-03-12T03:33:21.573003+0000","last_unstale":"2024-03-12T03:35:36.181292+0000","last_undegraded":"2024-03-12T03:35:36.181292+0000","last_fullsized":"2024-03-12T03:35:36.181292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:34.272759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.272759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:47.242996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":132,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183210+0000","last_change":"2024-03-12T03:35:06.754200+0000","last_active":"2024-03-12T03:35:36.183210+0000","last_peered":"2024-03-12T03:35:36.183210+0000","last_clean":"2024-03-12T03:35:36.183210+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:35:36.183210+0000","last_undegraded":"2024-03-12T03:35:36.183210+0000","last_fullsized":"2024-03-12T03:35:36.183210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:35:06.754088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:06.754088+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:12.920218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.174651+0000","last_change":"2024-03-12T03:34:28.572496+0000","last_active":"2024-03-12T03:35:36.174651+0000","last_peered":"2024-03-12T03:35:36.174651+0000","last_clean":"2024-03-12T03:35:36.174651+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:35:36.174651+0000","last_undegraded":"2024-03-12T03:35:36.174651+0000","last_fullsized":"2024-03-12T03:35:36.174651+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:28.572379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:28.572379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:58.234927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"101'2053","reported_seq":2071,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.761184+0000","last_change":"2024-03-12T03:35:23.781284+0000","last_active":"2024-03-12T03:35:45.761184+0000","last_peered":"2024-03-12T03:35:45.761184+0000","last_clean":"2024-03-12T03:35:45.761184+0000","last_became_active":"2024-03-12T03:35:23.780941+0000","last_became_peered":"2024-03-12T03:35:23.780941+0000","last_unstale":"2024-03-12T03:35:45.761184+0000","last_undegraded":"2024-03-12T03:35:45.761184+0000","last_fullsized":"2024-03-12T03:35:45.761184+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2053,"log_dups_size":0,"ondisk_log_size":2053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:26.636565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134545408,"num_objects":2053,"num_object_clones":0,"num_object_copies":6159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2053,"num_write_kb":131392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.181450+0000","last_change":"2024-03-12T03:34:51.734825+0000","last_active":"2024-03-12T03:35:36.181450+0000","last_peered":"2024-03-12T03:35:36.181450+0000","last_clean":"2024-03-12T03:35:36.181450+0000","last_became_active":"2024-03-12T03:33:27.194266+0000","last_became_peered":"2024-03-12T03:33:27.194266+0000","last_unstale":"2024-03-12T03:35:36.181450+0000","last_undegraded":"2024-03-12T03:35:36.181450+0000","last_fullsized":"2024-03-12T03:35:36.181450+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:51.734742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:51.734742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:12.279921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183547+0000","last_change":"2024-03-12T03:34:41.152579+0000","last_active":"2024-03-12T03:35:36.183547+0000","last_peered":"2024-03-12T03:35:36.183547+0000","last_clean":"2024-03-12T03:35:36.183547+0000","last_became_active":"2024-03-12T03:33:21.573251+0000","last_became_peered":"2024-03-12T03:33:21.573251+0000","last_unstale":"2024-03-12T03:35:36.183547+0000","last_undegraded":"2024-03-12T03:35:36.183547+0000","last_fullsized":"2024-03-12T03:35:36.183547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.152440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.152440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.371887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":111,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.180911+0000","last_change":"2024-03-12T03:34:35.282309+0000","last_active":"2024-03-12T03:35:36.180911+0000","last_peered":"2024-03-12T03:35:36.180911+0000","last_clean":"2024-03-12T03:35:36.180911+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:35:36.180911+0000","last_undegraded":"2024-03-12T03:35:36.180911+0000","last_fullsized":"2024-03-12T03:35:36.180911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:34:35.282203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:35.282203+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.121154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":117,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.647366+0000","last_change":"2024-03-12T03:34:30.767076+0000","last_active":"2024-03-12T03:35:36.647366+0000","last_peered":"2024-03-12T03:35:36.647366+0000","last_clean":"2024-03-12T03:35:36.647366+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:35:36.647366+0000","last_undegraded":"2024-03-12T03:35:36.647366+0000","last_fullsized":"2024-03-12T03:35:36.647366+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:30.766975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.766975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:45.275798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"101'2050","reported_seq":2068,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.811465+0000","last_change":"2024-03-12T03:35:23.778223+0000","last_active":"2024-03-12T03:35:45.811465+0000","last_peered":"2024-03-12T03:35:45.811465+0000","last_clean":"2024-03-12T03:35:45.811465+0000","last_became_active":"2024-03-12T03:35:23.777935+0000","last_became_peered":"2024-03-12T03:35:23.777935+0000","last_unstale":"2024-03-12T03:35:45.811465+0000","last_undegraded":"2024-03-12T03:35:45.811465+0000","last_fullsized":"2024-03-12T03:35:45.811465+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2050,"log_dups_size":0,"ondisk_log_size":2050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:51.033924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134348800,"num_objects":2050,"num_object_clones":0,"num_object_copies":6150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2050,"num_write_kb":131200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.187040+0000","last_change":"2024-03-12T03:34:44.750269+0000","last_active":"2024-03-12T03:35:36.187040+0000","last_peered":"2024-03-12T03:35:36.187040+0000","last_clean":"2024-03-12T03:35:36.187040+0000","last_became_active":"2024-03-12T03:33:27.383249+0000","last_became_peered":"2024-03-12T03:33:27.383249+0000","last_unstale":"2024-03-12T03:35:36.187040+0000","last_undegraded":"2024-03-12T03:35:36.187040+0000","last_fullsized":"2024-03-12T03:35:36.187040+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:44.750135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.750135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:55.084645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":89,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.181793+0000","last_change":"2024-03-12T03:34:56.779549+0000","last_active":"2024-03-12T03:35:36.181793+0000","last_peered":"2024-03-12T03:35:36.181793+0000","last_clean":"2024-03-12T03:35:36.181793+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:35:36.181793+0000","last_undegraded":"2024-03-12T03:35:36.181793+0000","last_fullsized":"2024-03-12T03:35:36.181793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:34:56.779379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:56.779379+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.372243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":117,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.177118+0000","last_change":"2024-03-12T03:34:47.043566+0000","last_active":"2024-03-12T03:35:36.177118+0000","last_peered":"2024-03-12T03:35:36.177118+0000","last_clean":"2024-03-12T03:35:36.177118+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:35:36.177118+0000","last_undegraded":"2024-03-12T03:35:36.177118+0000","last_fullsized":"2024-03-12T03:35:36.177118+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-03-12T03:34:47.043433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.043433+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.321445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":117,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.648178+0000","last_change":"2024-03-12T03:34:25.168128+0000","last_active":"2024-03-12T03:35:36.648178+0000","last_peered":"2024-03-12T03:35:36.648178+0000","last_clean":"2024-03-12T03:35:36.648178+0000","last_became_active":"2024-03-12T03:33:25.177311+0000","last_became_peered":"2024-03-12T03:33:25.177311+0000","last_unstale":"2024-03-12T03:35:36.648178+0000","last_undegraded":"2024-03-12T03:35:36.648178+0000","last_fullsized":"2024-03-12T03:35:36.648178+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:25.168002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.168002+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:51.036057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"101'2126","reported_seq":2144,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.820224+0000","last_change":"2024-03-12T03:35:23.778034+0000","last_active":"2024-03-12T03:35:45.820224+0000","last_peered":"2024-03-12T03:35:45.820224+0000","last_clean":"2024-03-12T03:35:45.820224+0000","last_became_active":"2024-03-12T03:35:23.777749+0000","last_became_peered":"2024-03-12T03:35:23.777749+0000","last_unstale":"2024-03-12T03:35:45.820224+0000","last_undegraded":"2024-03-12T03:35:45.820224+0000","last_fullsized":"2024-03-12T03:35:45.820224+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2126,"log_dups_size":0,"ondisk_log_size":2126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:32.320246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139329536,"num_objects":2126,"num_object_clones":0,"num_object_copies":6378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2126,"num_write_kb":136064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.175782+0000","last_change":"2024-03-12T03:34:58.085602+0000","last_active":"2024-03-12T03:35:36.175782+0000","last_peered":"2024-03-12T03:35:36.175782+0000","last_clean":"2024-03-12T03:35:36.175782+0000","last_became_active":"2024-03-12T03:33:27.187864+0000","last_became_peered":"2024-03-12T03:33:27.187864+0000","last_unstale":"2024-03-12T03:35:36.175782+0000","last_undegraded":"2024-03-12T03:35:36.175782+0000","last_fullsized":"2024-03-12T03:35:36.175782+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:58.085499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.085499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:03.220277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.191662+0000","last_change":"2024-03-12T03:34:47.155013+0000","last_active":"2024-03-12T03:35:36.191662+0000","last_peered":"2024-03-12T03:35:36.191662+0000","last_clean":"2024-03-12T03:35:36.191662+0000","last_became_active":"2024-03-12T03:33:21.104430+0000","last_became_peered":"2024-03-12T03:33:21.104430+0000","last_unstale":"2024-03-12T03:35:36.191662+0000","last_undegraded":"2024-03-12T03:35:36.191662+0000","last_fullsized":"2024-03-12T03:35:36.191662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.154925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.154925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:48.895337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":132,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.189878+0000","last_change":"2024-03-12T03:33:23.551957+0000","last_active":"2024-03-12T03:35:36.189878+0000","last_peered":"2024-03-12T03:35:36.189878+0000","last_clean":"2024-03-12T03:35:36.189878+0000","last_became_active":"2024-03-12T03:33:23.551623+0000","last_became_peered":"2024-03-12T03:33:23.551623+0000","last_unstale":"2024-03-12T03:35:36.189878+0000","last_undegraded":"2024-03-12T03:35:36.189878+0000","last_fullsized":"2024-03-12T03:35:36.189878+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:40.001529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.176770+0000","last_change":"2024-03-12T03:35:00.071368+0000","last_active":"2024-03-12T03:35:36.176770+0000","last_peered":"2024-03-12T03:35:36.176770+0000","last_clean":"2024-03-12T03:35:36.176770+0000","last_became_active":"2024-03-12T03:33:25.181020+0000","last_became_peered":"2024-03-12T03:33:25.181020+0000","last_unstale":"2024-03-12T03:35:36.176770+0000","last_undegraded":"2024-03-12T03:35:36.176770+0000","last_fullsized":"2024-03-12T03:35:36.176770+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:00.071245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.071245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.842153+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"101'2014","reported_seq":2032,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.757335+0000","last_change":"2024-03-12T03:35:23.785403+0000","last_active":"2024-03-12T03:35:45.757335+0000","last_peered":"2024-03-12T03:35:45.757335+0000","last_clean":"2024-03-12T03:35:45.757335+0000","last_became_active":"2024-03-12T03:35:23.785037+0000","last_became_peered":"2024-03-12T03:35:23.785037+0000","last_unstale":"2024-03-12T03:35:45.757335+0000","last_undegraded":"2024-03-12T03:35:45.757335+0000","last_fullsized":"2024-03-12T03:35:45.757335+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2014,"log_dups_size":0,"ondisk_log_size":2014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:51.241046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131989504,"num_objects":2014,"num_object_clones":0,"num_object_copies":6042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2014,"num_write_kb":128896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.183477+0000","last_change":"2024-03-12T03:35:17.142169+0000","last_active":"2024-03-12T03:35:36.183477+0000","last_peered":"2024-03-12T03:35:36.183477+0000","last_clean":"2024-03-12T03:35:36.183477+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:35:36.183477+0000","last_undegraded":"2024-03-12T03:35:36.183477+0000","last_fullsized":"2024-03-12T03:35:36.183477+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:17.142086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:17.142086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:34.053176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":117,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:41.648515+0000","last_change":"2024-03-12T03:35:20.138357+0000","last_active":"2024-03-12T03:35:41.648515+0000","last_peered":"2024-03-12T03:35:41.648515+0000","last_clean":"2024-03-12T03:35:41.648515+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:35:41.648515+0000","last_undegraded":"2024-03-12T03:35:41.648515+0000","last_fullsized":"2024-03-12T03:35:41.648515+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:35:20.138257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:20.138257+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:41.959809+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.189784+0000","last_change":"2024-03-12T03:35:13.030518+0000","last_active":"2024-03-12T03:35:36.189784+0000","last_peered":"2024-03-12T03:35:36.189784+0000","last_clean":"2024-03-12T03:35:36.189784+0000","last_became_active":"2024-03-12T03:33:21.104293+0000","last_became_peered":"2024-03-12T03:33:21.104293+0000","last_unstale":"2024-03-12T03:35:36.189784+0000","last_undegraded":"2024-03-12T03:35:36.189784+0000","last_fullsized":"2024-03-12T03:35:36.189784+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:13.030426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:13.030426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:27.030999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":79,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188417+0000","last_change":"2024-03-12T03:35:07.004248+0000","last_active":"2024-03-12T03:35:36.188417+0000","last_peered":"2024-03-12T03:35:36.188417+0000","last_clean":"2024-03-12T03:35:36.188417+0000","last_became_active":"2024-03-12T03:33:50.153240+0000","last_became_peered":"2024-03-12T03:33:50.153240+0000","last_unstale":"2024-03-12T03:35:36.188417+0000","last_undegraded":"2024-03-12T03:35:36.188417+0000","last_fullsized":"2024-03-12T03:35:36.188417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:35:07.004152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:07.004152+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:14.808273+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"101'2039","reported_seq":2057,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.744252+0000","last_change":"2024-03-12T03:35:23.783266+0000","last_active":"2024-03-12T03:35:45.744252+0000","last_peered":"2024-03-12T03:35:45.744252+0000","last_clean":"2024-03-12T03:35:45.744252+0000","last_became_active":"2024-03-12T03:35:23.782945+0000","last_became_peered":"2024-03-12T03:35:23.782945+0000","last_unstale":"2024-03-12T03:35:45.744252+0000","last_undegraded":"2024-03-12T03:35:45.744252+0000","last_fullsized":"2024-03-12T03:35:45.744252+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2039,"log_dups_size":0,"ondisk_log_size":2039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:44.162343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133627904,"num_objects":2039,"num_object_clones":0,"num_object_copies":6117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":130496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":68,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.190467+0000","last_change":"2024-03-12T03:33:27.381253+0000","last_active":"2024-03-12T03:35:36.190467+0000","last_peered":"2024-03-12T03:35:36.190467+0000","last_clean":"2024-03-12T03:35:36.190467+0000","last_became_active":"2024-03-12T03:33:27.380896+0000","last_became_peered":"2024-03-12T03:33:27.380896+0000","last_unstale":"2024-03-12T03:35:36.190467+0000","last_undegraded":"2024-03-12T03:35:36.190467+0000","last_fullsized":"2024-03-12T03:35:36.190467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:47.011091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.184201+0000","last_change":"2024-03-12T03:34:36.138975+0000","last_active":"2024-03-12T03:35:36.184201+0000","last_peered":"2024-03-12T03:35:36.184201+0000","last_clean":"2024-03-12T03:35:36.184201+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:35:36.184201+0000","last_undegraded":"2024-03-12T03:35:36.184201+0000","last_fullsized":"2024-03-12T03:35:36.184201+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.138847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.138847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.950469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.176872+0000","last_change":"2024-03-12T03:33:21.573917+0000","last_active":"2024-03-12T03:35:36.176872+0000","last_peered":"2024-03-12T03:35:36.176872+0000","last_clean":"2024-03-12T03:35:36.176872+0000","last_became_active":"2024-03-12T03:33:21.573257+0000","last_became_peered":"2024-03-12T03:33:21.573257+0000","last_unstale":"2024-03-12T03:35:36.176872+0000","last_undegraded":"2024-03-12T03:35:36.176872+0000","last_fullsized":"2024-03-12T03:35:36.176872+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"95'23","reported_seq":121,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.176771+0000","last_change":"2024-03-12T03:35:06.028109+0000","last_active":"2024-03-12T03:35:36.176771+0000","last_peered":"2024-03-12T03:35:36.176771+0000","last_clean":"2024-03-12T03:35:36.176771+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:35:36.176771+0000","last_undegraded":"2024-03-12T03:35:36.176771+0000","last_fullsized":"2024-03-12T03:35:36.176771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'23","last_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:35:06.028013+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:09.991579+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"101'2096","reported_seq":2114,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.781335+0000","last_change":"2024-03-12T03:35:23.777241+0000","last_active":"2024-03-12T03:35:45.781335+0000","last_peered":"2024-03-12T03:35:45.781335+0000","last_clean":"2024-03-12T03:35:45.781335+0000","last_became_active":"2024-03-12T03:35:23.776932+0000","last_became_peered":"2024-03-12T03:35:23.776932+0000","last_unstale":"2024-03-12T03:35:45.781335+0000","last_undegraded":"2024-03-12T03:35:45.781335+0000","last_fullsized":"2024-03-12T03:35:45.781335+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2096,"log_dups_size":0,"ondisk_log_size":2096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:41.400472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137363456,"num_objects":2096,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2096,"num_write_kb":134144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188634+0000","last_change":"2024-03-12T03:34:31.142859+0000","last_active":"2024-03-12T03:35:36.188634+0000","last_peered":"2024-03-12T03:35:36.188634+0000","last_clean":"2024-03-12T03:35:36.188634+0000","last_became_active":"2024-03-12T03:33:27.187815+0000","last_became_peered":"2024-03-12T03:33:27.187815+0000","last_unstale":"2024-03-12T03:35:36.188634+0000","last_undegraded":"2024-03-12T03:35:36.188634+0000","last_fullsized":"2024-03-12T03:35:36.188634+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:31.142765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:31.142765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:54.545296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":80,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.181438+0000","last_change":"2024-03-12T03:34:30.280067+0000","last_active":"2024-03-12T03:35:36.181438+0000","last_peered":"2024-03-12T03:35:36.181438+0000","last_clean":"2024-03-12T03:35:36.181438+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:35:36.181438+0000","last_undegraded":"2024-03-12T03:35:36.181438+0000","last_fullsized":"2024-03-12T03:35:36.181438+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:30.279966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.279966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:49.784484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.174511+0000","last_change":"2024-03-12T03:34:23.646936+0000","last_active":"2024-03-12T03:35:36.174511+0000","last_peered":"2024-03-12T03:35:36.174511+0000","last_clean":"2024-03-12T03:35:36.174511+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:35:36.174511+0000","last_undegraded":"2024-03-12T03:35:36.174511+0000","last_fullsized":"2024-03-12T03:35:36.174511+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.646819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":104,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.176764+0000","last_change":"2024-03-12T03:33:23.164300+0000","last_active":"2024-03-12T03:35:36.176764+0000","last_peered":"2024-03-12T03:35:36.176764+0000","last_clean":"2024-03-12T03:35:36.176764+0000","last_became_active":"2024-03-12T03:33:23.163871+0000","last_became_peered":"2024-03-12T03:33:23.163871+0000","last_unstale":"2024-03-12T03:35:36.176764+0000","last_undegraded":"2024-03-12T03:35:36.176764+0000","last_fullsized":"2024-03-12T03:35:36.176764+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"101'2134","reported_seq":2152,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:45.780895+0000","last_change":"2024-03-12T03:35:23.784927+0000","last_active":"2024-03-12T03:35:45.780895+0000","last_peered":"2024-03-12T03:35:45.780895+0000","last_clean":"2024-03-12T03:35:45.780895+0000","last_became_active":"2024-03-12T03:35:23.784627+0000","last_became_peered":"2024-03-12T03:35:23.784627+0000","last_unstale":"2024-03-12T03:35:45.780895+0000","last_undegraded":"2024-03-12T03:35:45.780895+0000","last_fullsized":"2024-03-12T03:35:45.780895+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":2134,"log_dups_size":0,"ondisk_log_size":2134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:37.248005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139853824,"num_objects":2134,"num_object_clones":0,"num_object_copies":6402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2134,"num_write_kb":136576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":68,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.176881+0000","last_change":"2024-03-12T03:33:27.184691+0000","last_active":"2024-03-12T03:35:36.176881+0000","last_peered":"2024-03-12T03:35:36.176881+0000","last_clean":"2024-03-12T03:35:36.176881+0000","last_became_active":"2024-03-12T03:33:27.184405+0000","last_became_peered":"2024-03-12T03:33:27.184405+0000","last_unstale":"2024-03-12T03:35:36.176881+0000","last_undegraded":"2024-03-12T03:35:36.176881+0000","last_fullsized":"2024-03-12T03:35:36.176881+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:48.127955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":117,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:40.788218+0000","last_change":"2024-03-12T03:34:55.906528+0000","last_active":"2024-03-12T03:35:40.788218+0000","last_peered":"2024-03-12T03:35:40.788218+0000","last_clean":"2024-03-12T03:35:40.788218+0000","last_became_active":"2024-03-12T03:33:25.177613+0000","last_became_peered":"2024-03-12T03:33:25.177613+0000","last_unstale":"2024-03-12T03:35:40.788218+0000","last_undegraded":"2024-03-12T03:35:40.788218+0000","last_fullsized":"2024-03-12T03:35:40.788218+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:55.906394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:55.906394+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:15.924626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":88,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.188026+0000","last_change":"2024-03-12T03:35:15.981464+0000","last_active":"2024-03-12T03:35:36.188026+0000","last_peered":"2024-03-12T03:35:36.188026+0000","last_clean":"2024-03-12T03:35:36.188026+0000","last_became_active":"2024-03-12T03:33:21.097304+0000","last_became_peered":"2024-03-12T03:33:21.097304+0000","last_unstale":"2024-03-12T03:35:36.188026+0000","last_undegraded":"2024-03-12T03:35:36.188026+0000","last_fullsized":"2024-03-12T03:35:36.188026+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.981378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.981378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:41.402156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":91,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-12T03:35:36.176419+0000","last_change":"2024-03-12T03:33:23.163246+0000","last_active":"2024-03-12T03:35:36.176419+0000","last_peered":"2024-03-12T03:35:36.176419+0000","last_clean":"2024-03-12T03:35:36.176419+0000","last_became_active":"2024-03-12T03:33:23.162657+0000","last_became_peered":"2024-03-12T03:33:23.162657+0000","last_unstale":"2024-03-12T03:35:36.176419+0000","last_undegraded":"2024-03-12T03:35:36.176419+0000","last_fullsized":"2024-03-12T03:35:36.176419+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2156789760,"num_objects":32910,"num_object_clones":0,"num_object_copies":98730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32910,"num_write_kb":2106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6473056256,"data_stored":6473056256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32910,"ondisk_log_size":32910,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":139,"num_read_kb":134,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":625,"ondisk_log_size":625,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070498,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":661500,"kb_used_data":626164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93083652,"statfs":{"total":95995035648,"available":95317659648,"internally_reserved":0,"allocated":641191936,"data_stored":640814698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332135,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654244,"kb_used_data":623068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93090908,"statfs":{"total":95995035648,"available":95325089792,"internally_reserved":0,"allocated":638021632,"data_stored":637667007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":819016,"kb_used_data":779456,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":92926136,"statfs":{"total":95995035648,"available":95156363264,"internally_reserved":0,"allocated":798162944,"data_stored":797772541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1051324,"kb_used_data":1002100,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92693828,"statfs":{"total":95995035648,"available":94918479872,"internally_reserved":0,"allocated":1026150400,"data_stored":1025772416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":797156,"kb_used_data":761820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":92947996,"statfs":{"total":95995035648,"available":95178747904,"internally_reserved":0,"allocated":780103680,"data_stored":779749744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117045,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1185352,"kb_used_data":1140288,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92559800,"statfs":{"total":95995035648,"available":94781235200,"internally_reserved":0,"allocated":1167654912,"data_stored":1167264206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953522,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":669116,"kb_used_data":633780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93076036,"statfs":{"total":95995035648,"available":95309860864,"internally_reserved":0,"allocated":648990720,"data_stored":648612478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659180,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":534064,"kb_used_data":507048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93211088,"statfs":{"total":95995035648,"available":95448154112,"internally_reserved":0,"allocated":519217152,"data_stored":518850366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1081278464,"data_stored":1081278464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":803799040,"data_stored":803799040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":813760512,"data_stored":813760512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1223294976,"data_stored":1223294976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":677511168,"data_stored":677511168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":538116096,"data_stored":538116096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":661651456,"data_stored":661651456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":673644544,"data_stored":673644544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:35:50.582 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-12T03:35:50.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:50.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:50.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:50.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:50.726 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:50.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:50.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:50.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:50.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:50.960 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:51.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:51.003 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:51.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:51.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:51.145 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:51.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:51.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:51 smithi169 ceph-mon[29344]: pgmap v76: 148 pgs: 148 active+clean; 2.0 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-12T03:35:51.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:51 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:35:51.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:51 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:35:51.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:51 smithi169 ceph-mon[39949]: pgmap v76: 148 pgs: 148 active+clean; 2.0 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-12T03:35:51.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:51 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:35:51.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:51 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:35:51.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:51.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:51.294 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:51.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:51.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:51.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:51.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:51.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:51.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:51 smithi173 ceph-mon[33544]: pgmap v76: 148 pgs: 148 active+clean; 2.0 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-12T03:35:51.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:51 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:35:51.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:51 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:35:51.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:51.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:51.563 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:35:51.561414+0000 min lat: 0.00118901 max lat: 0.163567 avg lat: 0.00751769 2024-03-12T03:35:51.563 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:35:51.563 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 20 16 42562 42546 132.932 107.812 0.00754521 0.00751769 2024-03-12T03:35:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:51.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:51.664 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:51.704 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:51.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:51.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:51.707 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:51.772 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-12T03:35:51.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:51.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:51.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:51.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:51.990 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:51.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:52.032 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:35:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:35:52.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:52.148 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:52.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:52 smithi173 ceph-mon[33544]: osdmap e102: 8 total, 8 up, 8 in 2024-03-12T03:35:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:52 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:35:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:52 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:35:52.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:52.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:52.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:52.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:52.288 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:52 smithi169 ceph-mon[39949]: osdmap e102: 8 total, 8 up, 8 in 2024-03-12T03:35:52.288 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:52 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:35:52.289 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:52 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:35:52.289 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:52 smithi169 ceph-mon[29344]: osdmap e102: 8 total, 8 up, 8 in 2024-03-12T03:35:52.289 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:52 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:35:52.289 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:52 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:35:52.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:52.425 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:52.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:52.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:52.562 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 21 16 44132 44116 131.274 98.125 0.00896382 0.00761167 2024-03-12T03:35:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:52.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:52.578 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:35:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:35:52.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:52.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:52.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:52.805 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:52.847 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:52.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:52.855 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:52 smithi173 systemd[1]: Stopping Ceph prometheus.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:35:52.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:52.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:52.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:52.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:53.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:53.122 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:53.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:53.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:53.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:53.261 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:53.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:53.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:53.362 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:53.404 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:53.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:53.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:53 smithi169 ceph-mon[29344]: pgmap v78: 148 pgs: 148 active+clean; 2.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-12T03:35:53.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:53 smithi169 ceph-mon[29344]: osdmap e103: 8 total, 8 up, 8 in 2024-03-12T03:35:53.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:53 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:53.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:53 smithi169 ceph-mon[39949]: pgmap v78: 148 pgs: 148 active+clean; 2.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-12T03:35:53.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:53 smithi169 ceph-mon[39949]: osdmap e103: 8 total, 8 up, 8 in 2024-03-12T03:35:53.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:53 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:53.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-mon[33544]: pgmap v78: 148 pgs: 148 active+clean; 2.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-12T03:35:53.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-mon[33544]: osdmap e103: 8 total, 8 up, 8 in 2024-03-12T03:35:53.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:35:53.486 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[62029]: ts=2024-03-12T03:35:53.186Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:35:53.487 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[62029]: ts=2024-03-12T03:35:53.186Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-12T03:35:53.487 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[62029]: ts=2024-03-12T03:35:53.186Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-12T03:35:53.487 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[62029]: ts=2024-03-12T03:35:53.187Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T03:35:53.487 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[62029]: ts=2024-03-12T03:35:53.187Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-12T03:35:53.487 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[62029]: ts=2024-03-12T03:35:53.187Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T03:35:53.487 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[62029]: ts=2024-03-12T03:35:53.187Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-12T03:35:53.487 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[62029]: ts=2024-03-12T03:35:53.187Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-12T03:35:53.487 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[62029]: ts=2024-03-12T03:35:53.187Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-12T03:35:53.487 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[62029]: ts=2024-03-12T03:35:53.187Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T03:35:53.487 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[62029]: ts=2024-03-12T03:35:53.187Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-12T03:35:53.487 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[62029]: ts=2024-03-12T03:35:53.187Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-12T03:35:53.487 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 podman[67869]: 2024-03-12 03:35:53.197042824 +0000 UTC m=+0.140993459 container died 3ae0da93deec0cd37ae345fb5881ca46d6de98549e96cd11f9cfcc8bc75575e2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:53.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:53.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:53.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:53.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:53.563 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 22 16 45659 45643 129.643 95.4375 0.00837764 0.00770704 2024-03-12T03:35:53.601 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:53.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:53.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:53.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:53.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:53.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:53.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:53.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:53.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:53.947 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:53.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:54.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:54.094 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:54.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:54.137 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 podman[67869]: 2024-03-12 03:35:53.87133277 +0000 UTC m=+0.815283403 container cleanup 3ae0da93deec0cd37ae345fb5881ca46d6de98549e96cd11f9cfcc8bc75575e2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:54.137 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:53 smithi173 bash[67869]: ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T03:35:54.194 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:35:54.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:54.242 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:54.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:54.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:54.296 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:35:54.342 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:35:54.342 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:54 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:35:54.342 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:54 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:35:54.342 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:54 smithi169 ceph-mon[29344]: osdmap e104: 8 total, 8 up, 8 in 2024-03-12T03:35:54.342 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:54 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/713672365' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:35:54.342 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:54 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:35:54.343 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:54 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:35:54.343 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:54 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:35:54.343 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:54 smithi169 ceph-mon[39949]: osdmap e104: 8 total, 8 up, 8 in 2024-03-12T03:35:54.343 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:54 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/713672365' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:35:54.343 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:54 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:35:54.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:54.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:54.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:54.383 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:35:54.426 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:35:54.477 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:35:54.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:54.486 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 podman[67880]: 2024-03-12 03:35:54.139668124 +0000 UTC m=+0.943583852 container remove 3ae0da93deec0cd37ae345fb5881ca46d6de98549e96cd11f9cfcc8bc75575e2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:54.486 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-12T03:35:54.486 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 systemd[1]: Stopped Ceph prometheus.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:35:54.486 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@prometheus.a.service: Consumed 1.296s CPU time. 2024-03-12T03:35:54.486 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 systemd[1]: Starting Ceph prometheus.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:35:54.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:35:54.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:35:54.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-mon[33544]: osdmap e104: 8 total, 8 up, 8 in 2024-03-12T03:35:54.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/713672365' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:35:54.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:35:54.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:54.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:54.516 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:35:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:35:54.563 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 23 16 47157 47141 128.074 93.625 0.00820525 0.00780219 2024-03-12T03:35:54.594 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:35:54.598 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:35:54.598 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:35:54.598 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:35:54.598 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:35:54.598 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:35:54.598 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:35:54.598 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:35:54.599 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:35:54.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:54.655 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:54.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:54.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:54.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:54.797 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:54.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:54.862 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 podman[68036]: 2024-03-12 03:35:54.478926173 +0000 UTC m=+0.027351347 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-12T03:35:54.862 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 podman[68036]: 2024-03-12 03:35:54.607688111 +0000 UTC m=+0.156113295 container create 48619c1583bf34b4bda7d98ccfdbed6f47506c1bc2a5f7826009531171ecbbfc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:54.862 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 podman[68036]: 2024-03-12 03:35:54.860671608 +0000 UTC m=+0.409096758 container init 48619c1583bf34b4bda7d98ccfdbed6f47506c1bc2a5f7826009531171ecbbfc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:54.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:54.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:54.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:55.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:55.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:55.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:55.139 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 podman[68036]: 2024-03-12 03:35:54.86553446 +0000 UTC m=+0.413959616 container start 48619c1583bf34b4bda7d98ccfdbed6f47506c1bc2a5f7826009531171ecbbfc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.899Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.899Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.899Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi173 (none))" 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.899Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.899Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.902Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.904Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.908Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.908Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.115µs 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.908Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.909Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.909Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-12T03:35:55.140 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.909Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.671µs wal_replay_duration=1.279604ms wbl_replay_duration=128ns total_replay_duration=1.396699ms 2024-03-12T03:35:55.141 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.910Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-12T03:35:55.141 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.910Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-12T03:35:55.141 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.911Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-12T03:35:55.141 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.911Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-12T03:35:55.141 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.911Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-12T03:35:55.141 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.940Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=29.62589ms db_storage=1.237µs remote_storage=1.269µs web_handler=356ns query_engine=794ns scrape=529.982µs scrape_sd=100.591µs notify=17.213µs notify_sd=25.104µs rules=28.048781ms tracing=8.239µs 2024-03-12T03:35:55.141 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.941Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-12T03:35:55.141 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:35:54.941Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-12T03:35:55.141 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 bash[68036]: 48619c1583bf34b4bda7d98ccfdbed6f47506c1bc2a5f7826009531171ecbbfc 2024-03-12T03:35:55.141 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:35:54 smithi173 systemd[1]: Started Ceph prometheus.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:35:55.172 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:55.213 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:55.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:55.231 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: pgmap v81: 148 pgs: 148 active+clean; 2.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-12T03:35:55.231 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:35:55.231 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:35:55.231 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: osdmap e105: 8 total, 8 up, 8 in 2024-03-12T03:35:55.231 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2407418442' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:35:55.231 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:55.231 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:55.231 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:35:55.232 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi169.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:35:55.232 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:55.232 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:35:55.232 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:35:55.232 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:55.232 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Bus STOPPING 2024-03-12T03:35:55.232 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:35:55.232 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Bus STOPPED 2024-03-12T03:35:55.232 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Bus STARTING 2024-03-12T03:35:55.232 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Serving on http://:::9283 2024-03-12T03:35:55.232 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Bus STARTED 2024-03-12T03:35:55.233 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Bus STOPPING 2024-03-12T03:35:55.233 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:35:55.233 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Bus STOPPED 2024-03-12T03:35:55.233 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: pgmap v81: 148 pgs: 148 active+clean; 2.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-12T03:35:55.233 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:35:55.233 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:35:55.233 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: osdmap e105: 8 total, 8 up, 8 in 2024-03-12T03:35:55.233 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2407418442' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:35:55.233 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:55.233 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:55.233 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:35:55.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi169.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:35:55.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:55.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:35:55.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:35:55.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:55.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:35:55.358 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:55.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:55.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:35:55.396 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: pgmap v81: 148 pgs: 148 active+clean; 2.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-12T03:35:55.396 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:35:55.396 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:35:55.396 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: osdmap e105: 8 total, 8 up, 8 in 2024-03-12T03:35:55.396 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2407418442' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:35:55.396 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:55.397 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:55.397 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:35:55.397 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi169.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:35:55.397 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:55.397 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:35:55.397 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:35:55.397 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:55 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:55.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:55.486 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Bus STARTING 2024-03-12T03:35:55.486 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Serving on http://:::9283 2024-03-12T03:35:55.486 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Bus STARTED 2024-03-12T03:35:55.486 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Bus STOPPING 2024-03-12T03:35:55.486 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:35:55.486 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Bus STOPPED 2024-03-12T03:35:55.486 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Bus STARTING 2024-03-12T03:35:55.486 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Serving on http://:::9283 2024-03-12T03:35:55.486 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:35:55] ENGINE Bus STARTED 2024-03-12T03:35:55.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:55.503 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:55.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:55.563 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 24 16 48619 48603 126.544 91.375 0.00902967 0.00789653 2024-03-12T03:35:55.583 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:35:55.583 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:35:55.583 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-03-12T03:35:55.583 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:35:55.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:55.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:55.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:55.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:55.742 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:55.805 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:55.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:55.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:55.908 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:55.957 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:55.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:56.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:56.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:56.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:56.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:56.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:56.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:56.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:56.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:56.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:56.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:56.406 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[29344]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:35:56.406 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[29344]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi169.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:35:56.406 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[29344]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:35:56.406 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[29344]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:35:56.406 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:35:56.406 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[29344]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:35:56.406 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:35:56.406 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[29344]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:35:56.406 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:56.407 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:35:56.407 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[39949]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:35:56.407 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[39949]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi169.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:35:56.407 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[39949]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:35:56.407 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[39949]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:35:56.407 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:35:56.407 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[39949]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:35:56.407 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:35:56.407 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[39949]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:35:56.407 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:56.407 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:56 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:35:56.408 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:35:55 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[81118]: ts=2024-03-12T03:35:55.937Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004268897s 2024-03-12T03:35:56.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:56.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:56 smithi173 ceph-mon[33544]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:35:56.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:56 smithi173 ceph-mon[33544]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi169.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:35:56.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:56 smithi173 ceph-mon[33544]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:35:56.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:56 smithi173 ceph-mon[33544]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:35:56.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:56 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:35:56.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:56 smithi173 ceph-mon[33544]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:35:56.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:56 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:35:56.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:56 smithi173 ceph-mon[33544]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:35:56.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:56 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:35:56.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:56 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:35:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:56.532 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:56.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:56.564 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 25 16 50079 50063 125.132 91.25 0.00737473 0.00798681 2024-03-12T03:35:56.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:56.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:56.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:56.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:56.806 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:56.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:56.908 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:56.958 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:56.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:57.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:35:57.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:57.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:57.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:35:57.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:57.247 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:57.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:57.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:57.376 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:57.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:57.381 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:57 smithi169 ceph-mon[29344]: pgmap v83: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 32 MiB/s wr, 507 op/s 2024-03-12T03:35:57.381 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:57 smithi169 ceph-mon[29344]: 6.f scrub starts 2024-03-12T03:35:57.381 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:57 smithi169 ceph-mon[29344]: 6.f scrub ok 2024-03-12T03:35:57.381 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:57 smithi169 ceph-mon[39949]: pgmap v83: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 32 MiB/s wr, 507 op/s 2024-03-12T03:35:57.381 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:57 smithi169 ceph-mon[39949]: 6.f scrub starts 2024-03-12T03:35:57.381 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:57 smithi169 ceph-mon[39949]: 6.f scrub ok 2024-03-12T03:35:57.433 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:35:57.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:57.482 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:57.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:57 smithi173 ceph-mon[33544]: pgmap v83: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 32 MiB/s wr, 507 op/s 2024-03-12T03:35:57.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:57 smithi173 ceph-mon[33544]: 6.f scrub starts 2024-03-12T03:35:57.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:57 smithi173 ceph-mon[33544]: 6.f scrub ok 2024-03-12T03:35:57.490 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:35:57.490 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:35:57.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:57.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:57.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:57.564 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 26 16 51483 51467 123.693 87.75 0.00835084 0.00807795 2024-03-12T03:35:57.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:57.661 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:57.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:57.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:57.813 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:57.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:57.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:57.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:57.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:57.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:58.049 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:58.075 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:58.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:58.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:58.216 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:58.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:58.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:58.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:58.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:58.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:58.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:58.482 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:58.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:58.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:58 smithi173 ceph-mon[33544]: 6.1c scrub starts 2024-03-12T03:35:58.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:58 smithi173 ceph-mon[33544]: 6.1c scrub ok 2024-03-12T03:35:58.565 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 27 16 52798 52782 122.156 82.1875 0.0083542 0.00818001 2024-03-12T03:35:58.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:35:58.629 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:58.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:35:58.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:58 smithi169 ceph-mon[39949]: 6.1c scrub starts 2024-03-12T03:35:58.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:58 smithi169 ceph-mon[39949]: 6.1c scrub ok 2024-03-12T03:35:58.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:58 smithi169 ceph-mon[29344]: 6.1c scrub starts 2024-03-12T03:35:58.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:58 smithi169 ceph-mon[29344]: 6.1c scrub ok 2024-03-12T03:35:58.731 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:58.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:58.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:58.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:35:58.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:58.905 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:58.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:35:58.906 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:35:59.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:59.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:59.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:59.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:59.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:59.190 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:59.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:59.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:59.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:59.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:59.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[29344]: pgmap v84: 148 pgs: 148 active+clean; 3.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-12T03:35:59.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[29344]: 3.10 scrub starts 2024-03-12T03:35:59.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[29344]: 4.14 scrub starts 2024-03-12T03:35:59.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[29344]: 3.10 scrub ok 2024-03-12T03:35:59.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[29344]: 4.14 scrub ok 2024-03-12T03:35:59.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[29344]: 4.c scrub starts 2024-03-12T03:35:59.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[29344]: 5.6 scrub starts 2024-03-12T03:35:59.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[29344]: 4.c scrub ok 2024-03-12T03:35:59.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[29344]: 5.6 scrub ok 2024-03-12T03:35:59.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[39949]: pgmap v84: 148 pgs: 148 active+clean; 3.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-12T03:35:59.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[39949]: 3.10 scrub starts 2024-03-12T03:35:59.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[39949]: 4.14 scrub starts 2024-03-12T03:35:59.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[39949]: 3.10 scrub ok 2024-03-12T03:35:59.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[39949]: 4.14 scrub ok 2024-03-12T03:35:59.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[39949]: 4.c scrub starts 2024-03-12T03:35:59.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[39949]: 5.6 scrub starts 2024-03-12T03:35:59.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[39949]: 4.c scrub ok 2024-03-12T03:35:59.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:35:59 smithi169 ceph-mon[39949]: 5.6 scrub ok 2024-03-12T03:35:59.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:35:59.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:59.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:59.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:35:59.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:59 smithi173 ceph-mon[33544]: pgmap v84: 148 pgs: 148 active+clean; 3.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-12T03:35:59.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:59 smithi173 ceph-mon[33544]: 3.10 scrub starts 2024-03-12T03:35:59.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:59 smithi173 ceph-mon[33544]: 4.14 scrub starts 2024-03-12T03:35:59.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:59 smithi173 ceph-mon[33544]: 3.10 scrub ok 2024-03-12T03:35:59.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:59 smithi173 ceph-mon[33544]: 4.14 scrub ok 2024-03-12T03:35:59.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:59 smithi173 ceph-mon[33544]: 4.c scrub starts 2024-03-12T03:35:59.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:59 smithi173 ceph-mon[33544]: 5.6 scrub starts 2024-03-12T03:35:59.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:59 smithi173 ceph-mon[33544]: 4.c scrub ok 2024-03-12T03:35:59.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:35:59 smithi173 ceph-mon[33544]: 5.6 scrub ok 2024-03-12T03:35:59.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:35:59.565 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 28 15 54149 54134 120.809 84.5 0.0148959 0.00827118 2024-03-12T03:35:59.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:59.600 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:35:59.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:35:59.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:35:59.729 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:59.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:59.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:35:59.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:59.860 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:59.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:35:59.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:35:59.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:35:59.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:35:59.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:00.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:00.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:00.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:00.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:00.230 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:00.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:00.273 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:00.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:00.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:00.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:00.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:00.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:00.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:36:00.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:36:00.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: 3.a scrub starts 2024-03-12T03:36:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: 3.a scrub ok 2024-03-12T03:36:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: 4.d scrub starts 2024-03-12T03:36:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: 4.d scrub ok 2024-03-12T03:36:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: 6.11 scrub starts 2024-03-12T03:36:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: 3.17 scrub starts 2024-03-12T03:36:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: 6.11 scrub ok 2024-03-12T03:36:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: 3.17 scrub ok 2024-03-12T03:36:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: osdmap e106: 8 total, 8 up, 8 in 2024-03-12T03:36:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: 3.16 scrub starts 2024-03-12T03:36:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: 3.16 scrub ok 2024-03-12T03:36:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:00.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:00.510 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:36:00.510 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:36:00.510 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: 3.a scrub starts 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: 3.a scrub ok 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: 4.d scrub starts 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: 4.d scrub ok 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: 6.11 scrub starts 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: 3.17 scrub starts 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: 6.11 scrub ok 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: 3.17 scrub ok 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: osdmap e106: 8 total, 8 up, 8 in 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: 3.16 scrub starts 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: 3.16 scrub ok 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:36:00.511 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:36:00.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: 3.a scrub starts 2024-03-12T03:36:00.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: 3.a scrub ok 2024-03-12T03:36:00.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: 4.d scrub starts 2024-03-12T03:36:00.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: 4.d scrub ok 2024-03-12T03:36:00.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: 6.11 scrub starts 2024-03-12T03:36:00.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: 3.17 scrub starts 2024-03-12T03:36:00.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: 6.11 scrub ok 2024-03-12T03:36:00.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: 3.17 scrub ok 2024-03-12T03:36:00.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: osdmap e106: 8 total, 8 up, 8 in 2024-03-12T03:36:00.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: 3.16 scrub starts 2024-03-12T03:36:00.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: 3.16 scrub ok 2024-03-12T03:36:00.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:00.546 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:00.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:00.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:00.565 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 29 16 55446 55430 119.436 81 0.0119814 0.00836794 2024-03-12T03:36:00.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:00.690 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:00.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:00.734 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:00.792 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:00.844 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:00.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:00.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:00.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:00.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:00.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:01.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:01.109 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:01.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:01.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:01.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:01.245 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:01.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:01.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:01.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:01.375 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:01.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:01.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:01.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:01 smithi173 ceph-mon[33544]: pgmap v86: 148 pgs: 148 active+clean; 3.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 769 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-12T03:36:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:01 smithi173 ceph-mon[33544]: 5.1f scrub starts 2024-03-12T03:36:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:01 smithi173 ceph-mon[33544]: 5.1f scrub ok 2024-03-12T03:36:01.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:01.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:01.541 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:01 smithi169 ceph-mon[29344]: pgmap v86: 148 pgs: 148 active+clean; 3.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 769 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-12T03:36:01.541 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:01 smithi169 ceph-mon[29344]: 5.1f scrub starts 2024-03-12T03:36:01.541 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:01 smithi169 ceph-mon[29344]: 5.1f scrub ok 2024-03-12T03:36:01.541 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:01 smithi169 ceph-mon[39949]: pgmap v86: 148 pgs: 148 active+clean; 3.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 769 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-12T03:36:01.541 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:01 smithi169 ceph-mon[39949]: 5.1f scrub starts 2024-03-12T03:36:01.541 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:01 smithi169 ceph-mon[39949]: 5.1f scrub ok 2024-03-12T03:36:01.565 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 30 16 56886 56870 118.455 90 0.00926164 0.00843587 2024-03-12T03:36:01.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:01.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:01.657 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:01.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:01.758 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:01.807 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:01.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:01.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:01.909 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:01.952 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:01.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:01.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:02.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:02.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:02.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:02.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:02.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:36:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:36:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:36:02.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:02.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:02.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:02.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:02.352 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:02.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:02.454 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:02.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:02.495 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:02.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:02.565 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 31 16 58300 58284 117.485 88.375 0.00723051 0.00850711 2024-03-12T03:36:02.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:02.638 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:02.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[29344]: 5.12 scrub starts 2024-03-12T03:36:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[29344]: 5.12 scrub ok 2024-03-12T03:36:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[29344]: 6.1e scrub starts 2024-03-12T03:36:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[29344]: 6.1e scrub ok 2024-03-12T03:36:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[29344]: 5.11 scrub starts 2024-03-12T03:36:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[29344]: 5.11 scrub ok 2024-03-12T03:36:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[39949]: 5.12 scrub starts 2024-03-12T03:36:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[39949]: 5.12 scrub ok 2024-03-12T03:36:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[39949]: 6.1e scrub starts 2024-03-12T03:36:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[39949]: 6.1e scrub ok 2024-03-12T03:36:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[39949]: 5.11 scrub starts 2024-03-12T03:36:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[39949]: 5.11 scrub ok 2024-03-12T03:36:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:02.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:02 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:02.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:36:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:36:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:02 smithi173 ceph-mon[33544]: 5.12 scrub starts 2024-03-12T03:36:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:02 smithi173 ceph-mon[33544]: 5.12 scrub ok 2024-03-12T03:36:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:02 smithi173 ceph-mon[33544]: 6.1e scrub starts 2024-03-12T03:36:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:02 smithi173 ceph-mon[33544]: 6.1e scrub ok 2024-03-12T03:36:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:02 smithi173 ceph-mon[33544]: 5.11 scrub starts 2024-03-12T03:36:02.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:02 smithi173 ceph-mon[33544]: 5.11 scrub ok 2024-03-12T03:36:02.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:02 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:02.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:02 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:02.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:02.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:02.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:02.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-12T03:36:02.871 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:02.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:02.934 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:02.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:03.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:03.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:03.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:03.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:03.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:03.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:03.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:03.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:03.339 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:03.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:03.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:03.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:03.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:03.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:03.565 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 32 16 59648 59632 116.446 84.25 0.0180296 0.00858262 2024-03-12T03:36:03.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:03.591 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[29344]: pgmap v87: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-12T03:36:03.591 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[29344]: 3.4 scrub starts 2024-03-12T03:36:03.591 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[29344]: 3.4 scrub ok 2024-03-12T03:36:03.591 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[29344]: 6.5 scrub starts 2024-03-12T03:36:03.591 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[29344]: 6.5 scrub ok 2024-03-12T03:36:03.591 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[29344]: 6.14 scrub starts 2024-03-12T03:36:03.592 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[29344]: 6.14 scrub ok 2024-03-12T03:36:03.592 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:03.592 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[39949]: pgmap v87: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-12T03:36:03.592 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[39949]: 3.4 scrub starts 2024-03-12T03:36:03.592 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[39949]: 3.4 scrub ok 2024-03-12T03:36:03.592 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[39949]: 6.5 scrub starts 2024-03-12T03:36:03.592 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[39949]: 6.5 scrub ok 2024-03-12T03:36:03.592 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[39949]: 6.14 scrub starts 2024-03-12T03:36:03.592 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[39949]: 6.14 scrub ok 2024-03-12T03:36:03.592 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:03 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:03.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:03.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:03.709 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:03 smithi173 ceph-mon[33544]: pgmap v87: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-12T03:36:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:03 smithi173 ceph-mon[33544]: 3.4 scrub starts 2024-03-12T03:36:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:03 smithi173 ceph-mon[33544]: 3.4 scrub ok 2024-03-12T03:36:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:03 smithi173 ceph-mon[33544]: 6.5 scrub starts 2024-03-12T03:36:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:03 smithi173 ceph-mon[33544]: 6.5 scrub ok 2024-03-12T03:36:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:03 smithi173 ceph-mon[33544]: 6.14 scrub starts 2024-03-12T03:36:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:03 smithi173 ceph-mon[33544]: 6.14 scrub ok 2024-03-12T03:36:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:03 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:03.766 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:03.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:03.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:03.914 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:03.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:04.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:04.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:04.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:04.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:04.161 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:04.187 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:04.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:04.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:04.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:04.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:04.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:04.321 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:36:04.333 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:36:04.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:04.468 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:04.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:04.565 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 33 16 60632 60616 114.781 61.5 0.00630155 0.00870672 2024-03-12T03:36:04.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:04.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:04.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:04.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[29344]: 5.19 scrub starts 2024-03-12T03:36:04.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[29344]: 5.19 scrub ok 2024-03-12T03:36:04.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[29344]: 4.18 scrub starts 2024-03-12T03:36:04.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[29344]: 4.18 scrub ok 2024-03-12T03:36:04.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[29344]: 4.6 scrub starts 2024-03-12T03:36:04.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[29344]: 4.6 scrub ok 2024-03-12T03:36:04.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[29344]: 4.5 scrub starts 2024-03-12T03:36:04.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[29344]: 4.5 scrub ok 2024-03-12T03:36:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[39949]: 5.19 scrub starts 2024-03-12T03:36:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[39949]: 5.19 scrub ok 2024-03-12T03:36:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[39949]: 4.18 scrub starts 2024-03-12T03:36:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[39949]: 4.18 scrub ok 2024-03-12T03:36:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[39949]: 4.6 scrub starts 2024-03-12T03:36:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[39949]: 4.6 scrub ok 2024-03-12T03:36:04.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[39949]: 4.5 scrub starts 2024-03-12T03:36:04.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:04 smithi169 ceph-mon[39949]: 4.5 scrub ok 2024-03-12T03:36:04.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:04.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:04.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:04 smithi173 ceph-mon[33544]: 5.19 scrub starts 2024-03-12T03:36:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:04 smithi173 ceph-mon[33544]: 5.19 scrub ok 2024-03-12T03:36:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:04 smithi173 ceph-mon[33544]: 4.18 scrub starts 2024-03-12T03:36:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:04 smithi173 ceph-mon[33544]: 4.18 scrub ok 2024-03-12T03:36:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:04 smithi173 ceph-mon[33544]: 4.6 scrub starts 2024-03-12T03:36:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:04 smithi173 ceph-mon[33544]: 4.6 scrub ok 2024-03-12T03:36:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:04 smithi173 ceph-mon[33544]: 4.5 scrub starts 2024-03-12T03:36:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:04 smithi173 ceph-mon[33544]: 4.5 scrub ok 2024-03-12T03:36:04.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:04.857 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:04.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:04.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:04.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:04.989 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:04.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:04.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:05.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:05.134 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:05.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:05.196 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-03-12T03:36:03.793894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3656456616,"num_objects":56014,"num_object_clones":0,"num_object_copies":168042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56014,"num_whiteouts":0,"num_read":1376,"num_read_kb":1190,"num_write":56591,"num_write_kb":3572637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56617,"ondisk_log_size":56617,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10898792,"kb_used_data":10524380,"kb_used_omap":12,"kb_used_meta":374259,"kb_avail":739062424,"statfs":{"total":767960285184,"available":756799922176,"internally_reserved":0,"allocated":10776965120,"data_stored":10773758240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":383241776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":951713792,"num_objects":14522,"num_object_clones":0,"num_object_copies":43566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14522,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14522,"num_write_kb":929408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.619363"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.222992+0000","last_change":"2024-03-12T03:35:21.106867+0000","last_active":"2024-03-12T03:35:59.222992+0000","last_peered":"2024-03-12T03:35:59.222992+0000","last_clean":"2024-03-12T03:35:59.222992+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:35:59.222992+0000","last_undegraded":"2024-03-12T03:35:59.222992+0000","last_fullsized":"2024-03-12T03:35:59.222992+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:21.106755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:21.106755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:24.765060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.233436+0000","last_change":"2024-03-12T03:34:29.244162+0000","last_active":"2024-03-12T03:35:59.233436+0000","last_peered":"2024-03-12T03:35:59.233436+0000","last_clean":"2024-03-12T03:35:59.233436+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:35:59.233436+0000","last_undegraded":"2024-03-12T03:35:59.233436+0000","last_fullsized":"2024-03-12T03:35:59.233436+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:29.244078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:29.244078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.235013+0000","last_change":"2024-03-12T03:34:48.829003+0000","last_active":"2024-03-12T03:35:59.235013+0000","last_peered":"2024-03-12T03:35:59.235013+0000","last_clean":"2024-03-12T03:35:59.235013+0000","last_became_active":"2024-03-12T03:33:21.573080+0000","last_became_peered":"2024-03-12T03:33:21.573080+0000","last_unstale":"2024-03-12T03:35:59.235013+0000","last_undegraded":"2024-03-12T03:35:59.235013+0000","last_fullsized":"2024-03-12T03:35:59.235013+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:48.828926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:48.828926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:15.319549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":114,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.222675+0000","last_change":"2024-03-12T03:33:23.164433+0000","last_active":"2024-03-12T03:35:59.222675+0000","last_peered":"2024-03-12T03:35:59.222675+0000","last_clean":"2024-03-12T03:35:59.222675+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:35:59.222675+0000","last_undegraded":"2024-03-12T03:35:59.222675+0000","last_fullsized":"2024-03-12T03:35:59.222675+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.226382+0000","last_change":"2024-03-12T03:33:27.382583+0000","last_active":"2024-03-12T03:35:59.226382+0000","last_peered":"2024-03-12T03:35:59.226382+0000","last_clean":"2024-03-12T03:35:59.226382+0000","last_became_active":"2024-03-12T03:33:27.382050+0000","last_became_peered":"2024-03-12T03:33:27.382050+0000","last_unstale":"2024-03-12T03:35:59.226382+0000","last_undegraded":"2024-03-12T03:35:59.226382+0000","last_fullsized":"2024-03-12T03:35:59.226382+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.222784+0000","last_change":"2024-03-12T03:35:17.100474+0000","last_active":"2024-03-12T03:35:59.222784+0000","last_peered":"2024-03-12T03:35:59.222784+0000","last_clean":"2024-03-12T03:35:59.222784+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:35:59.222784+0000","last_undegraded":"2024-03-12T03:35:59.222784+0000","last_fullsized":"2024-03-12T03:35:59.222784+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:17.100375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:17.100375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:39.744648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.226885+0000","last_change":"2024-03-12T03:34:41.143075+0000","last_active":"2024-03-12T03:35:59.226885+0000","last_peered":"2024-03-12T03:35:59.226885+0000","last_clean":"2024-03-12T03:35:59.226885+0000","last_became_active":"2024-03-12T03:33:21.572876+0000","last_became_peered":"2024-03-12T03:33:21.572876+0000","last_unstale":"2024-03-12T03:35:59.226885+0000","last_undegraded":"2024-03-12T03:35:59.226885+0000","last_fullsized":"2024-03-12T03:35:59.226885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.142994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"94'50","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.225294+0000","last_change":"2024-03-12T03:35:05.042142+0000","last_active":"2024-03-12T03:35:59.225294+0000","last_peered":"2024-03-12T03:35:59.225294+0000","last_clean":"2024-03-12T03:35:59.225294+0000","last_became_active":"2024-03-12T03:33:23.161549+0000","last_became_peered":"2024-03-12T03:33:23.161549+0000","last_unstale":"2024-03-12T03:35:59.225294+0000","last_undegraded":"2024-03-12T03:35:59.225294+0000","last_fullsized":"2024-03-12T03:35:59.225294+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.042030+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:12.264888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229534+0000","last_change":"2024-03-12T03:35:00.731143+0000","last_active":"2024-03-12T03:35:59.229534+0000","last_peered":"2024-03-12T03:35:59.229534+0000","last_clean":"2024-03-12T03:35:59.229534+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:35:59.229534+0000","last_undegraded":"2024-03-12T03:35:59.229534+0000","last_fullsized":"2024-03-12T03:35:59.229534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:00.731066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.731066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:14.276977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232365+0000","last_change":"2024-03-12T03:34:50.189337+0000","last_active":"2024-03-12T03:35:59.232365+0000","last_peered":"2024-03-12T03:35:59.232365+0000","last_clean":"2024-03-12T03:35:59.232365+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:35:59.232365+0000","last_undegraded":"2024-03-12T03:35:59.232365+0000","last_fullsized":"2024-03-12T03:35:59.232365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:50.189256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:50.189256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232605+0000","last_change":"2024-03-12T03:34:51.188063+0000","last_active":"2024-03-12T03:35:59.232605+0000","last_peered":"2024-03-12T03:35:59.232605+0000","last_clean":"2024-03-12T03:35:59.232605+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:35:59.232605+0000","last_undegraded":"2024-03-12T03:35:59.232605+0000","last_fullsized":"2024-03-12T03:35:59.232605+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:34:51.187976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:08.066983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'21","reported_seq":134,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223192+0000","last_change":"2024-03-12T03:35:19.084309+0000","last_active":"2024-03-12T03:35:59.223192+0000","last_peered":"2024-03-12T03:35:59.223192+0000","last_clean":"2024-03-12T03:35:59.223192+0000","last_became_active":"2024-03-12T03:33:23.162749+0000","last_became_peered":"2024-03-12T03:33:23.162749+0000","last_unstale":"2024-03-12T03:35:59.223192+0000","last_undegraded":"2024-03-12T03:35:59.223192+0000","last_fullsized":"2024-03-12T03:35:59.223192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:35:19.084191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:19.084191+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:29.257405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232029+0000","last_change":"2024-03-12T03:35:05.181569+0000","last_active":"2024-03-12T03:35:59.232029+0000","last_peered":"2024-03-12T03:35:59.232029+0000","last_clean":"2024-03-12T03:35:59.232029+0000","last_became_active":"2024-03-12T03:33:27.187943+0000","last_became_peered":"2024-03-12T03:33:27.187943+0000","last_unstale":"2024-03-12T03:35:59.232029+0000","last_undegraded":"2024-03-12T03:35:59.232029+0000","last_fullsized":"2024-03-12T03:35:59.232029+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:05.181454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.181454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:33.321570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230398+0000","last_change":"2024-03-12T03:35:11.909632+0000","last_active":"2024-03-12T03:35:59.230398+0000","last_peered":"2024-03-12T03:35:59.230398+0000","last_clean":"2024-03-12T03:35:59.230398+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:35:59.230398+0000","last_undegraded":"2024-03-12T03:35:59.230398+0000","last_fullsized":"2024-03-12T03:35:59.230398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:11.909537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.909537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:26.882852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232499+0000","last_change":"2024-03-12T03:34:49.175167+0000","last_active":"2024-03-12T03:35:59.232499+0000","last_peered":"2024-03-12T03:35:59.232499+0000","last_clean":"2024-03-12T03:35:59.232499+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:35:59.232499+0000","last_undegraded":"2024-03-12T03:35:59.232499+0000","last_fullsized":"2024-03-12T03:35:59.232499+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.175091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.175091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:59.268389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"86'14","reported_seq":133,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223801+0000","last_change":"2024-03-12T03:34:23.795189+0000","last_active":"2024-03-12T03:35:59.223801+0000","last_peered":"2024-03-12T03:35:59.223801+0000","last_clean":"2024-03-12T03:35:59.223801+0000","last_became_active":"2024-03-12T03:33:50.155908+0000","last_became_peered":"2024-03-12T03:33:50.155908+0000","last_unstale":"2024-03-12T03:35:59.223801+0000","last_undegraded":"2024-03-12T03:35:59.223801+0000","last_fullsized":"2024-03-12T03:35:59.223801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.795040+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223650+0000","last_change":"2024-03-12T03:33:27.380906+0000","last_active":"2024-03-12T03:35:59.223650+0000","last_peered":"2024-03-12T03:35:59.223650+0000","last_clean":"2024-03-12T03:35:59.223650+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:35:59.223650+0000","last_undegraded":"2024-03-12T03:35:59.223650+0000","last_fullsized":"2024-03-12T03:35:59.223650+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.234087+0000","last_change":"2024-03-12T03:34:26.233162+0000","last_active":"2024-03-12T03:35:59.234087+0000","last_peered":"2024-03-12T03:35:59.234087+0000","last_clean":"2024-03-12T03:35:59.234087+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:35:59.234087+0000","last_undegraded":"2024-03-12T03:35:59.234087+0000","last_fullsized":"2024-03-12T03:35:59.234087+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:26.233072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:26.233072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230552+0000","last_change":"2024-03-12T03:34:50.784072+0000","last_active":"2024-03-12T03:35:59.230552+0000","last_peered":"2024-03-12T03:35:59.230552+0000","last_clean":"2024-03-12T03:35:59.230552+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:35:59.230552+0000","last_undegraded":"2024-03-12T03:35:59.230552+0000","last_fullsized":"2024-03-12T03:35:59.230552+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:50.783950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:50.783950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":119,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.237585+0000","last_change":"2024-03-12T03:34:58.889062+0000","last_active":"2024-03-12T03:35:59.237585+0000","last_peered":"2024-03-12T03:35:59.237585+0000","last_clean":"2024-03-12T03:35:59.237585+0000","last_became_active":"2024-03-12T03:33:23.160702+0000","last_became_peered":"2024-03-12T03:33:23.160702+0000","last_unstale":"2024-03-12T03:35:59.237585+0000","last_undegraded":"2024-03-12T03:35:59.237585+0000","last_fullsized":"2024-03-12T03:35:59.237585+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:34:58.888971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.888971+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:28.745266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.224673+0000","last_change":"2024-03-12T03:33:27.381440+0000","last_active":"2024-03-12T03:35:59.224673+0000","last_peered":"2024-03-12T03:35:59.224673+0000","last_clean":"2024-03-12T03:35:59.224673+0000","last_became_active":"2024-03-12T03:33:27.380880+0000","last_became_peered":"2024-03-12T03:33:27.380880+0000","last_unstale":"2024-03-12T03:35:59.224673+0000","last_undegraded":"2024-03-12T03:35:59.224673+0000","last_fullsized":"2024-03-12T03:35:59.224673+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223636+0000","last_change":"2024-03-12T03:34:36.949207+0000","last_active":"2024-03-12T03:35:59.223636+0000","last_peered":"2024-03-12T03:35:59.223636+0000","last_clean":"2024-03-12T03:35:59.223636+0000","last_became_active":"2024-03-12T03:33:25.181087+0000","last_became_peered":"2024-03-12T03:33:25.181087+0000","last_unstale":"2024-03-12T03:35:59.223636+0000","last_undegraded":"2024-03-12T03:35:59.223636+0000","last_fullsized":"2024-03-12T03:35:59.223636+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.949124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.949124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223217+0000","last_change":"2024-03-12T03:34:44.029203+0000","last_active":"2024-03-12T03:35:59.223217+0000","last_peered":"2024-03-12T03:35:59.223217+0000","last_clean":"2024-03-12T03:35:59.223217+0000","last_became_active":"2024-03-12T03:33:21.573218+0000","last_became_peered":"2024-03-12T03:33:21.573218+0000","last_unstale":"2024-03-12T03:35:59.223217+0000","last_undegraded":"2024-03-12T03:35:59.223217+0000","last_fullsized":"2024-03-12T03:35:59.223217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:44.029060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.029060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":121,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.224089+0000","last_change":"2024-03-12T03:33:23.162704+0000","last_active":"2024-03-12T03:35:59.224089+0000","last_peered":"2024-03-12T03:35:59.224089+0000","last_clean":"2024-03-12T03:35:59.224089+0000","last_became_active":"2024-03-12T03:33:23.162427+0000","last_became_peered":"2024-03-12T03:33:23.162427+0000","last_unstale":"2024-03-12T03:35:59.224089+0000","last_undegraded":"2024-03-12T03:35:59.224089+0000","last_fullsized":"2024-03-12T03:35:59.224089+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.231274+0000","last_change":"2024-03-12T03:35:57.051016+0000","last_active":"2024-03-12T03:35:59.231274+0000","last_peered":"2024-03-12T03:35:59.231274+0000","last_clean":"2024-03-12T03:35:59.231274+0000","last_became_active":"2024-03-12T03:33:27.189152+0000","last_became_peered":"2024-03-12T03:33:27.189152+0000","last_unstale":"2024-03-12T03:35:59.231274+0000","last_undegraded":"2024-03-12T03:35:59.231274+0000","last_fullsized":"2024-03-12T03:35:59.231274+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:35:57.050605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:57.050605+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:06.936509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":141,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.063314+0000","last_change":"2024-03-12T03:36:00.063359+0000","last_active":"2024-03-12T03:36:00.063314+0000","last_peered":"2024-03-12T03:36:00.063314+0000","last_clean":"2024-03-12T03:36:00.063314+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:36:00.063314+0000","last_undegraded":"2024-03-12T03:36:00.063314+0000","last_fullsized":"2024-03-12T03:36:00.063314+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:00.063228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:00.063228+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:17.858770+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.233511+0000","last_change":"2024-03-12T03:34:43.324075+0000","last_active":"2024-03-12T03:35:59.233511+0000","last_peered":"2024-03-12T03:35:59.233511+0000","last_clean":"2024-03-12T03:35:59.233511+0000","last_became_active":"2024-03-12T03:33:21.097218+0000","last_became_peered":"2024-03-12T03:33:21.097218+0000","last_unstale":"2024-03-12T03:35:59.233511+0000","last_undegraded":"2024-03-12T03:35:59.233511+0000","last_fullsized":"2024-03-12T03:35:59.233511+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:34:43.323991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.323991+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:05.428737+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'17","reported_seq":122,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.233150+0000","last_change":"2024-03-12T03:35:03.718267+0000","last_active":"2024-03-12T03:35:59.233150+0000","last_peered":"2024-03-12T03:35:59.233150+0000","last_clean":"2024-03-12T03:35:59.233150+0000","last_became_active":"2024-03-12T03:33:23.552812+0000","last_became_peered":"2024-03-12T03:33:23.552812+0000","last_unstale":"2024-03-12T03:35:59.233150+0000","last_undegraded":"2024-03-12T03:35:59.233150+0000","last_fullsized":"2024-03-12T03:35:59.233150+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:35:03.718123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:03.718123+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.467204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.234286+0000","last_change":"2024-03-12T03:34:59.373439+0000","last_active":"2024-03-12T03:35:59.234286+0000","last_peered":"2024-03-12T03:35:59.234286+0000","last_clean":"2024-03-12T03:35:59.234286+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:35:59.234286+0000","last_undegraded":"2024-03-12T03:35:59.234286+0000","last_fullsized":"2024-03-12T03:35:59.234286+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:59.373323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:59.373323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:20.913803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232844+0000","last_change":"2024-03-12T03:34:47.832795+0000","last_active":"2024-03-12T03:35:59.232844+0000","last_peered":"2024-03-12T03:35:59.232844+0000","last_clean":"2024-03-12T03:35:59.232844+0000","last_became_active":"2024-03-12T03:33:25.178346+0000","last_became_peered":"2024-03-12T03:33:25.178346+0000","last_unstale":"2024-03-12T03:35:59.232844+0000","last_undegraded":"2024-03-12T03:35:59.232844+0000","last_fullsized":"2024-03-12T03:35:59.232844+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.832716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.832716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.392333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.224458+0000","last_change":"2024-03-12T03:33:21.095860+0000","last_active":"2024-03-12T03:35:59.224458+0000","last_peered":"2024-03-12T03:35:59.224458+0000","last_clean":"2024-03-12T03:35:59.224458+0000","last_became_active":"2024-03-12T03:33:21.095556+0000","last_became_peered":"2024-03-12T03:33:21.095556+0000","last_unstale":"2024-03-12T03:35:59.224458+0000","last_undegraded":"2024-03-12T03:35:59.224458+0000","last_fullsized":"2024-03-12T03:35:59.224458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"94'23","reported_seq":131,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.236677+0000","last_change":"2024-03-12T03:34:52.730098+0000","last_active":"2024-03-12T03:35:59.236677+0000","last_peered":"2024-03-12T03:35:59.236677+0000","last_clean":"2024-03-12T03:35:59.236677+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:35:59.236677+0000","last_undegraded":"2024-03-12T03:35:59.236677+0000","last_fullsized":"2024-03-12T03:35:59.236677+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'23","last_scrub_stamp":"2024-03-12T03:34:52.729983+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:34:52.729983+0000","last_clean_scrub_stamp":"2024-03-12T03:34:52.729983+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230316+0000","last_change":"2024-03-12T03:35:15.855752+0000","last_active":"2024-03-12T03:35:59.230316+0000","last_peered":"2024-03-12T03:35:59.230316+0000","last_clean":"2024-03-12T03:35:59.230316+0000","last_became_active":"2024-03-12T03:33:27.189696+0000","last_became_peered":"2024-03-12T03:33:27.189696+0000","last_unstale":"2024-03-12T03:35:59.230316+0000","last_undegraded":"2024-03-12T03:35:59.230316+0000","last_fullsized":"2024-03-12T03:35:59.230316+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.855641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.855641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:31.748344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.236148+0000","last_change":"2024-03-12T03:34:36.701692+0000","last_active":"2024-03-12T03:35:59.236148+0000","last_peered":"2024-03-12T03:35:59.236148+0000","last_clean":"2024-03-12T03:35:59.236148+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:35:59.236148+0000","last_undegraded":"2024-03-12T03:35:59.236148+0000","last_fullsized":"2024-03-12T03:35:59.236148+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.701611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.701611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":106,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.235159+0000","last_change":"2024-03-12T03:35:59.076304+0000","last_active":"2024-03-12T03:35:59.235159+0000","last_peered":"2024-03-12T03:35:59.235159+0000","last_clean":"2024-03-12T03:35:59.235159+0000","last_became_active":"2024-03-12T03:33:21.096560+0000","last_became_peered":"2024-03-12T03:33:21.096560+0000","last_unstale":"2024-03-12T03:35:59.235159+0000","last_undegraded":"2024-03-12T03:35:59.235159+0000","last_fullsized":"2024-03-12T03:35:59.235159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.076203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.076203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:18.990073+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"90'6","reported_seq":102,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.224549+0000","last_change":"2024-03-12T03:34:34.274478+0000","last_active":"2024-03-12T03:35:59.224549+0000","last_peered":"2024-03-12T03:35:59.224549+0000","last_clean":"2024-03-12T03:35:59.224549+0000","last_became_active":"2024-03-12T03:33:23.161801+0000","last_became_peered":"2024-03-12T03:33:23.161801+0000","last_unstale":"2024-03-12T03:35:59.224549+0000","last_undegraded":"2024-03-12T03:35:59.224549+0000","last_fullsized":"2024-03-12T03:35:59.224549+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.274353+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229882+0000","last_change":"2024-03-12T03:34:40.191618+0000","last_active":"2024-03-12T03:35:59.229882+0000","last_peered":"2024-03-12T03:35:59.229882+0000","last_clean":"2024-03-12T03:35:59.229882+0000","last_became_active":"2024-03-12T03:33:27.381110+0000","last_became_peered":"2024-03-12T03:33:27.381110+0000","last_unstale":"2024-03-12T03:35:59.229882+0000","last_undegraded":"2024-03-12T03:35:59.229882+0000","last_fullsized":"2024-03-12T03:35:59.229882+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.191538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.480916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230365+0000","last_change":"2024-03-12T03:34:53.878589+0000","last_active":"2024-03-12T03:35:59.230365+0000","last_peered":"2024-03-12T03:35:59.230365+0000","last_clean":"2024-03-12T03:35:59.230365+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:35:59.230365+0000","last_undegraded":"2024-03-12T03:35:59.230365+0000","last_fullsized":"2024-03-12T03:35:59.230365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:53.878484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:53.878484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.602882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":106,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.352359+0000","last_change":"2024-03-12T03:35:59.352393+0000","last_active":"2024-03-12T03:35:59.352359+0000","last_peered":"2024-03-12T03:35:59.352359+0000","last_clean":"2024-03-12T03:35:59.352359+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:35:59.352359+0000","last_undegraded":"2024-03-12T03:35:59.352359+0000","last_fullsized":"2024-03-12T03:35:59.352359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.352272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.352272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:00.463157+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'18","reported_seq":126,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.225248+0000","last_change":"2024-03-12T03:35:02.079317+0000","last_active":"2024-03-12T03:35:59.225248+0000","last_peered":"2024-03-12T03:35:59.225248+0000","last_clean":"2024-03-12T03:35:59.225248+0000","last_became_active":"2024-03-12T03:33:23.552210+0000","last_became_peered":"2024-03-12T03:33:23.552210+0000","last_unstale":"2024-03-12T03:35:59.225248+0000","last_undegraded":"2024-03-12T03:35:59.225248+0000","last_fullsized":"2024-03-12T03:35:59.225248+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:35:02.079224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:02.079224+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:19.194009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229248+0000","last_change":"2024-03-12T03:34:57.781185+0000","last_active":"2024-03-12T03:35:59.229248+0000","last_peered":"2024-03-12T03:35:59.229248+0000","last_clean":"2024-03-12T03:35:59.229248+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:35:59.229248+0000","last_undegraded":"2024-03-12T03:35:59.229248+0000","last_fullsized":"2024-03-12T03:35:59.229248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:57.781083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:57.781083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.508624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.226266+0000","last_change":"2024-03-12T03:34:45.089928+0000","last_active":"2024-03-12T03:35:59.226266+0000","last_peered":"2024-03-12T03:35:59.226266+0000","last_clean":"2024-03-12T03:35:59.226266+0000","last_became_active":"2024-03-12T03:33:25.183725+0000","last_became_peered":"2024-03-12T03:33:25.183725+0000","last_unstale":"2024-03-12T03:35:59.226266+0000","last_undegraded":"2024-03-12T03:35:59.226266+0000","last_fullsized":"2024-03-12T03:35:59.226266+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:45.089762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:45.089762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230647+0000","last_change":"2024-03-12T03:34:41.698397+0000","last_active":"2024-03-12T03:35:59.230647+0000","last_peered":"2024-03-12T03:35:59.230647+0000","last_clean":"2024-03-12T03:35:59.230647+0000","last_became_active":"2024-03-12T03:33:21.107037+0000","last_became_peered":"2024-03-12T03:33:21.107037+0000","last_unstale":"2024-03-12T03:35:59.230647+0000","last_undegraded":"2024-03-12T03:35:59.230647+0000","last_fullsized":"2024-03-12T03:35:59.230647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.698268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.698268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:05.134034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":110,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232588+0000","last_change":"2024-03-12T03:34:58.413300+0000","last_active":"2024-03-12T03:35:59.232588+0000","last_peered":"2024-03-12T03:35:59.232588+0000","last_clean":"2024-03-12T03:35:59.232588+0000","last_became_active":"2024-03-12T03:33:23.160818+0000","last_became_peered":"2024-03-12T03:33:23.160818+0000","last_unstale":"2024-03-12T03:35:59.232588+0000","last_undegraded":"2024-03-12T03:35:59.232588+0000","last_fullsized":"2024-03-12T03:35:59.232588+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-12T03:34:58.413172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.413172+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:35:59.604035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.225016+0000","last_change":"2024-03-12T03:35:59.076156+0000","last_active":"2024-03-12T03:35:59.225016+0000","last_peered":"2024-03-12T03:35:59.225016+0000","last_clean":"2024-03-12T03:35:59.225016+0000","last_became_active":"2024-03-12T03:33:27.191710+0000","last_became_peered":"2024-03-12T03:33:27.191710+0000","last_unstale":"2024-03-12T03:35:59.225016+0000","last_undegraded":"2024-03-12T03:35:59.225016+0000","last_fullsized":"2024-03-12T03:35:59.225016+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.075897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.075897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:18.991490+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232337+0000","last_change":"2024-03-12T03:34:33.258733+0000","last_active":"2024-03-12T03:35:59.232337+0000","last_peered":"2024-03-12T03:35:59.232337+0000","last_clean":"2024-03-12T03:35:59.232337+0000","last_became_active":"2024-03-12T03:33:25.182004+0000","last_became_peered":"2024-03-12T03:33:25.182004+0000","last_unstale":"2024-03-12T03:35:59.232337+0000","last_undegraded":"2024-03-12T03:35:59.232337+0000","last_fullsized":"2024-03-12T03:35:59.232337+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:33.258626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:33.258626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223387+0000","last_change":"2024-03-12T03:34:50.076895+0000","last_active":"2024-03-12T03:35:59.223387+0000","last_peered":"2024-03-12T03:35:59.223387+0000","last_clean":"2024-03-12T03:35:59.223387+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:35:59.223387+0000","last_undegraded":"2024-03-12T03:35:59.223387+0000","last_fullsized":"2024-03-12T03:35:59.223387+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:50.076814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:50.076814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":113,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223860+0000","last_change":"2024-03-12T03:34:34.953658+0000","last_active":"2024-03-12T03:35:59.223860+0000","last_peered":"2024-03-12T03:35:59.223860+0000","last_clean":"2024-03-12T03:35:59.223860+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:35:59.223860+0000","last_undegraded":"2024-03-12T03:35:59.223860+0000","last_fullsized":"2024-03-12T03:35:59.223860+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:34:34.953565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.953565+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.161677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229424+0000","last_change":"2024-03-12T03:34:34.718591+0000","last_active":"2024-03-12T03:35:59.229424+0000","last_peered":"2024-03-12T03:35:59.229424+0000","last_clean":"2024-03-12T03:35:59.229424+0000","last_became_active":"2024-03-12T03:33:27.194094+0000","last_became_peered":"2024-03-12T03:33:27.194094+0000","last_unstale":"2024-03-12T03:35:59.229424+0000","last_undegraded":"2024-03-12T03:35:59.229424+0000","last_fullsized":"2024-03-12T03:35:59.229424+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:34.718509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.718509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":130,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232638+0000","last_change":"2024-03-12T03:34:38.139709+0000","last_active":"2024-03-12T03:35:59.232638+0000","last_peered":"2024-03-12T03:35:59.232638+0000","last_clean":"2024-03-12T03:35:59.232638+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:35:59.232638+0000","last_undegraded":"2024-03-12T03:35:59.232638+0000","last_fullsized":"2024-03-12T03:35:59.232638+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:38.139582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.139582+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":104,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230786+0000","last_change":"2024-03-12T03:34:38.663634+0000","last_active":"2024-03-12T03:35:59.230786+0000","last_peered":"2024-03-12T03:35:59.230786+0000","last_clean":"2024-03-12T03:35:59.230786+0000","last_became_active":"2024-03-12T03:33:21.573244+0000","last_became_peered":"2024-03-12T03:33:21.573244+0000","last_unstale":"2024-03-12T03:35:59.230786+0000","last_undegraded":"2024-03-12T03:35:59.230786+0000","last_fullsized":"2024-03-12T03:35:59.230786+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:34:38.663542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.663542+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":133,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.225751+0000","last_change":"2024-03-12T03:35:58.079779+0000","last_active":"2024-03-12T03:35:59.225751+0000","last_peered":"2024-03-12T03:35:59.225751+0000","last_clean":"2024-03-12T03:35:59.225751+0000","last_became_active":"2024-03-12T03:33:23.551706+0000","last_became_peered":"2024-03-12T03:33:23.551706+0000","last_unstale":"2024-03-12T03:35:59.225751+0000","last_undegraded":"2024-03-12T03:35:59.225751+0000","last_fullsized":"2024-03-12T03:35:59.225751+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-03-12T03:35:58.079645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.079645+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.444037+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223814+0000","last_change":"2024-03-12T03:35:12.124255+0000","last_active":"2024-03-12T03:35:59.223814+0000","last_peered":"2024-03-12T03:35:59.223814+0000","last_clean":"2024-03-12T03:35:59.223814+0000","last_became_active":"2024-03-12T03:33:27.184916+0000","last_became_peered":"2024-03-12T03:33:27.184916+0000","last_unstale":"2024-03-12T03:35:59.223814+0000","last_undegraded":"2024-03-12T03:35:59.223814+0000","last_fullsized":"2024-03-12T03:35:59.223814+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:12.124167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:12.124167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:21.269917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":133,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.225919+0000","last_change":"2024-03-12T03:35:08.006530+0000","last_active":"2024-03-12T03:35:59.225919+0000","last_peered":"2024-03-12T03:35:59.225919+0000","last_clean":"2024-03-12T03:35:59.225919+0000","last_became_active":"2024-03-12T03:33:25.179158+0000","last_became_peered":"2024-03-12T03:33:25.179158+0000","last_unstale":"2024-03-12T03:35:59.225919+0000","last_undegraded":"2024-03-12T03:35:59.225919+0000","last_fullsized":"2024-03-12T03:35:59.225919+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:35:08.006375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:08.006375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:29.223748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229056+0000","last_change":"2024-03-12T03:34:40.753233+0000","last_active":"2024-03-12T03:35:59.229056+0000","last_peered":"2024-03-12T03:35:59.229056+0000","last_clean":"2024-03-12T03:35:59.229056+0000","last_became_active":"2024-03-12T03:33:21.101819+0000","last_became_peered":"2024-03-12T03:33:21.101819+0000","last_unstale":"2024-03-12T03:35:59.229056+0000","last_undegraded":"2024-03-12T03:35:59.229056+0000","last_fullsized":"2024-03-12T03:35:59.229056+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.753153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.816911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":122,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232764+0000","last_change":"2024-03-12T03:34:29.083183+0000","last_active":"2024-03-12T03:35:59.232764+0000","last_peered":"2024-03-12T03:35:59.232764+0000","last_clean":"2024-03-12T03:35:59.232764+0000","last_became_active":"2024-03-12T03:33:23.553480+0000","last_became_peered":"2024-03-12T03:33:23.553480+0000","last_unstale":"2024-03-12T03:35:59.232764+0000","last_undegraded":"2024-03-12T03:35:59.232764+0000","last_fullsized":"2024-03-12T03:35:59.232764+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:34:29.083063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:29.083063+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.237882+0000","last_change":"2024-03-12T03:34:43.039900+0000","last_active":"2024-03-12T03:35:59.237882+0000","last_peered":"2024-03-12T03:35:59.237882+0000","last_clean":"2024-03-12T03:35:59.237882+0000","last_became_active":"2024-03-12T03:33:27.187657+0000","last_became_peered":"2024-03-12T03:33:27.187657+0000","last_unstale":"2024-03-12T03:35:59.237882+0000","last_undegraded":"2024-03-12T03:35:59.237882+0000","last_fullsized":"2024-03-12T03:35:59.237882+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:43.039799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.039799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:01.927684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.224515+0000","last_change":"2024-03-12T03:35:04.007338+0000","last_active":"2024-03-12T03:35:59.224515+0000","last_peered":"2024-03-12T03:35:59.224515+0000","last_clean":"2024-03-12T03:35:59.224515+0000","last_became_active":"2024-03-12T03:33:25.174953+0000","last_became_peered":"2024-03-12T03:33:25.174953+0000","last_unstale":"2024-03-12T03:35:59.224515+0000","last_undegraded":"2024-03-12T03:35:59.224515+0000","last_fullsized":"2024-03-12T03:35:59.224515+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:35:04.007252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:31.186652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229986+0000","last_change":"2024-03-12T03:34:52.848468+0000","last_active":"2024-03-12T03:35:59.229986+0000","last_peered":"2024-03-12T03:35:59.229986+0000","last_clean":"2024-03-12T03:35:59.229986+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:35:59.229986+0000","last_undegraded":"2024-03-12T03:35:59.229986+0000","last_fullsized":"2024-03-12T03:35:59.229986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:52.848366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:52.848366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.609011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":134,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232682+0000","last_change":"2024-03-12T03:35:11.786139+0000","last_active":"2024-03-12T03:35:59.232682+0000","last_peered":"2024-03-12T03:35:59.232682+0000","last_clean":"2024-03-12T03:35:59.232682+0000","last_became_active":"2024-03-12T03:33:23.552164+0000","last_became_peered":"2024-03-12T03:33:23.552164+0000","last_unstale":"2024-03-12T03:35:59.232682+0000","last_undegraded":"2024-03-12T03:35:59.232682+0000","last_fullsized":"2024-03-12T03:35:59.232682+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:35:11.786040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.786040+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:24.463019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230564+0000","last_change":"2024-03-12T03:34:46.767546+0000","last_active":"2024-03-12T03:35:59.230564+0000","last_peered":"2024-03-12T03:35:59.230564+0000","last_clean":"2024-03-12T03:35:59.230564+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:35:59.230564+0000","last_undegraded":"2024-03-12T03:35:59.230564+0000","last_fullsized":"2024-03-12T03:35:59.230564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:46.767453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:46.767453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232916+0000","last_change":"2024-03-12T03:35:08.194100+0000","last_active":"2024-03-12T03:35:59.232916+0000","last_peered":"2024-03-12T03:35:59.232916+0000","last_clean":"2024-03-12T03:35:59.232916+0000","last_became_active":"2024-03-12T03:33:25.180752+0000","last_became_peered":"2024-03-12T03:33:25.180752+0000","last_unstale":"2024-03-12T03:35:59.232916+0000","last_undegraded":"2024-03-12T03:35:59.232916+0000","last_fullsized":"2024-03-12T03:35:59.232916+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:08.193996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:08.193996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:20.553560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":106,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.235791+0000","last_change":"2024-03-12T03:35:58.075845+0000","last_active":"2024-03-12T03:35:59.235791+0000","last_peered":"2024-03-12T03:35:59.235791+0000","last_clean":"2024-03-12T03:35:59.235791+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:35:59.235791+0000","last_undegraded":"2024-03-12T03:35:59.235791+0000","last_fullsized":"2024-03-12T03:35:59.235791+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:58.075754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.075754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:04.874641+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":115,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.224766+0000","last_change":"2024-03-12T03:35:00.079249+0000","last_active":"2024-03-12T03:35:59.224766+0000","last_peered":"2024-03-12T03:35:59.224766+0000","last_clean":"2024-03-12T03:35:59.224766+0000","last_became_active":"2024-03-12T03:33:23.551773+0000","last_became_peered":"2024-03-12T03:33:23.551773+0000","last_unstale":"2024-03-12T03:35:59.224766+0000","last_undegraded":"2024-03-12T03:35:59.224766+0000","last_fullsized":"2024-03-12T03:35:59.224766+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:35:00.079152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.079152+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:21.325882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"106'3380","reported_seq":3408,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.781627+0000","last_change":"2024-03-12T03:35:23.782770+0000","last_active":"2024-03-12T03:36:00.781627+0000","last_peered":"2024-03-12T03:36:00.781627+0000","last_clean":"2024-03-12T03:36:00.781627+0000","last_became_active":"2024-03-12T03:35:23.782112+0000","last_became_peered":"2024-03-12T03:35:23.782112+0000","last_unstale":"2024-03-12T03:36:00.781627+0000","last_undegraded":"2024-03-12T03:36:00.781627+0000","last_fullsized":"2024-03-12T03:36:00.781627+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3380,"log_dups_size":0,"ondisk_log_size":3380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:41.457014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221511680,"num_objects":3380,"num_object_clones":0,"num_object_copies":10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3380,"num_write_kb":216320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232160+0000","last_change":"2024-03-12T03:35:00.250874+0000","last_active":"2024-03-12T03:35:59.232160+0000","last_peered":"2024-03-12T03:35:59.232160+0000","last_clean":"2024-03-12T03:35:59.232160+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:35:59.232160+0000","last_undegraded":"2024-03-12T03:35:59.232160+0000","last_fullsized":"2024-03-12T03:35:59.232160+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:00.250780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.250780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:05.004387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":110,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.231297+0000","last_change":"2024-03-12T03:35:12.907083+0000","last_active":"2024-03-12T03:35:59.231297+0000","last_peered":"2024-03-12T03:35:59.231297+0000","last_clean":"2024-03-12T03:35:59.231297+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:35:59.231297+0000","last_undegraded":"2024-03-12T03:35:59.231297+0000","last_fullsized":"2024-03-12T03:35:59.231297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:35:12.906988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:12.906988+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:35.363926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"90'36","reported_seq":152,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232105+0000","last_change":"2024-03-12T03:34:39.184089+0000","last_active":"2024-03-12T03:35:59.232105+0000","last_peered":"2024-03-12T03:35:59.232105+0000","last_clean":"2024-03-12T03:35:59.232105+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:35:59.232105+0000","last_undegraded":"2024-03-12T03:35:59.232105+0000","last_fullsized":"2024-03-12T03:35:59.232105+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.183984+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":130,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230177+0000","last_change":"2024-03-12T03:34:39.659403+0000","last_active":"2024-03-12T03:35:59.230177+0000","last_peered":"2024-03-12T03:35:59.230177+0000","last_clean":"2024-03-12T03:35:59.230177+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:35:59.230177+0000","last_undegraded":"2024-03-12T03:35:59.230177+0000","last_fullsized":"2024-03-12T03:35:59.230177+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:39.659277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.659277+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:08.903088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"106'3524","reported_seq":3552,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.844293+0000","last_change":"2024-03-12T03:35:23.777596+0000","last_active":"2024-03-12T03:36:00.844293+0000","last_peered":"2024-03-12T03:36:00.844293+0000","last_clean":"2024-03-12T03:36:00.844293+0000","last_became_active":"2024-03-12T03:35:23.777272+0000","last_became_peered":"2024-03-12T03:35:23.777272+0000","last_unstale":"2024-03-12T03:36:00.844293+0000","last_undegraded":"2024-03-12T03:36:00.844293+0000","last_fullsized":"2024-03-12T03:36:00.844293+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:26.791230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230948864,"num_objects":3524,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3524,"num_write_kb":225536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.226426+0000","last_change":"2024-03-12T03:33:27.191091+0000","last_active":"2024-03-12T03:35:59.226426+0000","last_peered":"2024-03-12T03:35:59.226426+0000","last_clean":"2024-03-12T03:35:59.226426+0000","last_became_active":"2024-03-12T03:33:27.190753+0000","last_became_peered":"2024-03-12T03:33:27.190753+0000","last_unstale":"2024-03-12T03:35:59.226426+0000","last_undegraded":"2024-03-12T03:35:59.226426+0000","last_fullsized":"2024-03-12T03:35:59.226426+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230751+0000","last_change":"2024-03-12T03:34:23.842758+0000","last_active":"2024-03-12T03:35:59.230751+0000","last_peered":"2024-03-12T03:35:59.230751+0000","last_clean":"2024-03-12T03:35:59.230751+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:35:59.230751+0000","last_undegraded":"2024-03-12T03:35:59.230751+0000","last_fullsized":"2024-03-12T03:35:59.230751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:23.842628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:23.842628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":133,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223342+0000","last_change":"2024-03-12T03:33:23.163665+0000","last_active":"2024-03-12T03:35:59.223342+0000","last_peered":"2024-03-12T03:35:59.223342+0000","last_clean":"2024-03-12T03:35:59.223342+0000","last_became_active":"2024-03-12T03:33:23.163351+0000","last_became_peered":"2024-03-12T03:33:23.163351+0000","last_unstale":"2024-03-12T03:35:59.223342+0000","last_undegraded":"2024-03-12T03:35:59.223342+0000","last_fullsized":"2024-03-12T03:35:59.223342+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.236507+0000","last_change":"2024-03-12T03:34:51.958873+0000","last_active":"2024-03-12T03:35:59.236507+0000","last_peered":"2024-03-12T03:35:59.236507+0000","last_clean":"2024-03-12T03:35:59.236507+0000","last_became_active":"2024-03-12T03:33:25.185596+0000","last_became_peered":"2024-03-12T03:33:25.185596+0000","last_unstale":"2024-03-12T03:35:59.236507+0000","last_undegraded":"2024-03-12T03:35:59.236507+0000","last_fullsized":"2024-03-12T03:35:59.236507+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:51.958777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:51.958777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:20.123226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"106'3477","reported_seq":3505,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.766822+0000","last_change":"2024-03-12T03:35:23.778943+0000","last_active":"2024-03-12T03:36:00.766822+0000","last_peered":"2024-03-12T03:36:00.766822+0000","last_clean":"2024-03-12T03:36:00.766822+0000","last_became_active":"2024-03-12T03:35:23.778606+0000","last_became_peered":"2024-03-12T03:35:23.778606+0000","last_unstale":"2024-03-12T03:36:00.766822+0000","last_undegraded":"2024-03-12T03:36:00.766822+0000","last_fullsized":"2024-03-12T03:36:00.766822+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3477,"log_dups_size":0,"ondisk_log_size":3477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:50.273914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227868672,"num_objects":3477,"num_object_clones":0,"num_object_copies":10431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3477,"num_write_kb":222528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229924+0000","last_change":"2024-03-12T03:34:43.716123+0000","last_active":"2024-03-12T03:35:59.229924+0000","last_peered":"2024-03-12T03:35:59.229924+0000","last_clean":"2024-03-12T03:35:59.229924+0000","last_became_active":"2024-03-12T03:33:27.187955+0000","last_became_peered":"2024-03-12T03:33:27.187955+0000","last_unstale":"2024-03-12T03:35:59.229924+0000","last_undegraded":"2024-03-12T03:35:59.229924+0000","last_fullsized":"2024-03-12T03:35:59.229924+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:43.716021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.716021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:01.645227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.238585+0000","last_change":"2024-03-12T03:35:11.095883+0000","last_active":"2024-03-12T03:35:59.238585+0000","last_peered":"2024-03-12T03:35:59.238585+0000","last_clean":"2024-03-12T03:35:59.238585+0000","last_became_active":"2024-03-12T03:33:25.179593+0000","last_became_peered":"2024-03-12T03:33:25.179593+0000","last_unstale":"2024-03-12T03:35:59.238585+0000","last_undegraded":"2024-03-12T03:35:59.238585+0000","last_fullsized":"2024-03-12T03:35:59.238585+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:11.095766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.095766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:12.302418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.231034+0000","last_change":"2024-03-12T03:34:30.792781+0000","last_active":"2024-03-12T03:35:59.231034+0000","last_peered":"2024-03-12T03:35:59.231034+0000","last_clean":"2024-03-12T03:35:59.231034+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:35:59.231034+0000","last_undegraded":"2024-03-12T03:35:59.231034+0000","last_fullsized":"2024-03-12T03:35:59.231034+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:30.792702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.792702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":137,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.236134+0000","last_change":"2024-03-12T03:34:46.722408+0000","last_active":"2024-03-12T03:35:59.236134+0000","last_peered":"2024-03-12T03:35:59.236134+0000","last_clean":"2024-03-12T03:35:59.236134+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:35:59.236134+0000","last_undegraded":"2024-03-12T03:35:59.236134+0000","last_fullsized":"2024-03-12T03:35:59.236134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:34:46.722278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:46.722278+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.145172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"106'3483","reported_seq":3511,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.792452+0000","last_change":"2024-03-12T03:35:23.785560+0000","last_active":"2024-03-12T03:36:00.792452+0000","last_peered":"2024-03-12T03:36:00.792452+0000","last_clean":"2024-03-12T03:36:00.792452+0000","last_became_active":"2024-03-12T03:35:23.784701+0000","last_became_peered":"2024-03-12T03:35:23.784701+0000","last_unstale":"2024-03-12T03:36:00.792452+0000","last_undegraded":"2024-03-12T03:36:00.792452+0000","last_fullsized":"2024-03-12T03:36:00.792452+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:26.081129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228261888,"num_objects":3483,"num_object_clones":0,"num_object_copies":10449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":222912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229010+0000","last_change":"2024-03-12T03:34:37.737471+0000","last_active":"2024-03-12T03:35:59.229010+0000","last_peered":"2024-03-12T03:35:59.229010+0000","last_clean":"2024-03-12T03:35:59.229010+0000","last_became_active":"2024-03-12T03:33:27.189948+0000","last_became_peered":"2024-03-12T03:33:27.189948+0000","last_unstale":"2024-03-12T03:35:59.229010+0000","last_undegraded":"2024-03-12T03:35:59.229010+0000","last_fullsized":"2024-03-12T03:35:59.229010+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:37.737386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:37.737386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.387230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.238747+0000","last_change":"2024-03-12T03:35:04.986995+0000","last_active":"2024-03-12T03:35:59.238747+0000","last_peered":"2024-03-12T03:35:59.238747+0000","last_clean":"2024-03-12T03:35:59.238747+0000","last_became_active":"2024-03-12T03:33:25.184289+0000","last_became_peered":"2024-03-12T03:33:25.184289+0000","last_unstale":"2024-03-12T03:35:59.238747+0000","last_undegraded":"2024-03-12T03:35:59.238747+0000","last_fullsized":"2024-03-12T03:35:59.238747+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:04.986909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:04.986909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:09.694715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232398+0000","last_change":"2024-03-12T03:34:48.160481+0000","last_active":"2024-03-12T03:35:59.232398+0000","last_peered":"2024-03-12T03:35:59.232398+0000","last_clean":"2024-03-12T03:35:59.232398+0000","last_became_active":"2024-03-12T03:33:21.102410+0000","last_became_peered":"2024-03-12T03:33:21.102410+0000","last_unstale":"2024-03-12T03:35:59.232398+0000","last_undegraded":"2024-03-12T03:35:59.232398+0000","last_fullsized":"2024-03-12T03:35:59.232398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:34:48.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:17.839681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":122,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230369+0000","last_change":"2024-03-12T03:35:07.777251+0000","last_active":"2024-03-12T03:35:59.230369+0000","last_peered":"2024-03-12T03:35:59.230369+0000","last_clean":"2024-03-12T03:35:59.230369+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:35:59.230369+0000","last_undegraded":"2024-03-12T03:35:59.230369+0000","last_fullsized":"2024-03-12T03:35:59.230369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:35:07.777146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:07.777146+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:17.920452+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"106'3490","reported_seq":3518,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.767326+0000","last_change":"2024-03-12T03:35:23.784948+0000","last_active":"2024-03-12T03:36:00.767326+0000","last_peered":"2024-03-12T03:36:00.767326+0000","last_clean":"2024-03-12T03:36:00.767326+0000","last_became_active":"2024-03-12T03:35:23.784628+0000","last_became_peered":"2024-03-12T03:35:23.784628+0000","last_unstale":"2024-03-12T03:36:00.767326+0000","last_undegraded":"2024-03-12T03:36:00.767326+0000","last_fullsized":"2024-03-12T03:36:00.767326+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3490,"log_dups_size":0,"ondisk_log_size":3490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:33.288176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228720640,"num_objects":3490,"num_object_clones":0,"num_object_copies":10470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3490,"num_write_kb":223360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230966+0000","last_change":"2024-03-12T03:34:49.088183+0000","last_active":"2024-03-12T03:35:59.230966+0000","last_peered":"2024-03-12T03:35:59.230966+0000","last_clean":"2024-03-12T03:35:59.230966+0000","last_became_active":"2024-03-12T03:33:27.185041+0000","last_became_peered":"2024-03-12T03:33:27.185041+0000","last_unstale":"2024-03-12T03:35:59.230966+0000","last_undegraded":"2024-03-12T03:35:59.230966+0000","last_fullsized":"2024-03-12T03:35:59.230966+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.088098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.088098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.622130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":109,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232229+0000","last_change":"2024-03-12T03:34:22.168794+0000","last_active":"2024-03-12T03:35:59.232229+0000","last_peered":"2024-03-12T03:35:59.232229+0000","last_clean":"2024-03-12T03:35:59.232229+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:35:59.232229+0000","last_undegraded":"2024-03-12T03:35:59.232229+0000","last_fullsized":"2024-03-12T03:35:59.232229+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:34:22.168654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:22.168654+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223570+0000","last_change":"2024-03-12T03:34:59.105754+0000","last_active":"2024-03-12T03:35:59.223570+0000","last_peered":"2024-03-12T03:35:59.223570+0000","last_clean":"2024-03-12T03:35:59.223570+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:35:59.223570+0000","last_undegraded":"2024-03-12T03:35:59.223570+0000","last_fullsized":"2024-03-12T03:35:59.223570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:59.105634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:59.105634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:02.625520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'35","reported_seq":125,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232239+0000","last_change":"2024-03-12T03:34:54.301841+0000","last_active":"2024-03-12T03:35:59.232239+0000","last_peered":"2024-03-12T03:35:59.232239+0000","last_clean":"2024-03-12T03:35:59.232239+0000","last_became_active":"2024-03-12T03:33:50.151664+0000","last_became_peered":"2024-03-12T03:33:50.151664+0000","last_unstale":"2024-03-12T03:35:59.232239+0000","last_undegraded":"2024-03-12T03:35:59.232239+0000","last_fullsized":"2024-03-12T03:35:59.232239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'35","last_scrub_stamp":"2024-03-12T03:34:54.301725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:54.301725+0000","objects_scrubbed":11,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:23.962972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.231588+0000","last_change":"2024-03-12T03:34:38.698005+0000","last_active":"2024-03-12T03:35:59.231588+0000","last_peered":"2024-03-12T03:35:59.231588+0000","last_clean":"2024-03-12T03:35:59.231588+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:35:59.231588+0000","last_undegraded":"2024-03-12T03:35:59.231588+0000","last_fullsized":"2024-03-12T03:35:59.231588+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:38.697923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:38.697923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:07.086680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"106'3517","reported_seq":3544,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.746036+0000","last_change":"2024-03-12T03:35:23.776994+0000","last_active":"2024-03-12T03:36:00.746036+0000","last_peered":"2024-03-12T03:36:00.746036+0000","last_clean":"2024-03-12T03:36:00.746036+0000","last_became_active":"2024-03-12T03:35:23.776662+0000","last_became_peered":"2024-03-12T03:35:23.776662+0000","last_unstale":"2024-03-12T03:36:00.746036+0000","last_undegraded":"2024-03-12T03:36:00.746036+0000","last_fullsized":"2024-03-12T03:36:00.746036+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:37.352608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230490112,"num_objects":3517,"num_object_clones":0,"num_object_copies":10551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3517,"num_write_kb":225088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.231128+0000","last_change":"2024-03-12T03:34:40.663082+0000","last_active":"2024-03-12T03:35:59.231128+0000","last_peered":"2024-03-12T03:35:59.231128+0000","last_clean":"2024-03-12T03:35:59.231128+0000","last_became_active":"2024-03-12T03:33:27.380389+0000","last_became_peered":"2024-03-12T03:33:27.380389+0000","last_unstale":"2024-03-12T03:35:59.231128+0000","last_undegraded":"2024-03-12T03:35:59.231128+0000","last_fullsized":"2024-03-12T03:35:59.231128+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.663002+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.236722+0000","last_change":"2024-03-12T03:34:51.444303+0000","last_active":"2024-03-12T03:35:59.236722+0000","last_peered":"2024-03-12T03:35:59.236722+0000","last_clean":"2024-03-12T03:35:59.236722+0000","last_became_active":"2024-03-12T03:33:21.572911+0000","last_became_peered":"2024-03-12T03:33:21.572911+0000","last_unstale":"2024-03-12T03:35:59.236722+0000","last_undegraded":"2024-03-12T03:35:59.236722+0000","last_fullsized":"2024-03-12T03:35:59.236722+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:51.444181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:51.444181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.657671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":107,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.236279+0000","last_change":"2024-03-12T03:34:49.918898+0000","last_active":"2024-03-12T03:35:59.236279+0000","last_peered":"2024-03-12T03:35:59.236279+0000","last_clean":"2024-03-12T03:35:59.236279+0000","last_became_active":"2024-03-12T03:33:20.093064+0000","last_became_peered":"2024-03-12T03:33:20.093064+0000","last_unstale":"2024-03-12T03:35:59.236279+0000","last_undegraded":"2024-03-12T03:35:59.236279+0000","last_fullsized":"2024-03-12T03:35:59.236279+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.918802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.918802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.157514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000516994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":133,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232262+0000","last_change":"2024-03-12T03:34:44.301692+0000","last_active":"2024-03-12T03:35:59.232262+0000","last_peered":"2024-03-12T03:35:59.232262+0000","last_clean":"2024-03-12T03:35:59.232262+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:35:59.232262+0000","last_undegraded":"2024-03-12T03:35:59.232262+0000","last_fullsized":"2024-03-12T03:35:59.232262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-12T03:34:44.301556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.301556+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.466292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.237775+0000","last_change":"2024-03-12T03:35:58.091300+0000","last_active":"2024-03-12T03:35:59.237775+0000","last_peered":"2024-03-12T03:35:59.237775+0000","last_clean":"2024-03-12T03:35:59.237775+0000","last_became_active":"2024-03-12T03:33:25.179022+0000","last_became_peered":"2024-03-12T03:33:25.179022+0000","last_unstale":"2024-03-12T03:35:59.237775+0000","last_undegraded":"2024-03-12T03:35:59.237775+0000","last_fullsized":"2024-03-12T03:35:59.237775+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:58.091182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.091182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:04.769136+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"106'3450","reported_seq":3478,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.771552+0000","last_change":"2024-03-12T03:35:23.785163+0000","last_active":"2024-03-12T03:36:00.771552+0000","last_peered":"2024-03-12T03:36:00.771552+0000","last_clean":"2024-03-12T03:36:00.771552+0000","last_became_active":"2024-03-12T03:35:23.784880+0000","last_became_peered":"2024-03-12T03:35:23.784880+0000","last_unstale":"2024-03-12T03:36:00.771552+0000","last_undegraded":"2024-03-12T03:36:00.771552+0000","last_fullsized":"2024-03-12T03:36:00.771552+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3450,"log_dups_size":0,"ondisk_log_size":3450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:41.806437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226099200,"num_objects":3450,"num_object_clones":0,"num_object_copies":10350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3450,"num_write_kb":220800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232629+0000","last_change":"2024-03-12T03:34:58.277475+0000","last_active":"2024-03-12T03:35:59.232629+0000","last_peered":"2024-03-12T03:35:59.232629+0000","last_clean":"2024-03-12T03:35:59.232629+0000","last_became_active":"2024-03-12T03:33:27.188551+0000","last_became_peered":"2024-03-12T03:33:27.188551+0000","last_unstale":"2024-03-12T03:35:59.232629+0000","last_undegraded":"2024-03-12T03:35:59.232629+0000","last_fullsized":"2024-03-12T03:35:59.232629+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:58.277378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.277378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:16.176471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":353,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230127+0000","last_change":"2024-03-12T03:34:58.903289+0000","last_active":"2024-03-12T03:35:59.230127+0000","last_peered":"2024-03-12T03:35:59.230127+0000","last_clean":"2024-03-12T03:35:59.230127+0000","last_became_active":"2024-03-12T03:33:11.319888+0000","last_became_peered":"2024-03-12T03:33:11.319888+0000","last_unstale":"2024-03-12T03:35:59.230127+0000","last_undegraded":"2024-03-12T03:35:59.230127+0000","last_fullsized":"2024-03-12T03:35:59.230127+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:34:58.903163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.903163+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:07.967565+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230264+0000","last_change":"2024-03-12T03:35:14.996451+0000","last_active":"2024-03-12T03:35:59.230264+0000","last_peered":"2024-03-12T03:35:59.230264+0000","last_clean":"2024-03-12T03:35:59.230264+0000","last_became_active":"2024-03-12T03:33:21.100900+0000","last_became_peered":"2024-03-12T03:33:21.100900+0000","last_unstale":"2024-03-12T03:35:59.230264+0000","last_undegraded":"2024-03-12T03:35:59.230264+0000","last_fullsized":"2024-03-12T03:35:59.230264+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:14.996357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:14.996357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:35.315899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":148,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.236521+0000","last_change":"2024-03-12T03:34:35.664996+0000","last_active":"2024-03-12T03:35:59.236521+0000","last_peered":"2024-03-12T03:35:59.236521+0000","last_clean":"2024-03-12T03:35:59.236521+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:35:59.236521+0000","last_undegraded":"2024-03-12T03:35:59.236521+0000","last_fullsized":"2024-03-12T03:35:59.236521+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:34:35.664876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:35.664876+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.231295+0000","last_change":"2024-03-12T03:34:47.796670+0000","last_active":"2024-03-12T03:35:59.231295+0000","last_peered":"2024-03-12T03:35:59.231295+0000","last_clean":"2024-03-12T03:35:59.231295+0000","last_became_active":"2024-03-12T03:33:25.177762+0000","last_became_peered":"2024-03-12T03:33:25.177762+0000","last_unstale":"2024-03-12T03:35:59.231295+0000","last_undegraded":"2024-03-12T03:35:59.231295+0000","last_fullsized":"2024-03-12T03:35:59.231295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.796561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.796561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.258919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"106'3473","reported_seq":3501,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.773320+0000","last_change":"2024-03-12T03:35:23.786371+0000","last_active":"2024-03-12T03:36:00.773320+0000","last_peered":"2024-03-12T03:36:00.773320+0000","last_clean":"2024-03-12T03:36:00.773320+0000","last_became_active":"2024-03-12T03:35:23.786059+0000","last_became_peered":"2024-03-12T03:35:23.786059+0000","last_unstale":"2024-03-12T03:36:00.773320+0000","last_undegraded":"2024-03-12T03:36:00.773320+0000","last_fullsized":"2024-03-12T03:36:00.773320+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3473,"log_dups_size":0,"ondisk_log_size":3473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:44.824542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227606528,"num_objects":3473,"num_object_clones":0,"num_object_copies":10419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3473,"num_write_kb":222272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.234374+0000","last_change":"2024-03-12T03:34:47.352525+0000","last_active":"2024-03-12T03:35:59.234374+0000","last_peered":"2024-03-12T03:35:59.234374+0000","last_clean":"2024-03-12T03:35:59.234374+0000","last_became_active":"2024-03-12T03:33:27.191878+0000","last_became_peered":"2024-03-12T03:33:27.191878+0000","last_unstale":"2024-03-12T03:35:59.234374+0000","last_undegraded":"2024-03-12T03:35:59.234374+0000","last_fullsized":"2024-03-12T03:35:59.234374+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.352450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.352450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:01.666869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229252+0000","last_change":"2024-03-12T03:35:05.715931+0000","last_active":"2024-03-12T03:35:59.229252+0000","last_peered":"2024-03-12T03:35:59.229252+0000","last_clean":"2024-03-12T03:35:59.229252+0000","last_became_active":"2024-03-12T03:33:21.097086+0000","last_became_peered":"2024-03-12T03:33:21.097086+0000","last_unstale":"2024-03-12T03:35:59.229252+0000","last_undegraded":"2024-03-12T03:35:59.229252+0000","last_fullsized":"2024-03-12T03:35:59.229252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:05.715849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.715849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:19.094010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":264,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.345345+0000","last_change":"2024-03-12T03:34:21.799658+0000","last_active":"2024-03-12T03:35:59.345345+0000","last_peered":"2024-03-12T03:35:59.345345+0000","last_clean":"2024-03-12T03:35:59.345345+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:35:59.345345+0000","last_undegraded":"2024-03-12T03:35:59.345345+0000","last_fullsized":"2024-03-12T03:35:59.345345+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:34:21.799534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:34:21.799534+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":154,"num_read_kb":149,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":112,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229518+0000","last_change":"2024-03-12T03:34:39.707369+0000","last_active":"2024-03-12T03:35:59.229518+0000","last_peered":"2024-03-12T03:35:59.229518+0000","last_clean":"2024-03-12T03:35:59.229518+0000","last_became_active":"2024-03-12T03:33:23.168750+0000","last_became_peered":"2024-03-12T03:33:23.168750+0000","last_unstale":"2024-03-12T03:35:59.229518+0000","last_undegraded":"2024-03-12T03:35:59.229518+0000","last_fullsized":"2024-03-12T03:35:59.229518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:34:39.707264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.707264+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232852+0000","last_change":"2024-03-12T03:35:01.257832+0000","last_active":"2024-03-12T03:35:59.232852+0000","last_peered":"2024-03-12T03:35:59.232852+0000","last_clean":"2024-03-12T03:35:59.232852+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:35:59.232852+0000","last_undegraded":"2024-03-12T03:35:59.232852+0000","last_fullsized":"2024-03-12T03:35:59.232852+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:35:01.257725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:12.838678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"106'3453","reported_seq":3481,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.783368+0000","last_change":"2024-03-12T03:35:23.778292+0000","last_active":"2024-03-12T03:36:00.783368+0000","last_peered":"2024-03-12T03:36:00.783368+0000","last_clean":"2024-03-12T03:36:00.783368+0000","last_became_active":"2024-03-12T03:35:23.777945+0000","last_became_peered":"2024-03-12T03:35:23.777945+0000","last_unstale":"2024-03-12T03:36:00.783368+0000","last_undegraded":"2024-03-12T03:36:00.783368+0000","last_fullsized":"2024-03-12T03:36:00.783368+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3453,"log_dups_size":0,"ondisk_log_size":3453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:44.975408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226295808,"num_objects":3453,"num_object_clones":0,"num_object_copies":10359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3453,"num_write_kb":220992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.233353+0000","last_change":"2024-03-12T03:34:45.344064+0000","last_active":"2024-03-12T03:35:59.233353+0000","last_peered":"2024-03-12T03:35:59.233353+0000","last_clean":"2024-03-12T03:35:59.233353+0000","last_became_active":"2024-03-12T03:33:27.380458+0000","last_became_peered":"2024-03-12T03:33:27.380458+0000","last_unstale":"2024-03-12T03:35:59.233353+0000","last_undegraded":"2024-03-12T03:35:59.233353+0000","last_fullsized":"2024-03-12T03:35:59.233353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:45.343945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:45.343945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:10.292957+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.234643+0000","last_change":"2024-03-12T03:34:34.272895+0000","last_active":"2024-03-12T03:35:59.234643+0000","last_peered":"2024-03-12T03:35:59.234643+0000","last_clean":"2024-03-12T03:35:59.234643+0000","last_became_active":"2024-03-12T03:33:21.573003+0000","last_became_peered":"2024-03-12T03:33:21.573003+0000","last_unstale":"2024-03-12T03:35:59.234643+0000","last_undegraded":"2024-03-12T03:35:59.234643+0000","last_fullsized":"2024-03-12T03:35:59.234643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:34.272759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.272759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":142,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230630+0000","last_change":"2024-03-12T03:35:06.754200+0000","last_active":"2024-03-12T03:35:59.230630+0000","last_peered":"2024-03-12T03:35:59.230630+0000","last_clean":"2024-03-12T03:35:59.230630+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:35:59.230630+0000","last_undegraded":"2024-03-12T03:35:59.230630+0000","last_fullsized":"2024-03-12T03:35:59.230630+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:35:06.754088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:06.754088+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:12.920218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.236172+0000","last_change":"2024-03-12T03:34:28.572496+0000","last_active":"2024-03-12T03:35:59.236172+0000","last_peered":"2024-03-12T03:35:59.236172+0000","last_clean":"2024-03-12T03:35:59.236172+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:35:59.236172+0000","last_undegraded":"2024-03-12T03:35:59.236172+0000","last_fullsized":"2024-03-12T03:35:59.236172+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:28.572379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:28.572379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"106'3466","reported_seq":3494,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.734146+0000","last_change":"2024-03-12T03:35:23.781284+0000","last_active":"2024-03-12T03:36:00.734146+0000","last_peered":"2024-03-12T03:36:00.734146+0000","last_clean":"2024-03-12T03:36:00.734146+0000","last_became_active":"2024-03-12T03:35:23.780941+0000","last_became_peered":"2024-03-12T03:35:23.780941+0000","last_unstale":"2024-03-12T03:36:00.734146+0000","last_undegraded":"2024-03-12T03:36:00.734146+0000","last_fullsized":"2024-03-12T03:36:00.734146+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3466,"log_dups_size":0,"ondisk_log_size":3466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:26.636565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227147776,"num_objects":3466,"num_object_clones":0,"num_object_copies":10398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":221824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229361+0000","last_change":"2024-03-12T03:34:51.734825+0000","last_active":"2024-03-12T03:35:59.229361+0000","last_peered":"2024-03-12T03:35:59.229361+0000","last_clean":"2024-03-12T03:35:59.229361+0000","last_became_active":"2024-03-12T03:33:27.194266+0000","last_became_peered":"2024-03-12T03:33:27.194266+0000","last_unstale":"2024-03-12T03:35:59.229361+0000","last_undegraded":"2024-03-12T03:35:59.229361+0000","last_fullsized":"2024-03-12T03:35:59.229361+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:51.734742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:51.734742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:12.279921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232529+0000","last_change":"2024-03-12T03:34:41.152579+0000","last_active":"2024-03-12T03:35:59.232529+0000","last_peered":"2024-03-12T03:35:59.232529+0000","last_clean":"2024-03-12T03:35:59.232529+0000","last_became_active":"2024-03-12T03:33:21.573251+0000","last_became_peered":"2024-03-12T03:33:21.573251+0000","last_unstale":"2024-03-12T03:35:59.232529+0000","last_undegraded":"2024-03-12T03:35:59.232529+0000","last_fullsized":"2024-03-12T03:35:59.232529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.152440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.152440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":121,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232436+0000","last_change":"2024-03-12T03:34:35.282309+0000","last_active":"2024-03-12T03:35:59.232436+0000","last_peered":"2024-03-12T03:35:59.232436+0000","last_clean":"2024-03-12T03:35:59.232436+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:35:59.232436+0000","last_undegraded":"2024-03-12T03:35:59.232436+0000","last_fullsized":"2024-03-12T03:35:59.232436+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:34:35.282203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:35.282203+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":130,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229538+0000","last_change":"2024-03-12T03:34:30.767076+0000","last_active":"2024-03-12T03:35:59.229538+0000","last_peered":"2024-03-12T03:35:59.229538+0000","last_clean":"2024-03-12T03:35:59.229538+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:35:59.229538+0000","last_undegraded":"2024-03-12T03:35:59.229538+0000","last_fullsized":"2024-03-12T03:35:59.229538+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:30.766975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.766975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"106'3426","reported_seq":3454,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.842276+0000","last_change":"2024-03-12T03:35:23.778223+0000","last_active":"2024-03-12T03:36:00.842276+0000","last_peered":"2024-03-12T03:36:00.842276+0000","last_clean":"2024-03-12T03:36:00.842276+0000","last_became_active":"2024-03-12T03:35:23.777935+0000","last_became_peered":"2024-03-12T03:35:23.777935+0000","last_unstale":"2024-03-12T03:36:00.842276+0000","last_undegraded":"2024-03-12T03:36:00.842276+0000","last_fullsized":"2024-03-12T03:36:00.842276+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3426,"log_dups_size":0,"ondisk_log_size":3426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:51.033924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224526336,"num_objects":3426,"num_object_clones":0,"num_object_copies":10278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3426,"num_write_kb":219264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.230597+0000","last_change":"2024-03-12T03:34:44.750269+0000","last_active":"2024-03-12T03:35:59.230597+0000","last_peered":"2024-03-12T03:35:59.230597+0000","last_clean":"2024-03-12T03:35:59.230597+0000","last_became_active":"2024-03-12T03:33:27.383249+0000","last_became_peered":"2024-03-12T03:33:27.383249+0000","last_unstale":"2024-03-12T03:35:59.230597+0000","last_undegraded":"2024-03-12T03:35:59.230597+0000","last_fullsized":"2024-03-12T03:35:59.230597+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:44.750135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:44.750135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.229614+0000","last_change":"2024-03-12T03:34:56.779549+0000","last_active":"2024-03-12T03:35:59.229614+0000","last_peered":"2024-03-12T03:35:59.229614+0000","last_clean":"2024-03-12T03:35:59.229614+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:35:59.229614+0000","last_undegraded":"2024-03-12T03:35:59.229614+0000","last_fullsized":"2024-03-12T03:35:59.229614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:34:56.779379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:56.779379+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.372243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":127,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223228+0000","last_change":"2024-03-12T03:34:47.043566+0000","last_active":"2024-03-12T03:35:59.223228+0000","last_peered":"2024-03-12T03:35:59.223228+0000","last_clean":"2024-03-12T03:35:59.223228+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:35:59.223228+0000","last_undegraded":"2024-03-12T03:35:59.223228+0000","last_fullsized":"2024-03-12T03:35:59.223228+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-03-12T03:34:47.043433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.043433+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:04.321445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":130,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.224795+0000","last_change":"2024-03-12T03:34:25.168128+0000","last_active":"2024-03-12T03:35:59.224795+0000","last_peered":"2024-03-12T03:35:59.224795+0000","last_clean":"2024-03-12T03:35:59.224795+0000","last_became_active":"2024-03-12T03:33:25.177311+0000","last_became_peered":"2024-03-12T03:33:25.177311+0000","last_unstale":"2024-03-12T03:35:59.224795+0000","last_undegraded":"2024-03-12T03:35:59.224795+0000","last_fullsized":"2024-03-12T03:35:59.224795+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:25.168002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.168002+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"106'3521","reported_seq":3549,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.827851+0000","last_change":"2024-03-12T03:35:23.778034+0000","last_active":"2024-03-12T03:36:00.827851+0000","last_peered":"2024-03-12T03:36:00.827851+0000","last_clean":"2024-03-12T03:36:00.827851+0000","last_became_active":"2024-03-12T03:35:23.777749+0000","last_became_peered":"2024-03-12T03:35:23.777749+0000","last_unstale":"2024-03-12T03:36:00.827851+0000","last_undegraded":"2024-03-12T03:36:00.827851+0000","last_fullsized":"2024-03-12T03:36:00.827851+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3521,"log_dups_size":0,"ondisk_log_size":3521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:32.320246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230752256,"num_objects":3521,"num_object_clones":0,"num_object_copies":10563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3521,"num_write_kb":225344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223086+0000","last_change":"2024-03-12T03:34:58.085602+0000","last_active":"2024-03-12T03:35:59.223086+0000","last_peered":"2024-03-12T03:35:59.223086+0000","last_clean":"2024-03-12T03:35:59.223086+0000","last_became_active":"2024-03-12T03:33:27.187864+0000","last_became_peered":"2024-03-12T03:33:27.187864+0000","last_unstale":"2024-03-12T03:35:59.223086+0000","last_undegraded":"2024-03-12T03:35:59.223086+0000","last_fullsized":"2024-03-12T03:35:59.223086+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:58.085499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.085499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:03.220277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.224306+0000","last_change":"2024-03-12T03:34:47.155013+0000","last_active":"2024-03-12T03:35:59.224306+0000","last_peered":"2024-03-12T03:35:59.224306+0000","last_clean":"2024-03-12T03:35:59.224306+0000","last_became_active":"2024-03-12T03:33:21.104430+0000","last_became_peered":"2024-03-12T03:33:21.104430+0000","last_unstale":"2024-03-12T03:35:59.224306+0000","last_undegraded":"2024-03-12T03:35:59.224306+0000","last_fullsized":"2024-03-12T03:35:59.224306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.154925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.154925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":142,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.224709+0000","last_change":"2024-03-12T03:33:23.551957+0000","last_active":"2024-03-12T03:35:59.224709+0000","last_peered":"2024-03-12T03:35:59.224709+0000","last_clean":"2024-03-12T03:35:59.224709+0000","last_became_active":"2024-03-12T03:33:23.551623+0000","last_became_peered":"2024-03-12T03:33:23.551623+0000","last_unstale":"2024-03-12T03:35:59.224709+0000","last_undegraded":"2024-03-12T03:35:59.224709+0000","last_fullsized":"2024-03-12T03:35:59.224709+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:33:22.141170+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.222904+0000","last_change":"2024-03-12T03:35:00.071368+0000","last_active":"2024-03-12T03:35:59.222904+0000","last_peered":"2024-03-12T03:35:59.222904+0000","last_clean":"2024-03-12T03:35:59.222904+0000","last_became_active":"2024-03-12T03:33:25.181020+0000","last_became_peered":"2024-03-12T03:33:25.181020+0000","last_unstale":"2024-03-12T03:35:59.222904+0000","last_undegraded":"2024-03-12T03:35:59.222904+0000","last_fullsized":"2024-03-12T03:35:59.222904+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:00.071245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.071245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:13.842153+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"106'3522","reported_seq":3550,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.746785+0000","last_change":"2024-03-12T03:35:23.785403+0000","last_active":"2024-03-12T03:36:00.746785+0000","last_peered":"2024-03-12T03:36:00.746785+0000","last_clean":"2024-03-12T03:36:00.746785+0000","last_became_active":"2024-03-12T03:35:23.785037+0000","last_became_peered":"2024-03-12T03:35:23.785037+0000","last_unstale":"2024-03-12T03:36:00.746785+0000","last_undegraded":"2024-03-12T03:36:00.746785+0000","last_fullsized":"2024-03-12T03:36:00.746785+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3522,"log_dups_size":0,"ondisk_log_size":3522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:51.241046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230817792,"num_objects":3522,"num_object_clones":0,"num_object_copies":10566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3522,"num_write_kb":225408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232516+0000","last_change":"2024-03-12T03:35:17.142169+0000","last_active":"2024-03-12T03:35:59.232516+0000","last_peered":"2024-03-12T03:35:59.232516+0000","last_clean":"2024-03-12T03:35:59.232516+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:35:59.232516+0000","last_undegraded":"2024-03-12T03:35:59.232516+0000","last_fullsized":"2024-03-12T03:35:59.232516+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:17.142086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:17.142086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:34.053176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":130,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223867+0000","last_change":"2024-03-12T03:35:20.138357+0000","last_active":"2024-03-12T03:35:59.223867+0000","last_peered":"2024-03-12T03:35:59.223867+0000","last_clean":"2024-03-12T03:35:59.223867+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:35:59.223867+0000","last_undegraded":"2024-03-12T03:35:59.223867+0000","last_fullsized":"2024-03-12T03:35:59.223867+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:35:20.138257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:20.138257+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:41.959809+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.224632+0000","last_change":"2024-03-12T03:35:13.030518+0000","last_active":"2024-03-12T03:35:59.224632+0000","last_peered":"2024-03-12T03:35:59.224632+0000","last_clean":"2024-03-12T03:35:59.224632+0000","last_became_active":"2024-03-12T03:33:21.104293+0000","last_became_peered":"2024-03-12T03:33:21.104293+0000","last_unstale":"2024-03-12T03:35:59.224632+0000","last_undegraded":"2024-03-12T03:35:59.224632+0000","last_fullsized":"2024-03-12T03:35:59.224632+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:13.030426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:13.030426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:27.030999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":89,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.237269+0000","last_change":"2024-03-12T03:35:07.004248+0000","last_active":"2024-03-12T03:35:59.237269+0000","last_peered":"2024-03-12T03:35:59.237269+0000","last_clean":"2024-03-12T03:35:59.237269+0000","last_became_active":"2024-03-12T03:33:50.153240+0000","last_became_peered":"2024-03-12T03:33:50.153240+0000","last_unstale":"2024-03-12T03:35:59.237269+0000","last_undegraded":"2024-03-12T03:35:59.237269+0000","last_fullsized":"2024-03-12T03:35:59.237269+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:35:07.004152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:07.004152+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:14.808273+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"106'3426","reported_seq":3454,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.773182+0000","last_change":"2024-03-12T03:35:23.783266+0000","last_active":"2024-03-12T03:36:00.773182+0000","last_peered":"2024-03-12T03:36:00.773182+0000","last_clean":"2024-03-12T03:36:00.773182+0000","last_became_active":"2024-03-12T03:35:23.782945+0000","last_became_peered":"2024-03-12T03:35:23.782945+0000","last_unstale":"2024-03-12T03:36:00.773182+0000","last_undegraded":"2024-03-12T03:36:00.773182+0000","last_fullsized":"2024-03-12T03:36:00.773182+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3426,"log_dups_size":0,"ondisk_log_size":3426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:44.162343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224526336,"num_objects":3426,"num_object_clones":0,"num_object_copies":10278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3426,"num_write_kb":219264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.225519+0000","last_change":"2024-03-12T03:33:27.381253+0000","last_active":"2024-03-12T03:35:59.225519+0000","last_peered":"2024-03-12T03:35:59.225519+0000","last_clean":"2024-03-12T03:35:59.225519+0000","last_became_active":"2024-03-12T03:33:27.380896+0000","last_became_peered":"2024-03-12T03:33:27.380896+0000","last_unstale":"2024-03-12T03:35:59.225519+0000","last_undegraded":"2024-03-12T03:35:59.225519+0000","last_fullsized":"2024-03-12T03:35:59.225519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.232825+0000","last_change":"2024-03-12T03:34:36.138975+0000","last_active":"2024-03-12T03:35:59.232825+0000","last_peered":"2024-03-12T03:35:59.232825+0000","last_clean":"2024-03-12T03:35:59.232825+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:35:59.232825+0000","last_undegraded":"2024-03-12T03:35:59.232825+0000","last_fullsized":"2024-03-12T03:35:59.232825+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:36.138847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:36.138847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:00.950469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223010+0000","last_change":"2024-03-12T03:33:21.573917+0000","last_active":"2024-03-12T03:35:59.223010+0000","last_peered":"2024-03-12T03:35:59.223010+0000","last_clean":"2024-03-12T03:35:59.223010+0000","last_became_active":"2024-03-12T03:33:21.573257+0000","last_became_peered":"2024-03-12T03:33:21.573257+0000","last_unstale":"2024-03-12T03:35:59.223010+0000","last_undegraded":"2024-03-12T03:35:59.223010+0000","last_fullsized":"2024-03-12T03:35:59.223010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:33:20.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"95'23","reported_seq":131,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223852+0000","last_change":"2024-03-12T03:35:06.028109+0000","last_active":"2024-03-12T03:35:59.223852+0000","last_peered":"2024-03-12T03:35:59.223852+0000","last_clean":"2024-03-12T03:35:59.223852+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:35:59.223852+0000","last_undegraded":"2024-03-12T03:35:59.223852+0000","last_fullsized":"2024-03-12T03:35:59.223852+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'23","last_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:35:06.028013+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:09.991579+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"106'3579","reported_seq":3607,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.766506+0000","last_change":"2024-03-12T03:35:23.777241+0000","last_active":"2024-03-12T03:36:00.766506+0000","last_peered":"2024-03-12T03:36:00.766506+0000","last_clean":"2024-03-12T03:36:00.766506+0000","last_became_active":"2024-03-12T03:35:23.776932+0000","last_became_peered":"2024-03-12T03:35:23.776932+0000","last_unstale":"2024-03-12T03:36:00.766506+0000","last_undegraded":"2024-03-12T03:36:00.766506+0000","last_fullsized":"2024-03-12T03:36:00.766506+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3579,"log_dups_size":0,"ondisk_log_size":3579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:41.400472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234553344,"num_objects":3579,"num_object_clones":0,"num_object_copies":10737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3579,"num_write_kb":229056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.237473+0000","last_change":"2024-03-12T03:35:56.071430+0000","last_active":"2024-03-12T03:35:59.237473+0000","last_peered":"2024-03-12T03:35:59.237473+0000","last_clean":"2024-03-12T03:35:59.237473+0000","last_became_active":"2024-03-12T03:33:27.187815+0000","last_became_peered":"2024-03-12T03:33:27.187815+0000","last_unstale":"2024-03-12T03:35:59.237473+0000","last_undegraded":"2024-03-12T03:35:59.237473+0000","last_fullsized":"2024-03-12T03:35:59.237473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:56.071324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:56.071324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:05.061549+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.234792+0000","last_change":"2024-03-12T03:34:30.280067+0000","last_active":"2024-03-12T03:35:59.234792+0000","last_peered":"2024-03-12T03:35:59.234792+0000","last_clean":"2024-03-12T03:35:59.234792+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:35:59.234792+0000","last_undegraded":"2024-03-12T03:35:59.234792+0000","last_fullsized":"2024-03-12T03:35:59.234792+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:30.279966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:30.279966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":106,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.236065+0000","last_change":"2024-03-12T03:35:59.016620+0000","last_active":"2024-03-12T03:35:59.236065+0000","last_peered":"2024-03-12T03:35:59.236065+0000","last_clean":"2024-03-12T03:35:59.236065+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:35:59.236065+0000","last_undegraded":"2024-03-12T03:35:59.236065+0000","last_fullsized":"2024-03-12T03:35:59.236065+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.016489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.016489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:28.178945+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":122,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.224115+0000","last_change":"2024-03-12T03:35:59.053600+0000","last_active":"2024-03-12T03:35:59.224115+0000","last_peered":"2024-03-12T03:35:59.224115+0000","last_clean":"2024-03-12T03:35:59.224115+0000","last_became_active":"2024-03-12T03:33:23.163871+0000","last_became_peered":"2024-03-12T03:33:23.163871+0000","last_unstale":"2024-03-12T03:35:59.224115+0000","last_undegraded":"2024-03-12T03:35:59.224115+0000","last_fullsized":"2024-03-12T03:35:59.224115+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:35:59.053486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.053486+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:03.009127+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"106'3597","reported_seq":3625,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:36:00.757306+0000","last_change":"2024-03-12T03:35:23.784927+0000","last_active":"2024-03-12T03:36:00.757306+0000","last_peered":"2024-03-12T03:36:00.757306+0000","last_clean":"2024-03-12T03:36:00.757306+0000","last_became_active":"2024-03-12T03:35:23.784627+0000","last_became_peered":"2024-03-12T03:35:23.784627+0000","last_unstale":"2024-03-12T03:36:00.757306+0000","last_undegraded":"2024-03-12T03:36:00.757306+0000","last_fullsized":"2024-03-12T03:36:00.757306+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3597,"log_dups_size":0,"ondisk_log_size":3597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:37.248005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235732992,"num_objects":3597,"num_object_clones":0,"num_object_copies":10791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":230208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223954+0000","last_change":"2024-03-12T03:33:27.184691+0000","last_active":"2024-03-12T03:35:59.223954+0000","last_peered":"2024-03-12T03:35:59.223954+0000","last_clean":"2024-03-12T03:35:59.223954+0000","last_became_active":"2024-03-12T03:33:27.184405+0000","last_became_peered":"2024-03-12T03:33:27.184405+0000","last_unstale":"2024-03-12T03:35:59.223954+0000","last_undegraded":"2024-03-12T03:35:59.223954+0000","last_fullsized":"2024-03-12T03:35:59.223954+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":130,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.237352+0000","last_change":"2024-03-12T03:34:55.906528+0000","last_active":"2024-03-12T03:35:59.237352+0000","last_peered":"2024-03-12T03:35:59.237352+0000","last_clean":"2024-03-12T03:35:59.237352+0000","last_became_active":"2024-03-12T03:33:25.177613+0000","last_became_peered":"2024-03-12T03:33:25.177613+0000","last_unstale":"2024-03-12T03:35:59.237352+0000","last_undegraded":"2024-03-12T03:35:59.237352+0000","last_fullsized":"2024-03-12T03:35:59.237352+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:55.906394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:55.906394+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:15.924626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.226191+0000","last_change":"2024-03-12T03:35:15.981464+0000","last_active":"2024-03-12T03:35:59.226191+0000","last_peered":"2024-03-12T03:35:59.226191+0000","last_clean":"2024-03-12T03:35:59.226191+0000","last_became_active":"2024-03-12T03:33:21.097304+0000","last_became_peered":"2024-03-12T03:33:21.097304+0000","last_unstale":"2024-03-12T03:35:59.226191+0000","last_undegraded":"2024-03-12T03:35:59.226191+0000","last_fullsized":"2024-03-12T03:35:59.226191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.981378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.981378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:41.402156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":109,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-12T03:35:59.223541+0000","last_change":"2024-03-12T03:35:58.089544+0000","last_active":"2024-03-12T03:35:59.223541+0000","last_peered":"2024-03-12T03:35:59.223541+0000","last_clean":"2024-03-12T03:35:59.223541+0000","last_became_active":"2024-03-12T03:33:23.162657+0000","last_became_peered":"2024-03-12T03:33:23.162657+0000","last_unstale":"2024-03-12T03:35:59.223541+0000","last_undegraded":"2024-03-12T03:35:59.223541+0000","last_fullsized":"2024-03-12T03:35:59.223541+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-12T03:35:58.089405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.089405+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:25.792358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3655860224,"num_objects":55784,"num_object_clones":0,"num_object_copies":167352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55784,"num_write_kb":3570176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10967908352,"data_stored":10967908352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55784,"ondisk_log_size":55784,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":149,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":625,"ondisk_log_size":625,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070501,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1133180,"kb_used_data":1102000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92611972,"statfs":{"total":95995035648,"available":94834659328,"internally_reserved":0,"allocated":1128448000,"data_stored":1128043642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332138,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1112612,"kb_used_data":1085592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92632540,"statfs":{"total":95995035648,"available":94855720960,"internally_reserved":0,"allocated":1111646208,"data_stored":1111264463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1346884,"kb_used_data":1293436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92398268,"statfs":{"total":95995035648,"available":94615826432,"internally_reserved":0,"allocated":1324478464,"data_stored":1324060941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1814008,"kb_used_data":1760624,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":91931144,"statfs":{"total":95995035648,"available":94137491456,"internally_reserved":0,"allocated":1802878976,"data_stored":1802473872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1377376,"kb_used_data":1323992,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92367776,"statfs":{"total":95995035648,"available":94584602624,"internally_reserved":0,"allocated":1355767808,"data_stored":1355386752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117048,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2049796,"kb_used_data":1978364,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91695356,"statfs":{"total":95995035648,"available":93896044544,"internally_reserved":0,"allocated":2025844736,"data_stored":2025426910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953525,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1148732,"kb_used_data":1099504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92596420,"statfs":{"total":95995035648,"available":94818734080,"internally_reserved":0,"allocated":1125892096,"data_stored":1125486734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659183,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":916204,"kb_used_data":880868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":92828948,"statfs":{"total":95995035648,"available":95056842752,"internally_reserved":0,"allocated":902008832,"data_stored":901614926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1836056576,"data_stored":1836056576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1355874304,"data_stored":1355874304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1375928320,"data_stored":1375928320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2062155776,"data_stored":2062155776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1143537664,"data_stored":1143537664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":913113088,"data_stored":913113088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1132789760,"data_stored":1132789760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1148452864,"data_stored":1148452864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:36:05.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool set unique_pool_0 pgp_num 16 2024-03-12T03:36:05.236 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:05.285 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:05.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:05.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:05.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:05.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:05.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:05.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:05 smithi169 ceph-mon[29344]: pgmap v88: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-12T03:36:05.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:05 smithi169 ceph-mon[29344]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:36:05.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:05 smithi169 ceph-mon[39949]: pgmap v88: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-12T03:36:05.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:05 smithi169 ceph-mon[39949]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:36:05.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:05.566 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 34 16 61857 61841 113.655 76.5625 0.00851539 0.00879268 2024-03-12T03:36:05.576 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:05.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:05.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:05.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:05.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:05.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:05.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:05 smithi173 ceph-mon[33544]: pgmap v88: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-12T03:36:05.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:05 smithi173 ceph-mon[33544]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:36:05.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:05.852 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:05.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:05.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:05.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:05.984 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:05.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:05.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:06.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:06.126 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:06.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:06.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:06.268 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:06.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:06.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:06.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:06.409 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:06.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:06.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:06.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:06.550 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:06.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:06.566 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 35 16 62998 62982 112.445 71.3125 0.005659 0.0088882 2024-03-12T03:36:06.651 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:06.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:06 smithi169 ceph-mon[29344]: 5.c scrub starts 2024-03-12T03:36:06.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:06 smithi169 ceph-mon[29344]: 5.c scrub ok 2024-03-12T03:36:06.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:06 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2225082474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:36:06.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:06 smithi169 ceph-mon[39949]: 5.c scrub starts 2024-03-12T03:36:06.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:06 smithi169 ceph-mon[39949]: 5.c scrub ok 2024-03-12T03:36:06.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:06 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2225082474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:36:06.692 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:06.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:06 smithi173 ceph-mon[33544]: 5.c scrub starts 2024-03-12T03:36:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:06 smithi173 ceph-mon[33544]: 5.c scrub ok 2024-03-12T03:36:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:06 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2225082474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:36:06.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:06.829 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:06.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:06.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:06.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:06.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:06.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:07.045 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:36:07.045 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:36:07.046 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:36:07.046 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:36:07.046 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:36:07.046 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:36:07.046 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:36:07.046 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:36:07.046 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:36:07.046 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:36:07.046 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:36:07.046 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:36:07.046 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:36:07.046 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:36:07.047 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:36:07.047 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:36:07.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:07.104 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:07.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:07.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:07.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:07.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:07.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:07.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:07.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:07.445 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:07.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:07.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:07.566 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 36 15 64109 64094 111.252 69.5 0.0110859 0.0089822 2024-03-12T03:36:07.586 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:07.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:07.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:07 smithi169 ceph-mon[29344]: pgmap v89: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-12T03:36:07.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:07 smithi169 ceph-mon[29344]: 5.3 scrub starts 2024-03-12T03:36:07.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:07 smithi169 ceph-mon[29344]: 5.3 scrub ok 2024-03-12T03:36:07.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:07 smithi169 ceph-mon[29344]: 4.0 scrub starts 2024-03-12T03:36:07.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:07 smithi169 ceph-mon[29344]: 4.0 scrub ok 2024-03-12T03:36:07.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:07 smithi169 ceph-mon[39949]: pgmap v89: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-12T03:36:07.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:07 smithi169 ceph-mon[39949]: 5.3 scrub starts 2024-03-12T03:36:07.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:07 smithi169 ceph-mon[39949]: 5.3 scrub ok 2024-03-12T03:36:07.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:07 smithi169 ceph-mon[39949]: 4.0 scrub starts 2024-03-12T03:36:07.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:07 smithi169 ceph-mon[39949]: 4.0 scrub ok 2024-03-12T03:36:07.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:07.725 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:07.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:07.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:07 smithi173 ceph-mon[33544]: pgmap v89: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-12T03:36:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:07 smithi173 ceph-mon[33544]: 5.3 scrub starts 2024-03-12T03:36:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:07 smithi173 ceph-mon[33544]: 5.3 scrub ok 2024-03-12T03:36:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:07 smithi173 ceph-mon[33544]: 4.0 scrub starts 2024-03-12T03:36:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:07 smithi173 ceph-mon[33544]: 4.0 scrub ok 2024-03-12T03:36:07.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:07.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:07.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:07.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:07.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:07.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:07.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:08.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:08.124 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:08.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:08.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:08.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:08.286 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:08.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:08.387 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:08.441 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:08.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:08.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:08.566 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 37 16 65423 65407 110.463 82.0625 0.0079763 0.00904769 2024-03-12T03:36:08.583 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:08.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:08.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:08.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:08 smithi169 ceph-mon[29344]: 6.e scrub starts 2024-03-12T03:36:08.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:08 smithi169 ceph-mon[29344]: 6.e scrub ok 2024-03-12T03:36:08.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:08 smithi169 ceph-mon[29344]: 6.16 scrub starts 2024-03-12T03:36:08.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:08 smithi169 ceph-mon[29344]: 6.16 scrub ok 2024-03-12T03:36:08.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:08 smithi169 ceph-mon[29344]: 5.15 scrub starts 2024-03-12T03:36:08.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:08 smithi169 ceph-mon[29344]: 5.15 scrub ok 2024-03-12T03:36:08.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:08 smithi169 ceph-mon[39949]: 6.e scrub starts 2024-03-12T03:36:08.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:08 smithi169 ceph-mon[39949]: 6.e scrub ok 2024-03-12T03:36:08.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:08 smithi169 ceph-mon[39949]: 6.16 scrub starts 2024-03-12T03:36:08.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:08 smithi169 ceph-mon[39949]: 6.16 scrub ok 2024-03-12T03:36:08.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:08 smithi169 ceph-mon[39949]: 5.15 scrub starts 2024-03-12T03:36:08.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:08 smithi169 ceph-mon[39949]: 5.15 scrub ok 2024-03-12T03:36:08.685 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:08.731 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:08.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:08.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:08 smithi173 ceph-mon[33544]: 6.e scrub starts 2024-03-12T03:36:08.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:08 smithi173 ceph-mon[33544]: 6.e scrub ok 2024-03-12T03:36:08.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:08 smithi173 ceph-mon[33544]: 6.16 scrub starts 2024-03-12T03:36:08.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:08 smithi173 ceph-mon[33544]: 6.16 scrub ok 2024-03-12T03:36:08.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:08 smithi173 ceph-mon[33544]: 5.15 scrub starts 2024-03-12T03:36:08.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:08 smithi173 ceph-mon[33544]: 5.15 scrub ok 2024-03-12T03:36:08.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:08.859 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:08.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:08.960 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:08.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:08.999 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:08.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:09.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:09.144 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:09.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:09.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:09.286 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:09.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:09.387 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:09.430 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:09.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:09.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:09.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:09.566 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:09.566 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 38 16 66607 66591 109.503 74 0.0130464 0.00912725 2024-03-12T03:36:09.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:09.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:09.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[29344]: pgmap v90: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-12T03:36:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[29344]: 4.2 scrub starts 2024-03-12T03:36:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[29344]: 4.2 scrub ok 2024-03-12T03:36:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[29344]: 2.0 scrub starts 2024-03-12T03:36:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[29344]: 2.0 scrub ok 2024-03-12T03:36:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[29344]: 3.1b scrub starts 2024-03-12T03:36:09.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[29344]: 3.1b scrub ok 2024-03-12T03:36:09.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[39949]: pgmap v90: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-12T03:36:09.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[39949]: 4.2 scrub starts 2024-03-12T03:36:09.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[39949]: 4.2 scrub ok 2024-03-12T03:36:09.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[39949]: 2.0 scrub starts 2024-03-12T03:36:09.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[39949]: 2.0 scrub ok 2024-03-12T03:36:09.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[39949]: 3.1b scrub starts 2024-03-12T03:36:09.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:09 smithi169 ceph-mon[39949]: 3.1b scrub ok 2024-03-12T03:36:09.709 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:09.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:09.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:09 smithi173 ceph-mon[33544]: pgmap v90: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-12T03:36:09.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:09 smithi173 ceph-mon[33544]: 4.2 scrub starts 2024-03-12T03:36:09.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:09 smithi173 ceph-mon[33544]: 4.2 scrub ok 2024-03-12T03:36:09.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:09 smithi173 ceph-mon[33544]: 2.0 scrub starts 2024-03-12T03:36:09.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:09 smithi173 ceph-mon[33544]: 2.0 scrub ok 2024-03-12T03:36:09.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:09 smithi173 ceph-mon[33544]: 3.1b scrub starts 2024-03-12T03:36:09.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:09 smithi173 ceph-mon[33544]: 3.1b scrub ok 2024-03-12T03:36:09.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:09.842 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:09.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:09.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:09.986 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:09.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:09.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:10.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:10.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:10.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:10.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:10.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:10.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:10.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:10.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:10.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:10.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:10.483 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:10.522 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:10.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:10.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:10.566 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 39 15 67911 67896 108.787 81.5625 0.0099988 0.00918716 2024-03-12T03:36:10.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:10.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:10.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:10.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:10.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:10 smithi169 ceph-mon[29344]: 5.1d scrub starts 2024-03-12T03:36:10.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:10 smithi169 ceph-mon[29344]: 5.1d scrub ok 2024-03-12T03:36:10.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:10 smithi169 ceph-mon[29344]: 3.d scrub starts 2024-03-12T03:36:10.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:10 smithi169 ceph-mon[29344]: 3.d scrub ok 2024-03-12T03:36:10.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:10 smithi169 ceph-mon[39949]: 5.1d scrub starts 2024-03-12T03:36:10.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:10 smithi169 ceph-mon[39949]: 5.1d scrub ok 2024-03-12T03:36:10.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:10 smithi169 ceph-mon[39949]: 3.d scrub starts 2024-03-12T03:36:10.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:10 smithi169 ceph-mon[39949]: 3.d scrub ok 2024-03-12T03:36:10.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:10 smithi173 ceph-mon[33544]: 5.1d scrub starts 2024-03-12T03:36:10.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:10 smithi173 ceph-mon[33544]: 5.1d scrub ok 2024-03-12T03:36:10.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:10 smithi173 ceph-mon[33544]: 3.d scrub starts 2024-03-12T03:36:10.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:10 smithi173 ceph-mon[33544]: 3.d scrub ok 2024-03-12T03:36:10.758 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:10.798 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:10.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:10.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:10.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:10.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:10.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:10.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:11.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:11.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:11.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:11.164 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:11.208 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:11.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:11.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:11.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:11.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:11.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:11.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:11.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:11.566 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:36:11.565358+0000 min lat: 0.00118901 max lat: 0.163567 avg lat: 0.00925076 2024-03-12T03:36:11.566 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:36:11.567 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 40 16 69173 69157 108.037 78.8125 0.0116081 0.00925076 2024-03-12T03:36:11.575 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:11.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:11.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:11.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:11.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:11 smithi169 ceph-mon[29344]: pgmap v91: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 969 B/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-12T03:36:11.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:11 smithi169 ceph-mon[29344]: 4.9 scrub starts 2024-03-12T03:36:11.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:11 smithi169 ceph-mon[29344]: 4.9 scrub ok 2024-03-12T03:36:11.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:11 smithi169 ceph-mon[29344]: 6.9 scrub starts 2024-03-12T03:36:11.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:11 smithi169 ceph-mon[29344]: 6.9 scrub ok 2024-03-12T03:36:11.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:11 smithi169 ceph-mon[39949]: pgmap v91: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 969 B/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-12T03:36:11.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:11 smithi169 ceph-mon[39949]: 4.9 scrub starts 2024-03-12T03:36:11.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:11 smithi169 ceph-mon[39949]: 4.9 scrub ok 2024-03-12T03:36:11.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:11 smithi169 ceph-mon[39949]: 6.9 scrub starts 2024-03-12T03:36:11.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:11 smithi169 ceph-mon[39949]: 6.9 scrub ok 2024-03-12T03:36:11.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:11.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:11 smithi173 ceph-mon[33544]: pgmap v91: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 969 B/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-12T03:36:11.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:11 smithi173 ceph-mon[33544]: 4.9 scrub starts 2024-03-12T03:36:11.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:11 smithi173 ceph-mon[33544]: 4.9 scrub ok 2024-03-12T03:36:11.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:11 smithi173 ceph-mon[33544]: 6.9 scrub starts 2024-03-12T03:36:11.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:11 smithi173 ceph-mon[33544]: 6.9 scrub ok 2024-03-12T03:36:11.746 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:11.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:11.812 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:11.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:11.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:11.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:11.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:11.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:12.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:12.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:12.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:12.060 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:36:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:36:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:36:12.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:12.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:12.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:12.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:12.270 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:12.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:12.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:12.417 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:12.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:12.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:12.556 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:12.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:12.566 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 41 16 70423 70407 107.308 78.125 0.0070322 0.00931368 2024-03-12T03:36:12.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:12 smithi169 ceph-mon[29344]: 4.12 scrub starts 2024-03-12T03:36:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:12 smithi169 ceph-mon[29344]: 4.12 scrub ok 2024-03-12T03:36:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:12 smithi169 ceph-mon[29344]: 3.6 scrub starts 2024-03-12T03:36:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:12 smithi169 ceph-mon[29344]: 3.6 scrub ok 2024-03-12T03:36:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:12 smithi169 ceph-mon[29344]: 5.a scrub starts 2024-03-12T03:36:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:12 smithi169 ceph-mon[29344]: 5.a scrub ok 2024-03-12T03:36:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:12 smithi169 ceph-mon[39949]: 4.12 scrub starts 2024-03-12T03:36:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:12 smithi169 ceph-mon[39949]: 4.12 scrub ok 2024-03-12T03:36:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:12 smithi169 ceph-mon[39949]: 3.6 scrub starts 2024-03-12T03:36:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:12 smithi169 ceph-mon[39949]: 3.6 scrub ok 2024-03-12T03:36:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:12 smithi169 ceph-mon[39949]: 5.a scrub starts 2024-03-12T03:36:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:12 smithi169 ceph-mon[39949]: 5.a scrub ok 2024-03-12T03:36:12.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:12.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:12.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:12.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:36:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:36:12.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:12 smithi173 ceph-mon[33544]: 4.12 scrub starts 2024-03-12T03:36:12.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:12 smithi173 ceph-mon[33544]: 4.12 scrub ok 2024-03-12T03:36:12.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:12 smithi173 ceph-mon[33544]: 3.6 scrub starts 2024-03-12T03:36:12.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:12 smithi173 ceph-mon[33544]: 3.6 scrub ok 2024-03-12T03:36:12.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:12 smithi173 ceph-mon[33544]: 5.a scrub starts 2024-03-12T03:36:12.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:12 smithi173 ceph-mon[33544]: 5.a scrub ok 2024-03-12T03:36:12.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:12.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:12.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:12.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:12.968 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:12.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:12.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:13.070 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:13.111 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:13.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:13.212 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:13.260 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:13.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:13.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:13.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:36:13.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:13.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:13.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:13.429 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:13 smithi169 ceph-mon[29344]: pgmap v92: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-12T03:36:13.429 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:13 smithi169 ceph-mon[29344]: 4.15 scrub starts 2024-03-12T03:36:13.429 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:13 smithi169 ceph-mon[29344]: 4.15 scrub ok 2024-03-12T03:36:13.429 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:13 smithi169 ceph-mon[29344]: 4.1e scrub starts 2024-03-12T03:36:13.429 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:13 smithi169 ceph-mon[29344]: 4.1e scrub ok 2024-03-12T03:36:13.429 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:13 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:13.429 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:13 smithi169 ceph-mon[39949]: pgmap v92: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-12T03:36:13.429 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:13 smithi169 ceph-mon[39949]: 4.15 scrub starts 2024-03-12T03:36:13.430 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:13 smithi169 ceph-mon[39949]: 4.15 scrub ok 2024-03-12T03:36:13.430 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:13 smithi169 ceph-mon[39949]: 4.1e scrub starts 2024-03-12T03:36:13.430 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:13 smithi169 ceph-mon[39949]: 4.1e scrub ok 2024-03-12T03:36:13.430 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:13 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:13.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:13.537 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:13.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:13.566 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 42 16 71680 71664 106.623 78.5625 0.00994838 0.0093736 2024-03-12T03:36:13.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:13.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:13.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:13.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:13 smithi173 ceph-mon[33544]: pgmap v92: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-12T03:36:13.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:13 smithi173 ceph-mon[33544]: 4.15 scrub starts 2024-03-12T03:36:13.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:13 smithi173 ceph-mon[33544]: 4.15 scrub ok 2024-03-12T03:36:13.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:13 smithi173 ceph-mon[33544]: 4.1e scrub starts 2024-03-12T03:36:13.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:13 smithi173 ceph-mon[33544]: 4.1e scrub ok 2024-03-12T03:36:13.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:13 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:13.768 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:13.831 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:13.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:13.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:13.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:13.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:13.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:14.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:14.104 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:14.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:14.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:14.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:14.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:14.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:14.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:14.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:14.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:14.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:14.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:14.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:14.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:14.567 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 43 16 72882 72866 105.889 75.125 0.0159265 0.00943822 2024-03-12T03:36:14.600 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:14.637 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:14 smithi169 ceph-mon[29344]: 6.c scrub starts 2024-03-12T03:36:14.637 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:14 smithi169 ceph-mon[29344]: 6.c scrub ok 2024-03-12T03:36:14.637 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:14 smithi169 ceph-mon[29344]: 2.2 scrub starts 2024-03-12T03:36:14.637 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:14 smithi169 ceph-mon[29344]: 2.2 scrub ok 2024-03-12T03:36:14.637 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:14 smithi169 ceph-mon[29344]: 3.9 scrub starts 2024-03-12T03:36:14.637 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:14 smithi169 ceph-mon[29344]: 3.9 scrub ok 2024-03-12T03:36:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:14 smithi169 ceph-mon[39949]: 6.c scrub starts 2024-03-12T03:36:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:14 smithi169 ceph-mon[39949]: 6.c scrub ok 2024-03-12T03:36:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:14 smithi169 ceph-mon[39949]: 2.2 scrub starts 2024-03-12T03:36:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:14 smithi169 ceph-mon[39949]: 2.2 scrub ok 2024-03-12T03:36:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:14 smithi169 ceph-mon[39949]: 3.9 scrub starts 2024-03-12T03:36:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:14 smithi169 ceph-mon[39949]: 3.9 scrub ok 2024-03-12T03:36:14.644 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:14.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:14 smithi173 ceph-mon[33544]: 6.c scrub starts 2024-03-12T03:36:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:14 smithi173 ceph-mon[33544]: 6.c scrub ok 2024-03-12T03:36:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:14 smithi173 ceph-mon[33544]: 2.2 scrub starts 2024-03-12T03:36:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:14 smithi173 ceph-mon[33544]: 2.2 scrub ok 2024-03-12T03:36:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:14 smithi173 ceph-mon[33544]: 3.9 scrub starts 2024-03-12T03:36:14.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:14 smithi173 ceph-mon[33544]: 3.9 scrub ok 2024-03-12T03:36:14.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:14.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:14.786 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:14.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:14.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:14.939 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:14.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:14.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:15.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:15.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:15.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:15.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:15.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:15.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:15.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:15.284 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:36:15.284 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:36:15.284 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:36:15.284 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:36:15.284 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.284 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14156.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-03-12T03:36:15.284 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:31:06.365908+0000", 2024-03-12T03:36:15.284 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 308.74826328199998, 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.31863010800000002, 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14156", 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/197210445", 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 1 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.365908+0000", 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.285 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.365908+0000", 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.365909+0000", 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.339e-06 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.365916+0000", 2024-03-12T03:36:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7710000000000001e-06 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.365925+0000", 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.533e-06 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.365936+0000", 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0984e-05 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.287 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.366008+0000", 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1540999999999998e-05 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.366018+0000", 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0288999999999999e-05 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.683593+0000", 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.31757436999999999 2024-03-12T03:36:15.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.683711+0000", 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000118478 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,2", 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.683845+0000", 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013388500000000001 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:15.289 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.684219+0000", 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037372300000000001 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.684423+0000", 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020421799999999999 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.684472+0000", 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8965999999999998e-05 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.290 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.684507+0000", 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4904999999999999e-05 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.684538+0000", 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1106000000000003e-05 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14445.0:2 3.0 3:00000000::::head [pgnls start_epoch 67] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e67)", 2024-03-12T03:36:15.291 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:33:21.103943+0000", 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 174.01022791599999, 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46950572800000001, 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14445", 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/3748937004", 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 2 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103943+0000", 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103943+0000", 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103945+0000", 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5710000000000001e-06 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103951+0000", 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8420000000000001e-06 2024-03-12T03:36:15.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103961+0000", 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0139000000000001e-05 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103972+0000", 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1579e-05 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.104099+0000", 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000126329 2024-03-12T03:36:15.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.104106+0000", 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7100000000000007e-06 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.573181+0000", 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46907466599999997 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.573220+0000", 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9076999999999999e-05 2024-03-12T03:36:15.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.296 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.296 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.573449+0000", 2024-03-12T03:36:15.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000228815 2024-03-12T03:36:15.296 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.296 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.296 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.296 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14436.0:4 3.0 3:00000000::::head [pgnls start_epoch 67] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e67)", 2024-03-12T03:36:15.296 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:33:21.109874+0000", 2024-03-12T03:36:15.296 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 174.00429667899999, 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46447891499999999, 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14436", 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1557333783", 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109874+0000", 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109874+0000", 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109876+0000", 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.694e-06 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109882+0000", 2024-03-12T03:36:15.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.716e-06 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109894+0000", 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1758e-05 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109908+0000", 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4017e-05 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.299 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109959+0000", 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1094000000000002e-05 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109966+0000", 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5780000000000001e-06 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.574147+0000", 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46418114100000002 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.574184+0000", 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6148e-05 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.574353+0000", 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016976900000000001 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.14799.0:24233 7.b e101/97 ondisk, result = 0)", 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:40.731326+0000", 2024-03-12T03:36:15.301 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.382845549999999, 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.076566421999999995, 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.731326+0000", 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.731326+0000", 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.731326+0000", 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4300000000000003e-07 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.731333+0000", 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0589999999999997e-06 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.731342+0000", 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0759999999999992e-06 2024-03-12T03:36:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.731349+0000", 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5209999999999999e-06 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.731382+0000", 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3374999999999998e-05 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.731387+0000", 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5179999999999998e-06 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.807892+0000", 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.076505229999999994 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:24242 7.9 e101/97 7:9d0f356d:::benchmark_data_smithi169_79822_object24241:head v 101'1495, mlcod=101'1493)", 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:40.731864+0000", 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.382307492000002, 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.078054785000000002, 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.305 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.731864+0000", 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.731864+0000", 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.731864+0000", 2024-03-12T03:36:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.44e-07 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.731881+0000", 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7068999999999999e-05 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.807447+0000", 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.075565909000000001 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.807484+0000", 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6789000000000003e-05 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.807732+0000", 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00024772500000000002 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.807760+0000", 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8442e-05 2024-03-12T03:36:15.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.809820+0000", 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0020595470000000001 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.809918+0000", 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8759999999999996e-05 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.309 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:24244 7.0 e101/97 7:0be9df2a:::benchmark_data_smithi169_79822_object24243:head v 101'1571, mlcod=101'1569)", 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:40.732133+0000", 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.382038453, 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.077493840999999994, 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.732133+0000", 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.732133+0000", 2024-03-12T03:36:15.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.732133+0000", 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2499999999999995e-07 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.732239+0000", 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000105463 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.807492+0000", 2024-03-12T03:36:15.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.075253334000000005 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.807523+0000", 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0593000000000003e-05 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.808482+0000", 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00095982999999999995 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.808505+0000", 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2793e-05 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.809578+0000", 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0010731460000000001 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.809626+0000", 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8157e-05 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:36658 7.b 7:d8feefe7:::benchmark_data_smithi169_79822_object36657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:48.075310+0000", 2024-03-12T03:36:15.313 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.038861029, 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.059089586999999999, 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 36658 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.075310+0000", 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.075310+0000", 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.075312+0000", 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.539e-06 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.075336+0000", 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4108999999999999e-05 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.075346+0000", 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0249e-05 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.075354+0000", 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4409999999999993e-06 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.076266+0000", 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00091208499999999998 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.076393+0000", 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000126857 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.076521+0000", 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012747299999999999 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.079708+0000", 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0031875520000000002 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.084125+0000", 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044165760000000002 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.134253+0000", 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.050128059000000003 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.134342+0000", 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9103000000000004e-05 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.134400+0000", 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7544000000000003e-05 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:36660 7.b 7:da59840c:::benchmark_data_smithi169_79822_object36659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:48.075930+0000", 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.038241352, 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.058726184000000001, 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:15.319 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 36660 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.075930+0000", 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.075930+0000", 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.075931+0000", 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4200000000000005e-07 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.075949+0000", 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8318e-05 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.075962+0000", 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3089e-05 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.075970+0000", 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8520000000000004e-06 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.078082+0000", 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002112423 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.078226+0000", 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014354099999999999 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.078374+0000", 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014788299999999999 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.082984+0000", 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00460991 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.084129+0000", 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0011458499999999999 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.134510+0000", 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.050380302000000002 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.134614+0000", 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010430200000000001 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.134656+0000", 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1872000000000002e-05 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.324 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:36664 7.b 7:db503378:::benchmark_data_smithi169_79822_object36663:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:48.077213+0000", 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.036958079000001, 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.057727987000000001, 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 36664 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.077213+0000", 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.077213+0000", 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.077214+0000", 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1000000000000001e-06 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.077241+0000", 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6432e-05 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.077252+0000", 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1385e-05 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.077260+0000", 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4209999999999995e-06 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.081520+0000", 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042593090000000002 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.081640+0000", 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012003299999999999 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.081783+0000", 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014323800000000001 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.083987+0000", 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022036870000000002 2024-03-12T03:36:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.084450+0000", 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00046320299999999998 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.134753+0000", 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.050303410999999999 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.134897+0000", 2024-03-12T03:36:15.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014396400000000001 2024-03-12T03:36:15.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:48.134941+0000", 2024-03-12T03:36:15.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3804000000000003e-05 2024-03-12T03:36:15.330 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.330 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.330 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.330 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:63770 7.4 e106/97 7:2fe8a54c:::benchmark_data_smithi169_79822_object63769:head v 106'4016, mlcod=106'4014)", 2024-03-12T03:36:15.330 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.196491+0000", 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.917680013, 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.089125756, 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.196491+0000", 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.196491+0000", 2024-03-12T03:36:15.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.196492+0000", 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2600000000000002e-07 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.196515+0000", 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2971000000000001e-05 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.196524+0000", 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7650000000000005e-06 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.196532+0000", 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9510000000000006e-06 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.198570+0000", 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002038064 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.198591+0000", 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0469e-05 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.202691+0000", 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040997179999999996 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285617+0000", 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.082926191999999996 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.334 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:63771 7.9 e106/97 7:9e80b2ed:::benchmark_data_smithi169_79822_object63770:head v 106'4005, mlcod=106'4003)", 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.198440+0000", 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.9157314989999996, 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.090207783999999999, 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.198440+0000", 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.198440+0000", 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.198440+0000", 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0299999999999999e-07 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.198467+0000", 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6829e-05 2024-03-12T03:36:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.198479+0000", 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1962e-05 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.198487+0000", 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2689999999999999e-06 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200813+0000", 2024-03-12T03:36:15.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023260170000000001 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200837+0000", 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4144000000000001e-05 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.288257+0000", 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.087419620000000003 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.288647+0000", 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00039033999999999998 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:63775 7.b 7:da10d224:::benchmark_data_smithi169_79822_object63774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.200012+0000", 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.9141586469999998, 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.086313739, 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.339 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 63775 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200012+0000", 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200012+0000", 2024-03-12T03:36:15.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200015+0000", 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.181e-06 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200038+0000", 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2966999999999999e-05 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200050+0000", 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2333e-05 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200059+0000", 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9989999999999997e-06 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200127+0000", 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8066999999999997e-05 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200269+0000", 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014206700000000001 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200430+0000", 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000160466 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285089+0000", 2024-03-12T03:36:15.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.084659781000000003 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285118+0000", 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9142e-05 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285477+0000", 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00035846100000000002 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.286281+0000", 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00080441699999999998 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.286326+0000", 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4858000000000003e-05 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:63774 7.d e106/97 7:b9b713d6:::benchmark_data_smithi169_79822_object63773:head v 106'3937, mlcod=106'3935)", 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.200170+0000", 2024-03-12T03:36:15.345 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.914000938, 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.088209369999999995, 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200170+0000", 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200170+0000", 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200171+0000", 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5799999999999999e-07 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200246+0000", 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4808000000000006e-05 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200256+0000", 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0492000000000001e-05 2024-03-12T03:36:15.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200263+0000", 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6800000000000004e-06 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200333+0000", 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9863999999999994e-05 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.200352+0000", 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8872000000000002e-05 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.288290+0000", 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.087938395000000003 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.288380+0000", 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9600999999999998e-05 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.349 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.14799.0:63775 7.b e106/97 ondisk, result = 0)", 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.202717+0000", 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.911453699, 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.082381131999999996, 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.202717+0000", 2024-03-12T03:36:15.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.202717+0000", 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.202718+0000", 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1999999999999998e-07 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.202726+0000", 2024-03-12T03:36:15.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4660000000000002e-06 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.284758+0000", 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.082032077999999994 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.284775+0000", 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6663999999999998e-05 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285073+0000", 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00029808699999999999 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285083+0000", 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0128e-05 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285099+0000", 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5789000000000001e-05 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.353 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.14799.0:63775 7.b e106/97 ondisk, result = 0)", 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.202785+0000", 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.9113856010000001, 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.082336698999999999, 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.202785+0000", 2024-03-12T03:36:15.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.202785+0000", 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.202786+0000", 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1600000000000001e-07 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.202789+0000", 2024-03-12T03:36:15.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.444e-06 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.284814+0000", 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.082024267999999997 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.284821+0000", 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3039999999999999e-06 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285112+0000", 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00029121099999999999 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285116+0000", 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.534e-06 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285122+0000", 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4219999999999997e-06 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:63779 7.4 e106/97 7:2528b60e:::benchmark_data_smithi169_79822_object63778:head v 106'4017, mlcod=106'4014)", 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.205019+0000", 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.9091517289999995, 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.088460647000000003, 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.205019+0000", 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.205019+0000", 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.205020+0000", 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.015e-06 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.205047+0000", 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6587000000000001e-05 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285497+0000", 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.080450524999999995 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285508+0000", 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0172e-05 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285649+0000", 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014094800000000001 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285669+0000", 2024-03-12T03:36:15.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0432e-05 2024-03-12T03:36:15.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.293351+0000", 2024-03-12T03:36:15.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076821490000000001 2024-03-12T03:36:15.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.293480+0000", 2024-03-12T03:36:15.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012881899999999999 2024-03-12T03:36:15.361 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.361 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:63978 7.b 7:dcc2b84c:::benchmark_data_smithi169_79822_object63977:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.417790+0000", 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.6963806100000003, 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.059755733999999998, 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 63978 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.417790+0000", 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.417790+0000", 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.417792+0000", 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6509999999999999e-06 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.417816+0000", 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4150000000000001e-05 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.417829+0000", 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2801e-05 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.417843+0000", 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3828999999999999e-05 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.419981+0000", 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021384490000000002 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.420103+0000", 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121286 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.420230+0000", 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012721499999999999 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.424846+0000", 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046159529999999999 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.424885+0000", 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9666000000000003e-05 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.477227+0000", 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.052341866000000001 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.477485+0000", 2024-03-12T03:36:15.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00025750000000000002 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.477546+0000", 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1367999999999999e-05 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:63980 7.4 e106/97 7:25d5a71f:::benchmark_data_smithi169_79822_object63979:head v 106'4035, mlcod=106'4033)", 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.423192+0000", 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.6909787830000003, 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.059507354999999998, 2024-03-12T03:36:15.367 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.423192+0000", 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.423192+0000", 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.423193+0000", 2024-03-12T03:36:15.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9400000000000004e-07 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.423221+0000", 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7725e-05 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.423231+0000", 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0471e-05 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.423238+0000", 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7449999999999998e-06 2024-03-12T03:36:15.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.423292+0000", 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3712999999999999e-05 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.423314+0000", 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1778999999999999e-05 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.482625+0000", 2024-03-12T03:36:15.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.059311299999999997 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.482700+0000", 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4827999999999996e-05 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:63975 7.2 e106/97 7:4f7a6a45:::benchmark_data_smithi169_79822_object63974:head v 106'3953, mlcod=106'3949)", 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.425089+0000", 2024-03-12T03:36:15.371 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.6890823350000002, 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.060060884000000002, 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.425089+0000", 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.425089+0000", 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.425089+0000", 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7199999999999998e-07 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.425112+0000", 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3045999999999999e-05 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.425122+0000", 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1409999999999994e-06 2024-03-12T03:36:15.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.425128+0000", 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3890000000000002e-06 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.477639+0000", 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.052510797999999997 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.477975+0000", 2024-03-12T03:36:15.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00033640900000000002 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.484998+0000", 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0070227589999999999 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.485150+0000", 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015166999999999999 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:63988 7.b 7:d39bdd23:::benchmark_data_smithi169_79822_object63987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.426541+0000", 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.6876297740000004, 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.059754134, 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 63988 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.426541+0000", 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.426541+0000", 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.426543+0000", 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.996e-06 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.426566+0000", 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3096999999999999e-05 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.477161+0000", 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.050594932000000002 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.477182+0000", 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1021000000000001e-05 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.477720+0000", 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00053735999999999996 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.477898+0000", 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017869500000000001 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.478089+0000", 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019011000000000001 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.483038+0000", 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00494992 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.483074+0000", 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5197999999999997e-05 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.485972+0000", 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002898683 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:15.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.486247+0000", 2024-03-12T03:36:15.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00027512599999999999 2024-03-12T03:36:15.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:15.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:15.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:15.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.486295+0000", 2024-03-12T03:36:15.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7995999999999999e-05 2024-03-12T03:36:15.381 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.381 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.381 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.381 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:15.381 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:15.381 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:36:15.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:15.393 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:15 smithi169 ceph-mon[29344]: pgmap v93: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-12T03:36:15.393 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:15 smithi169 ceph-mon[29344]: 3.12 scrub starts 2024-03-12T03:36:15.393 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:15 smithi169 ceph-mon[29344]: 3.12 scrub ok 2024-03-12T03:36:15.394 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:15.394 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:15 smithi169 ceph-mon[29344]: 6.4 scrub starts 2024-03-12T03:36:15.394 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:15 smithi169 ceph-mon[29344]: 6.4 scrub ok 2024-03-12T03:36:15.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:15.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:15.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:15.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:15.561 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:15.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:15.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:15.568 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 44 16 73938 73922 104.98 66 0.0155474 0.00951768 2024-03-12T03:36:15.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:15.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:15 smithi169 ceph-mon[39949]: pgmap v93: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-12T03:36:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:15 smithi169 ceph-mon[39949]: 3.12 scrub starts 2024-03-12T03:36:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:15 smithi169 ceph-mon[39949]: 3.12 scrub ok 2024-03-12T03:36:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:15 smithi169 ceph-mon[39949]: 6.4 scrub starts 2024-03-12T03:36:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:15 smithi169 ceph-mon[39949]: 6.4 scrub ok 2024-03-12T03:36:15.716 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:15.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:15.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:15.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:15 smithi173 ceph-mon[33544]: pgmap v93: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-12T03:36:15.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:15 smithi173 ceph-mon[33544]: 3.12 scrub starts 2024-03-12T03:36:15.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:15 smithi173 ceph-mon[33544]: 3.12 scrub ok 2024-03-12T03:36:15.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:15.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:15 smithi173 ceph-mon[33544]: 6.4 scrub starts 2024-03-12T03:36:15.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:15 smithi173 ceph-mon[33544]: 6.4 scrub ok 2024-03-12T03:36:15.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:15.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:15.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:15.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:15.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:15.997 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:15.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:16.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:16.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:16.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:16.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:16.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:16.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:16.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:16.401 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:16.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:16.403 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:36:16.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:16.530 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:16.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:16.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:16.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-12T03:36:16.568 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 45 16 75108 75092 104.272 73.125 0.0142283 0.00958476 2024-03-12T03:36:16.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:16.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:16.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:16.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[29344]: 4.1b scrub starts 2024-03-12T03:36:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[29344]: 4.1b scrub ok 2024-03-12T03:36:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[29344]: 5.5 scrub starts 2024-03-12T03:36:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[29344]: 5.5 scrub ok 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[29344]: 4.8 scrub starts 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[29344]: 4.8 scrub ok 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[29344]: 2.1 scrub starts 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[29344]: 2.1 scrub ok 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2422890785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[29344]: 3.19 scrub starts 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[29344]: 3.19 scrub ok 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[39949]: 4.1b scrub starts 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[39949]: 4.1b scrub ok 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[39949]: 5.5 scrub starts 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[39949]: 5.5 scrub ok 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[39949]: 4.8 scrub starts 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[39949]: 4.8 scrub ok 2024-03-12T03:36:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[39949]: 2.1 scrub starts 2024-03-12T03:36:16.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[39949]: 2.1 scrub ok 2024-03-12T03:36:16.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2422890785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:36:16.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[39949]: 3.19 scrub starts 2024-03-12T03:36:16.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:16 smithi169 ceph-mon[39949]: 3.19 scrub ok 2024-03-12T03:36:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:16 smithi173 ceph-mon[33544]: 4.1b scrub starts 2024-03-12T03:36:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:16 smithi173 ceph-mon[33544]: 4.1b scrub ok 2024-03-12T03:36:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:16 smithi173 ceph-mon[33544]: 5.5 scrub starts 2024-03-12T03:36:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:16 smithi173 ceph-mon[33544]: 5.5 scrub ok 2024-03-12T03:36:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:16 smithi173 ceph-mon[33544]: 4.8 scrub starts 2024-03-12T03:36:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:16 smithi173 ceph-mon[33544]: 4.8 scrub ok 2024-03-12T03:36:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:16 smithi173 ceph-mon[33544]: 2.1 scrub starts 2024-03-12T03:36:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:16 smithi173 ceph-mon[33544]: 2.1 scrub ok 2024-03-12T03:36:16.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:16 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2422890785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:36:16.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:16 smithi173 ceph-mon[33544]: 3.19 scrub starts 2024-03-12T03:36:16.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:16 smithi173 ceph-mon[33544]: 3.19 scrub ok 2024-03-12T03:36:16.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:16.801 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:16.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:16.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:16.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:16.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:17.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:17.079 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:17.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:17.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:17.263 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:17.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:17.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:17.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:17.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:17.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:17.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:17.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:17.541 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:17.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:17.568 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 46 16 76221 76205 103.518 69.5625 0.00657039 0.00965462 2024-03-12T03:36:17.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:17.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[29344]: pgmap v94: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-12T03:36:17.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[29344]: 5.1b scrub starts 2024-03-12T03:36:17.677 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[29344]: 5.1b scrub ok 2024-03-12T03:36:17.677 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:36:17.677 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[29344]: Cluster is now healthy 2024-03-12T03:36:17.677 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2422890785' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:36:17.677 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[29344]: osdmap e107: 8 total, 8 up, 8 in 2024-03-12T03:36:17.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[39949]: pgmap v94: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-12T03:36:17.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[39949]: 5.1b scrub starts 2024-03-12T03:36:17.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[39949]: 5.1b scrub ok 2024-03-12T03:36:17.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:36:17.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[39949]: Cluster is now healthy 2024-03-12T03:36:17.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2422890785' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:36:17.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:17 smithi169 ceph-mon[39949]: osdmap e107: 8 total, 8 up, 8 in 2024-03-12T03:36:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:17.709 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:17.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:17 smithi173 ceph-mon[33544]: pgmap v94: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-12T03:36:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:17 smithi173 ceph-mon[33544]: 5.1b scrub starts 2024-03-12T03:36:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:17 smithi173 ceph-mon[33544]: 5.1b scrub ok 2024-03-12T03:36:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:17 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:36:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:17 smithi173 ceph-mon[33544]: Cluster is now healthy 2024-03-12T03:36:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:17 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2422890785' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:36:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:17 smithi173 ceph-mon[33544]: osdmap e107: 8 total, 8 up, 8 in 2024-03-12T03:36:17.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:17.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:17.870 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:17.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:17.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:18.018 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:18.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:18.095 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:18.119 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:18.189 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:18.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:18.233 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:18.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:18.327 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:18.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:18.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:18.451 INFO:teuthology.orchestra.run.smithi169.stderr:set pool 7 pgp_num to 16 2024-03-12T03:36:18.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:18.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:18.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:18.569 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 47 15 77237 77222 102.666 63.5625 0.00474159 0.00973499 2024-03-12T03:36:18.601 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:18.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:18.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[29344]: 3.e deep-scrub starts 2024-03-12T03:36:18.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[29344]: 3.e deep-scrub ok 2024-03-12T03:36:18.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[29344]: 3.5 scrub starts 2024-03-12T03:36:18.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[29344]: 3.5 scrub ok 2024-03-12T03:36:18.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/39276859' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-12T03:36:18.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-12T03:36:18.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[29344]: 6.1 scrub starts 2024-03-12T03:36:18.671 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[29344]: 6.1 scrub ok 2024-03-12T03:36:18.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[39949]: 3.e deep-scrub starts 2024-03-12T03:36:18.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[39949]: 3.e deep-scrub ok 2024-03-12T03:36:18.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[39949]: 3.5 scrub starts 2024-03-12T03:36:18.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[39949]: 3.5 scrub ok 2024-03-12T03:36:18.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/39276859' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-12T03:36:18.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-12T03:36:18.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[39949]: 6.1 scrub starts 2024-03-12T03:36:18.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:18 smithi169 ceph-mon[39949]: 6.1 scrub ok 2024-03-12T03:36:18.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:18.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:18.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:18.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:18 smithi173 ceph-mon[33544]: 3.e deep-scrub starts 2024-03-12T03:36:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:18 smithi173 ceph-mon[33544]: 3.e deep-scrub ok 2024-03-12T03:36:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:18 smithi173 ceph-mon[33544]: 3.5 scrub starts 2024-03-12T03:36:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:18 smithi173 ceph-mon[33544]: 3.5 scrub ok 2024-03-12T03:36:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:18 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/39276859' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-12T03:36:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:18 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-12T03:36:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:18 smithi173 ceph-mon[33544]: 6.1 scrub starts 2024-03-12T03:36:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:18 smithi173 ceph-mon[33544]: 6.1 scrub ok 2024-03-12T03:36:18.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:18.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:18.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:18.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:18.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:18.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:18.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:19.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:19.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:19.143 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:19.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:19.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:36:19.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:19.309 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:19.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:19.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:19.412 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:19.455 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:19.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:19.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:19.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:19.570 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 48 16 78379 78363 102.012 71.3125 0.011844 0.00979782 2024-03-12T03:36:19.586 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:19.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:19.688 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:19.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:19.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:19.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: pgmap v96: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-12T03:36:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: 6.10 deep-scrub starts 2024-03-12T03:36:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: 6.10 deep-scrub ok 2024-03-12T03:36:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: 3.1a scrub starts 2024-03-12T03:36:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: 3.1a scrub ok 2024-03-12T03:36:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: 3.1c scrub starts 2024-03-12T03:36:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: 3.1c scrub ok 2024-03-12T03:36:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: 5.b scrub starts 2024-03-12T03:36:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: 5.b scrub ok 2024-03-12T03:36:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-03-12T03:36:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: osdmap e108: 8 total, 8 up, 8 in 2024-03-12T03:36:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: 4.7 scrub starts 2024-03-12T03:36:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:19 smithi173 ceph-mon[33544]: 4.7 scrub ok 2024-03-12T03:36:19.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:19.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:19.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:19.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:19.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: pgmap v96: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-12T03:36:19.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: 6.10 deep-scrub starts 2024-03-12T03:36:19.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: 6.10 deep-scrub ok 2024-03-12T03:36:19.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: 3.1a scrub starts 2024-03-12T03:36:19.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: 3.1a scrub ok 2024-03-12T03:36:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: 3.1c scrub starts 2024-03-12T03:36:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: 3.1c scrub ok 2024-03-12T03:36:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: 5.b scrub starts 2024-03-12T03:36:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: 5.b scrub ok 2024-03-12T03:36:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-03-12T03:36:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: osdmap e108: 8 total, 8 up, 8 in 2024-03-12T03:36:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: 4.7 scrub starts 2024-03-12T03:36:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[29344]: 4.7 scrub ok 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: pgmap v96: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: 6.10 deep-scrub starts 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: 6.10 deep-scrub ok 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: 3.1a scrub starts 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: 3.1a scrub ok 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: 3.1c scrub starts 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: 3.1c scrub ok 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: 5.b scrub starts 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: 5.b scrub ok 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: osdmap e108: 8 total, 8 up, 8 in 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: 4.7 scrub starts 2024-03-12T03:36:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:19 smithi169 ceph-mon[39949]: 4.7 scrub ok 2024-03-12T03:36:19.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:19.991 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:19.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:20.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:20.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:20.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:20.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:20.270 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:20.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:20.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:20.414 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:20.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:20.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:20.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:20.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:20.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:20.569 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 49 16 79581 79565 101.463 75.125 0.00787783 0.00985074 2024-03-12T03:36:20.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:20.684 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:20.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:20.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:20 smithi173 ceph-mon[33544]: 6.3 scrub starts 2024-03-12T03:36:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:20 smithi173 ceph-mon[33544]: 6.3 scrub ok 2024-03-12T03:36:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:20 smithi173 ceph-mon[33544]: 5.f scrub starts 2024-03-12T03:36:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:20 smithi173 ceph-mon[33544]: 5.f scrub ok 2024-03-12T03:36:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:20 smithi173 ceph-mon[33544]: 6.1b scrub starts 2024-03-12T03:36:20.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:20 smithi173 ceph-mon[33544]: 6.1b scrub ok 2024-03-12T03:36:20.785 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:20.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:20.826 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:20.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:20 smithi169 ceph-mon[29344]: 6.3 scrub starts 2024-03-12T03:36:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:20 smithi169 ceph-mon[29344]: 6.3 scrub ok 2024-03-12T03:36:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:20 smithi169 ceph-mon[29344]: 5.f scrub starts 2024-03-12T03:36:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:20 smithi169 ceph-mon[29344]: 5.f scrub ok 2024-03-12T03:36:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:20 smithi169 ceph-mon[29344]: 6.1b scrub starts 2024-03-12T03:36:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:20 smithi169 ceph-mon[29344]: 6.1b scrub ok 2024-03-12T03:36:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:20 smithi169 ceph-mon[39949]: 6.3 scrub starts 2024-03-12T03:36:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:20 smithi169 ceph-mon[39949]: 6.3 scrub ok 2024-03-12T03:36:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:20 smithi169 ceph-mon[39949]: 5.f scrub starts 2024-03-12T03:36:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:20 smithi169 ceph-mon[39949]: 5.f scrub ok 2024-03-12T03:36:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:20 smithi169 ceph-mon[39949]: 6.1b scrub starts 2024-03-12T03:36:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:20 smithi169 ceph-mon[39949]: 6.1b scrub ok 2024-03-12T03:36:20.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:20.968 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:20.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:21.070 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:21.119 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:21.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:21.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:21.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:21.266 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:21.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:21.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:21.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:21.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:21.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:21.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:21.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:21.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:21.570 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 50 16 80754 80738 100.9 73.3125 0.010561 0.00990611 2024-03-12T03:36:21.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: pgmap v98: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 749 op/s 2024-03-12T03:36:21.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:36:21.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:36:21.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: 5.1c scrub starts 2024-03-12T03:36:21.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: 4.1f scrub starts 2024-03-12T03:36:21.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: 4.1f scrub ok 2024-03-12T03:36:21.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: 6.0 scrub starts 2024-03-12T03:36:21.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: 6.0 scrub ok 2024-03-12T03:36:21.586 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: 5.1c scrub ok 2024-03-12T03:36:21.586 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: 6.a deep-scrub starts 2024-03-12T03:36:21.586 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: 6.a deep-scrub ok 2024-03-12T03:36:21.586 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: osdmap e109: 8 total, 8 up, 8 in 2024-03-12T03:36:21.586 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: 3.0 scrub starts 2024-03-12T03:36:21.586 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[29344]: 3.0 scrub ok 2024-03-12T03:36:21.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: pgmap v98: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 749 op/s 2024-03-12T03:36:21.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:36:21.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:36:21.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: 5.1c scrub starts 2024-03-12T03:36:21.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: 4.1f scrub starts 2024-03-12T03:36:21.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: 4.1f scrub ok 2024-03-12T03:36:21.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: 6.0 scrub starts 2024-03-12T03:36:21.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: 6.0 scrub ok 2024-03-12T03:36:21.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: 5.1c scrub ok 2024-03-12T03:36:21.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: 6.a deep-scrub starts 2024-03-12T03:36:21.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: 6.a deep-scrub ok 2024-03-12T03:36:21.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: osdmap e109: 8 total, 8 up, 8 in 2024-03-12T03:36:21.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: 3.0 scrub starts 2024-03-12T03:36:21.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-mon[39949]: 3.0 scrub ok 2024-03-12T03:36:21.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:21.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:21.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: pgmap v98: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 749 op/s 2024-03-12T03:36:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:36:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:36:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: 5.1c scrub starts 2024-03-12T03:36:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: 4.1f scrub starts 2024-03-12T03:36:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: 4.1f scrub ok 2024-03-12T03:36:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: 6.0 scrub starts 2024-03-12T03:36:21.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: 6.0 scrub ok 2024-03-12T03:36:21.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: 5.1c scrub ok 2024-03-12T03:36:21.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: 6.a deep-scrub starts 2024-03-12T03:36:21.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: 6.a deep-scrub ok 2024-03-12T03:36:21.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: osdmap e109: 8 total, 8 up, 8 in 2024-03-12T03:36:21.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: 3.0 scrub starts 2024-03-12T03:36:21.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:21 smithi173 ceph-mon[33544]: 3.0 scrub ok 2024-03-12T03:36:21.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:21.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:21.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:21.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:21.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:36:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:36:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:36:21.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:21.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:21.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:22.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:22.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:22.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:22.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:22.205 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:22.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:22.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:22.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:22.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:22.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:22.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:22.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:22.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:22.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:22.570 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 51 16 81933 81917 100.366 73.6875 0.0190169 0.0099578 2024-03-12T03:36:22.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:22.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:22.705 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:22.734 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:22 smithi169 ceph-mon[29344]: 3.1d scrub starts 2024-03-12T03:36:22.734 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:22 smithi169 ceph-mon[29344]: 3.1d scrub ok 2024-03-12T03:36:22.734 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:22 smithi169 ceph-mon[29344]: 5.2 scrub starts 2024-03-12T03:36:22.734 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:22 smithi169 ceph-mon[29344]: 5.2 scrub ok 2024-03-12T03:36:22.734 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:22 smithi169 ceph-mon[29344]: 6.1d scrub starts 2024-03-12T03:36:22.735 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:22 smithi169 ceph-mon[29344]: 6.1d scrub ok 2024-03-12T03:36:22.735 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:22 smithi169 ceph-mon[39949]: 3.1d scrub starts 2024-03-12T03:36:22.735 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:22 smithi169 ceph-mon[39949]: 3.1d scrub ok 2024-03-12T03:36:22.735 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:22 smithi169 ceph-mon[39949]: 5.2 scrub starts 2024-03-12T03:36:22.735 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:22 smithi169 ceph-mon[39949]: 5.2 scrub ok 2024-03-12T03:36:22.735 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:22 smithi169 ceph-mon[39949]: 6.1d scrub starts 2024-03-12T03:36:22.735 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:22 smithi169 ceph-mon[39949]: 6.1d scrub ok 2024-03-12T03:36:22.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:36:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:36:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:22 smithi173 ceph-mon[33544]: 3.1d scrub starts 2024-03-12T03:36:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:22 smithi173 ceph-mon[33544]: 3.1d scrub ok 2024-03-12T03:36:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:22 smithi173 ceph-mon[33544]: 5.2 scrub starts 2024-03-12T03:36:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:22 smithi173 ceph-mon[33544]: 5.2 scrub ok 2024-03-12T03:36:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:22 smithi173 ceph-mon[33544]: 6.1d scrub starts 2024-03-12T03:36:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:22 smithi173 ceph-mon[33544]: 6.1d scrub ok 2024-03-12T03:36:22.763 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:22.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:22.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:22.910 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:22.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:22.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:23.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:23.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:23.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:23.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:23.182 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:23.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:23.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:23.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:23.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:23.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:23.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:23.461 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:23.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:23.562 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:23.571 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 52 16 83027 83011 99.7498 68.375 0.0136627 0.0100197 2024-03-12T03:36:23.616 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:23.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:23.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:23 smithi173 ceph-mon[33544]: pgmap v100: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-12T03:36:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:23 smithi173 ceph-mon[33544]: 4.3 scrub starts 2024-03-12T03:36:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:23 smithi173 ceph-mon[33544]: 4.3 scrub ok 2024-03-12T03:36:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:23 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:23 smithi173 ceph-mon[33544]: osdmap e110: 8 total, 8 up, 8 in 2024-03-12T03:36:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:23 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:23.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:23.748 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:23 smithi169 ceph-mon[29344]: pgmap v100: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-12T03:36:23.748 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:23 smithi169 ceph-mon[29344]: 4.3 scrub starts 2024-03-12T03:36:23.748 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:23 smithi169 ceph-mon[29344]: 4.3 scrub ok 2024-03-12T03:36:23.748 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:23 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:23.749 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:23 smithi169 ceph-mon[29344]: osdmap e110: 8 total, 8 up, 8 in 2024-03-12T03:36:23.749 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:23 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:23.749 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:23 smithi169 ceph-mon[39949]: pgmap v100: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-12T03:36:23.749 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:23 smithi169 ceph-mon[39949]: 4.3 scrub starts 2024-03-12T03:36:23.749 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:23 smithi169 ceph-mon[39949]: 4.3 scrub ok 2024-03-12T03:36:23.749 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:23 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:23.749 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:23 smithi169 ceph-mon[39949]: osdmap e110: 8 total, 8 up, 8 in 2024-03-12T03:36:23.749 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:23 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:23.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:23.771 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:36:23.837 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:36:23.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:23.868 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:23.887 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:36:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:23.935 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:23.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:23.951 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:36:23.974 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:36:23.974 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:36:23.993 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:36:24.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:24.039 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:36:24.080 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:36:24.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:24.086 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:24.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:24.149 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:36:24.165 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:36:24.165 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:36:24.165 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:36:24.165 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:36:24.165 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:36:24.165 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:36:24.165 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:36:24.165 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:36:24.187 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:24.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:24.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:24.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:24.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:24.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:24.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:24.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:24.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:24.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:24.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:24.571 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 53 16 83924 83908 98.9255 56.0625 0.00820304 0.0101029 2024-03-12T03:36:24.578 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:24.616 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:24.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:24.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:24.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:24 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:24 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/132196088' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:36:24.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:24.758 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:24.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:24.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:24.888 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:24.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:24.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:24 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:24 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/132196088' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:36:24.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:24 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:24.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:24 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/132196088' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:36:24.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:25.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:25.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:25.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:25.169 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:25.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:25.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:25.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:25.298 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:25.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:25.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:25.427 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:25.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:25.474 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:36:25.474 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:36:25.474 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-03-12T03:36:25.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:36:25.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:25.571 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 54 16 84972 84956 98.3065 65.5 0.0093718 0.0101669 2024-03-12T03:36:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:25.574 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:25.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:25.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:25.718 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:25.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:25.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:25.850 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:25.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:25.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:25.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[39949]: pgmap v102: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 59 MiB/s wr, 944 op/s 2024-03-12T03:36:25.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[39949]: 5.7 scrub starts 2024-03-12T03:36:25.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[39949]: 5.7 scrub ok 2024-03-12T03:36:25.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[39949]: 5.1e scrub starts 2024-03-12T03:36:25.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[39949]: 5.1e scrub ok 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[39949]: 4.f scrub starts 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[39949]: 4.f scrub ok 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[29344]: pgmap v102: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 59 MiB/s wr, 944 op/s 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[29344]: 5.7 scrub starts 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[29344]: 5.7 scrub ok 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[29344]: 5.1e scrub starts 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[29344]: 5.1e scrub ok 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[29344]: 4.f scrub starts 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[29344]: 4.f scrub ok 2024-03-12T03:36:25.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:25.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:25.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:25 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:25.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:25.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:25.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:25 smithi173 ceph-mon[33544]: pgmap v102: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 59 MiB/s wr, 944 op/s 2024-03-12T03:36:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:25 smithi173 ceph-mon[33544]: 5.7 scrub starts 2024-03-12T03:36:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:25 smithi173 ceph-mon[33544]: 5.7 scrub ok 2024-03-12T03:36:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:25 smithi173 ceph-mon[33544]: 5.1e scrub starts 2024-03-12T03:36:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:25 smithi173 ceph-mon[33544]: 5.1e scrub ok 2024-03-12T03:36:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:25 smithi173 ceph-mon[33544]: 4.f scrub starts 2024-03-12T03:36:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:25 smithi173 ceph-mon[33544]: 4.f scrub ok 2024-03-12T03:36:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:25 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:25 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:25.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:25 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:36:26.079 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:26.113 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:26.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:26.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:26.248 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:26.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:26.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:26.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:26.392 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:26.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:26.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:26.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:26.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:26.571 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 55 16 85921 85905 97.5975 59.3125 0.0135143 0.0102411 2024-03-12T03:36:26.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:26.669 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:26.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:26.770 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:26.809 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:26.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:26.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[39949]: 6.15 scrub starts 2024-03-12T03:36:26.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[39949]: 6.15 scrub ok 2024-03-12T03:36:26.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[39949]: 3.18 scrub starts 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[39949]: 3.18 scrub ok 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[39949]: 6.7 scrub starts 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[39949]: 6.7 scrub ok 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[39949]: 5.d scrub starts 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[39949]: 5.d scrub ok 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[29344]: 6.15 scrub starts 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[29344]: 6.15 scrub ok 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[29344]: 3.18 scrub starts 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[29344]: 3.18 scrub ok 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[29344]: 6.7 scrub starts 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[29344]: 6.7 scrub ok 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[29344]: 5.d scrub starts 2024-03-12T03:36:26.847 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:26 smithi169 ceph-mon[29344]: 5.d scrub ok 2024-03-12T03:36:26.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:26.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:26.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:26.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:26 smithi173 ceph-mon[33544]: 6.15 scrub starts 2024-03-12T03:36:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:26 smithi173 ceph-mon[33544]: 6.15 scrub ok 2024-03-12T03:36:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:26 smithi173 ceph-mon[33544]: 3.18 scrub starts 2024-03-12T03:36:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:26 smithi173 ceph-mon[33544]: 3.18 scrub ok 2024-03-12T03:36:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:26 smithi173 ceph-mon[33544]: 6.7 scrub starts 2024-03-12T03:36:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:26 smithi173 ceph-mon[33544]: 6.7 scrub ok 2024-03-12T03:36:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:26 smithi173 ceph-mon[33544]: 5.d scrub starts 2024-03-12T03:36:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:26 smithi173 ceph-mon[33544]: 5.d scrub ok 2024-03-12T03:36:27.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:27.084 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:27.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:27.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:27.220 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:27.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:27.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:27.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:27.360 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:27.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:27.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:27.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:27.497 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:27.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:27.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:27.571 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 56 15 87066 87051 97.1337 71.625 0.0135315 0.0102901 2024-03-12T03:36:27.600 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:27.647 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:27.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:27.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:27.793 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[29344]: pgmap v103: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-12T03:36:27.793 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[29344]: 6.18 scrub starts 2024-03-12T03:36:27.793 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[29344]: 6.18 scrub ok 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[29344]: 3.c scrub starts 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[29344]: 3.c scrub ok 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[29344]: 3.14 scrub starts 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[29344]: 3.14 scrub ok 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[29344]: 5.8 scrub starts 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[29344]: 5.8 scrub ok 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[39949]: pgmap v103: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[39949]: 6.18 scrub starts 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[39949]: 6.18 scrub ok 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[39949]: 3.c scrub starts 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[39949]: 3.c scrub ok 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[39949]: 3.14 scrub starts 2024-03-12T03:36:27.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[39949]: 3.14 scrub ok 2024-03-12T03:36:27.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[39949]: 5.8 scrub starts 2024-03-12T03:36:27.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:27 smithi169 ceph-mon[39949]: 5.8 scrub ok 2024-03-12T03:36:27.796 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:27.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:27.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-12T03:36:27.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:27.943 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:27.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:27.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:27 smithi173 ceph-mon[33544]: pgmap v103: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-12T03:36:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:27 smithi173 ceph-mon[33544]: 6.18 scrub starts 2024-03-12T03:36:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:27 smithi173 ceph-mon[33544]: 6.18 scrub ok 2024-03-12T03:36:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:27 smithi173 ceph-mon[33544]: 3.c scrub starts 2024-03-12T03:36:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:27 smithi173 ceph-mon[33544]: 3.c scrub ok 2024-03-12T03:36:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:27 smithi173 ceph-mon[33544]: 3.14 scrub starts 2024-03-12T03:36:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:27 smithi173 ceph-mon[33544]: 3.14 scrub ok 2024-03-12T03:36:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:27 smithi173 ceph-mon[33544]: 5.8 scrub starts 2024-03-12T03:36:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:27 smithi173 ceph-mon[33544]: 5.8 scrub ok 2024-03-12T03:36:28.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:28.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:28.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:28.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:28.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:28.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:28.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:28.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:36:28.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:28.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:28.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:28.394 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:28.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:28.490 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:28.518 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:28.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:28.571 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 57 15 87979 87964 96.4306 57.0625 0.0212313 0.0103646 2024-03-12T03:36:28.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:28.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:28 smithi169 ceph-mon[29344]: 4.a scrub starts 2024-03-12T03:36:28.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:28 smithi169 ceph-mon[29344]: 4.a scrub ok 2024-03-12T03:36:28.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:28 smithi169 ceph-mon[29344]: osdmap e111: 8 total, 8 up, 8 in 2024-03-12T03:36:28.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:28 smithi169 ceph-mon[39949]: 4.a scrub starts 2024-03-12T03:36:28.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:28 smithi169 ceph-mon[39949]: 4.a scrub ok 2024-03-12T03:36:28.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:28 smithi169 ceph-mon[39949]: osdmap e111: 8 total, 8 up, 8 in 2024-03-12T03:36:28.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:28.675 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:28.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:28.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:28.837 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:28.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:28.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:28.978 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:28.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:28.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:28 smithi173 ceph-mon[33544]: 4.a scrub starts 2024-03-12T03:36:28.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:28 smithi173 ceph-mon[33544]: 4.a scrub ok 2024-03-12T03:36:28.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:28 smithi173 ceph-mon[33544]: osdmap e111: 8 total, 8 up, 8 in 2024-03-12T03:36:29.079 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:29.115 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:29.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:29.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:29.255 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:29.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:29.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:29.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:29.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:29.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:29.529 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:29.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:29.551 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[29344]: pgmap v104: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[29344]: 3.13 scrub starts 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[29344]: 3.13 scrub ok 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[29344]: 4.1d scrub starts 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[29344]: 4.1d scrub ok 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[29344]: 3.1f deep-scrub starts 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[29344]: 3.1f deep-scrub ok 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/436228726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[39949]: pgmap v104: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[39949]: 3.13 scrub starts 2024-03-12T03:36:29.559 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[39949]: 3.13 scrub ok 2024-03-12T03:36:29.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[39949]: 4.1d scrub starts 2024-03-12T03:36:29.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[39949]: 4.1d scrub ok 2024-03-12T03:36:29.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[39949]: 3.1f deep-scrub starts 2024-03-12T03:36:29.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[39949]: 3.1f deep-scrub ok 2024-03-12T03:36:29.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:36:29.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:36:29.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:29 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/436228726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:36:29.571 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 58 16 88969 88953 95.8338 61.8125 0.00761491 0.010429 2024-03-12T03:36:29.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:29.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:29.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:29.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:29.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:29.830 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:29.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:29.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:29.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:29 smithi173 ceph-mon[33544]: pgmap v104: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-12T03:36:29.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:29 smithi173 ceph-mon[33544]: 3.13 scrub starts 2024-03-12T03:36:29.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:29 smithi173 ceph-mon[33544]: 3.13 scrub ok 2024-03-12T03:36:29.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:29 smithi173 ceph-mon[33544]: 4.1d scrub starts 2024-03-12T03:36:29.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:29 smithi173 ceph-mon[33544]: 4.1d scrub ok 2024-03-12T03:36:29.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:29 smithi173 ceph-mon[33544]: 3.1f deep-scrub starts 2024-03-12T03:36:29.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:29 smithi173 ceph-mon[33544]: 3.1f deep-scrub ok 2024-03-12T03:36:29.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:29 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:36:29.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:29 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:36:29.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:29 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/436228726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:36:29.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:29.972 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:29.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:30.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:30.100 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:30.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:30.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:30.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:30.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:30.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:30.332 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:30.370 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:30.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:30.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:30.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:30.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:30.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:30.571 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 59 16 90090 90074 95.3969 70.0625 0.00887974 0.0104776 2024-03-12T03:36:30.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:30.645 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:30.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:30.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:30.786 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:30.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:30.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:30.916 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:30.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:30.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[39949]: 4.b deep-scrub starts 2024-03-12T03:36:30.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[39949]: 5.16 scrub starts 2024-03-12T03:36:30.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[39949]: 5.16 scrub ok 2024-03-12T03:36:30.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[39949]: 4.b deep-scrub ok 2024-03-12T03:36:30.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:36:30.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/436228726' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:36:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[39949]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:36:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:30.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[29344]: 4.b deep-scrub starts 2024-03-12T03:36:30.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[29344]: 5.16 scrub starts 2024-03-12T03:36:30.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[29344]: 5.16 scrub ok 2024-03-12T03:36:30.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[29344]: 4.b deep-scrub ok 2024-03-12T03:36:30.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:36:30.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/436228726' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:36:30.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[29344]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:36:30.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:30 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:30 smithi173 ceph-mon[33544]: 4.b deep-scrub starts 2024-03-12T03:36:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:30 smithi173 ceph-mon[33544]: 5.16 scrub starts 2024-03-12T03:36:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:30 smithi173 ceph-mon[33544]: 5.16 scrub ok 2024-03-12T03:36:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:30 smithi173 ceph-mon[33544]: 4.b deep-scrub ok 2024-03-12T03:36:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:30 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:36:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:30 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/436228726' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:36:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:30 smithi173 ceph-mon[33544]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:36:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:30 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:31.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:31.062 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:31.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:31.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:31.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:31.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:31.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:31.335 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:31.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:31.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:31.485 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:31.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:31.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:31.572 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:36:31.570939+0000 min lat: 0.00118901 max lat: 0.163567 avg lat: 0.0105258 2024-03-12T03:36:31.572 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:36:31.572 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 60 16 91196 91180 94.9584 69.125 0.0203404 0.0105258 2024-03-12T03:36:31.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:31.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:31.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:31.718 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:31.757 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:31.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:31.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:31 smithi169 ceph-mon[29344]: pgmap v107: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 771 B/s rd, 50 MiB/s wr, 800 op/s 2024-03-12T03:36:31.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:31 smithi169 ceph-mon[39949]: pgmap v107: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 771 B/s rd, 50 MiB/s wr, 800 op/s 2024-03-12T03:36:31.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:31.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:31.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:31 smithi173 ceph-mon[33544]: pgmap v107: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 771 B/s rd, 50 MiB/s wr, 800 op/s 2024-03-12T03:36:31.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:32.048 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:32.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:32.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:32.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:36:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:36:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:36:32.201 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:32.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:32.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:32.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:32.343 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:32.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:32.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:32.487 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:32.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:32.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:32.572 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 61 16 92266 92250 94.4979 66.875 0.0210645 0.0105772 2024-03-12T03:36:32.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:32.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:32.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:32.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:32.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:32.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:36:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:36:32.757 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:32.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:32.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:32.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:32.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:32.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:33.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:36:33.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:33.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:33.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:33.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:33.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:33.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:33.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:33.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:33.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:33.432 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:33.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:33.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:33.575 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 62 16 93272 93256 93.9841 62.875 0.0111902 0.0106333 2024-03-12T03:36:33.591 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:33.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:33.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:33.693 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:33.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:33.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:33 smithi169 ceph-mon[29344]: pgmap v108: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-12T03:36:33.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:33 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:33.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:33 smithi169 ceph-mon[29344]: osdmap e113: 8 total, 8 up, 8 in 2024-03-12T03:36:33.784 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:33 smithi169 ceph-mon[39949]: pgmap v108: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-12T03:36:33.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:33 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:33.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:33 smithi169 ceph-mon[39949]: osdmap e113: 8 total, 8 up, 8 in 2024-03-12T03:36:33.804 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:33.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:33.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:33.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:33.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:33.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:33 smithi173 ceph-mon[33544]: pgmap v108: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-12T03:36:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:33 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:33 smithi173 ceph-mon[33544]: osdmap e113: 8 total, 8 up, 8 in 2024-03-12T03:36:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:34.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:34.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:34.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:34.135 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:34.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:34.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:34.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:34.266 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:36:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:34.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:34.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:34.276 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:36:34.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:34.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:34.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:34.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:34.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:34.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:34.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:34.574 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 63 16 94188 94172 93.401 57.25 0.012909 0.0107004 2024-03-12T03:36:34.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:34.677 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:34.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:34.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:34.816 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:34.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:34.917 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:34.960 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:34.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:35.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:35.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:35.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:35.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:35.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:35.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:35.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:35.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:35.357 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:35.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:35.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:35.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:35.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:35.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:35.575 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 64 16 95136 95120 92.8674 59.25 0.0155903 0.0107623 2024-03-12T03:36:35.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:35.627 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:35.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:35.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:35.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:35.769 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:35.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:35.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:35.912 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:35.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:35.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:35 smithi169 ceph-mon[29344]: pgmap v110: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 864 op/s 2024-03-12T03:36:35.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:35 smithi169 ceph-mon[29344]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:36:35.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:35 smithi169 ceph-mon[39949]: pgmap v110: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 864 op/s 2024-03-12T03:36:35.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:35 smithi169 ceph-mon[39949]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:36:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:35 smithi173 ceph-mon[33544]: pgmap v110: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 864 op/s 2024-03-12T03:36:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:35 smithi173 ceph-mon[33544]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:36:36.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:36.056 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:36.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:36.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:36.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:36.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:36.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:36.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:36.318 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:36.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:36.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:36.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:36.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:36.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:36.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:36.575 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 65 16 96061 96045 92.3282 57.8125 0.0144619 0.0108254 2024-03-12T03:36:36.603 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:36.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:36.704 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:36.750 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:36.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:36.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:36.889 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:36.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:36.990 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:37.015 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:37.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:37.117 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:37.169 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:37.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:37.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:37.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:37.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:37.435 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:37.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:37.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:37.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:37.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:37.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:37.575 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 66 16 97097 97081 91.9103 64.75 0.00699383 0.0108756 2024-03-12T03:36:37.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:37.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:37 smithi169 ceph-mon[29344]: pgmap v111: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 58 MiB/s wr, 922 op/s 2024-03-12T03:36:37.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:37 smithi169 ceph-mon[39949]: pgmap v111: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 58 MiB/s wr, 922 op/s 2024-03-12T03:36:37.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:37.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:37.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:37.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:37.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:37.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:37.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:37.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:37 smithi173 ceph-mon[33544]: pgmap v111: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 58 MiB/s wr, 922 op/s 2024-03-12T03:36:38.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:38.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:38.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:38.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:38.153 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:38.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:38.169 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-03-12T03:36:33.808481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5923281320,"num_objects":90603,"num_object_clones":0,"num_object_copies":271809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90603,"num_whiteouts":0,"num_read":1406,"num_read_kb":1220,"num_write":91180,"num_write_kb":5786333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86511,"ondisk_log_size":86511,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17723576,"kb_used_data":17214556,"kb_used_omap":12,"kb_used_meta":508915,"kb_avail":732237640,"statfs":{"total":767960285184,"available":749811343360,"internally_reserved":0,"allocated":17627705344,"data_stored":17624239712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":521129520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":339804160,"num_objects":5185,"num_object_clones":0,"num_object_copies":15555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5185,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5185,"num_write_kb":331840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002655"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.546648+0000","last_change":"2024-03-12T03:35:21.106867+0000","last_active":"2024-03-12T03:36:29.546648+0000","last_peered":"2024-03-12T03:36:29.546648+0000","last_clean":"2024-03-12T03:36:29.546648+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:36:29.546648+0000","last_undegraded":"2024-03-12T03:36:29.546648+0000","last_fullsized":"2024-03-12T03:36:29.546648+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:21.106755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:21.106755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.554321+0000","last_change":"2024-03-12T03:36:02.862199+0000","last_active":"2024-03-12T03:36:29.554321+0000","last_peered":"2024-03-12T03:36:29.554321+0000","last_clean":"2024-03-12T03:36:29.554321+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:36:29.554321+0000","last_undegraded":"2024-03-12T03:36:29.554321+0000","last_fullsized":"2024-03-12T03:36:29.554321+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:02.861854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:02.861854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:07.436631+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.562449+0000","last_change":"2024-03-12T03:36:28.208156+0000","last_active":"2024-03-12T03:36:29.562449+0000","last_peered":"2024-03-12T03:36:29.562449+0000","last_clean":"2024-03-12T03:36:29.562449+0000","last_became_active":"2024-03-12T03:33:21.573080+0000","last_became_peered":"2024-03-12T03:33:21.573080+0000","last_unstale":"2024-03-12T03:36:29.562449+0000","last_undegraded":"2024-03-12T03:36:29.562449+0000","last_fullsized":"2024-03-12T03:36:29.562449+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:28.208044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:28.208044+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.208044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:40.375427+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":134,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547445+0000","last_change":"2024-03-12T03:36:02.990432+0000","last_active":"2024-03-12T03:36:29.547445+0000","last_peered":"2024-03-12T03:36:29.547445+0000","last_clean":"2024-03-12T03:36:29.547445+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:36:29.547445+0000","last_undegraded":"2024-03-12T03:36:29.547445+0000","last_fullsized":"2024-03-12T03:36:29.547445+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:36:02.990324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:02.990324+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:22.888530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.552999+0000","last_change":"2024-03-12T03:36:19.237362+0000","last_active":"2024-03-12T03:36:29.552999+0000","last_peered":"2024-03-12T03:36:29.552999+0000","last_clean":"2024-03-12T03:36:29.552999+0000","last_became_active":"2024-03-12T03:33:27.382050+0000","last_became_peered":"2024-03-12T03:33:27.382050+0000","last_unstale":"2024-03-12T03:36:29.552999+0000","last_undegraded":"2024-03-12T03:36:29.552999+0000","last_fullsized":"2024-03-12T03:36:29.552999+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.236997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.236997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:39.350923+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547612+0000","last_change":"2024-03-12T03:35:17.100474+0000","last_active":"2024-03-12T03:36:29.547612+0000","last_peered":"2024-03-12T03:36:29.547612+0000","last_clean":"2024-03-12T03:36:29.547612+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:36:29.547612+0000","last_undegraded":"2024-03-12T03:36:29.547612+0000","last_fullsized":"2024-03-12T03:36:29.547612+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:17.100375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:17.100375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:39.744648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556099+0000","last_change":"2024-03-12T03:34:41.143075+0000","last_active":"2024-03-12T03:36:29.556099+0000","last_peered":"2024-03-12T03:36:29.556099+0000","last_clean":"2024-03-12T03:36:29.556099+0000","last_became_active":"2024-03-12T03:33:21.572876+0000","last_became_peered":"2024-03-12T03:33:21.572876+0000","last_unstale":"2024-03-12T03:36:29.556099+0000","last_undegraded":"2024-03-12T03:36:29.556099+0000","last_fullsized":"2024-03-12T03:36:29.556099+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.142994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"94'50","reported_seq":190,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.552939+0000","last_change":"2024-03-12T03:35:05.042142+0000","last_active":"2024-03-12T03:36:29.552939+0000","last_peered":"2024-03-12T03:36:29.552939+0000","last_clean":"2024-03-12T03:36:29.552939+0000","last_became_active":"2024-03-12T03:33:23.161549+0000","last_became_peered":"2024-03-12T03:33:23.161549+0000","last_unstale":"2024-03-12T03:36:29.552939+0000","last_undegraded":"2024-03-12T03:36:29.552939+0000","last_fullsized":"2024-03-12T03:36:29.552939+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.042030+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.555936+0000","last_change":"2024-03-12T03:36:26.107553+0000","last_active":"2024-03-12T03:36:29.555936+0000","last_peered":"2024-03-12T03:36:29.555936+0000","last_clean":"2024-03-12T03:36:29.555936+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:36:29.555936+0000","last_undegraded":"2024-03-12T03:36:29.555936+0000","last_fullsized":"2024-03-12T03:36:29.555936+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.107439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.107439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:55.895456+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.546952+0000","last_change":"2024-03-12T03:36:16.187277+0000","last_active":"2024-03-12T03:36:29.546952+0000","last_peered":"2024-03-12T03:36:29.546952+0000","last_clean":"2024-03-12T03:36:29.546952+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:36:29.546952+0000","last_undegraded":"2024-03-12T03:36:29.546952+0000","last_fullsized":"2024-03-12T03:36:29.546952+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:16.187145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:16.187145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:45.580351+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547760+0000","last_change":"2024-03-12T03:36:21.101794+0000","last_active":"2024-03-12T03:36:29.547760+0000","last_peered":"2024-03-12T03:36:29.547760+0000","last_clean":"2024-03-12T03:36:29.547760+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:36:29.547760+0000","last_undegraded":"2024-03-12T03:36:29.547760+0000","last_fullsized":"2024-03-12T03:36:29.547760+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:21.101693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:36:21.101693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:43.795511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'21","reported_seq":146,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.546884+0000","last_change":"2024-03-12T03:35:19.084309+0000","last_active":"2024-03-12T03:36:29.546884+0000","last_peered":"2024-03-12T03:36:29.546884+0000","last_clean":"2024-03-12T03:36:29.546884+0000","last_became_active":"2024-03-12T03:33:23.162749+0000","last_became_peered":"2024-03-12T03:33:23.162749+0000","last_unstale":"2024-03-12T03:36:29.546884+0000","last_undegraded":"2024-03-12T03:36:29.546884+0000","last_fullsized":"2024-03-12T03:36:29.546884+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:35:19.084191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:19.084191+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.546273+0000","last_change":"2024-03-12T03:35:05.181569+0000","last_active":"2024-03-12T03:36:29.546273+0000","last_peered":"2024-03-12T03:36:29.546273+0000","last_clean":"2024-03-12T03:36:29.546273+0000","last_became_active":"2024-03-12T03:33:27.187943+0000","last_became_peered":"2024-03-12T03:33:27.187943+0000","last_unstale":"2024-03-12T03:36:29.546273+0000","last_undegraded":"2024-03-12T03:36:29.546273+0000","last_fullsized":"2024-03-12T03:36:29.546273+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:05.181454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.181454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:33.321570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560767+0000","last_change":"2024-03-12T03:35:11.909632+0000","last_active":"2024-03-12T03:36:29.560767+0000","last_peered":"2024-03-12T03:36:29.560767+0000","last_clean":"2024-03-12T03:36:29.560767+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:36:29.560767+0000","last_undegraded":"2024-03-12T03:36:29.560767+0000","last_fullsized":"2024-03-12T03:36:29.560767+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:11.909537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.909537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547419+0000","last_change":"2024-03-12T03:36:18.202204+0000","last_active":"2024-03-12T03:36:29.547419+0000","last_peered":"2024-03-12T03:36:29.547419+0000","last_clean":"2024-03-12T03:36:29.547419+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:36:29.547419+0000","last_undegraded":"2024-03-12T03:36:29.547419+0000","last_fullsized":"2024-03-12T03:36:29.547419+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.202082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.202082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:32.588876+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"86'14","reported_seq":153,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.548074+0000","last_change":"2024-03-12T03:36:15.113589+0000","last_active":"2024-03-12T03:36:29.548074+0000","last_peered":"2024-03-12T03:36:29.548074+0000","last_clean":"2024-03-12T03:36:29.548074+0000","last_became_active":"2024-03-12T03:33:50.155908+0000","last_became_peered":"2024-03-12T03:33:50.155908+0000","last_unstale":"2024-03-12T03:36:29.548074+0000","last_undegraded":"2024-03-12T03:36:29.548074+0000","last_fullsized":"2024-03-12T03:36:29.548074+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:36:15.113450+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.113450+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:18.278802+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547690+0000","last_change":"2024-03-12T03:36:01.082667+0000","last_active":"2024-03-12T03:36:29.547690+0000","last_peered":"2024-03-12T03:36:29.547690+0000","last_clean":"2024-03-12T03:36:29.547690+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:36:29.547690+0000","last_undegraded":"2024-03-12T03:36:29.547690+0000","last_fullsized":"2024-03-12T03:36:29.547690+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:01.082531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:01.082531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:14.778425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.554087+0000","last_change":"2024-03-12T03:36:09.003751+0000","last_active":"2024-03-12T03:36:29.554087+0000","last_peered":"2024-03-12T03:36:29.554087+0000","last_clean":"2024-03-12T03:36:29.554087+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:36:29.554087+0000","last_undegraded":"2024-03-12T03:36:29.554087+0000","last_fullsized":"2024-03-12T03:36:29.554087+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:09.003476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:09.003476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:15.330848+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556496+0000","last_change":"2024-03-12T03:36:08.094593+0000","last_active":"2024-03-12T03:36:29.556496+0000","last_peered":"2024-03-12T03:36:29.556496+0000","last_clean":"2024-03-12T03:36:29.556496+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:36:29.556496+0000","last_undegraded":"2024-03-12T03:36:29.556496+0000","last_fullsized":"2024-03-12T03:36:29.556496+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:08.094475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:08.094475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:22.969182+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":131,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.568629+0000","last_change":"2024-03-12T03:34:58.889062+0000","last_active":"2024-03-12T03:36:29.568629+0000","last_peered":"2024-03-12T03:36:29.568629+0000","last_clean":"2024-03-12T03:36:29.568629+0000","last_became_active":"2024-03-12T03:33:23.160702+0000","last_became_peered":"2024-03-12T03:33:23.160702+0000","last_unstale":"2024-03-12T03:36:29.568629+0000","last_undegraded":"2024-03-12T03:36:29.568629+0000","last_fullsized":"2024-03-12T03:36:29.568629+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:34:58.888971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.888971+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.559255+0000","last_change":"2024-03-12T03:33:27.381440+0000","last_active":"2024-03-12T03:36:29.559255+0000","last_peered":"2024-03-12T03:36:29.559255+0000","last_clean":"2024-03-12T03:36:29.559255+0000","last_became_active":"2024-03-12T03:33:27.380880+0000","last_became_peered":"2024-03-12T03:33:27.380880+0000","last_unstale":"2024-03-12T03:36:29.559255+0000","last_undegraded":"2024-03-12T03:36:29.559255+0000","last_fullsized":"2024-03-12T03:36:29.559255+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547703+0000","last_change":"2024-03-12T03:36:20.142515+0000","last_active":"2024-03-12T03:36:29.547703+0000","last_peered":"2024-03-12T03:36:29.547703+0000","last_clean":"2024-03-12T03:36:29.547703+0000","last_became_active":"2024-03-12T03:33:25.181087+0000","last_became_peered":"2024-03-12T03:33:25.181087+0000","last_unstale":"2024-03-12T03:36:29.547703+0000","last_undegraded":"2024-03-12T03:36:29.547703+0000","last_fullsized":"2024-03-12T03:36:29.547703+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:20.142437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.142437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:49.273040+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.546982+0000","last_change":"2024-03-12T03:36:18.123144+0000","last_active":"2024-03-12T03:36:29.546982+0000","last_peered":"2024-03-12T03:36:29.546982+0000","last_clean":"2024-03-12T03:36:29.546982+0000","last_became_active":"2024-03-12T03:33:21.573218+0000","last_became_peered":"2024-03-12T03:33:21.573218+0000","last_unstale":"2024-03-12T03:36:29.546982+0000","last_undegraded":"2024-03-12T03:36:29.546982+0000","last_fullsized":"2024-03-12T03:36:29.546982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.123035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.123035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.552439+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":141,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.550965+0000","last_change":"2024-03-12T03:36:28.133412+0000","last_active":"2024-03-12T03:36:29.550965+0000","last_peered":"2024-03-12T03:36:29.550965+0000","last_clean":"2024-03-12T03:36:29.550965+0000","last_became_active":"2024-03-12T03:33:23.162427+0000","last_became_peered":"2024-03-12T03:33:23.162427+0000","last_unstale":"2024-03-12T03:36:29.550965+0000","last_undegraded":"2024-03-12T03:36:29.550965+0000","last_fullsized":"2024-03-12T03:36:29.550965+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:36:28.133261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.133261+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:51.305356+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.561201+0000","last_change":"2024-03-12T03:35:57.051016+0000","last_active":"2024-03-12T03:36:29.561201+0000","last_peered":"2024-03-12T03:36:29.561201+0000","last_clean":"2024-03-12T03:36:29.561201+0000","last_became_active":"2024-03-12T03:33:27.189152+0000","last_became_peered":"2024-03-12T03:33:27.189152+0000","last_unstale":"2024-03-12T03:36:29.561201+0000","last_undegraded":"2024-03-12T03:36:29.561201+0000","last_fullsized":"2024-03-12T03:36:29.561201+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:35:57.050605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:57.050605+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:06.936509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":159,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556638+0000","last_change":"2024-03-12T03:36:00.063359+0000","last_active":"2024-03-12T03:36:29.556638+0000","last_peered":"2024-03-12T03:36:29.556638+0000","last_clean":"2024-03-12T03:36:29.556638+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:36:29.556638+0000","last_undegraded":"2024-03-12T03:36:29.556638+0000","last_fullsized":"2024-03-12T03:36:29.556638+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:00.063228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:00.063228+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:17.858770+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.554400+0000","last_change":"2024-03-12T03:36:16.066119+0000","last_active":"2024-03-12T03:36:29.554400+0000","last_peered":"2024-03-12T03:36:29.554400+0000","last_clean":"2024-03-12T03:36:29.554400+0000","last_became_active":"2024-03-12T03:33:21.097218+0000","last_became_peered":"2024-03-12T03:33:21.097218+0000","last_unstale":"2024-03-12T03:36:29.554400+0000","last_undegraded":"2024-03-12T03:36:29.554400+0000","last_fullsized":"2024-03-12T03:36:29.554400+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:16.066002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:16.066002+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:39.793679+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'17","reported_seq":142,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560489+0000","last_change":"2024-03-12T03:36:12.216521+0000","last_active":"2024-03-12T03:36:29.560489+0000","last_peered":"2024-03-12T03:36:29.560489+0000","last_clean":"2024-03-12T03:36:29.560489+0000","last_became_active":"2024-03-12T03:33:23.552812+0000","last_became_peered":"2024-03-12T03:33:23.552812+0000","last_unstale":"2024-03-12T03:36:29.560489+0000","last_undegraded":"2024-03-12T03:36:29.560489+0000","last_fullsized":"2024-03-12T03:36:29.560489+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:36:12.216269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:12.216269+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:27.130146+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.554448+0000","last_change":"2024-03-12T03:36:22.163176+0000","last_active":"2024-03-12T03:36:29.554448+0000","last_peered":"2024-03-12T03:36:29.554448+0000","last_clean":"2024-03-12T03:36:29.554448+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:36:29.554448+0000","last_undegraded":"2024-03-12T03:36:29.554448+0000","last_fullsized":"2024-03-12T03:36:29.554448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:22.163050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:22.163050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.403724+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560757+0000","last_change":"2024-03-12T03:36:24.163956+0000","last_active":"2024-03-12T03:36:29.560757+0000","last_peered":"2024-03-12T03:36:29.560757+0000","last_clean":"2024-03-12T03:36:29.560757+0000","last_became_active":"2024-03-12T03:33:25.178346+0000","last_became_peered":"2024-03-12T03:33:25.178346+0000","last_unstale":"2024-03-12T03:36:29.560757+0000","last_undegraded":"2024-03-12T03:36:29.560757+0000","last_fullsized":"2024-03-12T03:36:29.560757+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:24.162965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.162965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:35.015647+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":116,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.551382+0000","last_change":"2024-03-12T03:36:25.092455+0000","last_active":"2024-03-12T03:36:29.551382+0000","last_peered":"2024-03-12T03:36:29.551382+0000","last_clean":"2024-03-12T03:36:29.551382+0000","last_became_active":"2024-03-12T03:33:21.095556+0000","last_became_peered":"2024-03-12T03:33:21.095556+0000","last_unstale":"2024-03-12T03:36:29.551382+0000","last_undegraded":"2024-03-12T03:36:29.551382+0000","last_fullsized":"2024-03-12T03:36:29.551382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:25.092313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.092313+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.705657+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"94'23","reported_seq":151,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556627+0000","last_change":"2024-03-12T03:36:20.132927+0000","last_active":"2024-03-12T03:36:29.556627+0000","last_peered":"2024-03-12T03:36:29.556627+0000","last_clean":"2024-03-12T03:36:29.556627+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:36:29.556627+0000","last_undegraded":"2024-03-12T03:36:29.556627+0000","last_fullsized":"2024-03-12T03:36:29.556627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'23","last_scrub_stamp":"2024-03-12T03:36:20.132683+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:34:52.729983+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.132683+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:42.640904+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560648+0000","last_change":"2024-03-12T03:35:15.855752+0000","last_active":"2024-03-12T03:36:29.560648+0000","last_peered":"2024-03-12T03:36:29.560648+0000","last_clean":"2024-03-12T03:36:29.560648+0000","last_became_active":"2024-03-12T03:33:27.189696+0000","last_became_peered":"2024-03-12T03:33:27.189696+0000","last_unstale":"2024-03-12T03:36:29.560648+0000","last_undegraded":"2024-03-12T03:36:29.560648+0000","last_fullsized":"2024-03-12T03:36:29.560648+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.855641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.855641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:31.748344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556163+0000","last_change":"2024-03-12T03:36:01.098836+0000","last_active":"2024-03-12T03:36:29.556163+0000","last_peered":"2024-03-12T03:36:29.556163+0000","last_clean":"2024-03-12T03:36:29.556163+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:36:29.556163+0000","last_undegraded":"2024-03-12T03:36:29.556163+0000","last_fullsized":"2024-03-12T03:36:29.556163+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:01.098726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:01.098726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:05.633644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.562911+0000","last_change":"2024-03-12T03:35:59.076304+0000","last_active":"2024-03-12T03:36:29.562911+0000","last_peered":"2024-03-12T03:36:29.562911+0000","last_clean":"2024-03-12T03:36:29.562911+0000","last_became_active":"2024-03-12T03:33:21.096560+0000","last_became_peered":"2024-03-12T03:33:21.096560+0000","last_unstale":"2024-03-12T03:36:29.562911+0000","last_undegraded":"2024-03-12T03:36:29.562911+0000","last_fullsized":"2024-03-12T03:36:29.562911+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.076203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.076203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:18.990073+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"90'6","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.551482+0000","last_change":"2024-03-12T03:34:34.274478+0000","last_active":"2024-03-12T03:36:29.551482+0000","last_peered":"2024-03-12T03:36:29.551482+0000","last_clean":"2024-03-12T03:36:29.551482+0000","last_became_active":"2024-03-12T03:33:23.161801+0000","last_became_peered":"2024-03-12T03:33:23.161801+0000","last_unstale":"2024-03-12T03:36:29.551482+0000","last_undegraded":"2024-03-12T03:36:29.551482+0000","last_fullsized":"2024-03-12T03:36:29.551482+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.274353+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.553261+0000","last_change":"2024-03-12T03:34:40.191618+0000","last_active":"2024-03-12T03:36:29.553261+0000","last_peered":"2024-03-12T03:36:29.553261+0000","last_clean":"2024-03-12T03:36:29.553261+0000","last_became_active":"2024-03-12T03:33:27.381110+0000","last_became_peered":"2024-03-12T03:33:27.381110+0000","last_unstale":"2024-03-12T03:36:29.553261+0000","last_undegraded":"2024-03-12T03:36:29.553261+0000","last_fullsized":"2024-03-12T03:36:29.553261+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.191538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560837+0000","last_change":"2024-03-12T03:34:53.878589+0000","last_active":"2024-03-12T03:36:29.560837+0000","last_peered":"2024-03-12T03:36:29.560837+0000","last_clean":"2024-03-12T03:36:29.560837+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:36:29.560837+0000","last_undegraded":"2024-03-12T03:36:29.560837+0000","last_fullsized":"2024-03-12T03:36:29.560837+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:53.878484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:53.878484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.548326+0000","last_change":"2024-03-12T03:35:59.352393+0000","last_active":"2024-03-12T03:36:29.548326+0000","last_peered":"2024-03-12T03:36:29.548326+0000","last_clean":"2024-03-12T03:36:29.548326+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:36:29.548326+0000","last_undegraded":"2024-03-12T03:36:29.548326+0000","last_fullsized":"2024-03-12T03:36:29.548326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.352272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.352272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:00.463157+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'18","reported_seq":138,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.552849+0000","last_change":"2024-03-12T03:35:02.079317+0000","last_active":"2024-03-12T03:36:29.552849+0000","last_peered":"2024-03-12T03:36:29.552849+0000","last_clean":"2024-03-12T03:36:29.552849+0000","last_became_active":"2024-03-12T03:33:23.552210+0000","last_became_peered":"2024-03-12T03:33:23.552210+0000","last_unstale":"2024-03-12T03:36:29.552849+0000","last_undegraded":"2024-03-12T03:36:29.552849+0000","last_fullsized":"2024-03-12T03:36:29.552849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:35:02.079224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:02.079224+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.555677+0000","last_change":"2024-03-12T03:36:18.113521+0000","last_active":"2024-03-12T03:36:29.555677+0000","last_peered":"2024-03-12T03:36:29.555677+0000","last_clean":"2024-03-12T03:36:29.555677+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:36:29.555677+0000","last_undegraded":"2024-03-12T03:36:29.555677+0000","last_fullsized":"2024-03-12T03:36:29.555677+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.113393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:18.113393+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.113393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:28.877769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.552923+0000","last_change":"2024-03-12T03:34:45.089928+0000","last_active":"2024-03-12T03:36:29.552923+0000","last_peered":"2024-03-12T03:36:29.552923+0000","last_clean":"2024-03-12T03:36:29.552923+0000","last_became_active":"2024-03-12T03:33:25.183725+0000","last_became_peered":"2024-03-12T03:33:25.183725+0000","last_unstale":"2024-03-12T03:36:29.552923+0000","last_undegraded":"2024-03-12T03:36:29.552923+0000","last_fullsized":"2024-03-12T03:36:29.552923+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:45.089762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:45.089762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.561089+0000","last_change":"2024-03-12T03:34:41.698397+0000","last_active":"2024-03-12T03:36:29.561089+0000","last_peered":"2024-03-12T03:36:29.561089+0000","last_clean":"2024-03-12T03:36:29.561089+0000","last_became_active":"2024-03-12T03:33:21.107037+0000","last_became_peered":"2024-03-12T03:33:21.107037+0000","last_unstale":"2024-03-12T03:36:29.561089+0000","last_undegraded":"2024-03-12T03:36:29.561089+0000","last_fullsized":"2024-03-12T03:36:29.561089+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.698268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.698268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":130,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.561865+0000","last_change":"2024-03-12T03:36:11.006658+0000","last_active":"2024-03-12T03:36:29.561865+0000","last_peered":"2024-03-12T03:36:29.561865+0000","last_clean":"2024-03-12T03:36:29.561865+0000","last_became_active":"2024-03-12T03:33:23.160818+0000","last_became_peered":"2024-03-12T03:33:23.160818+0000","last_unstale":"2024-03-12T03:36:29.561865+0000","last_undegraded":"2024-03-12T03:36:29.561865+0000","last_fullsized":"2024-03-12T03:36:29.561865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-12T03:36:11.006154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:11.006154+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.382856+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.552571+0000","last_change":"2024-03-12T03:35:59.076156+0000","last_active":"2024-03-12T03:36:29.552571+0000","last_peered":"2024-03-12T03:36:29.552571+0000","last_clean":"2024-03-12T03:36:29.552571+0000","last_became_active":"2024-03-12T03:33:27.191710+0000","last_became_peered":"2024-03-12T03:33:27.191710+0000","last_unstale":"2024-03-12T03:36:29.552571+0000","last_undegraded":"2024-03-12T03:36:29.552571+0000","last_fullsized":"2024-03-12T03:36:29.552571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.075897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.075897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:18.991490+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.561607+0000","last_change":"2024-03-12T03:36:00.842130+0000","last_active":"2024-03-12T03:36:29.561607+0000","last_peered":"2024-03-12T03:36:29.561607+0000","last_clean":"2024-03-12T03:36:29.561607+0000","last_became_active":"2024-03-12T03:33:25.182004+0000","last_became_peered":"2024-03-12T03:33:25.182004+0000","last_unstale":"2024-03-12T03:36:29.561607+0000","last_undegraded":"2024-03-12T03:36:29.561607+0000","last_fullsized":"2024-03-12T03:36:29.561607+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:00.841986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:00.841986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:23.143222+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547466+0000","last_change":"2024-03-12T03:36:26.141166+0000","last_active":"2024-03-12T03:36:29.547466+0000","last_peered":"2024-03-12T03:36:29.547466+0000","last_clean":"2024-03-12T03:36:29.547466+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:36:29.547466+0000","last_undegraded":"2024-03-12T03:36:29.547466+0000","last_fullsized":"2024-03-12T03:36:29.547466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.141036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.141036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.596277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":125,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.548270+0000","last_change":"2024-03-12T03:34:34.953658+0000","last_active":"2024-03-12T03:36:29.548270+0000","last_peered":"2024-03-12T03:36:29.548270+0000","last_clean":"2024-03-12T03:36:29.548270+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:36:29.548270+0000","last_undegraded":"2024-03-12T03:36:29.548270+0000","last_fullsized":"2024-03-12T03:36:29.548270+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:34:34.953565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.953565+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560159+0000","last_change":"2024-03-12T03:36:07.111872+0000","last_active":"2024-03-12T03:36:29.560159+0000","last_peered":"2024-03-12T03:36:29.560159+0000","last_clean":"2024-03-12T03:36:29.560159+0000","last_became_active":"2024-03-12T03:33:27.194094+0000","last_became_peered":"2024-03-12T03:33:27.194094+0000","last_unstale":"2024-03-12T03:36:29.560159+0000","last_undegraded":"2024-03-12T03:36:29.560159+0000","last_fullsized":"2024-03-12T03:36:29.560159+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:07.111722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.111722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:33.629653+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":156,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547837+0000","last_change":"2024-03-12T03:36:07.296952+0000","last_active":"2024-03-12T03:36:29.547837+0000","last_peered":"2024-03-12T03:36:29.547837+0000","last_clean":"2024-03-12T03:36:29.547837+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:36:29.547837+0000","last_undegraded":"2024-03-12T03:36:29.547837+0000","last_fullsized":"2024-03-12T03:36:29.547837+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:07.296796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.296796+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.190322+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560880+0000","last_change":"2024-03-12T03:36:27.955751+0000","last_active":"2024-03-12T03:36:29.560880+0000","last_peered":"2024-03-12T03:36:29.560880+0000","last_clean":"2024-03-12T03:36:29.560880+0000","last_became_active":"2024-03-12T03:33:21.573244+0000","last_became_peered":"2024-03-12T03:33:21.573244+0000","last_unstale":"2024-03-12T03:36:29.560880+0000","last_undegraded":"2024-03-12T03:36:29.560880+0000","last_fullsized":"2024-03-12T03:36:29.560880+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:27.955624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:27.955624+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:42.177558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":145,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.551894+0000","last_change":"2024-03-12T03:35:58.079779+0000","last_active":"2024-03-12T03:36:29.551894+0000","last_peered":"2024-03-12T03:36:29.551894+0000","last_clean":"2024-03-12T03:36:29.551894+0000","last_became_active":"2024-03-12T03:33:23.551706+0000","last_became_peered":"2024-03-12T03:33:23.551706+0000","last_unstale":"2024-03-12T03:36:29.551894+0000","last_undegraded":"2024-03-12T03:36:29.551894+0000","last_fullsized":"2024-03-12T03:36:29.551894+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-03-12T03:35:58.079645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.079645+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.444037+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.548571+0000","last_change":"2024-03-12T03:35:12.124255+0000","last_active":"2024-03-12T03:36:29.548571+0000","last_peered":"2024-03-12T03:36:29.548571+0000","last_clean":"2024-03-12T03:36:29.548571+0000","last_became_active":"2024-03-12T03:33:27.184916+0000","last_became_peered":"2024-03-12T03:33:27.184916+0000","last_unstale":"2024-03-12T03:36:29.548571+0000","last_undegraded":"2024-03-12T03:36:29.548571+0000","last_fullsized":"2024-03-12T03:36:29.548571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:12.124167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:12.124167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":151,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.552156+0000","last_change":"2024-03-12T03:35:08.006530+0000","last_active":"2024-03-12T03:36:29.552156+0000","last_peered":"2024-03-12T03:36:29.552156+0000","last_clean":"2024-03-12T03:36:29.552156+0000","last_became_active":"2024-03-12T03:33:25.179158+0000","last_became_peered":"2024-03-12T03:33:25.179158+0000","last_unstale":"2024-03-12T03:36:29.552156+0000","last_undegraded":"2024-03-12T03:36:29.552156+0000","last_fullsized":"2024-03-12T03:36:29.552156+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:35:08.006375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:08.006375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.555844+0000","last_change":"2024-03-12T03:36:14.197774+0000","last_active":"2024-03-12T03:36:29.555844+0000","last_peered":"2024-03-12T03:36:29.555844+0000","last_clean":"2024-03-12T03:36:29.555844+0000","last_became_active":"2024-03-12T03:33:21.101819+0000","last_became_peered":"2024-03-12T03:33:21.101819+0000","last_unstale":"2024-03-12T03:36:29.555844+0000","last_undegraded":"2024-03-12T03:36:29.555844+0000","last_fullsized":"2024-03-12T03:36:29.555844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:14.197649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:36:14.197649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:41.087756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":142,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547972+0000","last_change":"2024-03-12T03:36:12.092317+0000","last_active":"2024-03-12T03:36:29.547972+0000","last_peered":"2024-03-12T03:36:29.547972+0000","last_clean":"2024-03-12T03:36:29.547972+0000","last_became_active":"2024-03-12T03:33:23.553480+0000","last_became_peered":"2024-03-12T03:33:23.553480+0000","last_unstale":"2024-03-12T03:36:29.547972+0000","last_undegraded":"2024-03-12T03:36:29.547972+0000","last_fullsized":"2024-03-12T03:36:29.547972+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:36:12.092175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:12.092175+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.115981+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.569075+0000","last_change":"2024-03-12T03:36:02.155537+0000","last_active":"2024-03-12T03:36:29.569075+0000","last_peered":"2024-03-12T03:36:29.569075+0000","last_clean":"2024-03-12T03:36:29.569075+0000","last_became_active":"2024-03-12T03:33:27.187657+0000","last_became_peered":"2024-03-12T03:33:27.187657+0000","last_unstale":"2024-03-12T03:36:29.569075+0000","last_undegraded":"2024-03-12T03:36:29.569075+0000","last_fullsized":"2024-03-12T03:36:29.569075+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:02.155425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:02.155425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:24.813580+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.558993+0000","last_change":"2024-03-12T03:35:04.007338+0000","last_active":"2024-03-12T03:36:29.558993+0000","last_peered":"2024-03-12T03:36:29.558993+0000","last_clean":"2024-03-12T03:36:29.558993+0000","last_became_active":"2024-03-12T03:33:25.174953+0000","last_became_peered":"2024-03-12T03:33:25.174953+0000","last_unstale":"2024-03-12T03:36:29.558993+0000","last_undegraded":"2024-03-12T03:36:29.558993+0000","last_fullsized":"2024-03-12T03:36:29.558993+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:35:04.007252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:31.186652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560678+0000","last_change":"2024-03-12T03:34:52.848468+0000","last_active":"2024-03-12T03:36:29.560678+0000","last_peered":"2024-03-12T03:36:29.560678+0000","last_clean":"2024-03-12T03:36:29.560678+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:36:29.560678+0000","last_undegraded":"2024-03-12T03:36:29.560678+0000","last_fullsized":"2024-03-12T03:36:29.560678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:52.848366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:52.848366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":146,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560621+0000","last_change":"2024-03-12T03:35:11.786139+0000","last_active":"2024-03-12T03:36:29.560621+0000","last_peered":"2024-03-12T03:36:29.560621+0000","last_clean":"2024-03-12T03:36:29.560621+0000","last_became_active":"2024-03-12T03:33:23.552164+0000","last_became_peered":"2024-03-12T03:33:23.552164+0000","last_unstale":"2024-03-12T03:36:29.560621+0000","last_undegraded":"2024-03-12T03:36:29.560621+0000","last_fullsized":"2024-03-12T03:36:29.560621+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:35:11.786040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.786040+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.561034+0000","last_change":"2024-03-12T03:36:24.959007+0000","last_active":"2024-03-12T03:36:29.561034+0000","last_peered":"2024-03-12T03:36:29.561034+0000","last_clean":"2024-03-12T03:36:29.561034+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:36:29.561034+0000","last_undegraded":"2024-03-12T03:36:29.561034+0000","last_fullsized":"2024-03-12T03:36:29.561034+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:24.958880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.958880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:47.117767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.548399+0000","last_change":"2024-03-12T03:36:29.181191+0000","last_active":"2024-03-12T03:36:29.548399+0000","last_peered":"2024-03-12T03:36:29.548399+0000","last_clean":"2024-03-12T03:36:29.548399+0000","last_became_active":"2024-03-12T03:33:25.180752+0000","last_became_peered":"2024-03-12T03:33:25.180752+0000","last_unstale":"2024-03-12T03:36:29.548399+0000","last_undegraded":"2024-03-12T03:36:29.548399+0000","last_fullsized":"2024-03-12T03:36:29.548399+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:29.181010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:29.181010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:33.940875+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.555869+0000","last_change":"2024-03-12T03:35:58.075845+0000","last_active":"2024-03-12T03:36:29.555869+0000","last_peered":"2024-03-12T03:36:29.555869+0000","last_clean":"2024-03-12T03:36:29.555869+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:36:29.555869+0000","last_undegraded":"2024-03-12T03:36:29.555869+0000","last_fullsized":"2024-03-12T03:36:29.555869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:58.075754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.075754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:04.874641+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.559356+0000","last_change":"2024-03-12T03:35:00.079249+0000","last_active":"2024-03-12T03:36:29.559356+0000","last_peered":"2024-03-12T03:36:29.559356+0000","last_clean":"2024-03-12T03:36:29.559356+0000","last_became_active":"2024-03-12T03:33:23.551773+0000","last_became_peered":"2024-03-12T03:33:23.551773+0000","last_unstale":"2024-03-12T03:36:29.559356+0000","last_undegraded":"2024-03-12T03:36:29.559356+0000","last_fullsized":"2024-03-12T03:36:29.559356+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:35:00.079152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.079152+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"112'5572","reported_seq":5613,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.776700+0000","last_change":"2024-03-12T03:35:23.782770+0000","last_active":"2024-03-12T03:36:30.776700+0000","last_peered":"2024-03-12T03:36:30.776700+0000","last_clean":"2024-03-12T03:36:30.776700+0000","last_became_active":"2024-03-12T03:35:23.782112+0000","last_became_peered":"2024-03-12T03:35:23.782112+0000","last_unstale":"2024-03-12T03:36:30.776700+0000","last_undegraded":"2024-03-12T03:36:30.776700+0000","last_fullsized":"2024-03-12T03:36:30.776700+0000","mapping_epoch":97,"log_start":"100'500","ondisk_log_start":"100'500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:40.122707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365166592,"num_objects":5572,"num_object_clones":0,"num_object_copies":16716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5572,"num_write_kb":356608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.546544+0000","last_change":"2024-03-12T03:36:20.147626+0000","last_active":"2024-03-12T03:36:29.546544+0000","last_peered":"2024-03-12T03:36:29.546544+0000","last_clean":"2024-03-12T03:36:29.546544+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:36:29.546544+0000","last_undegraded":"2024-03-12T03:36:29.546544+0000","last_fullsized":"2024-03-12T03:36:29.546544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.147488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:44.684777+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":122,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.561251+0000","last_change":"2024-03-12T03:35:12.907083+0000","last_active":"2024-03-12T03:36:29.561251+0000","last_peered":"2024-03-12T03:36:29.561251+0000","last_clean":"2024-03-12T03:36:29.561251+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:36:29.561251+0000","last_undegraded":"2024-03-12T03:36:29.561251+0000","last_fullsized":"2024-03-12T03:36:29.561251+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:35:12.906988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:12.906988+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:35.363926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"90'36","reported_seq":172,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.546251+0000","last_change":"2024-03-12T03:36:15.181987+0000","last_active":"2024-03-12T03:36:29.546251+0000","last_peered":"2024-03-12T03:36:29.546251+0000","last_clean":"2024-03-12T03:36:29.546251+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:36:29.546251+0000","last_undegraded":"2024-03-12T03:36:29.546251+0000","last_fullsized":"2024-03-12T03:36:29.546251+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:36:15.181852+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.181852+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.159048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":148,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560811+0000","last_change":"2024-03-12T03:34:39.659403+0000","last_active":"2024-03-12T03:36:29.560811+0000","last_peered":"2024-03-12T03:36:29.560811+0000","last_clean":"2024-03-12T03:36:29.560811+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:36:29.560811+0000","last_undegraded":"2024-03-12T03:36:29.560811+0000","last_fullsized":"2024-03-12T03:36:29.560811+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:39.659277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.659277+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"112'5628","reported_seq":5669,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.846804+0000","last_change":"2024-03-12T03:35:23.777596+0000","last_active":"2024-03-12T03:36:30.846804+0000","last_peered":"2024-03-12T03:36:30.846804+0000","last_clean":"2024-03-12T03:36:30.846804+0000","last_became_active":"2024-03-12T03:35:23.777272+0000","last_became_peered":"2024-03-12T03:35:23.777272+0000","last_unstale":"2024-03-12T03:36:30.846804+0000","last_undegraded":"2024-03-12T03:36:30.846804+0000","last_fullsized":"2024-03-12T03:36:30.846804+0000","mapping_epoch":97,"log_start":"101'1000","ondisk_log_start":"101'1000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:43.470290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368836608,"num_objects":5628,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5628,"num_write_kb":360192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.553080+0000","last_change":"2024-03-12T03:33:27.191091+0000","last_active":"2024-03-12T03:36:29.553080+0000","last_peered":"2024-03-12T03:36:29.553080+0000","last_clean":"2024-03-12T03:36:29.553080+0000","last_became_active":"2024-03-12T03:33:27.190753+0000","last_became_peered":"2024-03-12T03:33:27.190753+0000","last_unstale":"2024-03-12T03:36:29.553080+0000","last_undegraded":"2024-03-12T03:36:29.553080+0000","last_fullsized":"2024-03-12T03:36:29.553080+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.561318+0000","last_change":"2024-03-12T03:36:17.051625+0000","last_active":"2024-03-12T03:36:29.561318+0000","last_peered":"2024-03-12T03:36:29.561318+0000","last_clean":"2024-03-12T03:36:29.561318+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:36:29.561318+0000","last_undegraded":"2024-03-12T03:36:29.561318+0000","last_fullsized":"2024-03-12T03:36:29.561318+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:36:17.051487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.698687+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":153,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547225+0000","last_change":"2024-03-12T03:36:10.043243+0000","last_active":"2024-03-12T03:36:29.547225+0000","last_peered":"2024-03-12T03:36:29.547225+0000","last_clean":"2024-03-12T03:36:29.547225+0000","last_became_active":"2024-03-12T03:33:23.163351+0000","last_became_peered":"2024-03-12T03:33:23.163351+0000","last_unstale":"2024-03-12T03:36:29.547225+0000","last_undegraded":"2024-03-12T03:36:29.547225+0000","last_fullsized":"2024-03-12T03:36:29.547225+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-03-12T03:36:10.043108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:10.043108+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.376152+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.571134+0000","last_change":"2024-03-12T03:36:26.332254+0000","last_active":"2024-03-12T03:36:29.571134+0000","last_peered":"2024-03-12T03:36:29.571134+0000","last_clean":"2024-03-12T03:36:29.571134+0000","last_became_active":"2024-03-12T03:33:25.185596+0000","last_became_peered":"2024-03-12T03:33:25.185596+0000","last_unstale":"2024-03-12T03:36:29.571134+0000","last_undegraded":"2024-03-12T03:36:29.571134+0000","last_fullsized":"2024-03-12T03:36:29.571134+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.332124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.332124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:45.657747+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"112'5688","reported_seq":5729,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.830496+0000","last_change":"2024-03-12T03:35:23.778943+0000","last_active":"2024-03-12T03:36:30.830496+0000","last_peered":"2024-03-12T03:36:30.830496+0000","last_clean":"2024-03-12T03:36:30.830496+0000","last_became_active":"2024-03-12T03:35:23.778606+0000","last_became_peered":"2024-03-12T03:35:23.778606+0000","last_unstale":"2024-03-12T03:36:30.830496+0000","last_undegraded":"2024-03-12T03:36:30.830496+0000","last_fullsized":"2024-03-12T03:36:30.830496+0000","mapping_epoch":97,"log_start":"100'100","ondisk_log_start":"100'100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5588,"log_dups_size":0,"ondisk_log_size":5588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:35.752669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372768768,"num_objects":5688,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5688,"num_write_kb":364032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560562+0000","last_change":"2024-03-12T03:34:43.716123+0000","last_active":"2024-03-12T03:36:29.560562+0000","last_peered":"2024-03-12T03:36:29.560562+0000","last_clean":"2024-03-12T03:36:29.560562+0000","last_became_active":"2024-03-12T03:33:27.187955+0000","last_became_peered":"2024-03-12T03:33:27.187955+0000","last_unstale":"2024-03-12T03:36:29.560562+0000","last_undegraded":"2024-03-12T03:36:29.560562+0000","last_fullsized":"2024-03-12T03:36:29.560562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:43.716021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.716021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.570977+0000","last_change":"2024-03-12T03:36:18.320863+0000","last_active":"2024-03-12T03:36:29.570977+0000","last_peered":"2024-03-12T03:36:29.570977+0000","last_clean":"2024-03-12T03:36:29.570977+0000","last_became_active":"2024-03-12T03:33:25.179593+0000","last_became_peered":"2024-03-12T03:33:25.179593+0000","last_unstale":"2024-03-12T03:36:29.570977+0000","last_undegraded":"2024-03-12T03:36:29.570977+0000","last_fullsized":"2024-03-12T03:36:29.570977+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.320728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.320728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.937998+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.561186+0000","last_change":"2024-03-12T03:36:09.026855+0000","last_active":"2024-03-12T03:36:29.561186+0000","last_peered":"2024-03-12T03:36:29.561186+0000","last_clean":"2024-03-12T03:36:29.561186+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:36:29.561186+0000","last_undegraded":"2024-03-12T03:36:29.561186+0000","last_fullsized":"2024-03-12T03:36:29.561186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:09.026725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:09.026725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.130351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":157,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556145+0000","last_change":"2024-03-12T03:36:27.151507+0000","last_active":"2024-03-12T03:36:29.556145+0000","last_peered":"2024-03-12T03:36:29.556145+0000","last_clean":"2024-03-12T03:36:29.556145+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:36:29.556145+0000","last_undegraded":"2024-03-12T03:36:29.556145+0000","last_fullsized":"2024-03-12T03:36:29.556145+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:36:27.151375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:27.151375+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.667242+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"112'5673","reported_seq":5714,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.851950+0000","last_change":"2024-03-12T03:35:23.785560+0000","last_active":"2024-03-12T03:36:30.851950+0000","last_peered":"2024-03-12T03:36:30.851950+0000","last_clean":"2024-03-12T03:36:30.851950+0000","last_became_active":"2024-03-12T03:35:23.784701+0000","last_became_peered":"2024-03-12T03:35:23.784701+0000","last_unstale":"2024-03-12T03:36:30.851950+0000","last_undegraded":"2024-03-12T03:36:30.851950+0000","last_fullsized":"2024-03-12T03:36:30.851950+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5673,"log_dups_size":0,"ondisk_log_size":5673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:50.631958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371785728,"num_objects":5673,"num_object_clones":0,"num_object_copies":17019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5673,"num_write_kb":363072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.553295+0000","last_change":"2024-03-12T03:36:10.178072+0000","last_active":"2024-03-12T03:36:29.553295+0000","last_peered":"2024-03-12T03:36:29.553295+0000","last_clean":"2024-03-12T03:36:29.553295+0000","last_became_active":"2024-03-12T03:33:27.189948+0000","last_became_peered":"2024-03-12T03:33:27.189948+0000","last_unstale":"2024-03-12T03:36:29.553295+0000","last_undegraded":"2024-03-12T03:36:29.553295+0000","last_fullsized":"2024-03-12T03:36:29.553295+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:10.177808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:10.177808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:35.471819+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.569723+0000","last_change":"2024-03-12T03:36:11.345502+0000","last_active":"2024-03-12T03:36:29.569723+0000","last_peered":"2024-03-12T03:36:29.569723+0000","last_clean":"2024-03-12T03:36:29.569723+0000","last_became_active":"2024-03-12T03:33:25.184289+0000","last_became_peered":"2024-03-12T03:33:25.184289+0000","last_unstale":"2024-03-12T03:36:29.569723+0000","last_undegraded":"2024-03-12T03:36:29.569723+0000","last_fullsized":"2024-03-12T03:36:29.569723+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:11.345381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:11.345381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:32.654030+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547223+0000","last_change":"2024-03-12T03:36:26.134473+0000","last_active":"2024-03-12T03:36:29.547223+0000","last_peered":"2024-03-12T03:36:29.547223+0000","last_clean":"2024-03-12T03:36:29.547223+0000","last_became_active":"2024-03-12T03:33:21.102410+0000","last_became_peered":"2024-03-12T03:33:21.102410+0000","last_unstale":"2024-03-12T03:36:29.547223+0000","last_undegraded":"2024-03-12T03:36:29.547223+0000","last_fullsized":"2024-03-12T03:36:29.547223+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.134337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.134337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:45.916621+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"111'20","reported_seq":147,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556293+0000","last_change":"2024-03-12T03:36:29.192473+0000","last_active":"2024-03-12T03:36:29.556293+0000","last_peered":"2024-03-12T03:36:29.556293+0000","last_clean":"2024-03-12T03:36:29.556293+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:36:29.556293+0000","last_undegraded":"2024-03-12T03:36:29.556293+0000","last_fullsized":"2024-03-12T03:36:29.556293+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'20","last_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:36:29.192310+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:32.549710+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"112'5671","reported_seq":5712,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.769740+0000","last_change":"2024-03-12T03:35:23.784948+0000","last_active":"2024-03-12T03:36:30.769740+0000","last_peered":"2024-03-12T03:36:30.769740+0000","last_clean":"2024-03-12T03:36:30.769740+0000","last_became_active":"2024-03-12T03:35:23.784628+0000","last_became_peered":"2024-03-12T03:35:23.784628+0000","last_unstale":"2024-03-12T03:36:30.769740+0000","last_undegraded":"2024-03-12T03:36:30.769740+0000","last_fullsized":"2024-03-12T03:36:30.769740+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371654656,"num_objects":5671,"num_object_clones":0,"num_object_copies":17013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5671,"num_write_kb":362944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556440+0000","last_change":"2024-03-12T03:34:49.088183+0000","last_active":"2024-03-12T03:36:29.556440+0000","last_peered":"2024-03-12T03:36:29.556440+0000","last_clean":"2024-03-12T03:36:29.556440+0000","last_became_active":"2024-03-12T03:33:27.185041+0000","last_became_peered":"2024-03-12T03:33:27.185041+0000","last_unstale":"2024-03-12T03:36:29.556440+0000","last_undegraded":"2024-03-12T03:36:29.556440+0000","last_fullsized":"2024-03-12T03:36:29.556440+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.088098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.088098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.546707+0000","last_change":"2024-03-12T03:36:13.134209+0000","last_active":"2024-03-12T03:36:29.546707+0000","last_peered":"2024-03-12T03:36:29.546707+0000","last_clean":"2024-03-12T03:36:29.546707+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:36:29.546707+0000","last_undegraded":"2024-03-12T03:36:29.546707+0000","last_fullsized":"2024-03-12T03:36:29.546707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:36:13.134087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.134087+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.167729+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547931+0000","last_change":"2024-03-12T03:34:59.105754+0000","last_active":"2024-03-12T03:36:29.547931+0000","last_peered":"2024-03-12T03:36:29.547931+0000","last_clean":"2024-03-12T03:36:29.547931+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:36:29.547931+0000","last_undegraded":"2024-03-12T03:36:29.547931+0000","last_fullsized":"2024-03-12T03:36:29.547931+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:59.105634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:59.105634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'35","reported_seq":143,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.546965+0000","last_change":"2024-03-12T03:34:54.301841+0000","last_active":"2024-03-12T03:36:29.546965+0000","last_peered":"2024-03-12T03:36:29.546965+0000","last_clean":"2024-03-12T03:36:29.546965+0000","last_became_active":"2024-03-12T03:33:50.151664+0000","last_became_peered":"2024-03-12T03:33:50.151664+0000","last_unstale":"2024-03-12T03:36:29.546965+0000","last_undegraded":"2024-03-12T03:36:29.546965+0000","last_fullsized":"2024-03-12T03:36:29.546965+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'35","last_scrub_stamp":"2024-03-12T03:34:54.301725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:54.301725+0000","objects_scrubbed":11,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.561349+0000","last_change":"2024-03-12T03:36:15.170135+0000","last_active":"2024-03-12T03:36:29.561349+0000","last_peered":"2024-03-12T03:36:29.561349+0000","last_clean":"2024-03-12T03:36:29.561349+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:36:29.561349+0000","last_undegraded":"2024-03-12T03:36:29.561349+0000","last_fullsized":"2024-03-12T03:36:29.561349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:15.170007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.170007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:15.186652+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"112'5701","reported_seq":5742,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.786966+0000","last_change":"2024-03-12T03:35:23.776994+0000","last_active":"2024-03-12T03:36:30.786966+0000","last_peered":"2024-03-12T03:36:30.786966+0000","last_clean":"2024-03-12T03:36:30.786966+0000","last_became_active":"2024-03-12T03:35:23.776662+0000","last_became_peered":"2024-03-12T03:35:23.776662+0000","last_unstale":"2024-03-12T03:36:30.786966+0000","last_undegraded":"2024-03-12T03:36:30.786966+0000","last_fullsized":"2024-03-12T03:36:30.786966+0000","mapping_epoch":97,"log_start":"100'100","ondisk_log_start":"100'100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:34.857076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373620736,"num_objects":5701,"num_object_clones":0,"num_object_copies":17103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5701,"num_write_kb":364864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.562115+0000","last_change":"2024-03-12T03:36:02.083741+0000","last_active":"2024-03-12T03:36:29.562115+0000","last_peered":"2024-03-12T03:36:29.562115+0000","last_clean":"2024-03-12T03:36:29.562115+0000","last_became_active":"2024-03-12T03:33:27.380389+0000","last_became_peered":"2024-03-12T03:33:27.380389+0000","last_unstale":"2024-03-12T03:36:29.562115+0000","last_undegraded":"2024-03-12T03:36:29.562115+0000","last_fullsized":"2024-03-12T03:36:29.562115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:36:02.083611+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:36:02.083611+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:13.347363+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.553971+0000","last_change":"2024-03-12T03:36:21.149936+0000","last_active":"2024-03-12T03:36:29.553971+0000","last_peered":"2024-03-12T03:36:29.553971+0000","last_clean":"2024-03-12T03:36:29.553971+0000","last_became_active":"2024-03-12T03:33:21.572911+0000","last_became_peered":"2024-03-12T03:33:21.572911+0000","last_unstale":"2024-03-12T03:36:29.553971+0000","last_undegraded":"2024-03-12T03:36:29.553971+0000","last_fullsized":"2024-03-12T03:36:29.553971+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:21.149686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:21.149686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:31.591082+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.569244+0000","last_change":"2024-03-12T03:36:15.348581+0000","last_active":"2024-03-12T03:36:29.569244+0000","last_peered":"2024-03-12T03:36:29.569244+0000","last_clean":"2024-03-12T03:36:29.569244+0000","last_became_active":"2024-03-12T03:33:20.093064+0000","last_became_peered":"2024-03-12T03:33:20.093064+0000","last_unstale":"2024-03-12T03:36:29.569244+0000","last_undegraded":"2024-03-12T03:36:29.569244+0000","last_fullsized":"2024-03-12T03:36:29.569244+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:15.348471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.348471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:21.623557+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.000516994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":153,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.553713+0000","last_change":"2024-03-12T03:36:19.124448+0000","last_active":"2024-03-12T03:36:29.553713+0000","last_peered":"2024-03-12T03:36:29.553713+0000","last_clean":"2024-03-12T03:36:29.553713+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:36:29.553713+0000","last_undegraded":"2024-03-12T03:36:29.553713+0000","last_fullsized":"2024-03-12T03:36:29.553713+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-12T03:36:19.124308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.124308+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:36.079349+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.569342+0000","last_change":"2024-03-12T03:35:58.091300+0000","last_active":"2024-03-12T03:36:29.569342+0000","last_peered":"2024-03-12T03:36:29.569342+0000","last_clean":"2024-03-12T03:36:29.569342+0000","last_became_active":"2024-03-12T03:33:25.179022+0000","last_became_peered":"2024-03-12T03:33:25.179022+0000","last_unstale":"2024-03-12T03:36:29.569342+0000","last_undegraded":"2024-03-12T03:36:29.569342+0000","last_fullsized":"2024-03-12T03:36:29.569342+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:58.091182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.091182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:04.769136+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"112'5581","reported_seq":5622,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.811585+0000","last_change":"2024-03-12T03:35:23.785163+0000","last_active":"2024-03-12T03:36:30.811585+0000","last_peered":"2024-03-12T03:36:30.811585+0000","last_clean":"2024-03-12T03:36:30.811585+0000","last_became_active":"2024-03-12T03:35:23.784880+0000","last_became_peered":"2024-03-12T03:35:23.784880+0000","last_unstale":"2024-03-12T03:36:30.811585+0000","last_undegraded":"2024-03-12T03:36:30.811585+0000","last_fullsized":"2024-03-12T03:36:30.811585+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:48.116983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365756416,"num_objects":5581,"num_object_clones":0,"num_object_copies":16743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":357184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547396+0000","last_change":"2024-03-12T03:36:25.119508+0000","last_active":"2024-03-12T03:36:29.547396+0000","last_peered":"2024-03-12T03:36:29.547396+0000","last_clean":"2024-03-12T03:36:29.547396+0000","last_became_active":"2024-03-12T03:33:27.188551+0000","last_became_peered":"2024-03-12T03:33:27.188551+0000","last_unstale":"2024-03-12T03:36:29.547396+0000","last_undegraded":"2024-03-12T03:36:29.547396+0000","last_fullsized":"2024-03-12T03:36:29.547396+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:25.119358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.119358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:54.029889+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":365,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560671+0000","last_change":"2024-03-12T03:34:58.903289+0000","last_active":"2024-03-12T03:36:29.560671+0000","last_peered":"2024-03-12T03:36:29.560671+0000","last_clean":"2024-03-12T03:36:29.560671+0000","last_became_active":"2024-03-12T03:33:11.319888+0000","last_became_peered":"2024-03-12T03:33:11.319888+0000","last_unstale":"2024-03-12T03:36:29.560671+0000","last_undegraded":"2024-03-12T03:36:29.560671+0000","last_fullsized":"2024-03-12T03:36:29.560671+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:34:58.903163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.903163+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.551556+0000","last_change":"2024-03-12T03:35:14.996451+0000","last_active":"2024-03-12T03:36:29.551556+0000","last_peered":"2024-03-12T03:36:29.551556+0000","last_clean":"2024-03-12T03:36:29.551556+0000","last_became_active":"2024-03-12T03:33:21.100900+0000","last_became_peered":"2024-03-12T03:33:21.100900+0000","last_unstale":"2024-03-12T03:36:29.551556+0000","last_undegraded":"2024-03-12T03:36:29.551556+0000","last_fullsized":"2024-03-12T03:36:29.551556+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:14.996357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:14.996357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:35.315899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":174,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556542+0000","last_change":"2024-03-12T03:36:03.118128+0000","last_active":"2024-03-12T03:36:29.556542+0000","last_peered":"2024-03-12T03:36:29.556542+0000","last_clean":"2024-03-12T03:36:29.556542+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:36:29.556542+0000","last_undegraded":"2024-03-12T03:36:29.556542+0000","last_fullsized":"2024-03-12T03:36:29.556542+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:36:03.117990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:03.117990+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:31.305573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.561349+0000","last_change":"2024-03-12T03:34:47.796670+0000","last_active":"2024-03-12T03:36:29.561349+0000","last_peered":"2024-03-12T03:36:29.561349+0000","last_clean":"2024-03-12T03:36:29.561349+0000","last_became_active":"2024-03-12T03:33:25.177762+0000","last_became_peered":"2024-03-12T03:33:25.177762+0000","last_unstale":"2024-03-12T03:36:29.561349+0000","last_undegraded":"2024-03-12T03:36:29.561349+0000","last_fullsized":"2024-03-12T03:36:29.561349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.796561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.796561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"112'5651","reported_seq":5692,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.761085+0000","last_change":"2024-03-12T03:35:23.786371+0000","last_active":"2024-03-12T03:36:30.761085+0000","last_peered":"2024-03-12T03:36:30.761085+0000","last_clean":"2024-03-12T03:36:30.761085+0000","last_became_active":"2024-03-12T03:35:23.786059+0000","last_became_peered":"2024-03-12T03:35:23.786059+0000","last_unstale":"2024-03-12T03:36:30.761085+0000","last_undegraded":"2024-03-12T03:36:30.761085+0000","last_fullsized":"2024-03-12T03:36:30.761085+0000","mapping_epoch":97,"log_start":"100'400","ondisk_log_start":"100'400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5251,"log_dups_size":0,"ondisk_log_size":5251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:52.195145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370343936,"num_objects":5651,"num_object_clones":0,"num_object_copies":16953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5651,"num_write_kb":361664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.555161+0000","last_change":"2024-03-12T03:36:15.042978+0000","last_active":"2024-03-12T03:36:29.555161+0000","last_peered":"2024-03-12T03:36:29.555161+0000","last_clean":"2024-03-12T03:36:29.555161+0000","last_became_active":"2024-03-12T03:33:27.191878+0000","last_became_peered":"2024-03-12T03:33:27.191878+0000","last_unstale":"2024-03-12T03:36:29.555161+0000","last_undegraded":"2024-03-12T03:36:29.555161+0000","last_fullsized":"2024-03-12T03:36:29.555161+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:15.042610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.042610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:21.860517+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.555293+0000","last_change":"2024-03-12T03:35:05.715931+0000","last_active":"2024-03-12T03:36:29.555293+0000","last_peered":"2024-03-12T03:36:29.555293+0000","last_clean":"2024-03-12T03:36:29.555293+0000","last_became_active":"2024-03-12T03:33:21.097086+0000","last_became_peered":"2024-03-12T03:33:21.097086+0000","last_unstale":"2024-03-12T03:36:29.555293+0000","last_undegraded":"2024-03-12T03:36:29.555293+0000","last_fullsized":"2024-03-12T03:36:29.555293+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:05.715849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.715849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":314,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560638+0000","last_change":"2024-03-12T03:36:08.059941+0000","last_active":"2024-03-12T03:36:29.560638+0000","last_peered":"2024-03-12T03:36:29.560638+0000","last_clean":"2024-03-12T03:36:29.560638+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:36:29.560638+0000","last_undegraded":"2024-03-12T03:36:29.560638+0000","last_fullsized":"2024-03-12T03:36:29.560638+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:36:08.059814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:36:08.059814+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:27.702586+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":184,"num_read_kb":179,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":132,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560252+0000","last_change":"2024-03-12T03:36:03.015860+0000","last_active":"2024-03-12T03:36:29.560252+0000","last_peered":"2024-03-12T03:36:29.560252+0000","last_clean":"2024-03-12T03:36:29.560252+0000","last_became_active":"2024-03-12T03:33:23.168750+0000","last_became_peered":"2024-03-12T03:33:23.168750+0000","last_unstale":"2024-03-12T03:36:29.560252+0000","last_undegraded":"2024-03-12T03:36:29.560252+0000","last_fullsized":"2024-03-12T03:36:29.560252+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:36:03.015565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:03.015565+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:05.935088+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547889+0000","last_change":"2024-03-12T03:36:24.109073+0000","last_active":"2024-03-12T03:36:29.547889+0000","last_peered":"2024-03-12T03:36:29.547889+0000","last_clean":"2024-03-12T03:36:29.547889+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:36:29.547889+0000","last_undegraded":"2024-03-12T03:36:29.547889+0000","last_fullsized":"2024-03-12T03:36:29.547889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:24.108984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.108984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.677961+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"112'5597","reported_seq":5638,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.750168+0000","last_change":"2024-03-12T03:35:23.778292+0000","last_active":"2024-03-12T03:36:30.750168+0000","last_peered":"2024-03-12T03:36:30.750168+0000","last_clean":"2024-03-12T03:36:30.750168+0000","last_became_active":"2024-03-12T03:35:23.777945+0000","last_became_peered":"2024-03-12T03:35:23.777945+0000","last_unstale":"2024-03-12T03:36:30.750168+0000","last_undegraded":"2024-03-12T03:36:30.750168+0000","last_fullsized":"2024-03-12T03:36:30.750168+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:39.883486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366804992,"num_objects":5597,"num_object_clones":0,"num_object_copies":16791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5597,"num_write_kb":358208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.554231+0000","last_change":"2024-03-12T03:36:18.066969+0000","last_active":"2024-03-12T03:36:29.554231+0000","last_peered":"2024-03-12T03:36:29.554231+0000","last_clean":"2024-03-12T03:36:29.554231+0000","last_became_active":"2024-03-12T03:33:27.380458+0000","last_became_peered":"2024-03-12T03:33:27.380458+0000","last_unstale":"2024-03-12T03:36:29.554231+0000","last_undegraded":"2024-03-12T03:36:29.554231+0000","last_fullsized":"2024-03-12T03:36:29.554231+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.066712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.066712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:47.296061+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.555377+0000","last_change":"2024-03-12T03:36:01.874384+0000","last_active":"2024-03-12T03:36:29.555377+0000","last_peered":"2024-03-12T03:36:29.555377+0000","last_clean":"2024-03-12T03:36:29.555377+0000","last_became_active":"2024-03-12T03:33:21.573003+0000","last_became_peered":"2024-03-12T03:33:21.573003+0000","last_unstale":"2024-03-12T03:36:29.555377+0000","last_undegraded":"2024-03-12T03:36:29.555377+0000","last_fullsized":"2024-03-12T03:36:29.555377+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:01.874124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:01.874124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:04.609067+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":162,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556629+0000","last_change":"2024-03-12T03:36:22.118169+0000","last_active":"2024-03-12T03:36:29.556629+0000","last_peered":"2024-03-12T03:36:29.556629+0000","last_clean":"2024-03-12T03:36:29.556629+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:36:29.556629+0000","last_undegraded":"2024-03-12T03:36:29.556629+0000","last_fullsized":"2024-03-12T03:36:29.556629+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:36:22.118034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:22.118034+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:23.200152+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556272+0000","last_change":"2024-03-12T03:36:21.140906+0000","last_active":"2024-03-12T03:36:29.556272+0000","last_peered":"2024-03-12T03:36:29.556272+0000","last_clean":"2024-03-12T03:36:29.556272+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:36:29.556272+0000","last_undegraded":"2024-03-12T03:36:29.556272+0000","last_fullsized":"2024-03-12T03:36:29.556272+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:21.140825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:21.140825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:25.002630+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"112'5547","reported_seq":5588,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.760575+0000","last_change":"2024-03-12T03:35:23.781284+0000","last_active":"2024-03-12T03:36:30.760575+0000","last_peered":"2024-03-12T03:36:30.760575+0000","last_clean":"2024-03-12T03:36:30.760575+0000","last_became_active":"2024-03-12T03:35:23.780941+0000","last_became_peered":"2024-03-12T03:35:23.780941+0000","last_unstale":"2024-03-12T03:36:30.760575+0000","last_undegraded":"2024-03-12T03:36:30.760575+0000","last_fullsized":"2024-03-12T03:36:30.760575+0000","mapping_epoch":97,"log_start":"100'700","ondisk_log_start":"100'700","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:35.124924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363528192,"num_objects":5547,"num_object_clones":0,"num_object_copies":16641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5547,"num_write_kb":355008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.555787+0000","last_change":"2024-03-12T03:36:20.138118+0000","last_active":"2024-03-12T03:36:29.555787+0000","last_peered":"2024-03-12T03:36:29.555787+0000","last_clean":"2024-03-12T03:36:29.555787+0000","last_became_active":"2024-03-12T03:33:27.194266+0000","last_became_peered":"2024-03-12T03:33:27.194266+0000","last_unstale":"2024-03-12T03:36:29.555787+0000","last_undegraded":"2024-03-12T03:36:29.555787+0000","last_fullsized":"2024-03-12T03:36:29.555787+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:20.138004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.138004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:34.538433+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547677+0000","last_change":"2024-03-12T03:36:17.238474+0000","last_active":"2024-03-12T03:36:29.547677+0000","last_peered":"2024-03-12T03:36:29.547677+0000","last_clean":"2024-03-12T03:36:29.547677+0000","last_became_active":"2024-03-12T03:33:21.573251+0000","last_became_peered":"2024-03-12T03:33:21.573251+0000","last_unstale":"2024-03-12T03:36:29.547677+0000","last_undegraded":"2024-03-12T03:36:29.547677+0000","last_fullsized":"2024-03-12T03:36:29.547677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:17.238386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:17.238386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:42.056483+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":141,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.561697+0000","last_change":"2024-03-12T03:36:07.967637+0000","last_active":"2024-03-12T03:36:29.561697+0000","last_peered":"2024-03-12T03:36:29.561697+0000","last_clean":"2024-03-12T03:36:29.561697+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:36:29.561697+0000","last_undegraded":"2024-03-12T03:36:29.561697+0000","last_fullsized":"2024-03-12T03:36:29.561697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:36:07.967492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.967492+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:29.190972+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":156,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556156+0000","last_change":"2024-03-12T03:36:06.069994+0000","last_active":"2024-03-12T03:36:29.556156+0000","last_peered":"2024-03-12T03:36:29.556156+0000","last_clean":"2024-03-12T03:36:29.556156+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:36:29.556156+0000","last_undegraded":"2024-03-12T03:36:29.556156+0000","last_fullsized":"2024-03-12T03:36:29.556156+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:06.069862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.069862+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:21.304322+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"112'5608","reported_seq":5649,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.877555+0000","last_change":"2024-03-12T03:35:23.778223+0000","last_active":"2024-03-12T03:36:30.877555+0000","last_peered":"2024-03-12T03:36:30.877555+0000","last_clean":"2024-03-12T03:36:30.877555+0000","last_became_active":"2024-03-12T03:35:23.777935+0000","last_became_peered":"2024-03-12T03:35:23.777935+0000","last_unstale":"2024-03-12T03:36:30.877555+0000","last_undegraded":"2024-03-12T03:36:30.877555+0000","last_fullsized":"2024-03-12T03:36:30.877555+0000","mapping_epoch":97,"log_start":"101'900","ondisk_log_start":"101'900","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4708,"log_dups_size":0,"ondisk_log_size":4708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:42.206407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367525888,"num_objects":5608,"num_object_clones":0,"num_object_copies":16824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":358912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560767+0000","last_change":"2024-03-12T03:36:19.049860+0000","last_active":"2024-03-12T03:36:29.560767+0000","last_peered":"2024-03-12T03:36:29.560767+0000","last_clean":"2024-03-12T03:36:29.560767+0000","last_became_active":"2024-03-12T03:33:27.383249+0000","last_became_peered":"2024-03-12T03:33:27.383249+0000","last_unstale":"2024-03-12T03:36:29.560767+0000","last_undegraded":"2024-03-12T03:36:29.560767+0000","last_fullsized":"2024-03-12T03:36:29.560767+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.049740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.049740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:43.452600+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.560342+0000","last_change":"2024-03-12T03:36:11.217211+0000","last_active":"2024-03-12T03:36:29.560342+0000","last_peered":"2024-03-12T03:36:29.560342+0000","last_clean":"2024-03-12T03:36:29.560342+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:36:29.560342+0000","last_undegraded":"2024-03-12T03:36:29.560342+0000","last_fullsized":"2024-03-12T03:36:29.560342+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:11.216960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:11.216960+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.390046+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":139,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547086+0000","last_change":"2024-03-12T03:34:47.043566+0000","last_active":"2024-03-12T03:36:29.547086+0000","last_peered":"2024-03-12T03:36:29.547086+0000","last_clean":"2024-03-12T03:36:29.547086+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:36:29.547086+0000","last_undegraded":"2024-03-12T03:36:29.547086+0000","last_fullsized":"2024-03-12T03:36:29.547086+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-03-12T03:34:47.043433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.043433+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":148,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.552368+0000","last_change":"2024-03-12T03:34:25.168128+0000","last_active":"2024-03-12T03:36:29.552368+0000","last_peered":"2024-03-12T03:36:29.552368+0000","last_clean":"2024-03-12T03:36:29.552368+0000","last_became_active":"2024-03-12T03:33:25.177311+0000","last_became_peered":"2024-03-12T03:33:25.177311+0000","last_unstale":"2024-03-12T03:36:29.552368+0000","last_undegraded":"2024-03-12T03:36:29.552368+0000","last_fullsized":"2024-03-12T03:36:29.552368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:25.168002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.168002+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"112'5700","reported_seq":5741,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.838327+0000","last_change":"2024-03-12T03:35:23.778034+0000","last_active":"2024-03-12T03:36:30.838327+0000","last_peered":"2024-03-12T03:36:30.838327+0000","last_clean":"2024-03-12T03:36:30.838327+0000","last_became_active":"2024-03-12T03:35:23.777749+0000","last_became_peered":"2024-03-12T03:35:23.777749+0000","last_unstale":"2024-03-12T03:36:30.838327+0000","last_undegraded":"2024-03-12T03:36:30.838327+0000","last_fullsized":"2024-03-12T03:36:30.838327+0000","mapping_epoch":97,"log_start":"101'1000","ondisk_log_start":"101'1000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:50.657515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373555200,"num_objects":5700,"num_object_clones":0,"num_object_copies":17100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5700,"num_write_kb":364800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.546744+0000","last_change":"2024-03-12T03:34:58.085602+0000","last_active":"2024-03-12T03:36:29.546744+0000","last_peered":"2024-03-12T03:36:29.546744+0000","last_clean":"2024-03-12T03:36:29.546744+0000","last_became_active":"2024-03-12T03:33:27.187864+0000","last_became_peered":"2024-03-12T03:33:27.187864+0000","last_unstale":"2024-03-12T03:36:29.546744+0000","last_undegraded":"2024-03-12T03:36:29.546744+0000","last_fullsized":"2024-03-12T03:36:29.546744+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:58.085499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.085499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.558086+0000","last_change":"2024-03-12T03:34:47.155013+0000","last_active":"2024-03-12T03:36:29.558086+0000","last_peered":"2024-03-12T03:36:29.558086+0000","last_clean":"2024-03-12T03:36:29.558086+0000","last_became_active":"2024-03-12T03:33:21.104430+0000","last_became_peered":"2024-03-12T03:33:21.104430+0000","last_unstale":"2024-03-12T03:36:29.558086+0000","last_undegraded":"2024-03-12T03:36:29.558086+0000","last_fullsized":"2024-03-12T03:36:29.558086+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.154925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.154925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":162,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.551676+0000","last_change":"2024-03-12T03:36:06.083361+0000","last_active":"2024-03-12T03:36:29.551676+0000","last_peered":"2024-03-12T03:36:29.551676+0000","last_clean":"2024-03-12T03:36:29.551676+0000","last_became_active":"2024-03-12T03:33:23.551623+0000","last_became_peered":"2024-03-12T03:33:23.551623+0000","last_unstale":"2024-03-12T03:36:29.551676+0000","last_undegraded":"2024-03-12T03:36:29.551676+0000","last_fullsized":"2024-03-12T03:36:29.551676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-03-12T03:36:06.083052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.083052+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:36.041591+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.546822+0000","last_change":"2024-03-12T03:35:00.071368+0000","last_active":"2024-03-12T03:36:29.546822+0000","last_peered":"2024-03-12T03:36:29.546822+0000","last_clean":"2024-03-12T03:36:29.546822+0000","last_became_active":"2024-03-12T03:33:25.181020+0000","last_became_peered":"2024-03-12T03:33:25.181020+0000","last_unstale":"2024-03-12T03:36:29.546822+0000","last_undegraded":"2024-03-12T03:36:29.546822+0000","last_fullsized":"2024-03-12T03:36:29.546822+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:00.071245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.071245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"112'5704","reported_seq":5740,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.765726+0000","last_change":"2024-03-12T03:35:23.785403+0000","last_active":"2024-03-12T03:36:30.765726+0000","last_peered":"2024-03-12T03:36:30.765726+0000","last_clean":"2024-03-12T03:36:30.765726+0000","last_became_active":"2024-03-12T03:35:23.785037+0000","last_became_peered":"2024-03-12T03:35:23.785037+0000","last_unstale":"2024-03-12T03:36:30.765726+0000","last_undegraded":"2024-03-12T03:36:30.765726+0000","last_fullsized":"2024-03-12T03:36:30.765726+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5704,"log_dups_size":0,"ondisk_log_size":5704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:51.830938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373817344,"num_objects":5704,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5704,"num_write_kb":365056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547579+0000","last_change":"2024-03-12T03:35:17.142169+0000","last_active":"2024-03-12T03:36:29.547579+0000","last_peered":"2024-03-12T03:36:29.547579+0000","last_clean":"2024-03-12T03:36:29.547579+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:36:29.547579+0000","last_undegraded":"2024-03-12T03:36:29.547579+0000","last_fullsized":"2024-03-12T03:36:29.547579+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:17.142086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:17.142086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:34.053176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":148,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547996+0000","last_change":"2024-03-12T03:35:20.138357+0000","last_active":"2024-03-12T03:36:29.547996+0000","last_peered":"2024-03-12T03:36:29.547996+0000","last_clean":"2024-03-12T03:36:29.547996+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:36:29.547996+0000","last_undegraded":"2024-03-12T03:36:29.547996+0000","last_fullsized":"2024-03-12T03:36:29.547996+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:35:20.138257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:20.138257+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:41.959809+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.551582+0000","last_change":"2024-03-12T03:35:13.030518+0000","last_active":"2024-03-12T03:36:29.551582+0000","last_peered":"2024-03-12T03:36:29.551582+0000","last_clean":"2024-03-12T03:36:29.551582+0000","last_became_active":"2024-03-12T03:33:21.104293+0000","last_became_peered":"2024-03-12T03:33:21.104293+0000","last_unstale":"2024-03-12T03:36:29.551582+0000","last_undegraded":"2024-03-12T03:36:29.551582+0000","last_fullsized":"2024-03-12T03:36:29.551582+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:13.030426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:13.030426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.568189+0000","last_change":"2024-03-12T03:36:24.342489+0000","last_active":"2024-03-12T03:36:29.568189+0000","last_peered":"2024-03-12T03:36:29.568189+0000","last_clean":"2024-03-12T03:36:29.568189+0000","last_became_active":"2024-03-12T03:33:50.153240+0000","last_became_peered":"2024-03-12T03:33:50.153240+0000","last_unstale":"2024-03-12T03:36:29.568189+0000","last_undegraded":"2024-03-12T03:36:29.568189+0000","last_fullsized":"2024-03-12T03:36:29.568189+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:36:24.342356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.342356+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.628737+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"112'5635","reported_seq":5676,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.753981+0000","last_change":"2024-03-12T03:35:23.783266+0000","last_active":"2024-03-12T03:36:30.753981+0000","last_peered":"2024-03-12T03:36:30.753981+0000","last_clean":"2024-03-12T03:36:30.753981+0000","last_became_active":"2024-03-12T03:35:23.782945+0000","last_became_peered":"2024-03-12T03:35:23.782945+0000","last_unstale":"2024-03-12T03:36:30.753981+0000","last_undegraded":"2024-03-12T03:36:30.753981+0000","last_fullsized":"2024-03-12T03:36:30.753981+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:52.222430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369295360,"num_objects":5635,"num_object_clones":0,"num_object_copies":16905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5635,"num_write_kb":360640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.551028+0000","last_change":"2024-03-12T03:36:13.114566+0000","last_active":"2024-03-12T03:36:29.551028+0000","last_peered":"2024-03-12T03:36:29.551028+0000","last_clean":"2024-03-12T03:36:29.551028+0000","last_became_active":"2024-03-12T03:33:27.380896+0000","last_became_peered":"2024-03-12T03:33:27.380896+0000","last_unstale":"2024-03-12T03:36:29.551028+0000","last_undegraded":"2024-03-12T03:36:29.551028+0000","last_fullsized":"2024-03-12T03:36:29.551028+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:13.114417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.114417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:37.138551+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547802+0000","last_change":"2024-03-12T03:36:19.182812+0000","last_active":"2024-03-12T03:36:29.547802+0000","last_peered":"2024-03-12T03:36:29.547802+0000","last_clean":"2024-03-12T03:36:29.547802+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:36:29.547802+0000","last_undegraded":"2024-03-12T03:36:29.547802+0000","last_fullsized":"2024-03-12T03:36:29.547802+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.182685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.182685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:25.003468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.546897+0000","last_change":"2024-03-12T03:36:13.157358+0000","last_active":"2024-03-12T03:36:29.546897+0000","last_peered":"2024-03-12T03:36:29.546897+0000","last_clean":"2024-03-12T03:36:29.546897+0000","last_became_active":"2024-03-12T03:33:21.573257+0000","last_became_peered":"2024-03-12T03:33:21.573257+0000","last_unstale":"2024-03-12T03:36:29.546897+0000","last_undegraded":"2024-03-12T03:36:29.546897+0000","last_fullsized":"2024-03-12T03:36:29.546897+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:13.157235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.157235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:31.473419+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"95'23","reported_seq":143,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547891+0000","last_change":"2024-03-12T03:35:06.028109+0000","last_active":"2024-03-12T03:36:29.547891+0000","last_peered":"2024-03-12T03:36:29.547891+0000","last_clean":"2024-03-12T03:36:29.547891+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:36:29.547891+0000","last_undegraded":"2024-03-12T03:36:29.547891+0000","last_fullsized":"2024-03-12T03:36:29.547891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'23","last_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:35:06.028013+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"112'5762","reported_seq":5803,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.847898+0000","last_change":"2024-03-12T03:35:23.777241+0000","last_active":"2024-03-12T03:36:30.847898+0000","last_peered":"2024-03-12T03:36:30.847898+0000","last_clean":"2024-03-12T03:36:30.847898+0000","last_became_active":"2024-03-12T03:35:23.776932+0000","last_became_peered":"2024-03-12T03:35:23.776932+0000","last_unstale":"2024-03-12T03:36:30.847898+0000","last_undegraded":"2024-03-12T03:36:30.847898+0000","last_fullsized":"2024-03-12T03:36:30.847898+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:42.893097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377618432,"num_objects":5762,"num_object_clones":0,"num_object_copies":17286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5762,"num_write_kb":368768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.568412+0000","last_change":"2024-03-12T03:35:56.071430+0000","last_active":"2024-03-12T03:36:29.568412+0000","last_peered":"2024-03-12T03:36:29.568412+0000","last_clean":"2024-03-12T03:36:29.568412+0000","last_became_active":"2024-03-12T03:33:27.187815+0000","last_became_peered":"2024-03-12T03:33:27.187815+0000","last_unstale":"2024-03-12T03:36:29.568412+0000","last_undegraded":"2024-03-12T03:36:29.568412+0000","last_fullsized":"2024-03-12T03:36:29.568412+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:56.071324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:56.071324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:05.061549+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.555527+0000","last_change":"2024-03-12T03:36:04.907090+0000","last_active":"2024-03-12T03:36:29.555527+0000","last_peered":"2024-03-12T03:36:29.555527+0000","last_clean":"2024-03-12T03:36:29.555527+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:36:29.555527+0000","last_undegraded":"2024-03-12T03:36:29.555527+0000","last_fullsized":"2024-03-12T03:36:29.555527+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:04.906798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:04.906798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:31.071522+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.556004+0000","last_change":"2024-03-12T03:35:59.016620+0000","last_active":"2024-03-12T03:36:29.556004+0000","last_peered":"2024-03-12T03:36:29.556004+0000","last_clean":"2024-03-12T03:36:29.556004+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:36:29.556004+0000","last_undegraded":"2024-03-12T03:36:29.556004+0000","last_fullsized":"2024-03-12T03:36:29.556004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.016489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.016489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:28.178945+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":134,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.548318+0000","last_change":"2024-03-12T03:35:59.053600+0000","last_active":"2024-03-12T03:36:29.548318+0000","last_peered":"2024-03-12T03:36:29.548318+0000","last_clean":"2024-03-12T03:36:29.548318+0000","last_became_active":"2024-03-12T03:33:23.163871+0000","last_became_peered":"2024-03-12T03:33:23.163871+0000","last_unstale":"2024-03-12T03:36:29.548318+0000","last_undegraded":"2024-03-12T03:36:29.548318+0000","last_fullsized":"2024-03-12T03:36:29.548318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:35:59.053486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.053486+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:03.009127+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"112'5655","reported_seq":5696,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:30.824546+0000","last_change":"2024-03-12T03:35:23.784927+0000","last_active":"2024-03-12T03:36:30.824546+0000","last_peered":"2024-03-12T03:36:30.824546+0000","last_clean":"2024-03-12T03:36:30.824546+0000","last_became_active":"2024-03-12T03:35:23.784627+0000","last_became_peered":"2024-03-12T03:35:23.784627+0000","last_unstale":"2024-03-12T03:36:30.824546+0000","last_undegraded":"2024-03-12T03:36:30.824546+0000","last_fullsized":"2024-03-12T03:36:30.824546+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5655,"log_dups_size":0,"ondisk_log_size":5655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:32.369495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370606080,"num_objects":5655,"num_object_clones":0,"num_object_copies":16965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":361920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.548091+0000","last_change":"2024-03-12T03:36:07.101180+0000","last_active":"2024-03-12T03:36:29.548091+0000","last_peered":"2024-03-12T03:36:29.548091+0000","last_clean":"2024-03-12T03:36:29.548091+0000","last_became_active":"2024-03-12T03:33:27.184405+0000","last_became_peered":"2024-03-12T03:33:27.184405+0000","last_unstale":"2024-03-12T03:36:29.548091+0000","last_undegraded":"2024-03-12T03:36:29.548091+0000","last_fullsized":"2024-03-12T03:36:29.548091+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:07.101098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.101098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:34.696849+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":156,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.571253+0000","last_change":"2024-03-12T03:36:25.346270+0000","last_active":"2024-03-12T03:36:29.571253+0000","last_peered":"2024-03-12T03:36:29.571253+0000","last_clean":"2024-03-12T03:36:29.571253+0000","last_became_active":"2024-03-12T03:33:25.177613+0000","last_became_peered":"2024-03-12T03:33:25.177613+0000","last_unstale":"2024-03-12T03:36:29.571253+0000","last_undegraded":"2024-03-12T03:36:29.571253+0000","last_fullsized":"2024-03-12T03:36:29.571253+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:25.346151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.346151+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:44.762442+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.552821+0000","last_change":"2024-03-12T03:35:15.981464+0000","last_active":"2024-03-12T03:36:29.552821+0000","last_peered":"2024-03-12T03:36:29.552821+0000","last_clean":"2024-03-12T03:36:29.552821+0000","last_became_active":"2024-03-12T03:33:21.097304+0000","last_became_peered":"2024-03-12T03:33:21.097304+0000","last_unstale":"2024-03-12T03:36:29.552821+0000","last_undegraded":"2024-03-12T03:36:29.552821+0000","last_fullsized":"2024-03-12T03:36:29.552821+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.981378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.981378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:36:41.402156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":121,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-12T03:36:29.547615+0000","last_change":"2024-03-12T03:35:58.089544+0000","last_active":"2024-03-12T03:36:29.547615+0000","last_peered":"2024-03-12T03:36:29.547615+0000","last_clean":"2024-03-12T03:36:29.547615+0000","last_became_active":"2024-03-12T03:33:23.162657+0000","last_became_peered":"2024-03-12T03:33:23.162657+0000","last_unstale":"2024-03-12T03:36:29.547615+0000","last_undegraded":"2024-03-12T03:36:29.547615+0000","last_fullsized":"2024-03-12T03:36:29.547615+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-12T03:35:58.089405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.089405+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:25.792358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5922684928,"num_objects":90373,"num_object_clones":0,"num_object_copies":271119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90373,"num_write_kb":5783872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17771659264,"data_stored":17771659264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85673,"ondisk_log_size":85673,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":184,"num_read_kb":179,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":630,"ondisk_log_size":630,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070507,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1847940,"kb_used_data":1790392,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":91897212,"statfs":{"total":95995035648,"available":94102745088,"internally_reserved":0,"allocated":1833361408,"data_stored":1832924706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332144,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1846128,"kb_used_data":1788576,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":91899024,"statfs":{"total":95995035648,"available":94104600576,"internally_reserved":0,"allocated":1831501824,"data_stored":1831087735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2213708,"kb_used_data":2146436,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91531444,"statfs":{"total":95995035648,"available":93728198656,"internally_reserved":0,"allocated":2197950464,"data_stored":2197500597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2945408,"kb_used_data":2882296,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":90799744,"statfs":{"total":95995035648,"available":92978937856,"internally_reserved":0,"allocated":2951471104,"data_stored":2951033656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2228200,"kb_used_data":2160928,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91516952,"statfs":{"total":95995035648,"available":93713358848,"internally_reserved":0,"allocated":2212790272,"data_stored":2212376872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117054,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3302420,"kb_used_data":3217092,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":90442732,"statfs":{"total":95995035648,"available":92613357568,"internally_reserved":0,"allocated":3294302208,"data_stored":3293852038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":87357895},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953531,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1853508,"kb_used_data":1795960,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":91891644,"statfs":{"total":95995035648,"available":94097043456,"internally_reserved":0,"allocated":1839063040,"data_stored":1838625334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659189,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1486264,"kb_used_data":1432876,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92258888,"statfs":{"total":95995035648,"available":94473101312,"internally_reserved":0,"allocated":1467265024,"data_stored":1466838774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2973040640,"data_stored":2973040640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2216427520,"data_stored":2216427520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2229927936,"data_stored":2229927936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3325231104,"data_stored":3325231104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1853947904,"data_stored":1853947904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1477902336,"data_stored":1477902336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1843527680,"data_stored":1843527680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1851654144,"data_stored":1851654144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:36:38.174 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-12T03:36:38.175 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool set unique_pool_0 pg_num 26 2024-03-12T03:36:38.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:38.308 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:38.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:38.333 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:38.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:38.445 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:36:38.445 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:36:38.479 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:38.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:38.575 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 67 16 98081 98065 91.4564 61.5 0.0229937 0.010929 2024-03-12T03:36:38.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:38.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:38.625 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:38.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:38.726 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:38.776 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:38.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:38.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:38.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:38.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:39.009 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:39.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:39.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:39.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:39.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:39.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:39.271 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:39.274 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:39 smithi169 ceph-mon[29344]: pgmap v112: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-12T03:36:39.274 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:39 smithi169 ceph-mon[29344]: osdmap e114: 8 total, 8 up, 8 in 2024-03-12T03:36:39.275 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:39 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:36:39.275 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:39 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:36:39.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:39 smithi169 ceph-mon[39949]: pgmap v112: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-12T03:36:39.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:39 smithi169 ceph-mon[39949]: osdmap e114: 8 total, 8 up, 8 in 2024-03-12T03:36:39.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:39 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:36:39.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:39 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:36:39.318 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:39.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:39.419 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:39.448 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:39.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:39.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:39 smithi173 ceph-mon[33544]: pgmap v112: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-12T03:36:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:39 smithi173 ceph-mon[33544]: osdmap e114: 8 total, 8 up, 8 in 2024-03-12T03:36:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:39 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:36:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:39 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:36:39.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-12T03:36:39.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:39.576 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 68 16 99129 99113 91.0743 65.5 0.00778496 0.0109751 2024-03-12T03:36:39.642 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:39.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:39.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:39.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:39.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:39.788 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:39.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:39.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:39.918 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:39.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:39.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:40.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:40.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:40.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:40.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:40.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:40.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:40.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:40.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:40.283 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:40.320 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:40.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:40.416 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:40 smithi169 ceph-mon[29344]: osdmap e115: 8 total, 8 up, 8 in 2024-03-12T03:36:40.416 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:40 smithi169 ceph-mon[39949]: osdmap e115: 8 total, 8 up, 8 in 2024-03-12T03:36:40.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:40.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:40.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:40 smithi173 ceph-mon[33544]: osdmap e115: 8 total, 8 up, 8 in 2024-03-12T03:36:40.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:40.575 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 69 15 100160 100145 90.6892 64.5 0.00850197 0.0110211 2024-03-12T03:36:40.580 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:40.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:40.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:40.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:40.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:40.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:40.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:40.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:40.865 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:40.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:40.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:41.013 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:41.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:41.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:41.157 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:41.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:41.226 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:36:41.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:41.271 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:36:41.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:41 smithi169 ceph-mon[29344]: pgmap v115: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 772 B/s rd, 47 MiB/s wr, 756 op/s 2024-03-12T03:36:41.272 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:41 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1484132592' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:36:41.272 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:41 smithi169 ceph-mon[39949]: pgmap v115: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 772 B/s rd, 47 MiB/s wr, 756 op/s 2024-03-12T03:36:41.272 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:41 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1484132592' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:36:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:41.314 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:41.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:41.321 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:36:41.361 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:36:41.403 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:36:41.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:41.464 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:36:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:41.482 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:41.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:41 smithi173 ceph-mon[33544]: pgmap v115: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 772 B/s rd, 47 MiB/s wr, 756 op/s 2024-03-12T03:36:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:41 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1484132592' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:36:41.513 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:36:41.554 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:36:41.572 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:36:41.572 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:36:41.572 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:36:41.572 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:36:41.572 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:36:41.572 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:36:41.572 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:36:41.572 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:36:41.575 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 70 16 101133 101117 90.2614 60.75 0.0104783 0.0110732 2024-03-12T03:36:41.583 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:41.624 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:41.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:41.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:41.754 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:41.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:41.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:41.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:41.891 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:41.900 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:41.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:41.974 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:42.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:42.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:42.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:42.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:42.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:42.176 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:36:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:36:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:36:42.184 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:42.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:42.240 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:36:42.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:42.320 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:42.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:42.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:42.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:42.459 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:42.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:42.474 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:42 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2774040859' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:36:42.474 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:42 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2774040859' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:36:42.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:36:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:36:42.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:42 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2774040859' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:36:42.562 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:42.576 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 71 16 102090 102074 89.8325 59.8125 0.0115013 0.0111265 2024-03-12T03:36:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:42.603 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:42.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:42.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:42.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:42.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:42.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:42.886 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:42.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:42.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:43.024 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:43.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:43.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:43.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:43.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:43.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:43.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:43.303 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:43.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:43.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:43.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:43.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:43 smithi169 ceph-mon[29344]: pgmap v116: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-12T03:36:43.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:43 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:36:43.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:43 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2774040859' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:36:43.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:43 smithi169 ceph-mon[29344]: osdmap e116: 8 total, 8 up, 8 in 2024-03-12T03:36:43.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:43 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:43.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:43 smithi169 ceph-mon[29344]: osdmap e117: 8 total, 8 up, 8 in 2024-03-12T03:36:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:43 smithi169 ceph-mon[39949]: pgmap v116: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-12T03:36:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:43 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:36:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:43 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2774040859' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:36:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:43 smithi169 ceph-mon[39949]: osdmap e116: 8 total, 8 up, 8 in 2024-03-12T03:36:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:43 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:43.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:43 smithi169 ceph-mon[39949]: osdmap e117: 8 total, 8 up, 8 in 2024-03-12T03:36:43.452 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:43.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:43.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:43 smithi173 ceph-mon[33544]: pgmap v116: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-12T03:36:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:43 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:36:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:43 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2774040859' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:36:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:43 smithi173 ceph-mon[33544]: osdmap e116: 8 total, 8 up, 8 in 2024-03-12T03:36:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:43 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:43 smithi173 ceph-mon[33544]: osdmap e117: 8 total, 8 up, 8 in 2024-03-12T03:36:43.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:43.576 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 72 16 103150 103134 89.505 66.25 0.0151068 0.011167 2024-03-12T03:36:43.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:43.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:43.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:43.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:43.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:43.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:43.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:43.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:43.949 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:43.991 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:43.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:44.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:44.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:44.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:44.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:44.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:44.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:44.248 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:44.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:44.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:44.392 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:44.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:44.494 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:44.522 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:44.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:44.576 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 73 16 104114 104098 89.1041 60.25 0.00884466 0.0112176 2024-03-12T03:36:44.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:44.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:44.656 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:44.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:44.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:44.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:44.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:44.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:36:44.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:44.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:44.943 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:44.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:45.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:45.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:45.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:45.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:45.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:45.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:45.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:45.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:45.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:45.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:45.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:45.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:45 smithi169 ceph-mon[29344]: pgmap v119: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 908 B/s rd, 56 MiB/s wr, 897 op/s 2024-03-12T03:36:45.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:45 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:45.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:45 smithi169 ceph-mon[39949]: pgmap v119: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 908 B/s rd, 56 MiB/s wr, 897 op/s 2024-03-12T03:36:45.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:45 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:45.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:45.484 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:45.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:45 smithi173 ceph-mon[33544]: pgmap v119: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 908 B/s rd, 56 MiB/s wr, 897 op/s 2024-03-12T03:36:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:45 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:36:45.586 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:45.595 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 74 16 105155 105139 88.7563 65.0625 0.00456769 0.0112611 2024-03-12T03:36:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:45.639 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:45.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:45.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:45.788 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:45.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:45.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:45.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:45.926 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:45.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:46.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:46.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:46.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:46.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:46.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:46.204 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:46.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:46.237 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:46.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:46.351 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:46.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:46.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:46.480 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:46.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:46.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:46.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:46.595 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 75 16 106141 106125 88.3945 61.625 0.0313873 0.0113081 2024-03-12T03:36:46.633 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:46.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:46.734 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:46.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:46.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:46.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:46.784 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:36:46.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:46.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:46.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:46.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:47.041 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:47.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:47.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:47.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:47.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:47.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:set pool 7 pg_num to 26 2024-03-12T03:36:47.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:47.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:47.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:47.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:47.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:47 smithi169 ceph-mon[29344]: pgmap v120: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-12T03:36:47.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:47 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1728140491' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-12T03:36:47.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:47 smithi169 ceph-mon[39949]: pgmap v120: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-12T03:36:47.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:47 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1728140491' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-12T03:36:47.443 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:47.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:47.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:47.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:47.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:47.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:47.595 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 76 16 107088 107072 88.0103 59.1875 0.0204567 0.011357 2024-03-12T03:36:47.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:47.718 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:47.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:47 smithi173 ceph-mon[33544]: pgmap v120: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-12T03:36:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:47 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1728140491' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-12T03:36:47.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:47.865 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:47.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:47.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:47.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:47.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:48.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:48.134 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:48.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:48.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:48.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:48.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:48.374 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:48.416 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:48.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:48.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:48.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:48.543 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:48.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:48.595 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 77 16 107740 107724 87.3967 40.75 0.314982 0.0113935 2024-03-12T03:36:48.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:48.667 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:48.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:48.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:48 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1728140491' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-03-12T03:36:48.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:48 smithi169 ceph-mon[39949]: osdmap e118: 8 total, 8 up, 8 in 2024-03-12T03:36:48.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:48 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-12T03:36:48.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:48 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-12T03:36:48.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:48 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-12T03:36:48.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:48 smithi169 ceph-mon[39949]: osdmap e119: 8 total, 8 up, 8 in 2024-03-12T03:36:48.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:48 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1728140491' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-03-12T03:36:48.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:48 smithi169 ceph-mon[29344]: osdmap e118: 8 total, 8 up, 8 in 2024-03-12T03:36:48.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:48 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-12T03:36:48.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:48 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-12T03:36:48.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:48 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-12T03:36:48.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:48 smithi169 ceph-mon[29344]: osdmap e119: 8 total, 8 up, 8 in 2024-03-12T03:36:48.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:48 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1728140491' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-03-12T03:36:48.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:48 smithi173 ceph-mon[33544]: osdmap e118: 8 total, 8 up, 8 in 2024-03-12T03:36:48.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:48 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-12T03:36:48.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:48 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-12T03:36:48.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:48 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-12T03:36:48.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:48 smithi173 ceph-mon[33544]: osdmap e119: 8 total, 8 up, 8 in 2024-03-12T03:36:48.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:48.792 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:48.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:48.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:48.919 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:48.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:48.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:48.943 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:48.972 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:36:48.972 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:36:48.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:36:48.972 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:36:48.972 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:23347 7.e 7:7a3cd58e:::benchmark_data_smithi169_79822_object23346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:40.220724+0000", 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 68.747835652999996, 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.085619522000000003, 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 23347 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.973 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.220724+0000", 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.220724+0000", 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.220725+0000", 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4890000000000001e-06 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.220747+0000", 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1979000000000001e-05 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.220756+0000", 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2019999999999993e-06 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.220764+0000", 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2060000000000001e-06 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:48.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.220823+0000", 2024-03-12T03:36:48.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9542e-05 2024-03-12T03:36:48.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:48.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.220937+0000", 2024-03-12T03:36:48.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000113977 2024-03-12T03:36:48.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:36:48.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.221020+0000", 2024-03-12T03:36:48.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2466000000000006e-05 2024-03-12T03:36:48.977 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.977 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:48.977 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.222407+0000", 2024-03-12T03:36:48.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013877270000000001 2024-03-12T03:36:48.977 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.977 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:48.977 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.222850+0000", 2024-03-12T03:36:48.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00044237299999999997 2024-03-12T03:36:48.977 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:48.978 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.306227+0000", 2024-03-12T03:36:48.978 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.083377609000000005 2024-03-12T03:36:48.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:48.978 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.306290+0000", 2024-03-12T03:36:48.978 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3021000000000005e-05 2024-03-12T03:36:48.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.306343+0000", 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2930999999999999e-05 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:23350 7.e 7:719c0058:::benchmark_data_smithi169_79822_object23349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:40.221470+0000", 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 68.747089470999995, 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.085640788999999995, 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:48.979 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 23350 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.221470+0000", 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.221470+0000", 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:48.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.221471+0000", 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2109999999999999e-06 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.221492+0000", 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0699999999999998e-05 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.221504+0000", 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2282e-05 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.221513+0000", 2024-03-12T03:36:48.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4579999999999993e-06 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.221755+0000", 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000242375 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.221844+0000", 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8873000000000001e-05 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:36:48.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.221955+0000", 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011089100000000001 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.222528+0000", 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00057318799999999997 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.222948+0000", 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000420402 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.307025+0000", 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.084077210999999999 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.307073+0000", 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7834000000000003e-05 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:40.307111+0000", 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7363999999999997e-05 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:48.984 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:48.985 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:48.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.985 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.985 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:25844 7.e 7:74e6ffde:::benchmark_data_smithi169_79822_object25843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:36:48.985 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:41.622004+0000", 2024-03-12T03:36:48.985 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 67.346555471000002, 2024-03-12T03:36:48.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.059941442999999997, 2024-03-12T03:36:48.985 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:48.985 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:48.985 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:48.985 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:48.986 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:48.986 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 25844 2024-03-12T03:36:48.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622004+0000", 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622004+0000", 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:48.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622004+0000", 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1900000000000003e-07 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622031+0000", 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6414000000000002e-05 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622038+0000", 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8090000000000004e-06 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622043+0000", 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8880000000000002e-06 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622098+0000", 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4280999999999997e-05 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622213+0000", 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115215 2024-03-12T03:36:48.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622350+0000", 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013736799999999999 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.678447+0000", 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.056096840000000002 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:48.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.678459+0000", 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1936999999999999e-05 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.681838+0000", 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033793500000000001 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.681896+0000", 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7701999999999997e-05 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.991 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.681945+0000", 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9119999999999997e-05 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:25849 7.e 7:77a4e8cb:::benchmark_data_smithi169_79822_object25848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:41.622111+0000", 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 67.346448238999997, 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.060210280999999997, 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:48.992 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 25849 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622111+0000", 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622111+0000", 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:48.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622112+0000", 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5600000000000004e-07 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622132+0000", 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0428000000000002e-05 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622143+0000", 2024-03-12T03:36:48.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0502e-05 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.622150+0000", 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.503e-06 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.623438+0000", 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012880140000000001 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.623554+0000", 2024-03-12T03:36:48.995 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115058 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.623679+0000", 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000125847 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.678466+0000", 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.054786804000000001 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.678643+0000", 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017657400000000001 2024-03-12T03:36:48.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.682240+0000", 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0035970960000000001 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.682287+0000", 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7129999999999998e-05 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:41.682321+0000", 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4369000000000002e-05 2024-03-12T03:36:48.997 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:27415 7.e 7:7d507fd2:::benchmark_data_smithi169_79822_object27414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:42.502008+0000", 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 66.466551710000005, 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.080853678999999998, 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:48.998 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 27415 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.502008+0000", 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.502008+0000", 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:48.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.502009+0000", 2024-03-12T03:36:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7699999999999992e-07 2024-03-12T03:36:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.502027+0000", 2024-03-12T03:36:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8739000000000001e-05 2024-03-12T03:36:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.502036+0000", 2024-03-12T03:36:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6470000000000004e-06 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.502042+0000", 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5639999999999996e-06 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.502095+0000", 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3927000000000001e-05 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.502244+0000", 2024-03-12T03:36:49.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014872600000000001 2024-03-12T03:36:49.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.002 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:36:49.002 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.502363+0000", 2024-03-12T03:36:49.002 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000118353 2024-03-12T03:36:49.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.002 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:49.002 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.503295+0000", 2024-03-12T03:36:49.002 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00093282100000000002 2024-03-12T03:36:49.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.503772+0000", 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00047678899999999998 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.582718+0000", 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.078946274999999996 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.582783+0000", 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4766999999999998e-05 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:42.582861+0000", 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8094000000000001e-05 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:29235 7.e 7:7ca074cb:::benchmark_data_smithi169_79822_object29234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:43.519729+0000", 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 65.448830749999999, 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13258550199999999, 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:49.004 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 29235 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519729+0000", 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519729+0000", 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.005 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.006 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519730+0000", 2024-03-12T03:36:49.006 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.559e-06 2024-03-12T03:36:49.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.006 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.006 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.006 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519753+0000", 2024-03-12T03:36:49.006 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3150999999999999e-05 2024-03-12T03:36:49.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.006 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.006 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.006 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519762+0000", 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8279999999999992e-06 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519769+0000", 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5270000000000004e-06 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519910+0000", 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014151999999999999 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.520044+0000", 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013344 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.520170+0000", 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000125931 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.522569+0000", 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002399834 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.522609+0000", 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9787999999999999e-05 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.652211+0000", 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12960175600000001 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.652267+0000", 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6144000000000003e-05 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.652314+0000", 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7024000000000002e-05 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:29237 7.e 7:728c8aa7:::benchmark_data_smithi169_79822_object29236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:43.519783+0000", 2024-03-12T03:36:49.010 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 65.448776670000001, 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13331062299999999, 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 29237 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519783+0000", 2024-03-12T03:36:49.011 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519783+0000", 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519783+0000", 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2800000000000002e-07 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519803+0000", 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9887000000000001e-05 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519809+0000", 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9120000000000003e-06 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.519814+0000", 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.293e-06 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.521276+0000", 2024-03-12T03:36:49.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014619699999999999 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.521492+0000", 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000215763 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.521623+0000", 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000130815 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.522757+0000", 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0011344700000000001 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.523636+0000", 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00087847200000000004 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.653004+0000", 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12936852099999999 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.653056+0000", 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1823000000000002e-05 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:43.653093+0000", 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7268999999999998e-05 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:36235 7.8 7:19f074ae:::benchmark_data_smithi169_79822_object36234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:35:47.715367+0000", 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 61.253192355000003, 2024-03-12T03:36:49.016 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13984633399999999, 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 36235 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.715367+0000", 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.017 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.715367+0000", 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.715368+0000", 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.95e-07 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.715384+0000", 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6402999999999999e-05 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.715396+0000", 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1362000000000001e-05 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.715401+0000", 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7270000000000004e-06 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.715443+0000", 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1143000000000002e-05 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.715518+0000", 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5846000000000005e-05 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.715598+0000", 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9468999999999994e-05 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.717003+0000", 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001405208 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.020 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.718312+0000", 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013090039999999999 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.855128+0000", 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13681547399999999 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.855184+0000", 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6453999999999999e-05 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:35:47.855213+0000", 2024-03-12T03:36:49.021 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9249000000000001e-05 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:63763 7.a e106/97 7:5ad6c598:::benchmark_data_smithi169_79822_object63762:head v 106'4028, mlcod=106'4025)", 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.195866+0000", 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 41.772692878000001, 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.089420745999999995, 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.195866+0000", 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.195866+0000", 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.195867+0000", 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9299999999999996e-07 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.195917+0000", 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9567000000000002e-05 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.195929+0000", 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2306999999999999e-05 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.195938+0000", 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7900000000000005e-06 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.195997+0000", 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8802999999999999e-05 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.196018+0000", 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1665e-05 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285240+0000", 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.089221493999999998 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.025 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.285287+0000", 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7227e-05 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:63772 7.5 e106/97 7:acf8b1b7:::benchmark_data_smithi169_79822_object63771:head v 106'3963, mlcod=106'3961)", 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:07.197532+0000", 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 41.771027410999999, 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.089865772999999996, 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:49.026 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.197532+0000", 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.197532+0000", 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.197532+0000", 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.68e-07 2024-03-12T03:36:49.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.197570+0000", 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7799999999999997e-05 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.197579+0000", 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1130000000000007e-06 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.197586+0000", 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7499999999999997e-06 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.197635+0000", 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9295999999999997e-05 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.197655+0000", 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9111999999999999e-05 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.287319+0000", 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.089664406000000002 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:07.287398+0000", 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8727999999999996e-05 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:66564 7.8 7:1c755348:::benchmark_data_smithi169_79822_object66563:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:09.489052+0000", 2024-03-12T03:36:49.030 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.479507013999999, 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062588167, 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 66564 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.489052+0000", 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.489052+0000", 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.489053+0000", 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8400000000000003e-07 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.489088+0000", 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4907999999999998e-05 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.489097+0000", 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4110000000000002e-06 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.489113+0000", 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5474000000000001e-05 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.496932+0000", 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078187689999999997 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.497053+0000", 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121218 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.497167+0000", 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011442699999999999 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.551456+0000", 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.054289086 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.551502+0000", 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5631999999999999e-05 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.551547+0000", 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4608999999999999e-05 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.551597+0000", 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0868999999999997e-05 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.551640+0000", 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2979999999999998e-05 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:66569 7.8 7:1e142e23:::benchmark_data_smithi169_79822_object66568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:09.492947+0000", 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.475611960000002, 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062063741999999998, 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:49.036 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 66569 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.492947+0000", 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.492947+0000", 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.492948+0000", 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2999999999999999e-07 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.492969+0000", 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1228000000000001e-05 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.492993+0000", 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3901999999999999e-05 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:49.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.493006+0000", 2024-03-12T03:36:49.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2186e-05 2024-03-12T03:36:49.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.546909+0000", 2024-03-12T03:36:49.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.053903635999999998 2024-03-12T03:36:49.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.547030+0000", 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000120469 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.547163+0000", 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000133356 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.554410+0000", 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072467520000000004 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.554464+0000", 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4101999999999998e-05 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.554479+0000", 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.482e-05 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.554948+0000", 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00046982099999999999 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.555011+0000", 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2639999999999997e-05 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:66577 7.e 7:706709fc:::benchmark_data_smithi169_79822_object66576:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:09.497706+0000", 2024-03-12T03:36:49.043 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.470853185999999, 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.060873363, 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 66577 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.497706+0000", 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.497706+0000", 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.497707+0000", 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1820000000000001e-06 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.497728+0000", 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.071e-05 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.497737+0000", 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7360000000000007e-06 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.497743+0000", 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6610000000000003e-06 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.497790+0000", 2024-03-12T03:36:49.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6258999999999999e-05 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.497903+0000", 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000113572 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.498035+0000", 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000131626 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.544554+0000", 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.046519304999999997 2024-03-12T03:36:49.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.544596+0000", 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1582000000000003e-05 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.558435+0000", 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013838961 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.558528+0000", 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2783999999999998e-05 2024-03-12T03:36:49.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:09.558579+0000", 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1984999999999998e-05 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:104033 7.e 7:7030fa54:::benchmark_data_smithi169_79822_object104032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:44.435815+0000", 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.5327445219999998, 2024-03-12T03:36:49.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.086704981, 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 104033 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.435815+0000", 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.435815+0000", 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.435816+0000", 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9899999999999999e-07 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.435844+0000", 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8112000000000001e-05 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.435854+0000", 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0144e-05 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.435861+0000", 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.153e-06 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.436797+0000", 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00093551400000000003 2024-03-12T03:36:49.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.436915+0000", 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000118486 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.437042+0000", 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012686500000000001 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.444616+0000", 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075745480000000004 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.444630+0000", 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3473e-05 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.522257+0000", 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.077627246999999996 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.522471+0000", 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000213406 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.522520+0000", 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9234000000000003e-05 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:104036 7.e 7:7f1d6192:::benchmark_data_smithi169_79822_object104035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:44.440065+0000", 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.5284945969999999, 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.085858806999999995, 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:49.055 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 104036 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.440065+0000", 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.440065+0000", 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.440066+0000", 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6619999999999999e-06 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.440086+0000", 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0035999999999999e-05 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.440105+0000", 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8182e-05 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.440112+0000", 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6199999999999999e-06 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.441053+0000", 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00094030199999999996 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.441185+0000", 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013283300000000001 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.441378+0000", 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019280799999999999 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.449796+0000", 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0084175550000000002 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.449842+0000", 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6224999999999999e-05 2024-03-12T03:36:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.525813+0000", 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.075971438000000002 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.525880+0000", 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6129000000000003e-05 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.525924+0000", 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4017000000000002e-05 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.060 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:104037 7.e 7:7b34774a:::benchmark_data_smithi169_79822_object104036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:44.440128+0000", 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.5284317569999999, 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.090351916000000004, 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 104037 2024-03-12T03:36:49.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.440128+0000", 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.440128+0000", 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.440128+0000", 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1999999999999999e-07 2024-03-12T03:36:49.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.440153+0000", 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4556e-05 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.440160+0000", 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.956e-06 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.440166+0000", 2024-03-12T03:36:49.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8549999999999998e-06 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.444676+0000", 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045100210000000003 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.444785+0000", 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109546 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.444898+0000", 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011285000000000001 2024-03-12T03:36:49.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.449823+0000", 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049252009999999997 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.449849+0000", 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5953999999999999e-05 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.530353+0000", 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.080504276 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.530426+0000", 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2939000000000001e-05 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.530479+0000", 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3142e-05 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:104043 7.e 7:74b73fe6:::benchmark_data_smithi169_79822_object104042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:44.442340+0000", 2024-03-12T03:36:49.066 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.5262188840000004, 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.096684297000000002, 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 104043 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.442340+0000", 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.442340+0000", 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.442342+0000", 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.53e-06 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.442373+0000", 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1488000000000001e-05 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.442387+0000", 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3481e-05 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.442395+0000", 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4030000000000007e-06 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.447852+0000", 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054569880000000003 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:36:49.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.447954+0000", 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101587 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.448047+0000", 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3379000000000002e-05 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.449886+0000", 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018390489999999999 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.449973+0000", 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6725999999999996e-05 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.538932+0000", 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.088958713999999994 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.538984+0000", 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2002000000000001e-05 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:44.539025+0000", 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0949999999999999e-05 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:107704 7.2 e118/97)", 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:48.181217+0000", 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.78734216300000004, 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.123368435, 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181217+0000", 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181217+0000", 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181218+0000", 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1900000000000001e-07 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181241+0000", 2024-03-12T03:36:49.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3400999999999998e-05 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181254+0000", 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2379e-05 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181262+0000", 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0390000000000005e-06 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.304562+0000", 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.123299664 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.304586+0000", 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4032999999999998e-05 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:107724 7.3 e118/97)", 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:48.181816+0000", 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.78674316399999999, 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14563426900000001, 2024-03-12T03:36:49.075 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181816+0000", 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181816+0000", 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181817+0000", 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8800000000000002e-07 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181844+0000", 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6902000000000001e-05 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181854+0000", 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0237999999999999e-05 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181862+0000", 2024-03-12T03:36:49.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6499999999999996e-06 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.327391+0000", 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.145529713 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.327450+0000", 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9077999999999999e-05 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.078 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:107729 7.8 7.8765df8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:48.194419+0000", 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.774139935, 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14236865200000001, 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 107729 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.194419+0000", 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.194419+0000", 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.194422+0000", 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3829999999999999e-06 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.194457+0000", 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4898999999999999e-05 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.194474+0000", 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7079999999999999e-05 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.194485+0000", 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0757999999999999e-05 2024-03-12T03:36:49.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.336774+0000", 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14228902399999999 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.336788+0000", 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4508000000000001e-05 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:36:49.082 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:36:49.109 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:49.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:49.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:49.236 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:49.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:49.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:49.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:49.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:49.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:49.505 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:49.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:49.595 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 78 16 108108 108092 86.5713 23 0.0104515 0.0115459 2024-03-12T03:36:49.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:49.635 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:49.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:49.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:49.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:49 smithi169 ceph-mon[29344]: pgmap v122: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s 2024-03-12T03:36:49.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:49 smithi169 ceph-mon[39949]: pgmap v122: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s 2024-03-12T03:36:49.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:49 smithi173 ceph-mon[33544]: pgmap v122: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s 2024-03-12T03:36:49.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:49.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:49.777 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:49.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:49.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:49.916 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:49.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:49.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:50.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:50.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:50.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:50.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:50.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:50.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:50.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:50.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:50.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:50.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:50.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:50.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:50 smithi169 ceph-mon[29344]: osdmap e120: 8 total, 8 up, 8 in 2024-03-12T03:36:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:50 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-12T03:36:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:50 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-12T03:36:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:50 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3533096854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:36:50.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:50 smithi169 ceph-mon[39949]: osdmap e120: 8 total, 8 up, 8 in 2024-03-12T03:36:50.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:50 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-12T03:36:50.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:50 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-12T03:36:50.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:50 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3533096854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:36:50.436 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:50.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:50.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:50.561 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:50.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:50.595 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 79 16 108995 108979 86.1773 55.4375 0.00233005 0.0115562 2024-03-12T03:36:50.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:50.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:50.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:50 smithi173 ceph-mon[33544]: osdmap e120: 8 total, 8 up, 8 in 2024-03-12T03:36:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:50 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-12T03:36:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:50 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-12T03:36:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:50 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3533096854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:36:50.743 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:36:50.743 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:36:50.743 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:36:50.743 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:36:50.743 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:36:50.743 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:36:50.743 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:36:50.743 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:36:50.744 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:36:50.744 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:36:50.744 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:36:50.744 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:36:50.744 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:36:50.744 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:36:50.744 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:36:50.744 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:36:50.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:50.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:50.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:50.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:50.941 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:50.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:51.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:51.067 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:51.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:51.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:51.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:51.194 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:51.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:51.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:51.320 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:51.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:51.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:51.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:51 smithi169 ceph-mon[39949]: pgmap v125: 158 pgs: 10 unknown, 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s 2024-03-12T03:36:51.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:51 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-12T03:36:51.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:51 smithi169 ceph-mon[39949]: osdmap e121: 8 total, 8 up, 8 in 2024-03-12T03:36:51.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:51 smithi169 ceph-mon[29344]: pgmap v125: 158 pgs: 10 unknown, 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s 2024-03-12T03:36:51.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:51 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-12T03:36:51.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:51 smithi169 ceph-mon[29344]: osdmap e121: 8 total, 8 up, 8 in 2024-03-12T03:36:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:51.448 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:51.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:51.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:51.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:51.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:51.596 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:36:51.594780+0000 min lat: 0.00118901 max lat: 1.19705 avg lat: 0.0115562 2024-03-12T03:36:51.596 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:36:51.596 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 80 16 108995 108979 85.1005 0 - 0.0115562 2024-03-12T03:36:51.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:51.699 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:51.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:51 smithi173 ceph-mon[33544]: pgmap v125: 158 pgs: 10 unknown, 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s 2024-03-12T03:36:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:51 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-12T03:36:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:51 smithi173 ceph-mon[33544]: osdmap e121: 8 total, 8 up, 8 in 2024-03-12T03:36:51.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:51.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:51.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:51.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:51.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:51.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:51.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:52.052 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:52.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-12T03:36:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:52.080 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:52.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:52.105 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:36:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:36:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:36:52.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:52.206 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:52.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:52.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:52.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:52.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:52.368 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:52 smithi169 ceph-mon[29344]: osdmap e122: 8 total, 8 up, 8 in 2024-03-12T03:36:52.368 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:52 smithi169 ceph-mon[29344]: osdmap e123: 8 total, 8 up, 8 in 2024-03-12T03:36:52.369 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:52 smithi169 ceph-mon[39949]: osdmap e122: 8 total, 8 up, 8 in 2024-03-12T03:36:52.369 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:52 smithi169 ceph-mon[39949]: osdmap e123: 8 total, 8 up, 8 in 2024-03-12T03:36:52.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:52.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:52.477 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:52.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:52.578 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:52.596 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 81 16 109128 109112 84.1524 4.15625 0.0179725 0.0118697 2024-03-12T03:36:52.618 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:52.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:52.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:52.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:52.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:52 smithi173 ceph-mon[33544]: osdmap e122: 8 total, 8 up, 8 in 2024-03-12T03:36:52.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:52 smithi173 ceph-mon[33544]: osdmap e123: 8 total, 8 up, 8 in 2024-03-12T03:36:52.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:36:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:36:52.764 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:52.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:52.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:52.913 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:52.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:53.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:53.057 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:53.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:53.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:53.159 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:53.186 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:53.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:53.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:53.322 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:36:53.322 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:36:53.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:36:53.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:53.333 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:53.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:53.420 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: pgmap v128: 158 pgs: 10 unknown, 148 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-03-12T03:36:53.420 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.420 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.420 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 120'6772 2024-03-12T03:36:53.420 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 120'6821 2024-03-12T03:36:53.420 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 120'6735 2024-03-12T03:36:53.420 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 120'6745 2024-03-12T03:36:53.420 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 120'6700 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 120'6707 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 120'6824 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 120'6772 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 120'6745 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 120'6745 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 120'6824 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.17 starting backfill to osd.7 from (0'0,0'0] MAX to 120'6824 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[29344]: 7.19 starting backfill to osd.7 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: pgmap v128: 158 pgs: 10 unknown, 148 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-03-12T03:36:53.421 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 120'6772 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 120'6821 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 120'6735 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 120'6745 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 120'6700 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 120'6707 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 120'6824 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 120'6772 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 120'6745 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 120'6745 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 120'6824 2024-03-12T03:36:53.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.17 starting backfill to osd.7 from (0'0,0'0] MAX to 120'6824 2024-03-12T03:36:53.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:53.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:53 smithi169 ceph-mon[39949]: 7.19 starting backfill to osd.7 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:53.492 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:53.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:53.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:36:53.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:53.598 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 82 16 110079 110063 83.8498 59.4375 0.0276802 0.0119213 2024-03-12T03:36:53.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:53.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:53.721 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:53.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: pgmap v128: 158 pgs: 10 unknown, 148 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-03-12T03:36:53.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 120'6772 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 120'6821 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 120'6735 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 120'6745 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 120'6700 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 120'6707 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 120'6824 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 120'6772 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 120'6745 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 120'6745 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 120'6824 2024-03-12T03:36:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.17 starting backfill to osd.7 from (0'0,0'0] MAX to 120'6824 2024-03-12T03:36:53.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:53.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:53 smithi173 ceph-mon[33544]: 7.19 starting backfill to osd.7 from (0'0,0'0] MAX to 120'6795 2024-03-12T03:36:53.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:53.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:53.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:53.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:53.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:53.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:53.979 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:54.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:54.035 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:54.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:54.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:54.164 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:54.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:54.228 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:54.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:54.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:54.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:54.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:54.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:54.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:54.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:54.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:54.597 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 83 16 110847 110831 83.4179 48 0.00727754 0.0119825 2024-03-12T03:36:54.602 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:54 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:36:54.602 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:54 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:36:54.602 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:54 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:36:54.602 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:54 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:36:54.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:54.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:54.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:54.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:54 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:36:54.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:54 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:36:54.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:54.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:54.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:54.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:54.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:54.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:54.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:55.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:55.116 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:36:55.119 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:55.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:55.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:55.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:55.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:55.281 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:55.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:55.373 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:36:55.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:55.410 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:55.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:55.513 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:55.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:55.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:55.598 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 84 16 111860 111844 83.1786 63.3125 0.00465574 0.0120172 2024-03-12T03:36:55.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:55.653 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:55 smithi169 ceph-mon[29344]: pgmap v130: 158 pgs: 10 unknown, 148 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 28 KiB/s, 0 objects/s recovering 2024-03-12T03:36:55.653 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:55 smithi169 ceph-mon[29344]: osdmap e124: 8 total, 8 up, 8 in 2024-03-12T03:36:55.653 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:55 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/960988528' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:36:55.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:55 smithi169 ceph-mon[39949]: pgmap v130: 158 pgs: 10 unknown, 148 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 28 KiB/s, 0 objects/s recovering 2024-03-12T03:36:55.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:55 smithi169 ceph-mon[39949]: osdmap e124: 8 total, 8 up, 8 in 2024-03-12T03:36:55.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:55 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/960988528' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:36:55.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:55.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:55.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:55.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:55 smithi173 ceph-mon[33544]: pgmap v130: 158 pgs: 10 unknown, 148 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 28 KiB/s, 0 objects/s recovering 2024-03-12T03:36:55.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:55 smithi173 ceph-mon[33544]: osdmap e124: 8 total, 8 up, 8 in 2024-03-12T03:36:55.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:55 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/960988528' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:36:55.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:55.806 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:55.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:55.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:55.907 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:55.950 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:55.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:56.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:56.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:56.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:56.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:56.232 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:56.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:56.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:56.379 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:56.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:56.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:56.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:56.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:56.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:56.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:56.598 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 85 16 112867 112851 82.9405 62.9375 0.0108807 0.0120518 2024-03-12T03:36:56.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:56.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:56.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:56.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:56.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:56 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:36:56.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:56 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/960988528' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:36:56.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:56 smithi169 ceph-mon[29344]: osdmap e125: 8 total, 8 up, 8 in 2024-03-12T03:36:56.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:56 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:36:56.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:56 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/960988528' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:36:56.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:56 smithi169 ceph-mon[39949]: osdmap e125: 8 total, 8 up, 8 in 2024-03-12T03:36:56.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:56 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:36:56.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:56 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/960988528' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:36:56.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:56 smithi173 ceph-mon[33544]: osdmap e125: 8 total, 8 up, 8 in 2024-03-12T03:36:56.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:56.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:56.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:56.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:56.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:56.926 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:56.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:57.027 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:57.070 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:57.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:57.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:57.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:57.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:57.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:57.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:57.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:57.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:57.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:57.474 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:57.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:57.576 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:57.598 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 86 16 113805 113789 82.6578 58.625 0.00288792 0.0120888 2024-03-12T03:36:57.599 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:57.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:57.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:57.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:57 smithi169 ceph-mon[29344]: pgmap v133: 158 pgs: 148 active+clean, 1 active+remapped+backfilling, 9 unknown; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 12 MiB/s wr, 194 op/s; 2990/248625 objects misplaced (1.203%); 7.3 MiB/s, 116 objects/s recovering 2024-03-12T03:36:57.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:57 smithi169 ceph-mon[39949]: pgmap v133: 158 pgs: 148 active+clean, 1 active+remapped+backfilling, 9 unknown; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 12 MiB/s wr, 194 op/s; 2990/248625 objects misplaced (1.203%); 7.3 MiB/s, 116 objects/s recovering 2024-03-12T03:36:57.701 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:57.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:57 smithi173 ceph-mon[33544]: pgmap v133: 158 pgs: 148 active+clean, 1 active+remapped+backfilling, 9 unknown; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 12 MiB/s wr, 194 op/s; 2990/248625 objects misplaced (1.203%); 7.3 MiB/s, 116 objects/s recovering 2024-03-12T03:36:57.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:57.743 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:57.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:57.844 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:57.873 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:57.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:57.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:57.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:36:58.008 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:58.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:36:58.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:58.141 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:58.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:58.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:58.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:36:58.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:58.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:58.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:36:58.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:58.416 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:58.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:58.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:36:58.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:58 smithi169 ceph-mon[29344]: 7.1 scrub starts 2024-03-12T03:36:58.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:58 smithi169 ceph-mon[39949]: 7.1 scrub starts 2024-03-12T03:36:58.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:58.548 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:58.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:58.598 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 87 16 114611 114595 82.2868 50.375 0.00617024 0.0121439 2024-03-12T03:36:58.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:58.688 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:58.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:58.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:58.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:58 smithi173 ceph-mon[33544]: 7.1 scrub starts 2024-03-12T03:36:58.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:58.824 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:58.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:58.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:58.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:58.966 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:58.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:59.067 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:59.109 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:59.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:59.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:36:59.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:59.243 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:59.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:36:59.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:36:59.387 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:59.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:36:59.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:36:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:59.533 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:59.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:36:59.598 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 88 16 115555 115539 82.0223 59 0.00948469 0.0121871 2024-03-12T03:36:59.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:36:59.637 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:59 smithi173 ceph-mon[33544]: pgmap v134: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 316 MiB/s wr, 5.06k op/s; 63965/337092 objects misplaced (18.976%); 14 MiB/s, 216 objects/s recovering 2024-03-12T03:36:59.637 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:59 smithi173 ceph-mon[33544]: 7.1 scrub ok 2024-03-12T03:36:59.637 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:36:59 smithi173 ceph-mon[33544]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:36:59.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:59 smithi169 ceph-mon[29344]: pgmap v134: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 316 MiB/s wr, 5.06k op/s; 63965/337092 objects misplaced (18.976%); 14 MiB/s, 216 objects/s recovering 2024-03-12T03:36:59.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:59 smithi169 ceph-mon[29344]: 7.1 scrub ok 2024-03-12T03:36:59.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:36:59 smithi169 ceph-mon[29344]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:36:59.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:59 smithi169 ceph-mon[39949]: pgmap v134: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 316 MiB/s wr, 5.06k op/s; 63965/337092 objects misplaced (18.976%); 14 MiB/s, 216 objects/s recovering 2024-03-12T03:36:59.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:59 smithi169 ceph-mon[39949]: 7.1 scrub ok 2024-03-12T03:36:59.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:36:59 smithi169 ceph-mon[39949]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:36:59.677 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:59.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:59.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:36:59.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:36:59.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:36:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:59.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:36:59.911 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:36:59.952 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:36:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:36:59.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:00.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:00.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:00.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:00.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:00.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:00.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:00.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:00.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:00.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:00.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:00.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:00.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:00.501 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:00.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:00.598 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 89 16 116557 116541 81.8043 62.625 0.00821856 0.0122195 2024-03-12T03:37:00.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:00.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:00.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:00.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:00.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:00.780 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:00.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:00.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:00.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:00.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:00.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:01.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:01.050 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:01.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:01.152 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:01.176 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:01.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:01.277 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:01.326 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:01.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:01.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:01 smithi169 ceph-mon[29344]: pgmap v135: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 679 B/s rd, 252 MiB/s wr, 4.03k op/s; 63965/337092 objects misplaced (18.976%); 11 MiB/s, 172 objects/s recovering 2024-03-12T03:37:01.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:01 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:01.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:01 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:01.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:01 smithi169 ceph-mon[29344]: 3.3 scrub starts 2024-03-12T03:37:01.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:01 smithi169 ceph-mon[29344]: 3.3 scrub ok 2024-03-12T03:37:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:01 smithi169 ceph-mon[39949]: pgmap v135: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 679 B/s rd, 252 MiB/s wr, 4.03k op/s; 63965/337092 objects misplaced (18.976%); 11 MiB/s, 172 objects/s recovering 2024-03-12T03:37:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:01 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:01 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:01 smithi169 ceph-mon[39949]: 3.3 scrub starts 2024-03-12T03:37:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:01 smithi169 ceph-mon[39949]: 3.3 scrub ok 2024-03-12T03:37:01.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:01.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:01.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:01.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:01 smithi173 ceph-mon[33544]: pgmap v135: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 679 B/s rd, 252 MiB/s wr, 4.03k op/s; 63965/337092 objects misplaced (18.976%); 11 MiB/s, 172 objects/s recovering 2024-03-12T03:37:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:01 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:01 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:01 smithi173 ceph-mon[33544]: 3.3 scrub starts 2024-03-12T03:37:01.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:01 smithi173 ceph-mon[33544]: 3.3 scrub ok 2024-03-12T03:37:01.562 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:01.595 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:01.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:01.598 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Total time run: 90.0245 2024-03-12T03:37:01.598 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Total writes made: 117414 2024-03-12T03:37:01.598 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Write size: 65536 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Object size: 65536 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Bandwidth (MB/sec): 81.5153 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev Bandwidth: 35.5793 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max bandwidth (MB/sec): 202.625 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min bandwidth (MB/sec): 0 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Average IOPS: 1304 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev IOPS: 569.281 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max IOPS: 3242 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min IOPS: 0 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Average Latency(s): 0.0122619 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev Latency(s): 0.0290292 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max latency(s): 2.2788 2024-03-12T03:37:01.599 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min latency(s): 0.00118901 2024-03-12T03:37:01.603 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-03-12T03:37:01.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:01.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:01.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:01.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:01.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:01.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:01.885 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:01.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:01.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:02.028 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:37:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:37:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:37:02.037 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:02.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:02.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:02.139 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:37:02.139 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:37:02.188 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:02.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:02.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:02.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:02.291 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:02 smithi169 ceph-mon[29344]: 6.2 scrub starts 2024-03-12T03:37:02.291 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:02 smithi169 ceph-mon[29344]: 6.2 scrub ok 2024-03-12T03:37:02.291 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:02 smithi169 ceph-mon[39949]: 6.2 scrub starts 2024-03-12T03:37:02.291 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:02 smithi169 ceph-mon[39949]: 6.2 scrub ok 2024-03-12T03:37:02.320 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:02.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:02.365 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:02 smithi173 ceph-mon[33544]: 6.2 scrub starts 2024-03-12T03:37:02.365 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:02 smithi173 ceph-mon[33544]: 6.2 scrub ok 2024-03-12T03:37:02.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:02.460 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:02.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:02.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:02.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:37:02.562 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:02.616 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:02.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:02.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:02.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:37:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:37:02.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:02.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:02.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:02.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:02.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:02.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:02.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-12T03:37:02.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:03.028 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:03.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:03.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:03.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:03.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:03.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:03 smithi169 ceph-mon[29344]: pgmap v136: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 246 MiB/s wr, 3.94k op/s; 62938/351255 objects misplaced (17.918%); 25 MiB/s, 393 objects/s recovering 2024-03-12T03:37:03.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:03 smithi169 ceph-mon[29344]: 4.13 scrub starts 2024-03-12T03:37:03.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:03 smithi169 ceph-mon[29344]: 4.13 scrub ok 2024-03-12T03:37:03.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:03 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:03.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:03 smithi169 ceph-mon[39949]: pgmap v136: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 246 MiB/s wr, 3.94k op/s; 62938/351255 objects misplaced (17.918%); 25 MiB/s, 393 objects/s recovering 2024-03-12T03:37:03.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:03 smithi169 ceph-mon[39949]: 4.13 scrub starts 2024-03-12T03:37:03.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:03 smithi169 ceph-mon[39949]: 4.13 scrub ok 2024-03-12T03:37:03.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:03 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:03.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:03.267 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:37:03.313 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:03.313 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:37:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:03.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:03.353 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:37:03.395 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:37:03.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:03.438 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:37:03.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:03.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:03.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:03.482 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:37:03.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:03 smithi173 ceph-mon[33544]: pgmap v136: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 246 MiB/s wr, 3.94k op/s; 62938/351255 objects misplaced (17.918%); 25 MiB/s, 393 objects/s recovering 2024-03-12T03:37:03.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:03 smithi173 ceph-mon[33544]: 4.13 scrub starts 2024-03-12T03:37:03.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:03 smithi173 ceph-mon[33544]: 4.13 scrub ok 2024-03-12T03:37:03.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:03 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:03.519 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:37:03.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:03.567 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:37:03.576 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:37:03.576 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:37:03.576 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:37:03.576 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:37:03.576 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:37:03.576 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:37:03.576 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:37:03.576 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:37:03.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:03.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:03.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:03.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:03.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:03.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:03.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:03.858 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:03.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:03.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:03.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:03.988 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:03.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:04.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:04.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:04.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:04.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:04.209 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:04 smithi169 ceph-mon[29344]: 4.1 scrub starts 2024-03-12T03:37:04.210 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:04 smithi169 ceph-mon[29344]: 4.1 scrub ok 2024-03-12T03:37:04.210 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:04 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2252770350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:37:04.210 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:04 smithi169 ceph-mon[39949]: 4.1 scrub starts 2024-03-12T03:37:04.210 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:04 smithi169 ceph-mon[39949]: 4.1 scrub ok 2024-03-12T03:37:04.211 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:04 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2252770350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:37:04.216 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:37:04.219 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:37:04.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:04.267 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:04.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:04.368 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:04.405 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:04.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:04.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:04 smithi173 ceph-mon[33544]: 4.1 scrub starts 2024-03-12T03:37:04.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:04 smithi173 ceph-mon[33544]: 4.1 scrub ok 2024-03-12T03:37:04.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:04 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2252770350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:37:04.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:04.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:04.553 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:04.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:04.639 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:37:04.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:04.658 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:37:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:04.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:04.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:04.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:04.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:04.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:04.917 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:04.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:04.975 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:04.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:05.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:05.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:05.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:05.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:05.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:05.335 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:05.371 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:05 smithi169 ceph-mon[29344]: pgmap v137: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 208 MiB/s wr, 3.33k op/s; 62938/351255 objects misplaced (17.918%); 21 MiB/s, 332 objects/s recovering 2024-03-12T03:37:05.371 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:05 smithi169 ceph-mon[29344]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:37:05.372 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:05 smithi169 ceph-mon[39949]: pgmap v137: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 208 MiB/s wr, 3.33k op/s; 62938/351255 objects misplaced (17.918%); 21 MiB/s, 332 objects/s recovering 2024-03-12T03:37:05.372 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:05 smithi169 ceph-mon[39949]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:37:05.378 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:05.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:05.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:05.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:05 smithi173 ceph-mon[33544]: pgmap v137: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 208 MiB/s wr, 3.33k op/s; 62938/351255 objects misplaced (17.918%); 21 MiB/s, 332 objects/s recovering 2024-03-12T03:37:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:05 smithi173 ceph-mon[33544]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:37:05.508 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:05.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:05.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:05.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:05.660 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:05.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:05.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:05.810 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:05.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:05.911 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:05.956 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:05.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:06.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:06.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:06.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:06.120 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-03-12T03:37:03.817155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7658805672,"num_objects":117085,"num_object_clones":0,"num_object_copies":351255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":62938,"num_objects_unfound":0,"num_objects_dirty":117085,"num_whiteouts":0,"num_read":1436,"num_read_kb":1250,"num_write":114100,"num_write_kb":7253213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3177,"num_bytes_recovered":208210082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147444,"ondisk_log_size":147444,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":493,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":23463036,"kb_used_data":22680572,"kb_used_omap":12,"kb_used_meta":782387,"kb_avail":726498180,"statfs":{"total":767960285184,"available":743934136320,"internally_reserved":0,"allocated":23224905728,"data_stored":23220894992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":801164848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2299265024,"num_objects":35085,"num_object_clones":0,"num_object_copies":105255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":62938,"num_objects_unfound":0,"num_objects_dirty":35085,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":31523,"num_write_kb":2017408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3148,"num_bytes_recovered":206307328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.472434"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.359318+0000","last_change":"2024-03-12T03:35:21.106867+0000","last_active":"2024-03-12T03:36:55.359318+0000","last_peered":"2024-03-12T03:36:55.359318+0000","last_clean":"2024-03-12T03:36:55.359318+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:36:55.359318+0000","last_undegraded":"2024-03-12T03:36:55.359318+0000","last_fullsized":"2024-03-12T03:36:55.359318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:21.106755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:21.106755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.380115+0000","last_change":"2024-03-12T03:36:02.862199+0000","last_active":"2024-03-12T03:36:55.380115+0000","last_peered":"2024-03-12T03:36:55.380115+0000","last_clean":"2024-03-12T03:36:55.380115+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:36:55.380115+0000","last_undegraded":"2024-03-12T03:36:55.380115+0000","last_fullsized":"2024-03-12T03:36:55.380115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:02.861854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:02.861854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:07.436631+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.374658+0000","last_change":"2024-03-12T03:36:28.208156+0000","last_active":"2024-03-12T03:36:55.374658+0000","last_peered":"2024-03-12T03:36:55.374658+0000","last_clean":"2024-03-12T03:36:55.374658+0000","last_became_active":"2024-03-12T03:33:21.573080+0000","last_became_peered":"2024-03-12T03:33:21.573080+0000","last_unstale":"2024-03-12T03:36:55.374658+0000","last_undegraded":"2024-03-12T03:36:55.374658+0000","last_fullsized":"2024-03-12T03:36:55.374658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:28.208044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:28.208044+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.208044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:40.375427+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":160,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.359301+0000","last_change":"2024-03-12T03:36:02.990432+0000","last_active":"2024-03-12T03:36:55.359301+0000","last_peered":"2024-03-12T03:36:55.359301+0000","last_clean":"2024-03-12T03:36:55.359301+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:36:55.359301+0000","last_undegraded":"2024-03-12T03:36:55.359301+0000","last_fullsized":"2024-03-12T03:36:55.359301+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:36:02.990324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:02.990324+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:22.888530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.366436+0000","last_change":"2024-03-12T03:36:19.237362+0000","last_active":"2024-03-12T03:36:55.366436+0000","last_peered":"2024-03-12T03:36:55.366436+0000","last_clean":"2024-03-12T03:36:55.366436+0000","last_became_active":"2024-03-12T03:33:27.382050+0000","last_became_peered":"2024-03-12T03:33:27.382050+0000","last_unstale":"2024-03-12T03:36:55.366436+0000","last_undegraded":"2024-03-12T03:36:55.366436+0000","last_fullsized":"2024-03-12T03:36:55.366436+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.236997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.236997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:39.350923+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.359484+0000","last_change":"2024-03-12T03:35:17.100474+0000","last_active":"2024-03-12T03:36:55.359484+0000","last_peered":"2024-03-12T03:36:55.359484+0000","last_clean":"2024-03-12T03:36:55.359484+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:36:55.359484+0000","last_undegraded":"2024-03-12T03:36:55.359484+0000","last_fullsized":"2024-03-12T03:36:55.359484+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:17.100375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:17.100375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363733+0000","last_change":"2024-03-12T03:34:41.143075+0000","last_active":"2024-03-12T03:36:55.363733+0000","last_peered":"2024-03-12T03:36:55.363733+0000","last_clean":"2024-03-12T03:36:55.363733+0000","last_became_active":"2024-03-12T03:33:21.572876+0000","last_became_peered":"2024-03-12T03:33:21.572876+0000","last_unstale":"2024-03-12T03:36:55.363733+0000","last_undegraded":"2024-03-12T03:36:55.363733+0000","last_fullsized":"2024-03-12T03:36:55.363733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.142994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"94'50","reported_seq":216,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.365608+0000","last_change":"2024-03-12T03:35:05.042142+0000","last_active":"2024-03-12T03:36:55.365608+0000","last_peered":"2024-03-12T03:36:55.365608+0000","last_clean":"2024-03-12T03:36:55.365608+0000","last_became_active":"2024-03-12T03:33:23.161549+0000","last_became_peered":"2024-03-12T03:33:23.161549+0000","last_unstale":"2024-03-12T03:36:55.365608+0000","last_undegraded":"2024-03-12T03:36:55.365608+0000","last_fullsized":"2024-03-12T03:36:55.365608+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.042030+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"125'7041","reported_seq":9797,"reported_epoch":125,"state":"active+remapped+backfilling","last_fresh":"2024-03-12T03:37:00.924826+0000","last_change":"2024-03-12T03:36:52.437816+0000","last_active":"2024-03-12T03:37:00.924826+0000","last_peered":"2024-03-12T03:37:00.924826+0000","last_clean":"2024-03-12T03:36:50.220133+0000","last_became_active":"2024-03-12T03:36:52.433270+0000","last_became_peered":"2024-03-12T03:36:52.433270+0000","last_unstale":"2024-03-12T03:37:00.924826+0000","last_undegraded":"2024-03-12T03:37:00.924826+0000","last_fullsized":"2024-03-12T03:37:00.924826+0000","mapping_epoch":122,"log_start":"101'1100","ondisk_log_start":"101'1100","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240156672,"num_objects":3665,"num_object_clones":0,"num_object_copies":10995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8931,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":234528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":668,"num_bytes_recovered":43778048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[0,1,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":3664},{"shards":"7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371772+0000","last_change":"2024-03-12T03:36:26.107553+0000","last_active":"2024-03-12T03:36:55.371772+0000","last_peered":"2024-03-12T03:36:55.371772+0000","last_clean":"2024-03-12T03:36:55.371772+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:36:55.371772+0000","last_undegraded":"2024-03-12T03:36:55.371772+0000","last_fullsized":"2024-03-12T03:36:55.371772+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.107439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.107439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:55.895456+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.362084+0000","last_change":"2024-03-12T03:36:16.187277+0000","last_active":"2024-03-12T03:36:55.362084+0000","last_peered":"2024-03-12T03:36:55.362084+0000","last_clean":"2024-03-12T03:36:55.362084+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:36:55.362084+0000","last_undegraded":"2024-03-12T03:36:55.362084+0000","last_fullsized":"2024-03-12T03:36:55.362084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:16.187145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:16.187145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:45.580351+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.362271+0000","last_change":"2024-03-12T03:36:21.101794+0000","last_active":"2024-03-12T03:36:55.362271+0000","last_peered":"2024-03-12T03:36:55.362271+0000","last_clean":"2024-03-12T03:36:55.362271+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:36:55.362271+0000","last_undegraded":"2024-03-12T03:36:55.362271+0000","last_fullsized":"2024-03-12T03:36:55.362271+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:21.101693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:36:21.101693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:43.795511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'21","reported_seq":172,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.359729+0000","last_change":"2024-03-12T03:35:19.084309+0000","last_active":"2024-03-12T03:36:55.359729+0000","last_peered":"2024-03-12T03:36:55.359729+0000","last_clean":"2024-03-12T03:36:55.359729+0000","last_became_active":"2024-03-12T03:33:23.162749+0000","last_became_peered":"2024-03-12T03:33:23.162749+0000","last_unstale":"2024-03-12T03:36:55.359729+0000","last_undegraded":"2024-03-12T03:36:55.359729+0000","last_fullsized":"2024-03-12T03:36:55.359729+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:35:19.084191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:19.084191+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"125'6969","reported_seq":7549,"reported_epoch":125,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-12T03:37:00.971493+0000","last_change":"2024-03-12T03:36:52.607201+0000","last_active":"2024-03-12T03:37:00.971493+0000","last_peered":"2024-03-12T03:37:00.971493+0000","last_clean":"2024-03-12T03:36:50.232406+0000","last_became_active":"2024-03-12T03:36:52.548937+0000","last_became_peered":"2024-03-12T03:36:52.548937+0000","last_unstale":"2024-03-12T03:37:00.971493+0000","last_undegraded":"2024-03-12T03:37:00.971493+0000","last_fullsized":"2024-03-12T03:36:52.286167+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6969,"log_dups_size":0,"ondisk_log_size":6969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236584960,"num_objects":3610,"num_object_clones":0,"num_object_copies":10830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6534,"num_objects_unfound":0,"num_objects_dirty":3610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":231040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":10944512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[2,7],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":3267},{"shards":"2,7","objects":343}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.361702+0000","last_change":"2024-03-12T03:35:05.181569+0000","last_active":"2024-03-12T03:36:55.361702+0000","last_peered":"2024-03-12T03:36:55.361702+0000","last_clean":"2024-03-12T03:36:55.361702+0000","last_became_active":"2024-03-12T03:33:27.187943+0000","last_became_peered":"2024-03-12T03:33:27.187943+0000","last_unstale":"2024-03-12T03:36:55.361702+0000","last_undegraded":"2024-03-12T03:36:55.361702+0000","last_fullsized":"2024-03-12T03:36:55.361702+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:05.181454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.181454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371028+0000","last_change":"2024-03-12T03:35:11.909632+0000","last_active":"2024-03-12T03:36:55.371028+0000","last_peered":"2024-03-12T03:36:55.371028+0000","last_clean":"2024-03-12T03:36:55.371028+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:36:55.371028+0000","last_undegraded":"2024-03-12T03:36:55.371028+0000","last_fullsized":"2024-03-12T03:36:55.371028+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:11.909537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.909537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363064+0000","last_change":"2024-03-12T03:36:18.202204+0000","last_active":"2024-03-12T03:36:55.363064+0000","last_peered":"2024-03-12T03:36:55.363064+0000","last_clean":"2024-03-12T03:36:55.363064+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:36:55.363064+0000","last_undegraded":"2024-03-12T03:36:55.363064+0000","last_fullsized":"2024-03-12T03:36:55.363064+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.202082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.202082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:32.588876+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"86'14","reported_seq":179,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.360449+0000","last_change":"2024-03-12T03:36:15.113589+0000","last_active":"2024-03-12T03:36:55.360449+0000","last_peered":"2024-03-12T03:36:55.360449+0000","last_clean":"2024-03-12T03:36:55.360449+0000","last_became_active":"2024-03-12T03:33:50.155908+0000","last_became_peered":"2024-03-12T03:33:50.155908+0000","last_unstale":"2024-03-12T03:36:55.360449+0000","last_undegraded":"2024-03-12T03:36:55.360449+0000","last_fullsized":"2024-03-12T03:36:55.360449+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:36:15.113450+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.113450+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:18.278802+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.360822+0000","last_change":"2024-03-12T03:36:01.082667+0000","last_active":"2024-03-12T03:36:55.360822+0000","last_peered":"2024-03-12T03:36:55.360822+0000","last_clean":"2024-03-12T03:36:55.360822+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:36:55.360822+0000","last_undegraded":"2024-03-12T03:36:55.360822+0000","last_fullsized":"2024-03-12T03:36:55.360822+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:01.082531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:01.082531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:14.778425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.380815+0000","last_change":"2024-03-12T03:36:09.003751+0000","last_active":"2024-03-12T03:36:55.380815+0000","last_peered":"2024-03-12T03:36:55.380815+0000","last_clean":"2024-03-12T03:36:55.380815+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:36:55.380815+0000","last_undegraded":"2024-03-12T03:36:55.380815+0000","last_fullsized":"2024-03-12T03:36:55.380815+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:09.003476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:09.003476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:15.330848+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.370976+0000","last_change":"2024-03-12T03:36:08.094593+0000","last_active":"2024-03-12T03:36:55.370976+0000","last_peered":"2024-03-12T03:36:55.370976+0000","last_clean":"2024-03-12T03:36:55.370976+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:36:55.370976+0000","last_undegraded":"2024-03-12T03:36:55.370976+0000","last_fullsized":"2024-03-12T03:36:55.370976+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:08.094475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:08.094475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:22.969182+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":157,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.373511+0000","last_change":"2024-03-12T03:34:58.889062+0000","last_active":"2024-03-12T03:36:55.373511+0000","last_peered":"2024-03-12T03:36:55.373511+0000","last_clean":"2024-03-12T03:36:55.373511+0000","last_became_active":"2024-03-12T03:33:23.160702+0000","last_became_peered":"2024-03-12T03:33:23.160702+0000","last_unstale":"2024-03-12T03:36:55.373511+0000","last_undegraded":"2024-03-12T03:36:55.373511+0000","last_fullsized":"2024-03-12T03:36:55.373511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:34:58.888971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.888971+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.367828+0000","last_change":"2024-03-12T03:33:27.381440+0000","last_active":"2024-03-12T03:36:55.367828+0000","last_peered":"2024-03-12T03:36:55.367828+0000","last_clean":"2024-03-12T03:36:55.367828+0000","last_became_active":"2024-03-12T03:33:27.380880+0000","last_became_peered":"2024-03-12T03:33:27.380880+0000","last_unstale":"2024-03-12T03:36:55.367828+0000","last_undegraded":"2024-03-12T03:36:55.367828+0000","last_fullsized":"2024-03-12T03:36:55.367828+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.360093+0000","last_change":"2024-03-12T03:36:20.142515+0000","last_active":"2024-03-12T03:36:55.360093+0000","last_peered":"2024-03-12T03:36:55.360093+0000","last_clean":"2024-03-12T03:36:55.360093+0000","last_became_active":"2024-03-12T03:33:25.181087+0000","last_became_peered":"2024-03-12T03:33:25.181087+0000","last_unstale":"2024-03-12T03:36:55.360093+0000","last_undegraded":"2024-03-12T03:36:55.360093+0000","last_fullsized":"2024-03-12T03:36:55.360093+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:20.142437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.142437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:49.273040+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.359837+0000","last_change":"2024-03-12T03:36:18.123144+0000","last_active":"2024-03-12T03:36:55.359837+0000","last_peered":"2024-03-12T03:36:55.359837+0000","last_clean":"2024-03-12T03:36:55.359837+0000","last_became_active":"2024-03-12T03:33:21.573218+0000","last_became_peered":"2024-03-12T03:33:21.573218+0000","last_unstale":"2024-03-12T03:36:55.359837+0000","last_undegraded":"2024-03-12T03:36:55.359837+0000","last_fullsized":"2024-03-12T03:36:55.359837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.123035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.123035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.552439+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":167,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363755+0000","last_change":"2024-03-12T03:36:28.133412+0000","last_active":"2024-03-12T03:36:55.363755+0000","last_peered":"2024-03-12T03:36:55.363755+0000","last_clean":"2024-03-12T03:36:55.363755+0000","last_became_active":"2024-03-12T03:33:23.162427+0000","last_became_peered":"2024-03-12T03:33:23.162427+0000","last_unstale":"2024-03-12T03:36:55.363755+0000","last_undegraded":"2024-03-12T03:36:55.363755+0000","last_fullsized":"2024-03-12T03:36:55.363755+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:36:28.133261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.133261+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:51.305356+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371538+0000","last_change":"2024-03-12T03:35:57.051016+0000","last_active":"2024-03-12T03:36:55.371538+0000","last_peered":"2024-03-12T03:36:55.371538+0000","last_clean":"2024-03-12T03:36:55.371538+0000","last_became_active":"2024-03-12T03:33:27.189152+0000","last_became_peered":"2024-03-12T03:33:27.189152+0000","last_unstale":"2024-03-12T03:36:55.371538+0000","last_undegraded":"2024-03-12T03:36:55.371538+0000","last_fullsized":"2024-03-12T03:36:55.371538+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:35:57.050605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:57.050605+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:06.936509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":191,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.369291+0000","last_change":"2024-03-12T03:36:00.063359+0000","last_active":"2024-03-12T03:36:55.369291+0000","last_peered":"2024-03-12T03:36:55.369291+0000","last_clean":"2024-03-12T03:36:55.369291+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:36:55.369291+0000","last_undegraded":"2024-03-12T03:36:55.369291+0000","last_fullsized":"2024-03-12T03:36:55.369291+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:00.063228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:00.063228+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:17.858770+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":145,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.379986+0000","last_change":"2024-03-12T03:36:16.066119+0000","last_active":"2024-03-12T03:36:55.379986+0000","last_peered":"2024-03-12T03:36:55.379986+0000","last_clean":"2024-03-12T03:36:55.379986+0000","last_became_active":"2024-03-12T03:33:21.097218+0000","last_became_peered":"2024-03-12T03:33:21.097218+0000","last_unstale":"2024-03-12T03:36:55.379986+0000","last_undegraded":"2024-03-12T03:36:55.379986+0000","last_fullsized":"2024-03-12T03:36:55.379986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:16.066002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:16.066002+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:39.793679+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'17","reported_seq":168,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.379066+0000","last_change":"2024-03-12T03:36:12.216521+0000","last_active":"2024-03-12T03:36:55.379066+0000","last_peered":"2024-03-12T03:36:55.379066+0000","last_clean":"2024-03-12T03:36:55.379066+0000","last_became_active":"2024-03-12T03:33:23.552812+0000","last_became_peered":"2024-03-12T03:33:23.552812+0000","last_unstale":"2024-03-12T03:36:55.379066+0000","last_undegraded":"2024-03-12T03:36:55.379066+0000","last_fullsized":"2024-03-12T03:36:55.379066+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:36:12.216269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:12.216269+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:27.130146+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.381025+0000","last_change":"2024-03-12T03:36:22.163176+0000","last_active":"2024-03-12T03:36:55.381025+0000","last_peered":"2024-03-12T03:36:55.381025+0000","last_clean":"2024-03-12T03:36:55.381025+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:36:55.381025+0000","last_undegraded":"2024-03-12T03:36:55.381025+0000","last_fullsized":"2024-03-12T03:36:55.381025+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:22.163050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:22.163050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.403724+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.379517+0000","last_change":"2024-03-12T03:36:24.163956+0000","last_active":"2024-03-12T03:36:55.379517+0000","last_peered":"2024-03-12T03:36:55.379517+0000","last_clean":"2024-03-12T03:36:55.379517+0000","last_became_active":"2024-03-12T03:33:25.178346+0000","last_became_peered":"2024-03-12T03:33:25.178346+0000","last_unstale":"2024-03-12T03:36:55.379517+0000","last_undegraded":"2024-03-12T03:36:55.379517+0000","last_fullsized":"2024-03-12T03:36:55.379517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:24.162965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.162965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:35.015647+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":142,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363334+0000","last_change":"2024-03-12T03:36:25.092455+0000","last_active":"2024-03-12T03:36:55.363334+0000","last_peered":"2024-03-12T03:36:55.363334+0000","last_clean":"2024-03-12T03:36:55.363334+0000","last_became_active":"2024-03-12T03:33:21.095556+0000","last_became_peered":"2024-03-12T03:33:21.095556+0000","last_unstale":"2024-03-12T03:36:55.363334+0000","last_undegraded":"2024-03-12T03:36:55.363334+0000","last_fullsized":"2024-03-12T03:36:55.363334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:25.092313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.092313+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.705657+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"94'23","reported_seq":177,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.369141+0000","last_change":"2024-03-12T03:36:20.132927+0000","last_active":"2024-03-12T03:36:55.369141+0000","last_peered":"2024-03-12T03:36:55.369141+0000","last_clean":"2024-03-12T03:36:55.369141+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:36:55.369141+0000","last_undegraded":"2024-03-12T03:36:55.369141+0000","last_fullsized":"2024-03-12T03:36:55.369141+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'23","last_scrub_stamp":"2024-03-12T03:36:20.132683+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:34:52.729983+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.132683+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:42.640904+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.13","version":"125'7066","reported_seq":7148,"reported_epoch":125,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-12T03:37:00.944806+0000","last_change":"2024-03-12T03:36:52.373199+0000","last_active":"2024-03-12T03:37:00.944806+0000","last_peered":"2024-03-12T03:37:00.944806+0000","last_clean":"2024-03-12T03:36:50.229510+0000","last_became_active":"2024-03-12T03:36:52.356636+0000","last_became_peered":"2024-03-12T03:36:52.356636+0000","last_unstale":"2024-03-12T03:37:00.944806+0000","last_undegraded":"2024-03-12T03:37:00.944806+0000","last_fullsized":"2024-03-12T03:37:00.944806+0000","mapping_epoch":122,"log_start":"101'2100","ondisk_log_start":"101'2100","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4966,"log_dups_size":0,"ondisk_log_size":4966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241893376,"num_objects":3691,"num_object_clones":0,"num_object_copies":11073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7382,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3691,"num_write_kb":236224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[2,3,4],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"2,3,4","objects":3691}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.370988+0000","last_change":"2024-03-12T03:35:15.855752+0000","last_active":"2024-03-12T03:36:55.370988+0000","last_peered":"2024-03-12T03:36:55.370988+0000","last_clean":"2024-03-12T03:36:55.370988+0000","last_became_active":"2024-03-12T03:33:27.189696+0000","last_became_peered":"2024-03-12T03:33:27.189696+0000","last_unstale":"2024-03-12T03:36:55.370988+0000","last_undegraded":"2024-03-12T03:36:55.370988+0000","last_fullsized":"2024-03-12T03:36:55.370988+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.855641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.855641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.368573+0000","last_change":"2024-03-12T03:36:01.098836+0000","last_active":"2024-03-12T03:36:55.368573+0000","last_peered":"2024-03-12T03:36:55.368573+0000","last_clean":"2024-03-12T03:36:55.368573+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:36:55.368573+0000","last_undegraded":"2024-03-12T03:36:55.368573+0000","last_fullsized":"2024-03-12T03:36:55.368573+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:01.098726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:01.098726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:05.633644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.375039+0000","last_change":"2024-03-12T03:35:59.076304+0000","last_active":"2024-03-12T03:36:55.375039+0000","last_peered":"2024-03-12T03:36:55.375039+0000","last_clean":"2024-03-12T03:36:55.375039+0000","last_became_active":"2024-03-12T03:33:21.096560+0000","last_became_peered":"2024-03-12T03:33:21.096560+0000","last_unstale":"2024-03-12T03:36:55.375039+0000","last_undegraded":"2024-03-12T03:36:55.375039+0000","last_fullsized":"2024-03-12T03:36:55.375039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.076203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.076203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:18.990073+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"90'6","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363428+0000","last_change":"2024-03-12T03:34:34.274478+0000","last_active":"2024-03-12T03:36:55.363428+0000","last_peered":"2024-03-12T03:36:55.363428+0000","last_clean":"2024-03-12T03:36:55.363428+0000","last_became_active":"2024-03-12T03:33:23.161801+0000","last_became_peered":"2024-03-12T03:33:23.161801+0000","last_unstale":"2024-03-12T03:36:55.363428+0000","last_undegraded":"2024-03-12T03:36:55.363428+0000","last_fullsized":"2024-03-12T03:36:55.363428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.274353+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"125'7013","reported_seq":7093,"reported_epoch":125,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-12T03:37:00.858338+0000","last_change":"2024-03-12T03:36:52.397358+0000","last_active":"2024-03-12T03:37:00.858338+0000","last_peered":"2024-03-12T03:37:00.858338+0000","last_clean":"2024-03-12T03:36:50.159748+0000","last_became_active":"2024-03-12T03:36:52.374031+0000","last_became_peered":"2024-03-12T03:36:52.374031+0000","last_unstale":"2024-03-12T03:37:00.858338+0000","last_undegraded":"2024-03-12T03:37:00.858338+0000","last_fullsized":"2024-03-12T03:37:00.858338+0000","mapping_epoch":122,"log_start":"101'2100","ondisk_log_start":"101'2100","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239271936,"num_objects":3651,"num_object_clones":0,"num_object_copies":10953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7302,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3651,"num_write_kb":233664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[1,2,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":3651}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.366736+0000","last_change":"2024-03-12T03:34:40.191618+0000","last_active":"2024-03-12T03:36:55.366736+0000","last_peered":"2024-03-12T03:36:55.366736+0000","last_clean":"2024-03-12T03:36:55.366736+0000","last_became_active":"2024-03-12T03:33:27.381110+0000","last_became_peered":"2024-03-12T03:33:27.381110+0000","last_unstale":"2024-03-12T03:36:55.366736+0000","last_undegraded":"2024-03-12T03:36:55.366736+0000","last_fullsized":"2024-03-12T03:36:55.366736+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.191538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.370794+0000","last_change":"2024-03-12T03:34:53.878589+0000","last_active":"2024-03-12T03:36:55.370794+0000","last_peered":"2024-03-12T03:36:55.370794+0000","last_clean":"2024-03-12T03:36:55.370794+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:36:55.370794+0000","last_undegraded":"2024-03-12T03:36:55.370794+0000","last_fullsized":"2024-03-12T03:36:55.370794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:53.878484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:53.878484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.362800+0000","last_change":"2024-03-12T03:35:59.352393+0000","last_active":"2024-03-12T03:36:55.362800+0000","last_peered":"2024-03-12T03:36:55.362800+0000","last_clean":"2024-03-12T03:36:55.362800+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:36:55.362800+0000","last_undegraded":"2024-03-12T03:36:55.362800+0000","last_fullsized":"2024-03-12T03:36:55.362800+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.352272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.352272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:00.463157+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'18","reported_seq":164,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.365812+0000","last_change":"2024-03-12T03:35:02.079317+0000","last_active":"2024-03-12T03:36:55.365812+0000","last_peered":"2024-03-12T03:36:55.365812+0000","last_clean":"2024-03-12T03:36:55.365812+0000","last_became_active":"2024-03-12T03:33:23.552210+0000","last_became_peered":"2024-03-12T03:33:23.552210+0000","last_unstale":"2024-03-12T03:36:55.365812+0000","last_undegraded":"2024-03-12T03:36:55.365812+0000","last_fullsized":"2024-03-12T03:36:55.365812+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:35:02.079224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:02.079224+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"125'6935","reported_seq":7089,"reported_epoch":125,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-12T03:37:00.943370+0000","last_change":"2024-03-12T03:36:52.576477+0000","last_active":"2024-03-12T03:37:00.943370+0000","last_peered":"2024-03-12T03:37:00.943370+0000","last_clean":"2024-03-12T03:36:50.164308+0000","last_became_active":"2024-03-12T03:36:52.342472+0000","last_became_peered":"2024-03-12T03:36:52.342472+0000","last_unstale":"2024-03-12T03:37:00.943370+0000","last_undegraded":"2024-03-12T03:37:00.943370+0000","last_fullsized":"2024-03-12T03:37:00.943370+0000","mapping_epoch":122,"log_start":"101'1800","ondisk_log_start":"101'1800","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5135,"log_dups_size":0,"ondisk_log_size":5135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236453888,"num_objects":3608,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3608,"num_write_kb":230912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":2424832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":3608}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371550+0000","last_change":"2024-03-12T03:36:18.113521+0000","last_active":"2024-03-12T03:36:55.371550+0000","last_peered":"2024-03-12T03:36:55.371550+0000","last_clean":"2024-03-12T03:36:55.371550+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:36:55.371550+0000","last_undegraded":"2024-03-12T03:36:55.371550+0000","last_fullsized":"2024-03-12T03:36:55.371550+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.113393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:18.113393+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.113393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:28.877769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.366349+0000","last_change":"2024-03-12T03:34:45.089928+0000","last_active":"2024-03-12T03:36:55.366349+0000","last_peered":"2024-03-12T03:36:55.366349+0000","last_clean":"2024-03-12T03:36:55.366349+0000","last_became_active":"2024-03-12T03:33:25.183725+0000","last_became_peered":"2024-03-12T03:33:25.183725+0000","last_unstale":"2024-03-12T03:36:55.366349+0000","last_undegraded":"2024-03-12T03:36:55.366349+0000","last_fullsized":"2024-03-12T03:36:55.366349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:45.089762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:45.089762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.372009+0000","last_change":"2024-03-12T03:34:41.698397+0000","last_active":"2024-03-12T03:36:55.372009+0000","last_peered":"2024-03-12T03:36:55.372009+0000","last_clean":"2024-03-12T03:36:55.372009+0000","last_became_active":"2024-03-12T03:33:21.107037+0000","last_became_peered":"2024-03-12T03:33:21.107037+0000","last_unstale":"2024-03-12T03:36:55.372009+0000","last_undegraded":"2024-03-12T03:36:55.372009+0000","last_fullsized":"2024-03-12T03:36:55.372009+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.698268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.698268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":156,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.378017+0000","last_change":"2024-03-12T03:36:11.006658+0000","last_active":"2024-03-12T03:36:55.378017+0000","last_peered":"2024-03-12T03:36:55.378017+0000","last_clean":"2024-03-12T03:36:55.378017+0000","last_became_active":"2024-03-12T03:33:23.160818+0000","last_became_peered":"2024-03-12T03:33:23.160818+0000","last_unstale":"2024-03-12T03:36:55.378017+0000","last_undegraded":"2024-03-12T03:36:55.378017+0000","last_fullsized":"2024-03-12T03:36:55.378017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-12T03:36:11.006154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:11.006154+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.382856+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.10","version":"125'6949","reported_seq":7721,"reported_epoch":125,"state":"active+remapped+backfilling","last_fresh":"2024-03-12T03:37:00.962723+0000","last_change":"2024-03-12T03:36:52.565904+0000","last_active":"2024-03-12T03:37:00.962723+0000","last_peered":"2024-03-12T03:37:00.962723+0000","last_clean":"2024-03-12T03:36:50.171417+0000","last_became_active":"2024-03-12T03:36:52.547003+0000","last_became_peered":"2024-03-12T03:36:52.547003+0000","last_unstale":"2024-03-12T03:37:00.962723+0000","last_undegraded":"2024-03-12T03:37:00.962723+0000","last_fullsized":"2024-03-12T03:37:00.962723+0000","mapping_epoch":122,"log_start":"100'800","ondisk_log_start":"100'800","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6149,"log_dups_size":0,"ondisk_log_size":6149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236879872,"num_objects":3615,"num_object_clones":0,"num_object_copies":10845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3252,"num_objects_unfound":0,"num_objects_dirty":3615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3615,"num_write_kb":231328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":22675456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1,5],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":3615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.362999+0000","last_change":"2024-03-12T03:35:59.076156+0000","last_active":"2024-03-12T03:36:55.362999+0000","last_peered":"2024-03-12T03:36:55.362999+0000","last_clean":"2024-03-12T03:36:55.362999+0000","last_became_active":"2024-03-12T03:33:27.191710+0000","last_became_peered":"2024-03-12T03:33:27.191710+0000","last_unstale":"2024-03-12T03:36:55.362999+0000","last_undegraded":"2024-03-12T03:36:55.362999+0000","last_fullsized":"2024-03-12T03:36:55.362999+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.075897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.075897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:18.991490+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.377774+0000","last_change":"2024-03-12T03:36:00.842130+0000","last_active":"2024-03-12T03:36:55.377774+0000","last_peered":"2024-03-12T03:36:55.377774+0000","last_clean":"2024-03-12T03:36:55.377774+0000","last_became_active":"2024-03-12T03:33:25.182004+0000","last_became_peered":"2024-03-12T03:33:25.182004+0000","last_unstale":"2024-03-12T03:36:55.377774+0000","last_undegraded":"2024-03-12T03:36:55.377774+0000","last_fullsized":"2024-03-12T03:36:55.377774+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:00.841986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:00.841986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:23.143222+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.359850+0000","last_change":"2024-03-12T03:36:26.141166+0000","last_active":"2024-03-12T03:36:55.359850+0000","last_peered":"2024-03-12T03:36:55.359850+0000","last_clean":"2024-03-12T03:36:55.359850+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:36:55.359850+0000","last_undegraded":"2024-03-12T03:36:55.359850+0000","last_fullsized":"2024-03-12T03:36:55.359850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.141036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.141036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.596277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":151,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.360596+0000","last_change":"2024-03-12T03:34:34.953658+0000","last_active":"2024-03-12T03:36:55.360596+0000","last_peered":"2024-03-12T03:36:55.360596+0000","last_clean":"2024-03-12T03:36:55.360596+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:36:55.360596+0000","last_undegraded":"2024-03-12T03:36:55.360596+0000","last_fullsized":"2024-03-12T03:36:55.360596+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:34:34.953565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.953565+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"125'7059","reported_seq":9451,"reported_epoch":125,"state":"active+remapped+backfilling","last_fresh":"2024-03-12T03:37:00.925711+0000","last_change":"2024-03-12T03:36:52.434219+0000","last_active":"2024-03-12T03:37:00.925711+0000","last_peered":"2024-03-12T03:37:00.925711+0000","last_clean":"2024-03-12T03:36:50.240631+0000","last_became_active":"2024-03-12T03:36:52.424496+0000","last_became_peered":"2024-03-12T03:36:52.424496+0000","last_unstale":"2024-03-12T03:37:00.925711+0000","last_undegraded":"2024-03-12T03:37:00.925711+0000","last_fullsized":"2024-03-12T03:37:00.925711+0000","mapping_epoch":122,"log_start":"100'500","ondisk_log_start":"100'500","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6559,"log_dups_size":0,"ondisk_log_size":6559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240123904,"num_objects":3664,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9207,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":234496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":37879808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[0,3,6],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3,6","objects":3664}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.378703+0000","last_change":"2024-03-12T03:36:07.111872+0000","last_active":"2024-03-12T03:36:55.378703+0000","last_peered":"2024-03-12T03:36:55.378703+0000","last_clean":"2024-03-12T03:36:55.378703+0000","last_became_active":"2024-03-12T03:33:27.194094+0000","last_became_peered":"2024-03-12T03:33:27.194094+0000","last_unstale":"2024-03-12T03:36:55.378703+0000","last_undegraded":"2024-03-12T03:36:55.378703+0000","last_fullsized":"2024-03-12T03:36:55.378703+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:07.111722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.111722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:33.629653+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":188,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.362397+0000","last_change":"2024-03-12T03:36:07.296952+0000","last_active":"2024-03-12T03:36:55.362397+0000","last_peered":"2024-03-12T03:36:55.362397+0000","last_clean":"2024-03-12T03:36:55.362397+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:36:55.362397+0000","last_undegraded":"2024-03-12T03:36:55.362397+0000","last_fullsized":"2024-03-12T03:36:55.362397+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:07.296796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.296796+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.190322+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":150,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371183+0000","last_change":"2024-03-12T03:36:27.955751+0000","last_active":"2024-03-12T03:36:55.371183+0000","last_peered":"2024-03-12T03:36:55.371183+0000","last_clean":"2024-03-12T03:36:55.371183+0000","last_became_active":"2024-03-12T03:33:21.573244+0000","last_became_peered":"2024-03-12T03:33:21.573244+0000","last_unstale":"2024-03-12T03:36:55.371183+0000","last_undegraded":"2024-03-12T03:36:55.371183+0000","last_fullsized":"2024-03-12T03:36:55.371183+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:27.955624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:27.955624+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:42.177558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":171,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.366839+0000","last_change":"2024-03-12T03:35:58.079779+0000","last_active":"2024-03-12T03:36:55.366839+0000","last_peered":"2024-03-12T03:36:55.366839+0000","last_clean":"2024-03-12T03:36:55.366839+0000","last_became_active":"2024-03-12T03:33:23.551706+0000","last_became_peered":"2024-03-12T03:33:23.551706+0000","last_unstale":"2024-03-12T03:36:55.366839+0000","last_undegraded":"2024-03-12T03:36:55.366839+0000","last_fullsized":"2024-03-12T03:36:55.366839+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-03-12T03:35:58.079645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.079645+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.444037+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"125'6967","reported_seq":7045,"reported_epoch":125,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-12T03:37:00.783242+0000","last_change":"2024-03-12T03:36:52.402066+0000","last_active":"2024-03-12T03:37:00.783242+0000","last_peered":"2024-03-12T03:37:00.783242+0000","last_clean":"2024-03-12T03:36:50.275996+0000","last_became_active":"2024-03-12T03:36:52.375928+0000","last_became_peered":"2024-03-12T03:36:52.375928+0000","last_unstale":"2024-03-12T03:37:00.783242+0000","last_undegraded":"2024-03-12T03:37:00.783242+0000","last_fullsized":"2024-03-12T03:37:00.783242+0000","mapping_epoch":122,"log_start":"101'1100","ondisk_log_start":"101'1100","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5867,"log_dups_size":0,"ondisk_log_size":5867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237109248,"num_objects":3618,"num_object_clones":0,"num_object_copies":10854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3618,"num_objects_unfound":0,"num_objects_dirty":3618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3618,"num_write_kb":231552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":3618}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.362356+0000","last_change":"2024-03-12T03:35:12.124255+0000","last_active":"2024-03-12T03:36:55.362356+0000","last_peered":"2024-03-12T03:36:55.362356+0000","last_clean":"2024-03-12T03:36:55.362356+0000","last_became_active":"2024-03-12T03:33:27.184916+0000","last_became_peered":"2024-03-12T03:33:27.184916+0000","last_unstale":"2024-03-12T03:36:55.362356+0000","last_undegraded":"2024-03-12T03:36:55.362356+0000","last_fullsized":"2024-03-12T03:36:55.362356+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:12.124167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:12.124167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":183,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.367019+0000","last_change":"2024-03-12T03:35:08.006530+0000","last_active":"2024-03-12T03:36:55.367019+0000","last_peered":"2024-03-12T03:36:55.367019+0000","last_clean":"2024-03-12T03:36:55.367019+0000","last_became_active":"2024-03-12T03:33:25.179158+0000","last_became_peered":"2024-03-12T03:33:25.179158+0000","last_unstale":"2024-03-12T03:36:55.367019+0000","last_undegraded":"2024-03-12T03:36:55.367019+0000","last_fullsized":"2024-03-12T03:36:55.367019+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:35:08.006375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:08.006375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371314+0000","last_change":"2024-03-12T03:36:14.197774+0000","last_active":"2024-03-12T03:36:55.371314+0000","last_peered":"2024-03-12T03:36:55.371314+0000","last_clean":"2024-03-12T03:36:55.371314+0000","last_became_active":"2024-03-12T03:33:21.101819+0000","last_became_peered":"2024-03-12T03:33:21.101819+0000","last_unstale":"2024-03-12T03:36:55.371314+0000","last_undegraded":"2024-03-12T03:36:55.371314+0000","last_fullsized":"2024-03-12T03:36:55.371314+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:14.197649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:36:14.197649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:41.087756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":168,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.362498+0000","last_change":"2024-03-12T03:36:12.092317+0000","last_active":"2024-03-12T03:36:55.362498+0000","last_peered":"2024-03-12T03:36:55.362498+0000","last_clean":"2024-03-12T03:36:55.362498+0000","last_became_active":"2024-03-12T03:33:23.553480+0000","last_became_peered":"2024-03-12T03:33:23.553480+0000","last_unstale":"2024-03-12T03:36:55.362498+0000","last_undegraded":"2024-03-12T03:36:55.362498+0000","last_fullsized":"2024-03-12T03:36:55.362498+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:36:12.092175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:12.092175+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.115981+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.15","version":"125'7063","reported_seq":9846,"reported_epoch":125,"state":"active+remapped+backfilling","last_fresh":"2024-03-12T03:37:00.796782+0000","last_change":"2024-03-12T03:36:52.359377+0000","last_active":"2024-03-12T03:37:00.796782+0000","last_peered":"2024-03-12T03:37:00.796782+0000","last_clean":"2024-03-12T03:36:50.241653+0000","last_became_active":"2024-03-12T03:36:52.348833+0000","last_became_peered":"2024-03-12T03:36:52.348833+0000","last_unstale":"2024-03-12T03:37:00.796782+0000","last_undegraded":"2024-03-12T03:37:00.796782+0000","last_fullsized":"2024-03-12T03:37:00.796782+0000","mapping_epoch":122,"log_start":"101'1600","ondisk_log_start":"101'1600","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5463,"log_dups_size":0,"ondisk_log_size":5463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240451584,"num_objects":3669,"num_object_clones":0,"num_object_copies":11007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2268,"num_objects_unfound":0,"num_objects_dirty":3669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3669,"num_write_kb":234816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1351,"num_bytes_recovered":88539136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,2,0],"avail_no_missing":["7","0","2"],"object_location_counts":[{"shards":"0,2,7","objects":3669}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.373805+0000","last_change":"2024-03-12T03:36:02.155537+0000","last_active":"2024-03-12T03:36:55.373805+0000","last_peered":"2024-03-12T03:36:55.373805+0000","last_clean":"2024-03-12T03:36:55.373805+0000","last_became_active":"2024-03-12T03:33:27.187657+0000","last_became_peered":"2024-03-12T03:33:27.187657+0000","last_unstale":"2024-03-12T03:36:55.373805+0000","last_undegraded":"2024-03-12T03:36:55.373805+0000","last_fullsized":"2024-03-12T03:36:55.373805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:02.155425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:02.155425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:24.813580+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.367716+0000","last_change":"2024-03-12T03:35:04.007338+0000","last_active":"2024-03-12T03:36:55.367716+0000","last_peered":"2024-03-12T03:36:55.367716+0000","last_clean":"2024-03-12T03:36:55.367716+0000","last_became_active":"2024-03-12T03:33:25.174953+0000","last_became_peered":"2024-03-12T03:33:25.174953+0000","last_unstale":"2024-03-12T03:36:55.367716+0000","last_undegraded":"2024-03-12T03:36:55.367716+0000","last_fullsized":"2024-03-12T03:36:55.367716+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:35:04.007252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371039+0000","last_change":"2024-03-12T03:34:52.848468+0000","last_active":"2024-03-12T03:36:55.371039+0000","last_peered":"2024-03-12T03:36:55.371039+0000","last_clean":"2024-03-12T03:36:55.371039+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:36:55.371039+0000","last_undegraded":"2024-03-12T03:36:55.371039+0000","last_fullsized":"2024-03-12T03:36:55.371039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:52.848366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:52.848366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":172,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.379283+0000","last_change":"2024-03-12T03:35:11.786139+0000","last_active":"2024-03-12T03:36:55.379283+0000","last_peered":"2024-03-12T03:36:55.379283+0000","last_clean":"2024-03-12T03:36:55.379283+0000","last_became_active":"2024-03-12T03:33:23.552164+0000","last_became_peered":"2024-03-12T03:33:23.552164+0000","last_unstale":"2024-03-12T03:36:55.379283+0000","last_undegraded":"2024-03-12T03:36:55.379283+0000","last_fullsized":"2024-03-12T03:36:55.379283+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:35:11.786040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.786040+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"125'6981","reported_seq":7064,"reported_epoch":125,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-12T03:37:00.814529+0000","last_change":"2024-03-12T03:36:52.413890+0000","last_active":"2024-03-12T03:37:00.814529+0000","last_peered":"2024-03-12T03:37:00.814529+0000","last_clean":"2024-03-12T03:36:50.262948+0000","last_became_active":"2024-03-12T03:36:52.402968+0000","last_became_peered":"2024-03-12T03:36:52.402968+0000","last_unstale":"2024-03-12T03:37:00.814529+0000","last_undegraded":"2024-03-12T03:37:00.814529+0000","last_fullsized":"2024-03-12T03:37:00.814529+0000","mapping_epoch":122,"log_start":"101'1100","ondisk_log_start":"101'1100","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5881,"log_dups_size":0,"ondisk_log_size":5881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237469696,"num_objects":3624,"num_object_clones":0,"num_object_copies":10872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10872,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3624,"num_write_kb":231904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[0,1,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":3624}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371909+0000","last_change":"2024-03-12T03:36:24.959007+0000","last_active":"2024-03-12T03:36:55.371909+0000","last_peered":"2024-03-12T03:36:55.371909+0000","last_clean":"2024-03-12T03:36:55.371909+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:36:55.371909+0000","last_undegraded":"2024-03-12T03:36:55.371909+0000","last_fullsized":"2024-03-12T03:36:55.371909+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:24.958880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.958880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:47.117767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.362882+0000","last_change":"2024-03-12T03:36:29.181191+0000","last_active":"2024-03-12T03:36:55.362882+0000","last_peered":"2024-03-12T03:36:55.362882+0000","last_clean":"2024-03-12T03:36:55.362882+0000","last_became_active":"2024-03-12T03:33:25.180752+0000","last_became_peered":"2024-03-12T03:33:25.180752+0000","last_unstale":"2024-03-12T03:36:55.362882+0000","last_undegraded":"2024-03-12T03:36:55.362882+0000","last_fullsized":"2024-03-12T03:36:55.362882+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:29.181010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:29.181010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:33.940875+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.368238+0000","last_change":"2024-03-12T03:35:58.075845+0000","last_active":"2024-03-12T03:36:55.368238+0000","last_peered":"2024-03-12T03:36:55.368238+0000","last_clean":"2024-03-12T03:36:55.368238+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:36:55.368238+0000","last_undegraded":"2024-03-12T03:36:55.368238+0000","last_fullsized":"2024-03-12T03:36:55.368238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:58.075754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.075754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:04.874641+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":153,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.367921+0000","last_change":"2024-03-12T03:35:00.079249+0000","last_active":"2024-03-12T03:36:55.367921+0000","last_peered":"2024-03-12T03:36:55.367921+0000","last_clean":"2024-03-12T03:36:55.367921+0000","last_became_active":"2024-03-12T03:33:23.551773+0000","last_became_peered":"2024-03-12T03:33:23.551773+0000","last_unstale":"2024-03-12T03:36:55.367921+0000","last_undegraded":"2024-03-12T03:36:55.367921+0000","last_fullsized":"2024-03-12T03:36:55.367921+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:35:00.079152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.079152+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"125'7160","reported_seq":7230,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.884395+0000","last_change":"2024-03-12T03:35:23.782770+0000","last_active":"2024-03-12T03:37:00.884395+0000","last_peered":"2024-03-12T03:37:00.884395+0000","last_clean":"2024-03-12T03:37:00.884395+0000","last_became_active":"2024-03-12T03:35:23.782112+0000","last_became_peered":"2024-03-12T03:35:23.782112+0000","last_unstale":"2024-03-12T03:37:00.884395+0000","last_undegraded":"2024-03-12T03:37:00.884395+0000","last_fullsized":"2024-03-12T03:37:00.884395+0000","mapping_epoch":97,"log_start":"102'2534","ondisk_log_start":"102'2534","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4626,"log_dups_size":0,"ondisk_log_size":4626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469237760,"num_objects":7160,"num_object_clones":0,"num_object_copies":21480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7160,"num_write_kb":458240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363444+0000","last_change":"2024-03-12T03:36:20.147626+0000","last_active":"2024-03-12T03:36:55.363444+0000","last_peered":"2024-03-12T03:36:55.363444+0000","last_clean":"2024-03-12T03:36:55.363444+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:36:55.363444+0000","last_undegraded":"2024-03-12T03:36:55.363444+0000","last_fullsized":"2024-03-12T03:36:55.363444+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.147488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:44.684777+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371727+0000","last_change":"2024-03-12T03:35:12.907083+0000","last_active":"2024-03-12T03:36:55.371727+0000","last_peered":"2024-03-12T03:36:55.371727+0000","last_clean":"2024-03-12T03:36:55.371727+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:36:55.371727+0000","last_undegraded":"2024-03-12T03:36:55.371727+0000","last_fullsized":"2024-03-12T03:36:55.371727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:35:12.906988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:12.906988+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"90'36","reported_seq":198,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.362929+0000","last_change":"2024-03-12T03:36:15.181987+0000","last_active":"2024-03-12T03:36:55.362929+0000","last_peered":"2024-03-12T03:36:55.362929+0000","last_clean":"2024-03-12T03:36:55.362929+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:36:55.362929+0000","last_undegraded":"2024-03-12T03:36:55.362929+0000","last_fullsized":"2024-03-12T03:36:55.362929+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:36:15.181852+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.181852+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.159048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":180,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371197+0000","last_change":"2024-03-12T03:34:39.659403+0000","last_active":"2024-03-12T03:36:55.371197+0000","last_peered":"2024-03-12T03:36:55.371197+0000","last_clean":"2024-03-12T03:36:55.371197+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:36:55.371197+0000","last_undegraded":"2024-03-12T03:36:55.371197+0000","last_fullsized":"2024-03-12T03:36:55.371197+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:39.659277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.659277+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"125'7349","reported_seq":7419,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.930870+0000","last_change":"2024-03-12T03:35:23.777596+0000","last_active":"2024-03-12T03:37:00.930870+0000","last_peered":"2024-03-12T03:37:00.930870+0000","last_clean":"2024-03-12T03:37:00.930870+0000","last_became_active":"2024-03-12T03:35:23.777272+0000","last_became_peered":"2024-03-12T03:35:23.777272+0000","last_unstale":"2024-03-12T03:37:00.930870+0000","last_undegraded":"2024-03-12T03:37:00.930870+0000","last_fullsized":"2024-03-12T03:37:00.930870+0000","mapping_epoch":97,"log_start":"105'3122","ondisk_log_start":"105'3122","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4227,"log_dups_size":0,"ondisk_log_size":4227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481624064,"num_objects":7349,"num_object_clones":0,"num_object_copies":22047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7349,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.366531+0000","last_change":"2024-03-12T03:33:27.191091+0000","last_active":"2024-03-12T03:36:55.366531+0000","last_peered":"2024-03-12T03:36:55.366531+0000","last_clean":"2024-03-12T03:36:55.366531+0000","last_became_active":"2024-03-12T03:33:27.190753+0000","last_became_peered":"2024-03-12T03:33:27.190753+0000","last_unstale":"2024-03-12T03:36:55.366531+0000","last_undegraded":"2024-03-12T03:36:55.366531+0000","last_fullsized":"2024-03-12T03:36:55.366531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371726+0000","last_change":"2024-03-12T03:36:17.051625+0000","last_active":"2024-03-12T03:36:55.371726+0000","last_peered":"2024-03-12T03:36:55.371726+0000","last_clean":"2024-03-12T03:36:55.371726+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:36:55.371726+0000","last_undegraded":"2024-03-12T03:36:55.371726+0000","last_fullsized":"2024-03-12T03:36:55.371726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:36:17.051487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.698687+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":179,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.360207+0000","last_change":"2024-03-12T03:36:10.043243+0000","last_active":"2024-03-12T03:36:55.360207+0000","last_peered":"2024-03-12T03:36:55.360207+0000","last_clean":"2024-03-12T03:36:55.360207+0000","last_became_active":"2024-03-12T03:33:23.163351+0000","last_became_peered":"2024-03-12T03:33:23.163351+0000","last_unstale":"2024-03-12T03:36:55.360207+0000","last_undegraded":"2024-03-12T03:36:55.360207+0000","last_fullsized":"2024-03-12T03:36:55.360207+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-03-12T03:36:10.043108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:10.043108+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.376152+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.374299+0000","last_change":"2024-03-12T03:36:26.332254+0000","last_active":"2024-03-12T03:36:55.374299+0000","last_peered":"2024-03-12T03:36:55.374299+0000","last_clean":"2024-03-12T03:36:55.374299+0000","last_became_active":"2024-03-12T03:33:25.185596+0000","last_became_peered":"2024-03-12T03:33:25.185596+0000","last_unstale":"2024-03-12T03:36:55.374299+0000","last_undegraded":"2024-03-12T03:36:55.374299+0000","last_fullsized":"2024-03-12T03:36:55.374299+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.332124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.332124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:45.657747+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"125'7056","reported_seq":7132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.881266+0000","last_change":"2024-03-12T03:36:49.287952+0000","last_active":"2024-03-12T03:37:00.881266+0000","last_peered":"2024-03-12T03:37:00.881266+0000","last_clean":"2024-03-12T03:37:00.881266+0000","last_became_active":"2024-03-12T03:36:49.278858+0000","last_became_peered":"2024-03-12T03:36:49.278858+0000","last_unstale":"2024-03-12T03:37:00.881266+0000","last_undegraded":"2024-03-12T03:37:00.881266+0000","last_fullsized":"2024-03-12T03:37:00.881266+0000","mapping_epoch":97,"log_start":"101'1100","ondisk_log_start":"101'1100","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5956,"log_dups_size":0,"ondisk_log_size":5956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241139712,"num_objects":3679,"num_object_clones":0,"num_object_copies":11037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3679,"num_write_kb":235488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.370564+0000","last_change":"2024-03-12T03:34:43.716123+0000","last_active":"2024-03-12T03:36:55.370564+0000","last_peered":"2024-03-12T03:36:55.370564+0000","last_clean":"2024-03-12T03:36:55.370564+0000","last_became_active":"2024-03-12T03:33:27.187955+0000","last_became_peered":"2024-03-12T03:33:27.187955+0000","last_unstale":"2024-03-12T03:36:55.370564+0000","last_undegraded":"2024-03-12T03:36:55.370564+0000","last_fullsized":"2024-03-12T03:36:55.370564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:43.716021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.716021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.373012+0000","last_change":"2024-03-12T03:36:18.320863+0000","last_active":"2024-03-12T03:36:55.373012+0000","last_peered":"2024-03-12T03:36:55.373012+0000","last_clean":"2024-03-12T03:36:55.373012+0000","last_became_active":"2024-03-12T03:33:25.179593+0000","last_became_peered":"2024-03-12T03:33:25.179593+0000","last_unstale":"2024-03-12T03:36:55.373012+0000","last_undegraded":"2024-03-12T03:36:55.373012+0000","last_fullsized":"2024-03-12T03:36:55.373012+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.320728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.320728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.937998+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.372187+0000","last_change":"2024-03-12T03:36:09.026855+0000","last_active":"2024-03-12T03:36:55.372187+0000","last_peered":"2024-03-12T03:36:55.372187+0000","last_clean":"2024-03-12T03:36:55.372187+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:36:55.372187+0000","last_undegraded":"2024-03-12T03:36:55.372187+0000","last_fullsized":"2024-03-12T03:36:55.372187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:09.026725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:09.026725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.130351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":183,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.368546+0000","last_change":"2024-03-12T03:36:27.151507+0000","last_active":"2024-03-12T03:36:55.368546+0000","last_peered":"2024-03-12T03:36:55.368546+0000","last_clean":"2024-03-12T03:36:55.368546+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:36:55.368546+0000","last_undegraded":"2024-03-12T03:36:55.368546+0000","last_fullsized":"2024-03-12T03:36:55.368546+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:36:27.151375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:27.151375+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.667242+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"125'6997","reported_seq":7073,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.915655+0000","last_change":"2024-03-12T03:36:49.308754+0000","last_active":"2024-03-12T03:37:00.915655+0000","last_peered":"2024-03-12T03:37:00.915655+0000","last_clean":"2024-03-12T03:37:00.915655+0000","last_became_active":"2024-03-12T03:36:49.305916+0000","last_became_peered":"2024-03-12T03:36:49.305916+0000","last_unstale":"2024-03-12T03:37:00.915655+0000","last_undegraded":"2024-03-12T03:37:00.915655+0000","last_fullsized":"2024-03-12T03:37:00.915655+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6997,"log_dups_size":0,"ondisk_log_size":6997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:01.279495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238419968,"num_objects":3638,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3638,"num_write_kb":232832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371928+0000","last_change":"2024-03-12T03:36:10.178072+0000","last_active":"2024-03-12T03:36:55.371928+0000","last_peered":"2024-03-12T03:36:55.371928+0000","last_clean":"2024-03-12T03:36:55.371928+0000","last_became_active":"2024-03-12T03:33:27.189948+0000","last_became_peered":"2024-03-12T03:33:27.189948+0000","last_unstale":"2024-03-12T03:36:55.371928+0000","last_undegraded":"2024-03-12T03:36:55.371928+0000","last_fullsized":"2024-03-12T03:36:55.371928+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:10.177808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:10.177808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:35.471819+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.374499+0000","last_change":"2024-03-12T03:36:11.345502+0000","last_active":"2024-03-12T03:36:55.374499+0000","last_peered":"2024-03-12T03:36:55.374499+0000","last_clean":"2024-03-12T03:36:55.374499+0000","last_became_active":"2024-03-12T03:33:25.184289+0000","last_became_peered":"2024-03-12T03:33:25.184289+0000","last_unstale":"2024-03-12T03:36:55.374499+0000","last_undegraded":"2024-03-12T03:36:55.374499+0000","last_fullsized":"2024-03-12T03:36:55.374499+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:11.345381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:11.345381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:32.654030+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.362807+0000","last_change":"2024-03-12T03:36:26.134473+0000","last_active":"2024-03-12T03:36:55.362807+0000","last_peered":"2024-03-12T03:36:55.362807+0000","last_clean":"2024-03-12T03:36:55.362807+0000","last_became_active":"2024-03-12T03:33:21.102410+0000","last_became_peered":"2024-03-12T03:33:21.102410+0000","last_unstale":"2024-03-12T03:36:55.362807+0000","last_undegraded":"2024-03-12T03:36:55.362807+0000","last_fullsized":"2024-03-12T03:36:55.362807+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.134337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.134337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:45.916621+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"111'20","reported_seq":173,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.370794+0000","last_change":"2024-03-12T03:36:29.192473+0000","last_active":"2024-03-12T03:36:55.370794+0000","last_peered":"2024-03-12T03:36:55.370794+0000","last_clean":"2024-03-12T03:36:55.370794+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:36:55.370794+0000","last_undegraded":"2024-03-12T03:36:55.370794+0000","last_fullsized":"2024-03-12T03:36:55.370794+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'20","last_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:36:29.192310+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:32.549710+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"125'7084","reported_seq":7159,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.700564+0000","last_change":"2024-03-12T03:36:49.295573+0000","last_active":"2024-03-12T03:37:00.700564+0000","last_peered":"2024-03-12T03:37:00.700564+0000","last_clean":"2024-03-12T03:37:00.700564+0000","last_became_active":"2024-03-12T03:36:49.292644+0000","last_became_peered":"2024-03-12T03:36:49.292644+0000","last_unstale":"2024-03-12T03:37:00.700564+0000","last_undegraded":"2024-03-12T03:37:00.700564+0000","last_fullsized":"2024-03-12T03:37:00.700564+0000","mapping_epoch":97,"log_start":"100'500","ondisk_log_start":"100'500","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6584,"log_dups_size":0,"ondisk_log_size":6584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:01.257141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241762304,"num_objects":3689,"num_object_clones":0,"num_object_copies":11067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3689,"num_write_kb":236096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.366324+0000","last_change":"2024-03-12T03:34:49.088183+0000","last_active":"2024-03-12T03:36:55.366324+0000","last_peered":"2024-03-12T03:36:55.366324+0000","last_clean":"2024-03-12T03:36:55.366324+0000","last_became_active":"2024-03-12T03:33:27.185041+0000","last_became_peered":"2024-03-12T03:33:27.185041+0000","last_unstale":"2024-03-12T03:36:55.366324+0000","last_undegraded":"2024-03-12T03:36:55.366324+0000","last_fullsized":"2024-03-12T03:36:55.366324+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.088098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.088098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":155,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363537+0000","last_change":"2024-03-12T03:36:13.134209+0000","last_active":"2024-03-12T03:36:55.363537+0000","last_peered":"2024-03-12T03:36:55.363537+0000","last_clean":"2024-03-12T03:36:55.363537+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:36:55.363537+0000","last_undegraded":"2024-03-12T03:36:55.363537+0000","last_fullsized":"2024-03-12T03:36:55.363537+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:36:13.134087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.134087+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.167729+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.206458+0000","last_change":"2024-03-12T03:37:00.206483+0000","last_active":"2024-03-12T03:37:00.206458+0000","last_peered":"2024-03-12T03:37:00.206458+0000","last_clean":"2024-03-12T03:37:00.206458+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:37:00.206458+0000","last_undegraded":"2024-03-12T03:37:00.206458+0000","last_fullsized":"2024-03-12T03:37:00.206458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:00.206383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:37:00.206383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:20.230781+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'35","reported_seq":175,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.361749+0000","last_change":"2024-03-12T03:34:54.301841+0000","last_active":"2024-03-12T03:36:55.361749+0000","last_peered":"2024-03-12T03:36:55.361749+0000","last_clean":"2024-03-12T03:36:55.361749+0000","last_became_active":"2024-03-12T03:33:50.151664+0000","last_became_peered":"2024-03-12T03:33:50.151664+0000","last_unstale":"2024-03-12T03:36:55.361749+0000","last_undegraded":"2024-03-12T03:36:55.361749+0000","last_fullsized":"2024-03-12T03:36:55.361749+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'35","last_scrub_stamp":"2024-03-12T03:34:54.301725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:54.301725+0000","objects_scrubbed":11,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.382706+0000","last_change":"2024-03-12T03:36:15.170135+0000","last_active":"2024-03-12T03:36:55.382706+0000","last_peered":"2024-03-12T03:36:55.382706+0000","last_clean":"2024-03-12T03:36:55.382706+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:36:55.382706+0000","last_undegraded":"2024-03-12T03:36:55.382706+0000","last_fullsized":"2024-03-12T03:36:55.382706+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:15.170007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.170007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:15.186652+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"125'7006","reported_seq":7082,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.842347+0000","last_change":"2024-03-12T03:36:49.289418+0000","last_active":"2024-03-12T03:37:00.842347+0000","last_peered":"2024-03-12T03:37:00.842347+0000","last_clean":"2024-03-12T03:37:00.842347+0000","last_became_active":"2024-03-12T03:36:49.281680+0000","last_became_peered":"2024-03-12T03:36:49.281680+0000","last_unstale":"2024-03-12T03:37:00.842347+0000","last_undegraded":"2024-03-12T03:37:00.842347+0000","last_fullsized":"2024-03-12T03:37:00.842347+0000","mapping_epoch":97,"log_start":"101'1100","ondisk_log_start":"101'1100","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5906,"log_dups_size":0,"ondisk_log_size":5906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239108096,"num_objects":3648,"num_object_clones":0,"num_object_copies":10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3648,"num_write_kb":233504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":134,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.372237+0000","last_change":"2024-03-12T03:36:02.083741+0000","last_active":"2024-03-12T03:36:55.372237+0000","last_peered":"2024-03-12T03:36:55.372237+0000","last_clean":"2024-03-12T03:36:55.372237+0000","last_became_active":"2024-03-12T03:33:27.380389+0000","last_became_peered":"2024-03-12T03:33:27.380389+0000","last_unstale":"2024-03-12T03:36:55.372237+0000","last_undegraded":"2024-03-12T03:36:55.372237+0000","last_fullsized":"2024-03-12T03:36:55.372237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:36:02.083611+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:36:02.083611+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:13.347363+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.375795+0000","last_change":"2024-03-12T03:36:21.149936+0000","last_active":"2024-03-12T03:36:55.375795+0000","last_peered":"2024-03-12T03:36:55.375795+0000","last_clean":"2024-03-12T03:36:55.375795+0000","last_became_active":"2024-03-12T03:33:21.572911+0000","last_became_peered":"2024-03-12T03:33:21.572911+0000","last_unstale":"2024-03-12T03:36:55.375795+0000","last_undegraded":"2024-03-12T03:36:55.375795+0000","last_fullsized":"2024-03-12T03:36:55.375795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:21.149686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:21.149686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:31.591082+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":153,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.372505+0000","last_change":"2024-03-12T03:36:15.348581+0000","last_active":"2024-03-12T03:36:55.372505+0000","last_peered":"2024-03-12T03:36:55.372505+0000","last_clean":"2024-03-12T03:36:55.372505+0000","last_became_active":"2024-03-12T03:33:20.093064+0000","last_became_peered":"2024-03-12T03:33:20.093064+0000","last_unstale":"2024-03-12T03:36:55.372505+0000","last_undegraded":"2024-03-12T03:36:55.372505+0000","last_fullsized":"2024-03-12T03:36:55.372505+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:15.348471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.348471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:21.623557+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.000516994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":179,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.376734+0000","last_change":"2024-03-12T03:36:19.124448+0000","last_active":"2024-03-12T03:36:55.376734+0000","last_peered":"2024-03-12T03:36:55.376734+0000","last_clean":"2024-03-12T03:36:55.376734+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:36:55.376734+0000","last_undegraded":"2024-03-12T03:36:55.376734+0000","last_fullsized":"2024-03-12T03:36:55.376734+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-12T03:36:19.124308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.124308+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:36.079349+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.383072+0000","last_change":"2024-03-12T03:35:58.091300+0000","last_active":"2024-03-12T03:36:55.383072+0000","last_peered":"2024-03-12T03:36:55.383072+0000","last_clean":"2024-03-12T03:36:55.383072+0000","last_became_active":"2024-03-12T03:33:25.179022+0000","last_became_peered":"2024-03-12T03:33:25.179022+0000","last_unstale":"2024-03-12T03:36:55.383072+0000","last_undegraded":"2024-03-12T03:36:55.383072+0000","last_fullsized":"2024-03-12T03:36:55.383072+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:58.091182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.091182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:04.769136+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"125'6969","reported_seq":7047,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.814329+0000","last_change":"2024-03-12T03:36:49.322938+0000","last_active":"2024-03-12T03:37:00.814329+0000","last_peered":"2024-03-12T03:37:00.814329+0000","last_clean":"2024-03-12T03:37:00.814329+0000","last_became_active":"2024-03-12T03:36:49.303137+0000","last_became_peered":"2024-03-12T03:36:49.303137+0000","last_unstale":"2024-03-12T03:37:00.814329+0000","last_undegraded":"2024-03-12T03:37:00.814329+0000","last_fullsized":"2024-03-12T03:37:00.814329+0000","mapping_epoch":97,"log_start":"101'1100","ondisk_log_start":"101'1100","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237240320,"num_objects":3620,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":231680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363293+0000","last_change":"2024-03-12T03:36:25.119508+0000","last_active":"2024-03-12T03:36:55.363293+0000","last_peered":"2024-03-12T03:36:55.363293+0000","last_clean":"2024-03-12T03:36:55.363293+0000","last_became_active":"2024-03-12T03:33:27.188551+0000","last_became_peered":"2024-03-12T03:33:27.188551+0000","last_unstale":"2024-03-12T03:36:55.363293+0000","last_undegraded":"2024-03-12T03:36:55.363293+0000","last_fullsized":"2024-03-12T03:36:55.363293+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:25.119358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.119358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:54.029889+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":391,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.370666+0000","last_change":"2024-03-12T03:34:58.903289+0000","last_active":"2024-03-12T03:36:55.370666+0000","last_peered":"2024-03-12T03:36:55.370666+0000","last_clean":"2024-03-12T03:36:55.370666+0000","last_became_active":"2024-03-12T03:33:11.319888+0000","last_became_peered":"2024-03-12T03:33:11.319888+0000","last_unstale":"2024-03-12T03:36:55.370666+0000","last_undegraded":"2024-03-12T03:36:55.370666+0000","last_fullsized":"2024-03-12T03:36:55.370666+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:34:58.903163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.903163+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.368010+0000","last_change":"2024-03-12T03:35:14.996451+0000","last_active":"2024-03-12T03:36:55.368010+0000","last_peered":"2024-03-12T03:36:55.368010+0000","last_clean":"2024-03-12T03:36:55.368010+0000","last_became_active":"2024-03-12T03:33:21.100900+0000","last_became_peered":"2024-03-12T03:33:21.100900+0000","last_unstale":"2024-03-12T03:36:55.368010+0000","last_undegraded":"2024-03-12T03:36:55.368010+0000","last_fullsized":"2024-03-12T03:36:55.368010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:14.996357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:14.996357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":206,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.369138+0000","last_change":"2024-03-12T03:36:03.118128+0000","last_active":"2024-03-12T03:36:55.369138+0000","last_peered":"2024-03-12T03:36:55.369138+0000","last_clean":"2024-03-12T03:36:55.369138+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:36:55.369138+0000","last_undegraded":"2024-03-12T03:36:55.369138+0000","last_fullsized":"2024-03-12T03:36:55.369138+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:36:03.117990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:03.117990+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:31.305573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371568+0000","last_change":"2024-03-12T03:34:47.796670+0000","last_active":"2024-03-12T03:36:55.371568+0000","last_peered":"2024-03-12T03:36:55.371568+0000","last_clean":"2024-03-12T03:36:55.371568+0000","last_became_active":"2024-03-12T03:33:25.177762+0000","last_became_peered":"2024-03-12T03:33:25.177762+0000","last_unstale":"2024-03-12T03:36:55.371568+0000","last_undegraded":"2024-03-12T03:36:55.371568+0000","last_fullsized":"2024-03-12T03:36:55.371568+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.796561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.796561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"125'7093","reported_seq":7169,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.710102+0000","last_change":"2024-03-12T03:36:49.285675+0000","last_active":"2024-03-12T03:37:00.710102+0000","last_peered":"2024-03-12T03:37:00.710102+0000","last_clean":"2024-03-12T03:37:00.710102+0000","last_became_active":"2024-03-12T03:36:49.284030+0000","last_became_peered":"2024-03-12T03:36:49.284030+0000","last_unstale":"2024-03-12T03:37:00.710102+0000","last_undegraded":"2024-03-12T03:37:00.710102+0000","last_fullsized":"2024-03-12T03:37:00.710102+0000","mapping_epoch":97,"log_start":"101'1600","ondisk_log_start":"101'1600","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5493,"log_dups_size":0,"ondisk_log_size":5493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:02.024488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242417664,"num_objects":3699,"num_object_clones":0,"num_object_copies":11097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3699,"num_write_kb":236736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.376696+0000","last_change":"2024-03-12T03:36:15.042978+0000","last_active":"2024-03-12T03:36:55.376696+0000","last_peered":"2024-03-12T03:36:55.376696+0000","last_clean":"2024-03-12T03:36:55.376696+0000","last_became_active":"2024-03-12T03:33:27.191878+0000","last_became_peered":"2024-03-12T03:33:27.191878+0000","last_unstale":"2024-03-12T03:36:55.376696+0000","last_undegraded":"2024-03-12T03:36:55.376696+0000","last_fullsized":"2024-03-12T03:36:55.376696+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:15.042610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.042610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:21.860517+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371237+0000","last_change":"2024-03-12T03:35:05.715931+0000","last_active":"2024-03-12T03:36:55.371237+0000","last_peered":"2024-03-12T03:36:55.371237+0000","last_clean":"2024-03-12T03:36:55.371237+0000","last_became_active":"2024-03-12T03:33:21.097086+0000","last_became_peered":"2024-03-12T03:33:21.097086+0000","last_unstale":"2024-03-12T03:36:55.371237+0000","last_undegraded":"2024-03-12T03:36:55.371237+0000","last_fullsized":"2024-03-12T03:36:55.371237+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:05.715849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.715849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":370,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:59.413255+0000","last_change":"2024-03-12T03:36:08.059941+0000","last_active":"2024-03-12T03:36:59.413255+0000","last_peered":"2024-03-12T03:36:59.413255+0000","last_clean":"2024-03-12T03:36:59.413255+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:36:59.413255+0000","last_undegraded":"2024-03-12T03:36:59.413255+0000","last_fullsized":"2024-03-12T03:36:59.413255+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:36:08.059814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:36:08.059814+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:27.702586+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":214,"num_read_kb":209,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":158,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.378798+0000","last_change":"2024-03-12T03:36:03.015860+0000","last_active":"2024-03-12T03:36:55.378798+0000","last_peered":"2024-03-12T03:36:55.378798+0000","last_clean":"2024-03-12T03:36:55.378798+0000","last_became_active":"2024-03-12T03:33:23.168750+0000","last_became_peered":"2024-03-12T03:33:23.168750+0000","last_unstale":"2024-03-12T03:36:55.378798+0000","last_undegraded":"2024-03-12T03:36:55.378798+0000","last_fullsized":"2024-03-12T03:36:55.378798+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:36:03.015565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:03.015565+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:05.935088+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363744+0000","last_change":"2024-03-12T03:36:24.109073+0000","last_active":"2024-03-12T03:36:55.363744+0000","last_peered":"2024-03-12T03:36:55.363744+0000","last_clean":"2024-03-12T03:36:55.363744+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:36:55.363744+0000","last_undegraded":"2024-03-12T03:36:55.363744+0000","last_fullsized":"2024-03-12T03:36:55.363744+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:24.108984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.108984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.677961+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"125'6928","reported_seq":7004,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.796622+0000","last_change":"2024-03-12T03:36:49.288153+0000","last_active":"2024-03-12T03:37:00.796622+0000","last_peered":"2024-03-12T03:37:00.796622+0000","last_clean":"2024-03-12T03:37:00.796622+0000","last_became_active":"2024-03-12T03:36:49.286111+0000","last_became_peered":"2024-03-12T03:36:49.286111+0000","last_unstale":"2024-03-12T03:37:00.796622+0000","last_undegraded":"2024-03-12T03:37:00.796622+0000","last_fullsized":"2024-03-12T03:37:00.796622+0000","mapping_epoch":97,"log_start":"100'800","ondisk_log_start":"100'800","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6128,"log_dups_size":0,"ondisk_log_size":6128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:02.176582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235503616,"num_objects":3593,"num_object_clones":0,"num_object_copies":10779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3593,"num_write_kb":229984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.380230+0000","last_change":"2024-03-12T03:36:18.066969+0000","last_active":"2024-03-12T03:36:55.380230+0000","last_peered":"2024-03-12T03:36:55.380230+0000","last_clean":"2024-03-12T03:36:55.380230+0000","last_became_active":"2024-03-12T03:33:27.380458+0000","last_became_peered":"2024-03-12T03:33:27.380458+0000","last_unstale":"2024-03-12T03:36:55.380230+0000","last_undegraded":"2024-03-12T03:36:55.380230+0000","last_fullsized":"2024-03-12T03:36:55.380230+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.066712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.066712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:47.296061+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.376893+0000","last_change":"2024-03-12T03:36:01.874384+0000","last_active":"2024-03-12T03:36:55.376893+0000","last_peered":"2024-03-12T03:36:55.376893+0000","last_clean":"2024-03-12T03:36:55.376893+0000","last_became_active":"2024-03-12T03:33:21.573003+0000","last_became_peered":"2024-03-12T03:33:21.573003+0000","last_unstale":"2024-03-12T03:36:55.376893+0000","last_undegraded":"2024-03-12T03:36:55.376893+0000","last_fullsized":"2024-03-12T03:36:55.376893+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:01.874124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:01.874124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:04.609067+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":188,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371054+0000","last_change":"2024-03-12T03:36:22.118169+0000","last_active":"2024-03-12T03:36:55.371054+0000","last_peered":"2024-03-12T03:36:55.371054+0000","last_clean":"2024-03-12T03:36:55.371054+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:36:55.371054+0000","last_undegraded":"2024-03-12T03:36:55.371054+0000","last_fullsized":"2024-03-12T03:36:55.371054+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:36:22.118034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:22.118034+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:23.200152+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.368695+0000","last_change":"2024-03-12T03:36:21.140906+0000","last_active":"2024-03-12T03:36:55.368695+0000","last_peered":"2024-03-12T03:36:55.368695+0000","last_clean":"2024-03-12T03:36:55.368695+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:36:55.368695+0000","last_undegraded":"2024-03-12T03:36:55.368695+0000","last_fullsized":"2024-03-12T03:36:55.368695+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:21.140825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:21.140825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:25.002630+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"125'6937","reported_seq":7054,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.768132+0000","last_change":"2024-03-12T03:36:58.116826+0000","last_active":"2024-03-12T03:37:00.768132+0000","last_peered":"2024-03-12T03:37:00.768132+0000","last_clean":"2024-03-12T03:37:00.768132+0000","last_became_active":"2024-03-12T03:36:49.303476+0000","last_became_peered":"2024-03-12T03:36:49.303476+0000","last_unstale":"2024-03-12T03:37:00.768132+0000","last_undegraded":"2024-03-12T03:37:00.768132+0000","last_fullsized":"2024-03-12T03:37:00.768132+0000","mapping_epoch":97,"log_start":"101'1800","ondisk_log_start":"101'1800","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'6867","last_scrub_stamp":"2024-03-12T03:36:58.116574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:36:58.116574+0000","objects_scrubbed":3562,"log_size":5137,"log_dups_size":0,"ondisk_log_size":5137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:04.811799+0000","scrub_duration":900,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239009792,"num_objects":3647,"num_object_clones":0,"num_object_copies":10941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":5440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371643+0000","last_change":"2024-03-12T03:36:20.138118+0000","last_active":"2024-03-12T03:36:55.371643+0000","last_peered":"2024-03-12T03:36:55.371643+0000","last_clean":"2024-03-12T03:36:55.371643+0000","last_became_active":"2024-03-12T03:33:27.194266+0000","last_became_peered":"2024-03-12T03:33:27.194266+0000","last_unstale":"2024-03-12T03:36:55.371643+0000","last_undegraded":"2024-03-12T03:36:55.371643+0000","last_fullsized":"2024-03-12T03:36:55.371643+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:20.138004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.138004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:34.538433+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.362186+0000","last_change":"2024-03-12T03:36:17.238474+0000","last_active":"2024-03-12T03:36:55.362186+0000","last_peered":"2024-03-12T03:36:55.362186+0000","last_clean":"2024-03-12T03:36:55.362186+0000","last_became_active":"2024-03-12T03:33:21.573251+0000","last_became_peered":"2024-03-12T03:33:21.573251+0000","last_unstale":"2024-03-12T03:36:55.362186+0000","last_undegraded":"2024-03-12T03:36:55.362186+0000","last_fullsized":"2024-03-12T03:36:55.362186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:17.238386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:17.238386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:42.056483+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":167,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.377867+0000","last_change":"2024-03-12T03:36:07.967637+0000","last_active":"2024-03-12T03:36:55.377867+0000","last_peered":"2024-03-12T03:36:55.377867+0000","last_clean":"2024-03-12T03:36:55.377867+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:36:55.377867+0000","last_undegraded":"2024-03-12T03:36:55.377867+0000","last_fullsized":"2024-03-12T03:36:55.377867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:36:07.967492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.967492+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:29.190972+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":188,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.370695+0000","last_change":"2024-03-12T03:36:06.069994+0000","last_active":"2024-03-12T03:36:55.370695+0000","last_peered":"2024-03-12T03:36:55.370695+0000","last_clean":"2024-03-12T03:36:55.370695+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:36:55.370695+0000","last_undegraded":"2024-03-12T03:36:55.370695+0000","last_fullsized":"2024-03-12T03:36:55.370695+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:06.069862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.069862+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:21.304322+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"125'6994","reported_seq":7072,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.940971+0000","last_change":"2024-03-12T03:36:49.340201+0000","last_active":"2024-03-12T03:37:00.940971+0000","last_peered":"2024-03-12T03:37:00.940971+0000","last_clean":"2024-03-12T03:37:00.940971+0000","last_became_active":"2024-03-12T03:36:49.321727+0000","last_became_peered":"2024-03-12T03:36:49.321727+0000","last_unstale":"2024-03-12T03:37:00.940971+0000","last_undegraded":"2024-03-12T03:37:00.940971+0000","last_fullsized":"2024-03-12T03:37:00.940971+0000","mapping_epoch":97,"log_start":"101'2100","ondisk_log_start":"101'2100","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:02.040654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238026752,"num_objects":3632,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3632,"num_write_kb":232448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.371133+0000","last_change":"2024-03-12T03:36:19.049860+0000","last_active":"2024-03-12T03:36:55.371133+0000","last_peered":"2024-03-12T03:36:55.371133+0000","last_clean":"2024-03-12T03:36:55.371133+0000","last_became_active":"2024-03-12T03:33:27.383249+0000","last_became_peered":"2024-03-12T03:33:27.383249+0000","last_unstale":"2024-03-12T03:36:55.371133+0000","last_undegraded":"2024-03-12T03:36:55.371133+0000","last_fullsized":"2024-03-12T03:36:55.371133+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.049740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.049740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:43.452600+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":145,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.378917+0000","last_change":"2024-03-12T03:36:11.217211+0000","last_active":"2024-03-12T03:36:55.378917+0000","last_peered":"2024-03-12T03:36:55.378917+0000","last_clean":"2024-03-12T03:36:55.378917+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:36:55.378917+0000","last_undegraded":"2024-03-12T03:36:55.378917+0000","last_fullsized":"2024-03-12T03:36:55.378917+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:11.216960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:11.216960+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.390046+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":165,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.360029+0000","last_change":"2024-03-12T03:34:47.043566+0000","last_active":"2024-03-12T03:36:55.360029+0000","last_peered":"2024-03-12T03:36:55.360029+0000","last_clean":"2024-03-12T03:36:55.360029+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:36:55.360029+0000","last_undegraded":"2024-03-12T03:36:55.360029+0000","last_fullsized":"2024-03-12T03:36:55.360029+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-03-12T03:34:47.043433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.043433+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":180,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363737+0000","last_change":"2024-03-12T03:34:25.168128+0000","last_active":"2024-03-12T03:36:55.363737+0000","last_peered":"2024-03-12T03:36:55.363737+0000","last_clean":"2024-03-12T03:36:55.363737+0000","last_became_active":"2024-03-12T03:33:25.177311+0000","last_became_peered":"2024-03-12T03:33:25.177311+0000","last_unstale":"2024-03-12T03:36:55.363737+0000","last_undegraded":"2024-03-12T03:36:55.363737+0000","last_fullsized":"2024-03-12T03:36:55.363737+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:25.168002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.168002+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"125'7019","reported_seq":7094,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.890008+0000","last_change":"2024-03-12T03:36:49.332704+0000","last_active":"2024-03-12T03:37:00.890008+0000","last_peered":"2024-03-12T03:37:00.890008+0000","last_clean":"2024-03-12T03:37:00.890008+0000","last_became_active":"2024-03-12T03:36:49.327962+0000","last_became_peered":"2024-03-12T03:36:49.327962+0000","last_unstale":"2024-03-12T03:37:00.890008+0000","last_undegraded":"2024-03-12T03:37:00.890008+0000","last_fullsized":"2024-03-12T03:37:00.890008+0000","mapping_epoch":97,"log_start":"101'2100","ondisk_log_start":"101'2100","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4919,"log_dups_size":0,"ondisk_log_size":4919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:02.040676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238813184,"num_objects":3644,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3644,"num_write_kb":233216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.359435+0000","last_change":"2024-03-12T03:34:58.085602+0000","last_active":"2024-03-12T03:36:55.359435+0000","last_peered":"2024-03-12T03:36:55.359435+0000","last_clean":"2024-03-12T03:36:55.359435+0000","last_became_active":"2024-03-12T03:33:27.187864+0000","last_became_peered":"2024-03-12T03:33:27.187864+0000","last_unstale":"2024-03-12T03:36:55.359435+0000","last_undegraded":"2024-03-12T03:36:55.359435+0000","last_fullsized":"2024-03-12T03:36:55.359435+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:58.085499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.085499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.367563+0000","last_change":"2024-03-12T03:34:47.155013+0000","last_active":"2024-03-12T03:36:55.367563+0000","last_peered":"2024-03-12T03:36:55.367563+0000","last_clean":"2024-03-12T03:36:55.367563+0000","last_became_active":"2024-03-12T03:33:21.104430+0000","last_became_peered":"2024-03-12T03:33:21.104430+0000","last_unstale":"2024-03-12T03:36:55.367563+0000","last_undegraded":"2024-03-12T03:36:55.367563+0000","last_fullsized":"2024-03-12T03:36:55.367563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.154925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.154925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":188,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363600+0000","last_change":"2024-03-12T03:36:06.083361+0000","last_active":"2024-03-12T03:36:55.363600+0000","last_peered":"2024-03-12T03:36:55.363600+0000","last_clean":"2024-03-12T03:36:55.363600+0000","last_became_active":"2024-03-12T03:33:23.551623+0000","last_became_peered":"2024-03-12T03:33:23.551623+0000","last_unstale":"2024-03-12T03:36:55.363600+0000","last_undegraded":"2024-03-12T03:36:55.363600+0000","last_fullsized":"2024-03-12T03:36:55.363600+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-03-12T03:36:06.083052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.083052+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:36.041591+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.359602+0000","last_change":"2024-03-12T03:35:00.071368+0000","last_active":"2024-03-12T03:36:55.359602+0000","last_peered":"2024-03-12T03:36:55.359602+0000","last_clean":"2024-03-12T03:36:55.359602+0000","last_became_active":"2024-03-12T03:33:25.181020+0000","last_became_peered":"2024-03-12T03:33:25.181020+0000","last_unstale":"2024-03-12T03:36:55.359602+0000","last_undegraded":"2024-03-12T03:36:55.359602+0000","last_fullsized":"2024-03-12T03:36:55.359602+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:00.071245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.071245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"125'7344","reported_seq":7414,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.793019+0000","last_change":"2024-03-12T03:35:23.785403+0000","last_active":"2024-03-12T03:37:00.793019+0000","last_peered":"2024-03-12T03:37:00.793019+0000","last_clean":"2024-03-12T03:37:00.793019+0000","last_became_active":"2024-03-12T03:35:23.785037+0000","last_became_peered":"2024-03-12T03:35:23.785037+0000","last_unstale":"2024-03-12T03:37:00.793019+0000","last_undegraded":"2024-03-12T03:37:00.793019+0000","last_fullsized":"2024-03-12T03:37:00.793019+0000","mapping_epoch":97,"log_start":"101'1677","ondisk_log_start":"101'1677","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481296384,"num_objects":7344,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7344,"num_write_kb":470016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.362078+0000","last_change":"2024-03-12T03:35:17.142169+0000","last_active":"2024-03-12T03:36:55.362078+0000","last_peered":"2024-03-12T03:36:55.362078+0000","last_clean":"2024-03-12T03:36:55.362078+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:36:55.362078+0000","last_undegraded":"2024-03-12T03:36:55.362078+0000","last_fullsized":"2024-03-12T03:36:55.362078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:17.142086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:17.142086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":180,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.361002+0000","last_change":"2024-03-12T03:35:20.138357+0000","last_active":"2024-03-12T03:36:55.361002+0000","last_peered":"2024-03-12T03:36:55.361002+0000","last_clean":"2024-03-12T03:36:55.361002+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:36:55.361002+0000","last_undegraded":"2024-03-12T03:36:55.361002+0000","last_fullsized":"2024-03-12T03:36:55.361002+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:35:20.138257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:20.138257+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363516+0000","last_change":"2024-03-12T03:35:13.030518+0000","last_active":"2024-03-12T03:36:55.363516+0000","last_peered":"2024-03-12T03:36:55.363516+0000","last_clean":"2024-03-12T03:36:55.363516+0000","last_became_active":"2024-03-12T03:33:21.104293+0000","last_became_peered":"2024-03-12T03:33:21.104293+0000","last_unstale":"2024-03-12T03:36:55.363516+0000","last_undegraded":"2024-03-12T03:36:55.363516+0000","last_fullsized":"2024-03-12T03:36:55.363516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:13.030426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:13.030426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.373897+0000","last_change":"2024-03-12T03:36:24.342489+0000","last_active":"2024-03-12T03:36:55.373897+0000","last_peered":"2024-03-12T03:36:55.373897+0000","last_clean":"2024-03-12T03:36:55.373897+0000","last_became_active":"2024-03-12T03:33:50.153240+0000","last_became_peered":"2024-03-12T03:33:50.153240+0000","last_unstale":"2024-03-12T03:36:55.373897+0000","last_undegraded":"2024-03-12T03:36:55.373897+0000","last_fullsized":"2024-03-12T03:36:55.373897+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:36:24.342356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.342356+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.628737+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"125'7332","reported_seq":7402,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.750066+0000","last_change":"2024-03-12T03:35:23.783266+0000","last_active":"2024-03-12T03:37:00.750066+0000","last_peered":"2024-03-12T03:37:00.750066+0000","last_clean":"2024-03-12T03:37:00.750066+0000","last_became_active":"2024-03-12T03:35:23.782945+0000","last_became_peered":"2024-03-12T03:35:23.782945+0000","last_unstale":"2024-03-12T03:37:00.750066+0000","last_undegraded":"2024-03-12T03:37:00.750066+0000","last_fullsized":"2024-03-12T03:37:00.750066+0000","mapping_epoch":97,"log_start":"101'1649","ondisk_log_start":"101'1649","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5683,"log_dups_size":0,"ondisk_log_size":5683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480509952,"num_objects":7332,"num_object_clones":0,"num_object_copies":21996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7332,"num_write_kb":469248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.366647+0000","last_change":"2024-03-12T03:36:13.114566+0000","last_active":"2024-03-12T03:36:55.366647+0000","last_peered":"2024-03-12T03:36:55.366647+0000","last_clean":"2024-03-12T03:36:55.366647+0000","last_became_active":"2024-03-12T03:33:27.380896+0000","last_became_peered":"2024-03-12T03:33:27.380896+0000","last_unstale":"2024-03-12T03:36:55.366647+0000","last_undegraded":"2024-03-12T03:36:55.366647+0000","last_fullsized":"2024-03-12T03:36:55.366647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:13.114417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.114417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:37.138551+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.363658+0000","last_change":"2024-03-12T03:36:19.182812+0000","last_active":"2024-03-12T03:36:55.363658+0000","last_peered":"2024-03-12T03:36:55.363658+0000","last_clean":"2024-03-12T03:36:55.363658+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:36:55.363658+0000","last_undegraded":"2024-03-12T03:36:55.363658+0000","last_fullsized":"2024-03-12T03:36:55.363658+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.182685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.182685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:25.003468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.359742+0000","last_change":"2024-03-12T03:36:13.157358+0000","last_active":"2024-03-12T03:36:55.359742+0000","last_peered":"2024-03-12T03:36:55.359742+0000","last_clean":"2024-03-12T03:36:55.359742+0000","last_became_active":"2024-03-12T03:33:21.573257+0000","last_became_peered":"2024-03-12T03:33:21.573257+0000","last_unstale":"2024-03-12T03:36:55.359742+0000","last_undegraded":"2024-03-12T03:36:55.359742+0000","last_fullsized":"2024-03-12T03:36:55.359742+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:13.157235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.157235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:31.473419+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"95'23","reported_seq":169,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.360907+0000","last_change":"2024-03-12T03:35:06.028109+0000","last_active":"2024-03-12T03:36:55.360907+0000","last_peered":"2024-03-12T03:36:55.360907+0000","last_clean":"2024-03-12T03:36:55.360907+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:36:55.360907+0000","last_undegraded":"2024-03-12T03:36:55.360907+0000","last_fullsized":"2024-03-12T03:36:55.360907+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'23","last_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:35:06.028013+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"125'7400","reported_seq":7469,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.972625+0000","last_change":"2024-03-12T03:35:23.777241+0000","last_active":"2024-03-12T03:37:00.972625+0000","last_peered":"2024-03-12T03:37:00.972625+0000","last_clean":"2024-03-12T03:37:00.972625+0000","last_became_active":"2024-03-12T03:35:23.776932+0000","last_became_peered":"2024-03-12T03:35:23.776932+0000","last_unstale":"2024-03-12T03:37:00.972625+0000","last_undegraded":"2024-03-12T03:37:00.972625+0000","last_fullsized":"2024-03-12T03:37:00.972625+0000","mapping_epoch":97,"log_start":"101'1628","ondisk_log_start":"101'1628","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484966400,"num_objects":7400,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7400,"num_write_kb":473600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.374189+0000","last_change":"2024-03-12T03:35:56.071430+0000","last_active":"2024-03-12T03:36:55.374189+0000","last_peered":"2024-03-12T03:36:55.374189+0000","last_clean":"2024-03-12T03:36:55.374189+0000","last_became_active":"2024-03-12T03:33:27.187815+0000","last_became_peered":"2024-03-12T03:33:27.187815+0000","last_unstale":"2024-03-12T03:36:55.374189+0000","last_undegraded":"2024-03-12T03:36:55.374189+0000","last_fullsized":"2024-03-12T03:36:55.374189+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:56.071324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:56.071324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:05.061549+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.377091+0000","last_change":"2024-03-12T03:36:04.907090+0000","last_active":"2024-03-12T03:36:55.377091+0000","last_peered":"2024-03-12T03:36:55.377091+0000","last_clean":"2024-03-12T03:36:55.377091+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:36:55.377091+0000","last_undegraded":"2024-03-12T03:36:55.377091+0000","last_fullsized":"2024-03-12T03:36:55.377091+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:04.906798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:04.906798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:31.071522+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.368437+0000","last_change":"2024-03-12T03:35:59.016620+0000","last_active":"2024-03-12T03:36:55.368437+0000","last_peered":"2024-03-12T03:36:55.368437+0000","last_clean":"2024-03-12T03:36:55.368437+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:36:55.368437+0000","last_undegraded":"2024-03-12T03:36:55.368437+0000","last_fullsized":"2024-03-12T03:36:55.368437+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.016489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.016489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:28.178945+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":160,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.360597+0000","last_change":"2024-03-12T03:35:59.053600+0000","last_active":"2024-03-12T03:36:55.360597+0000","last_peered":"2024-03-12T03:36:55.360597+0000","last_clean":"2024-03-12T03:36:55.360597+0000","last_became_active":"2024-03-12T03:33:23.163871+0000","last_became_peered":"2024-03-12T03:33:23.163871+0000","last_unstale":"2024-03-12T03:36:55.360597+0000","last_undegraded":"2024-03-12T03:36:55.360597+0000","last_fullsized":"2024-03-12T03:36:55.360597+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:35:59.053486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.053486+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:03.009127+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"125'7366","reported_seq":7435,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:37:00.916655+0000","last_change":"2024-03-12T03:35:23.784927+0000","last_active":"2024-03-12T03:37:00.916655+0000","last_peered":"2024-03-12T03:37:00.916655+0000","last_clean":"2024-03-12T03:37:00.916655+0000","last_became_active":"2024-03-12T03:35:23.784627+0000","last_became_peered":"2024-03-12T03:35:23.784627+0000","last_unstale":"2024-03-12T03:37:00.916655+0000","last_undegraded":"2024-03-12T03:37:00.916655+0000","last_fullsized":"2024-03-12T03:37:00.916655+0000","mapping_epoch":97,"log_start":"101'2461","ondisk_log_start":"101'2461","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4905,"log_dups_size":0,"ondisk_log_size":4905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482738176,"num_objects":7366,"num_object_clones":0,"num_object_copies":22098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7366,"num_write_kb":471424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":124,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.361000+0000","last_change":"2024-03-12T03:36:07.101180+0000","last_active":"2024-03-12T03:36:55.361000+0000","last_peered":"2024-03-12T03:36:55.361000+0000","last_clean":"2024-03-12T03:36:55.361000+0000","last_became_active":"2024-03-12T03:33:27.184405+0000","last_became_peered":"2024-03-12T03:33:27.184405+0000","last_unstale":"2024-03-12T03:36:55.361000+0000","last_undegraded":"2024-03-12T03:36:55.361000+0000","last_fullsized":"2024-03-12T03:36:55.361000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:07.101098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.101098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:34.696849+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":188,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.372786+0000","last_change":"2024-03-12T03:36:25.346270+0000","last_active":"2024-03-12T03:36:55.372786+0000","last_peered":"2024-03-12T03:36:55.372786+0000","last_clean":"2024-03-12T03:36:55.372786+0000","last_became_active":"2024-03-12T03:33:25.177613+0000","last_became_peered":"2024-03-12T03:33:25.177613+0000","last_unstale":"2024-03-12T03:36:55.372786+0000","last_undegraded":"2024-03-12T03:36:55.372786+0000","last_fullsized":"2024-03-12T03:36:55.372786+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:25.346151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.346151+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:44.762442+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.366253+0000","last_change":"2024-03-12T03:35:15.981464+0000","last_active":"2024-03-12T03:36:55.366253+0000","last_peered":"2024-03-12T03:36:55.366253+0000","last_clean":"2024-03-12T03:36:55.366253+0000","last_became_active":"2024-03-12T03:33:21.097304+0000","last_became_peered":"2024-03-12T03:33:21.097304+0000","last_unstale":"2024-03-12T03:36:55.366253+0000","last_undegraded":"2024-03-12T03:36:55.366253+0000","last_fullsized":"2024-03-12T03:36:55.366253+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.981378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.981378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":147,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-12T03:36:55.360003+0000","last_change":"2024-03-12T03:35:58.089544+0000","last_active":"2024-03-12T03:36:55.360003+0000","last_peered":"2024-03-12T03:36:55.360003+0000","last_clean":"2024-03-12T03:36:55.360003+0000","last_became_active":"2024-03-12T03:33:23.162657+0000","last_became_peered":"2024-03-12T03:33:23.162657+0000","last_unstale":"2024-03-12T03:36:55.360003+0000","last_undegraded":"2024-03-12T03:36:55.360003+0000","last_fullsized":"2024-03-12T03:36:55.360003+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-12T03:35:58.089405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.089405+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:25.792358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":7658209280,"num_objects":116855,"num_object_clones":0,"num_object_copies":350565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":62938,"num_objects_unfound":0,"num_objects_dirty":116855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113293,"num_write_kb":7250752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3150,"num_bytes_recovered":206438400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23371120640,"data_stored":23371120640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146606,"ondisk_log_size":146606,"up":78,"acting":77,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":214,"num_read_kb":209,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":630,"ondisk_log_size":630,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070513,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2507996,"kb_used_data":2400468,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91237156,"statfs":{"total":95995035648,"available":93426847744,"internally_reserved":0,"allocated":2458079232,"data_stored":2457574392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332150,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2451140,"kb_used_data":2383868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91294012,"statfs":{"total":95995035648,"available":93485068288,"internally_reserved":0,"allocated":2441080832,"data_stored":2440598605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2976552,"kb_used_data":2860704,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":90768600,"statfs":{"total":95995035648,"available":92947046400,"internally_reserved":0,"allocated":2929360896,"data_stored":2928842891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3876828,"kb_used_data":3765140,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":89868324,"statfs":{"total":95995035648,"available":92025163776,"internally_reserved":0,"allocated":3855503360,"data_stored":3854997774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3012100,"kb_used_data":2900412,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":90733052,"statfs":{"total":95995035648,"available":92910645248,"internally_reserved":0,"allocated":2970021888,"data_stored":2969540350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117060,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4299304,"kb_used_data":4177888,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":89445848,"statfs":{"total":95995035648,"available":91592548352,"internally_reserved":0,"allocated":4278157312,"data_stored":4277639004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124320199},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953537,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2380316,"kb_used_data":2304724,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91364836,"statfs":{"total":95995035648,"available":93557592064,"internally_reserved":0,"allocated":2360037376,"data_stored":2359531532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659195,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1958800,"kb_used_data":1887368,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91786352,"statfs":{"total":95995035648,"available":93989224448,"internally_reserved":0,"allocated":1932664832,"data_stored":1932170444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3871473664,"data_stored":3871473664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2950823936,"data_stored":2950823936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2992635904,"data_stored":2992635904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4307156992,"data_stored":4307156992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2369716224,"data_stored":2369716224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1944387584,"data_stored":1944387584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2454585344,"data_stored":2454585344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2480340992,"data_stored":2480340992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:37:06.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg force-recovery 7.18 2024-03-12T03:37:06.189 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:06.270 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:06.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:06.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:06.423 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:06.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:06.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:06 smithi169 ceph-mon[29344]: 4.e scrub starts 2024-03-12T03:37:06.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:06 smithi169 ceph-mon[29344]: 4.e scrub ok 2024-03-12T03:37:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:06 smithi169 ceph-mon[39949]: 4.e scrub starts 2024-03-12T03:37:06.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:06 smithi169 ceph-mon[39949]: 4.e scrub ok 2024-03-12T03:37:06.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:37:06.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:06 smithi173 ceph-mon[33544]: 4.e scrub starts 2024-03-12T03:37:06.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:06 smithi173 ceph-mon[33544]: 4.e scrub ok 2024-03-12T03:37:06.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:06.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:06.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:06.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:06.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:06.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:06.722 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:06.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:06.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:06.846 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:06.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:06.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:06.990 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:06.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:07.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:07.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:07.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:07.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:07.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:07.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:07.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:07.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:07.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:07.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:07.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:07.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:07 smithi169 ceph-mon[29344]: pgmap v138: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 190 MiB/s wr, 3.13k op/s; 62155/348645 objects misplaced (17.828%); 23 MiB/s, 372 objects/s recovering 2024-03-12T03:37:07.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:07 smithi169 ceph-mon[39949]: pgmap v138: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 190 MiB/s wr, 3.13k op/s; 62155/348645 objects misplaced (17.828%); 23 MiB/s, 372 objects/s recovering 2024-03-12T03:37:07.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:07.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:07 smithi173 ceph-mon[33544]: pgmap v138: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 190 MiB/s wr, 3.13k op/s; 62155/348645 objects misplaced (17.828%); 23 MiB/s, 372 objects/s recovering 2024-03-12T03:37:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:07.522 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:07.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:07.623 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:07.662 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:07.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:07.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:07.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:07.791 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:07.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:07.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:07.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:07.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:07.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:08.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:08.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:08.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:08.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:08.178 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:08.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:08.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:08.280 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:08.321 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:08.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:08.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:08.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:08.447 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:08.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:08.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:08.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:08.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:08.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:08.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:08.762 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:08.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:08.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:08.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:08.893 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:08.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:08.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:09.040 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:09.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:09.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:09.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:09.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:09.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:09.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:09.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:09.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:09.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:37:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:37:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[29344]: pgmap v139: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 938 B/s rd, 163 MiB/s wr, 2.93k op/s; 57963/341259 objects misplaced (16.985%); 23 MiB/s, 366 objects/s recovering 2024-03-12T03:37:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[29344]: osdmap e126: 8 total, 8 up, 8 in 2024-03-12T03:37:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[29344]: osdmap e127: 8 total, 8 up, 8 in 2024-03-12T03:37:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[29344]: 5.1 scrub starts 2024-03-12T03:37:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[29344]: 5.1 scrub ok 2024-03-12T03:37:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:37:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:37:09.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[39949]: pgmap v139: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 938 B/s rd, 163 MiB/s wr, 2.93k op/s; 57963/341259 objects misplaced (16.985%); 23 MiB/s, 366 objects/s recovering 2024-03-12T03:37:09.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[39949]: osdmap e126: 8 total, 8 up, 8 in 2024-03-12T03:37:09.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[39949]: osdmap e127: 8 total, 8 up, 8 in 2024-03-12T03:37:09.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[39949]: 5.1 scrub starts 2024-03-12T03:37:09.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:09 smithi169 ceph-mon[39949]: 5.1 scrub ok 2024-03-12T03:37:09.447 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:09.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:09.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:09 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:37:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:09 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:37:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:09 smithi173 ceph-mon[33544]: pgmap v139: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 938 B/s rd, 163 MiB/s wr, 2.93k op/s; 57963/341259 objects misplaced (16.985%); 23 MiB/s, 366 objects/s recovering 2024-03-12T03:37:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:09 smithi173 ceph-mon[33544]: osdmap e126: 8 total, 8 up, 8 in 2024-03-12T03:37:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:09 smithi173 ceph-mon[33544]: osdmap e127: 8 total, 8 up, 8 in 2024-03-12T03:37:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:09 smithi173 ceph-mon[33544]: 5.1 scrub starts 2024-03-12T03:37:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:09 smithi173 ceph-mon[33544]: 5.1 scrub ok 2024-03-12T03:37:09.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:09.577 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:09.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:09.679 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:09.722 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:09.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:09.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:09.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:09.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:09.949 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:09.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:09.984 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:09.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:10.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:10.109 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:10.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:10.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:10.249 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:10.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:10.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:10.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:10.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:10.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:10.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:10.508 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:10.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:10.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:10.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:10.650 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:10.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:10.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:10.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:10.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:10.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:10.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:10.922 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:10.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:11.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:11.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:11.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:11.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:11.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:11.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:11.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:11.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:11.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:11.328 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:11.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:11.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:11 smithi169 ceph-mon[39949]: pgmap v142: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 4.7 MiB/s wr, 566 op/s; 57963/341259 objects misplaced (16.985%); 14 MiB/s, 222 objects/s recovering 2024-03-12T03:37:11.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:11 smithi169 ceph-mon[39949]: 6.17 scrub starts 2024-03-12T03:37:11.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:11 smithi169 ceph-mon[39949]: 6.17 scrub ok 2024-03-12T03:37:11.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:11 smithi169 ceph-mon[29344]: pgmap v142: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 4.7 MiB/s wr, 566 op/s; 57963/341259 objects misplaced (16.985%); 14 MiB/s, 222 objects/s recovering 2024-03-12T03:37:11.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:11 smithi169 ceph-mon[29344]: 6.17 scrub starts 2024-03-12T03:37:11.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:11 smithi169 ceph-mon[29344]: 6.17 scrub ok 2024-03-12T03:37:11.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:11.456 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:11.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:11.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:11.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:11 smithi173 ceph-mon[33544]: pgmap v142: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 4.7 MiB/s wr, 566 op/s; 57963/341259 objects misplaced (16.985%); 14 MiB/s, 222 objects/s recovering 2024-03-12T03:37:11.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:11 smithi173 ceph-mon[33544]: 6.17 scrub starts 2024-03-12T03:37:11.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:11 smithi173 ceph-mon[33544]: 6.17 scrub ok 2024-03-12T03:37:11.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:11.604 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:11.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:11.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:11.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:11.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:11.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:11.884 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:11.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:11.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:12.008 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:12.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:12.110 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:12.144 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:12.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:12.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:37:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:37:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:37:12.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:12.285 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:12.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:12.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:12.426 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:12.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:12.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:12.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:12.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:12.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:12.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:12.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:12.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:37:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:37:12.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:12.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:12.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:12.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:12.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:12.971 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:12.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:12.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:13.073 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:13.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:13.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:13.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:13.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:13.265 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:13.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:13.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:13.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:13.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:13.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:13 smithi169 ceph-mon[29344]: pgmap v143: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 4.7 MiB/s wr, 1.05k op/s; 53049/329559 objects misplaced (16.097%); 28 MiB/s, 443 objects/s recovering 2024-03-12T03:37:13.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:13 smithi169 ceph-mon[29344]: 6.1a scrub starts 2024-03-12T03:37:13.426 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:13 smithi169 ceph-mon[29344]: 6.1a scrub ok 2024-03-12T03:37:13.426 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:13 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:13.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:13 smithi169 ceph-mon[39949]: pgmap v143: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 4.7 MiB/s wr, 1.05k op/s; 53049/329559 objects misplaced (16.097%); 28 MiB/s, 443 objects/s recovering 2024-03-12T03:37:13.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:13 smithi169 ceph-mon[39949]: 6.1a scrub starts 2024-03-12T03:37:13.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:13 smithi169 ceph-mon[39949]: 6.1a scrub ok 2024-03-12T03:37:13.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:13 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:13 smithi173 ceph-mon[33544]: pgmap v143: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 4.7 MiB/s wr, 1.05k op/s; 53049/329559 objects misplaced (16.097%); 28 MiB/s, 443 objects/s recovering 2024-03-12T03:37:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:13 smithi173 ceph-mon[33544]: 6.1a scrub starts 2024-03-12T03:37:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:13 smithi173 ceph-mon[33544]: 6.1a scrub ok 2024-03-12T03:37:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:13 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:13.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:13.543 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:13.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:13.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:13.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:13.715 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:13.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:13.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:13.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:13.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:13.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:13.916 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:37:13.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:13.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:13.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:14.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:14.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:14.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:14.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:14.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:14.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:14.242 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:37:14.242 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:37:14.242 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-03-12T03:37:14.242 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-03-12T03:37:14.242 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:37:14.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:14.344 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:14 smithi169 ceph-mon[29344]: 4.1a scrub starts 2024-03-12T03:37:14.344 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:14 smithi169 ceph-mon[29344]: 4.1a scrub ok 2024-03-12T03:37:14.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:14 smithi169 ceph-mon[39949]: 4.1a scrub starts 2024-03-12T03:37:14.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:14 smithi169 ceph-mon[39949]: 4.1a scrub ok 2024-03-12T03:37:14.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:14.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:14.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:14.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:14.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:14 smithi173 ceph-mon[33544]: 4.1a scrub starts 2024-03-12T03:37:14.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:14 smithi173 ceph-mon[33544]: 4.1a scrub ok 2024-03-12T03:37:14.504 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:14.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:14.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:14.646 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:14.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:14.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:14.785 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:14.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:14.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:14.930 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:14.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:14.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:15.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:15.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:15.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:15.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:15.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:15.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:15.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:15.238 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:37:15.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:15.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:15.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:15.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:15 smithi169 ceph-mon[39949]: pgmap v144: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 3.1 MiB/s wr, 895 op/s; 53049/329559 objects misplaced (16.097%); 22 MiB/s, 350 objects/s recovering 2024-03-12T03:37:15.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:15 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/113638337' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:37:15.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:15 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:37:15.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:15 smithi169 ceph-mon[29344]: pgmap v144: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 3.1 MiB/s wr, 895 op/s; 53049/329559 objects misplaced (16.097%); 22 MiB/s, 350 objects/s recovering 2024-03-12T03:37:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:15 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/113638337' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:37:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:15 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:37:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:15.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:15.469 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:15.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:15.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:15 smithi173 ceph-mon[33544]: pgmap v144: 158 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 3.1 MiB/s wr, 895 op/s; 53049/329559 objects misplaced (16.097%); 22 MiB/s, 350 objects/s recovering 2024-03-12T03:37:15.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:15 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/113638337' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:37:15.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:15 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:37:15.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:15.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-12T03:37:15.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:15.629 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:15.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:15.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:15.756 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:15.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:15.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:15.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:15.894 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:15.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:15.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:16.031 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:16.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:16.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:16.165 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:16.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:16.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:16.182 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:37:16.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:16.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:16.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:16.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:16.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:16.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:16.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:16 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:37:16.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:16 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:37:16.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:16 smithi173 ceph-mon[33544]: osdmap e128: 8 total, 8 up, 8 in 2024-03-12T03:37:16.547 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:16.590 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:16.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:16.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:16 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:37:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:16 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:37:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:16 smithi169 ceph-mon[29344]: osdmap e128: 8 total, 8 up, 8 in 2024-03-12T03:37:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:16 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:37:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:16 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:37:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:16 smithi169 ceph-mon[39949]: osdmap e128: 8 total, 8 up, 8 in 2024-03-12T03:37:16.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:16.721 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:16.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:16.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:16.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:16.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:16.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:16.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:16.979 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:16.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:16.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:17.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:17.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:17.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:17.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:17.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:17.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:17.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:17.387 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:17.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:17.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:17.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:17 smithi173 ceph-mon[33544]: pgmap v146: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 671 op/s; 52456/325767 objects misplaced (16.102%); 19 MiB/s, 300 objects/s recovering 2024-03-12T03:37:17.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:17 smithi173 ceph-mon[33544]: osdmap e129: 8 total, 8 up, 8 in 2024-03-12T03:37:17.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:17.533 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:17.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:17.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:17.607 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:17 smithi169 ceph-mon[29344]: pgmap v146: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 671 op/s; 52456/325767 objects misplaced (16.102%); 19 MiB/s, 300 objects/s recovering 2024-03-12T03:37:17.607 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:17 smithi169 ceph-mon[29344]: osdmap e129: 8 total, 8 up, 8 in 2024-03-12T03:37:17.607 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:17 smithi169 ceph-mon[39949]: pgmap v146: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 671 op/s; 52456/325767 objects misplaced (16.102%); 19 MiB/s, 300 objects/s recovering 2024-03-12T03:37:17.607 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:17 smithi169 ceph-mon[39949]: osdmap e129: 8 total, 8 up, 8 in 2024-03-12T03:37:17.616 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:37:17.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:17.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:17.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:17.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:17.787 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:17.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:17.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:17.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:17.932 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:17.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:18.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:18.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:18.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:18.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:18.208 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:18.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:18.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:18.335 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:18.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:18.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:18.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:18.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:18.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:18.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:18.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:18.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:18.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:18 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:37:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:18 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:37:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:18 smithi169 ceph-mon[29344]: osdmap e130: 8 total, 8 up, 8 in 2024-03-12T03:37:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:18 smithi169 ceph-mon[29344]: osdmap e131: 8 total, 8 up, 8 in 2024-03-12T03:37:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:18 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:37:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:18 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:37:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:18 smithi169 ceph-mon[39949]: osdmap e130: 8 total, 8 up, 8 in 2024-03-12T03:37:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:18 smithi169 ceph-mon[39949]: osdmap e131: 8 total, 8 up, 8 in 2024-03-12T03:37:18.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:18.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:18.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:18 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:37:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:18 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:37:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:18 smithi173 ceph-mon[33544]: osdmap e130: 8 total, 8 up, 8 in 2024-03-12T03:37:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:18 smithi173 ceph-mon[33544]: osdmap e131: 8 total, 8 up, 8 in 2024-03-12T03:37:18.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:18.881 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:18.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:18.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:19.019 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:19.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:19.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:19.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:19.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:19.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:19.255 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:19.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:19.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:19.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:19.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:19.437 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:19.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:19.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:19.578 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:19.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:19.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:19.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:37:19.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:19 smithi169 ceph-mon[39949]: pgmap v149: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 725 op/s; 47416/316509 objects misplaced (14.981%); 19 MiB/s, 302 objects/s recovering 2024-03-12T03:37:19.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:19 smithi169 ceph-mon[39949]: 5.18 deep-scrub starts 2024-03-12T03:37:19.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:19 smithi169 ceph-mon[39949]: 5.18 deep-scrub ok 2024-03-12T03:37:19.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:19 smithi169 ceph-mon[39949]: 7.b deep-scrub starts 2024-03-12T03:37:19.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:19 smithi169 ceph-mon[29344]: pgmap v149: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 725 op/s; 47416/316509 objects misplaced (14.981%); 19 MiB/s, 302 objects/s recovering 2024-03-12T03:37:19.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:19 smithi169 ceph-mon[29344]: 5.18 deep-scrub starts 2024-03-12T03:37:19.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:19 smithi169 ceph-mon[29344]: 5.18 deep-scrub ok 2024-03-12T03:37:19.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:19 smithi169 ceph-mon[29344]: 7.b deep-scrub starts 2024-03-12T03:37:19.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:19.733 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:19.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:19 smithi173 ceph-mon[33544]: pgmap v149: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 725 op/s; 47416/316509 objects misplaced (14.981%); 19 MiB/s, 302 objects/s recovering 2024-03-12T03:37:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:19 smithi173 ceph-mon[33544]: 5.18 deep-scrub starts 2024-03-12T03:37:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:19 smithi173 ceph-mon[33544]: 5.18 deep-scrub ok 2024-03-12T03:37:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:19 smithi173 ceph-mon[33544]: 7.b deep-scrub starts 2024-03-12T03:37:19.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:19.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:19.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:19.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:19.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:20.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:20.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:20.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:20.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:20.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:20.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:20.261 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:20.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:20.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:20.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:20.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:20.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:20.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:20 smithi169 ceph-mon[29344]: 3.4 scrub starts 2024-03-12T03:37:20.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:20 smithi169 ceph-mon[29344]: 3.4 scrub ok 2024-03-12T03:37:20.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:20 smithi169 ceph-mon[39949]: 3.4 scrub starts 2024-03-12T03:37:20.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:20 smithi169 ceph-mon[39949]: 3.4 scrub ok 2024-03-12T03:37:20.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:20.517 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:20.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:20.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:20.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:20.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:20 smithi173 ceph-mon[33544]: 3.4 scrub starts 2024-03-12T03:37:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:20 smithi173 ceph-mon[33544]: 3.4 scrub ok 2024-03-12T03:37:20.751 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:20.793 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:20.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:20.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:20.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:20.936 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:20.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:21.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:21.079 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:21.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:21.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:21.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:21.225 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:21.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:21.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:21.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:21.377 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:21.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:21.478 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:21.516 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:21.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:21.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:21.583 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:21 smithi169 ceph-mon[29344]: pgmap v151: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 946 op/s; 47416/316509 objects misplaced (14.981%); 25 MiB/s, 394 objects/s recovering 2024-03-12T03:37:21.584 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:21 smithi169 ceph-mon[39949]: pgmap v151: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 946 op/s; 47416/316509 objects misplaced (14.981%); 25 MiB/s, 394 objects/s recovering 2024-03-12T03:37:21.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:21.646 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:21.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:21.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:21 smithi173 ceph-mon[33544]: pgmap v151: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 946 op/s; 47416/316509 objects misplaced (14.981%); 25 MiB/s, 394 objects/s recovering 2024-03-12T03:37:21.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:21.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:21.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:21.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:21.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:21.920 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:21.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:21.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:21.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:37:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:37:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:37:22.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:22.060 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:22.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:22.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:22.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:22.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:22.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:22.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:22.338 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:22.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:22.439 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:22.500 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:22.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:22.573 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:22 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:37:22.573 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:22 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:37:22.573 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:22 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:37:22.573 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:22 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:37:22.573 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:22 smithi169 ceph-mon[39949]: osdmap e132: 8 total, 8 up, 8 in 2024-03-12T03:37:22.574 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:22 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:37:22.574 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:22 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:37:22.574 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:22 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:37:22.574 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:22 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:37:22.574 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:22 smithi169 ceph-mon[29344]: osdmap e132: 8 total, 8 up, 8 in 2024-03-12T03:37:22.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:22.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:22.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:22.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:22.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:22.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:37:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:37:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:22 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:37:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:22 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:37:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:22 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:37:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:22 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:37:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:22 smithi173 ceph-mon[33544]: osdmap e132: 8 total, 8 up, 8 in 2024-03-12T03:37:22.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:22.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:22.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:22.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:22.891 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:22.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:22.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:23.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:23.052 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:23.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:23.145 INFO:teuthology.orchestra.run.smithi169.stderr:instructing pg(s) [7.18] on osd.2 to force-recovery; 2024-03-12T03:37:23.146 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-12T03:37:23.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:23.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:23.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:23.339 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:23.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:23.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:23.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:23.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:23.592 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:23.606 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:23 smithi169 ceph-mon[29344]: pgmap v153: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 916 B/s rd, 0 B/s wr, 697 op/s; 43022/304824 objects misplaced (14.114%); 13 MiB/s, 208 objects/s recovering 2024-03-12T03:37:23.606 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:23 smithi169 ceph-mon[29344]: 5.e scrub starts 2024-03-12T03:37:23.606 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:23 smithi169 ceph-mon[29344]: 5.e scrub ok 2024-03-12T03:37:23.606 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:23 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:23.606 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:23 smithi169 ceph-mon[29344]: osdmap e133: 8 total, 8 up, 8 in 2024-03-12T03:37:23.606 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:23 smithi169 ceph-mon[39949]: pgmap v153: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 916 B/s rd, 0 B/s wr, 697 op/s; 43022/304824 objects misplaced (14.114%); 13 MiB/s, 208 objects/s recovering 2024-03-12T03:37:23.606 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:23 smithi169 ceph-mon[39949]: 5.e scrub starts 2024-03-12T03:37:23.607 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:23 smithi169 ceph-mon[39949]: 5.e scrub ok 2024-03-12T03:37:23.607 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:23 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:23.607 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:23 smithi169 ceph-mon[39949]: osdmap e133: 8 total, 8 up, 8 in 2024-03-12T03:37:23.646 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:23.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:23 smithi173 ceph-mon[33544]: pgmap v153: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 916 B/s rd, 0 B/s wr, 697 op/s; 43022/304824 objects misplaced (14.114%); 13 MiB/s, 208 objects/s recovering 2024-03-12T03:37:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:23 smithi173 ceph-mon[33544]: 5.e scrub starts 2024-03-12T03:37:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:23 smithi173 ceph-mon[33544]: 5.e scrub ok 2024-03-12T03:37:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:23 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:23 smithi173 ceph-mon[33544]: osdmap e133: 8 total, 8 up, 8 in 2024-03-12T03:37:23.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:23.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:23.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:23.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:23.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:23.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:23.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:24.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:24.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:24.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:24.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:24.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:24.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:24.187 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:24.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:24.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:24.329 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:24.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:24.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:24.473 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:24.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:24.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:24.646 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:24 smithi169 ceph-mon[29344]: from='client.24724 -' entity='client.admin' cmd=[{"prefix": "pg force-recovery", "pgid": ["7.18"], "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:37:24.646 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:24 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:37:24.646 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:24 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:37:24.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:24 smithi169 ceph-mon[39949]: from='client.24724 -' entity='client.admin' cmd=[{"prefix": "pg force-recovery", "pgid": ["7.18"], "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:37:24.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:24 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:37:24.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:24 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:37:24.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:24.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:24 smithi173 ceph-mon[33544]: from='client.24724 -' entity='client.admin' cmd=[{"prefix": "pg force-recovery", "pgid": ["7.18"], "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:37:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:24 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:37:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:24 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:37:24.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:24.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:24.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:24.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:24.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:24.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:24.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:25.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:25.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:25.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:25.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:25.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:25.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:25.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:25.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:25.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:25.301 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:25.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:25.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:25.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:25.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:25.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:25.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:25.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:25.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:25.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:25.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:25.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:25 smithi169 ceph-mon[39949]: pgmap v155: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 649 op/s; 43022/304824 objects misplaced (14.114%); 12 MiB/s, 194 objects/s recovering 2024-03-12T03:37:25.628 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:25 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:37:25.631 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:25 smithi169 ceph-mon[29344]: pgmap v155: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 649 op/s; 43022/304824 objects misplaced (14.114%); 12 MiB/s, 194 objects/s recovering 2024-03-12T03:37:25.631 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:25 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:37:25.666 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:37:25.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:25 smithi173 ceph-mon[33544]: pgmap v155: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 649 op/s; 43022/304824 objects misplaced (14.114%); 12 MiB/s, 194 objects/s recovering 2024-03-12T03:37:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:25 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:37:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:25.741 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:25.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:25.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:25.884 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:25.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:25.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:26.014 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:37:26.032 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:26.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:26.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:26.064 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:37:26.117 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:37:26.135 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:26.160 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:37:26.199 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:37:26.209 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:26.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:26.256 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:37:26.298 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:37:26.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:26.335 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:37:26.342 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:26.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:26.346 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:37:26.346 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:37:26.346 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:37:26.346 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:37:26.346 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:37:26.346 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:37:26.346 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:37:26.347 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:37:26.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:26.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:26.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:26.567 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:26 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4255050536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:37:26.567 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:26 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4255050536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:37:26.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:26.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:26.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:26.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:26.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:26 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4255050536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:37:26.748 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:26.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:26.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:26.891 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:26.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:26.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:27.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:27.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:27.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:27.186 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:27.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:27.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:27.325 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:27.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:27.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:27.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:27.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:27.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:27.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:27.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:27.611 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:27.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:27.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:27 smithi169 ceph-mon[29344]: pgmap v156: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 655 op/s; 43022/301521 objects misplaced (14.268%); 9.5 MiB/s, 152 objects/s recovering 2024-03-12T03:37:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:27 smithi169 ceph-mon[39949]: pgmap v156: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 655 op/s; 43022/301521 objects misplaced (14.268%); 9.5 MiB/s, 152 objects/s recovering 2024-03-12T03:37:27.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:27 smithi173 ceph-mon[33544]: pgmap v156: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 148 active+clean, 3 active+remapped+backfilling; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 655 op/s; 43022/301521 objects misplaced (14.268%); 9.5 MiB/s, 152 objects/s recovering 2024-03-12T03:37:27.745 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:27.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:27.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:27.891 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:27.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:27.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:28.033 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:28.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:28.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:28.135 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:28.181 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:28.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:28.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:28.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:28.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:28.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:28.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:28.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:28.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:28.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:28 smithi173 ceph-mon[33544]: 5.19 deep-scrub starts 2024-03-12T03:37:28.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:28 smithi173 ceph-mon[33544]: 5.19 deep-scrub ok 2024-03-12T03:37:28.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:28.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:28.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:28.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:28.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:28.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:28 smithi169 ceph-mon[29344]: 5.19 deep-scrub starts 2024-03-12T03:37:28.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:28 smithi169 ceph-mon[29344]: 5.19 deep-scrub ok 2024-03-12T03:37:28.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:28 smithi169 ceph-mon[39949]: 5.19 deep-scrub starts 2024-03-12T03:37:28.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:28 smithi169 ceph-mon[39949]: 5.19 deep-scrub ok 2024-03-12T03:37:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:28.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:28.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:28.802 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:28.847 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:28.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:28.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:28.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:28.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:28.975 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:28.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:29.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:29.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:29.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:29.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:29.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:29.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:29.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:29.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:29.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:29.365 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:29.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:29.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:29.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:29.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:29.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:29.624 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:29.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:29.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:29 smithi169 ceph-mon[29344]: pgmap v157: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 3 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 982 op/s; 38334/292944 objects misplaced (13.086%); 21 MiB/s, 343 objects/s recovering 2024-03-12T03:37:29.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:29 smithi169 ceph-mon[29344]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:37:29.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:29 smithi169 ceph-mon[39949]: pgmap v157: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 3 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 982 op/s; 38334/292944 objects misplaced (13.086%); 21 MiB/s, 343 objects/s recovering 2024-03-12T03:37:29.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:29 smithi169 ceph-mon[39949]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:37:29.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:29.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:29 smithi173 ceph-mon[33544]: pgmap v157: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 3 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 982 op/s; 38334/292944 objects misplaced (13.086%); 21 MiB/s, 343 objects/s recovering 2024-03-12T03:37:29.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:29 smithi173 ceph-mon[33544]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:37:29.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:29.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:29.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:29.884 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:29.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:29.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:30.026 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:30.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:30.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:30.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:30.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:30.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:30.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:30.293 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:30.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:30.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:30.437 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:30.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:30.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:30.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:30.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:30.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:30.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:30.691 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:37:30.691 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:37:30.691 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-12T03:37:30.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:37:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:30.709 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:30.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:30.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:30.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:30.856 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:30.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:30.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:31.013 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:31.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:31.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:31.158 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:31.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:31.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:31.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:31.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:31.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:31.334 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:31 smithi173 ceph-mon[33544]: pgmap v158: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 3 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 601 B/s rd, 0 B/s wr, 465 op/s; 38334/292944 objects misplaced (13.086%); 12 MiB/s, 185 objects/s recovering 2024-03-12T03:37:31.334 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:31 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:31.335 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:31.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:31.417 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:37:31.417 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:37:31.420 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:31 smithi169 ceph-mon[29344]: pgmap v158: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 3 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 601 B/s rd, 0 B/s wr, 465 op/s; 38334/292944 objects misplaced (13.086%); 12 MiB/s, 185 objects/s recovering 2024-03-12T03:37:31.420 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:31 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:31.420 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:31.420 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:31 smithi169 ceph-mon[39949]: pgmap v158: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 3 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 601 B/s rd, 0 B/s wr, 465 op/s; 38334/292944 objects misplaced (13.086%); 12 MiB/s, 185 objects/s recovering 2024-03-12T03:37:31.420 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:31 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:31.421 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:31.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:31.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:31.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:31.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:31.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:31.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:31.661 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:31.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:31.699 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:31.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:31.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:31.824 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:31.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:31.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:31.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:31.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:31.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:32.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:32.096 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:32.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:32.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:37:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:37:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:37:32.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:32.227 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:32.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:32.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:32.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:32.367 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:32.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:32.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:32.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:32.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:32.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:32.644 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:32.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:32.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:37:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:37:32.745 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:32.784 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:32.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:32.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:32.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:32.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:32.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:33.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:33.076 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:33.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:33.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:33.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:33.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:33.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:33.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:33.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:33.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:33.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-12T03:37:33.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:33.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:33.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:33.603 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:33.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:33.659 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:33.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:33.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:33 smithi169 ceph-mon[29344]: pgmap v159: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_wait+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 757 op/s; 34096/282117 objects misplaced (12.086%); 20 MiB/s, 319 objects/s recovering 2024-03-12T03:37:33.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:33 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:33.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:33 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:33.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:33 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:33.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:33 smithi169 ceph-mon[39949]: pgmap v159: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_wait+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 757 op/s; 34096/282117 objects misplaced (12.086%); 20 MiB/s, 319 objects/s recovering 2024-03-12T03:37:33.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:33 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:33.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:33 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:33.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:33 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:33 smithi173 ceph-mon[33544]: pgmap v159: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_wait+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 757 op/s; 34096/282117 objects misplaced (12.086%); 20 MiB/s, 319 objects/s recovering 2024-03-12T03:37:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:33 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:33 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:33 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:33.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:33.764 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:37:33.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:33.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:33.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:33.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:33.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:34.006 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:37:34.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:34.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:34.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:34.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:34.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:34.204 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:34.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:34.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:34.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:34.394 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:34.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:34.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:34.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:34.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:34.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:34.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:34.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:34.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:34.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:34.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:34.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:34.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:34.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:34.924 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:34.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:35.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:35.033 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:37:35.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:35.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:35.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:35.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:35.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:35.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:35.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:35.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:35.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:35.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:35.440 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:35.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:35.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:35.508 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:37:35.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:35.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:35.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:35.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:35.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:35 smithi173 ceph-mon[33544]: pgmap v160: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_wait+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 962 B/s rd, 0 B/s wr, 712 op/s; 34096/282117 objects misplaced (12.086%); 19 MiB/s, 300 objects/s recovering 2024-03-12T03:37:35.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:35 smithi173 ceph-mon[33544]: osdmap e134: 8 total, 8 up, 8 in 2024-03-12T03:37:35.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:35 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1136965607' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:37:35.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:35 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:37:35.718 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:35.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:35.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:35.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:35.871 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:37:35.871 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:37:35.881 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:35.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:35 smithi169 ceph-mon[29344]: pgmap v160: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_wait+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 962 B/s rd, 0 B/s wr, 712 op/s; 34096/282117 objects misplaced (12.086%); 19 MiB/s, 300 objects/s recovering 2024-03-12T03:37:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:35 smithi169 ceph-mon[29344]: osdmap e134: 8 total, 8 up, 8 in 2024-03-12T03:37:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:35 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1136965607' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:37:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:35 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:37:35.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:35 smithi169 ceph-mon[39949]: pgmap v160: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_wait+backfill_toofull, 148 active+clean, 4 active+remapped+backfilling; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 962 B/s rd, 0 B/s wr, 712 op/s; 34096/282117 objects misplaced (12.086%); 19 MiB/s, 300 objects/s recovering 2024-03-12T03:37:35.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:35 smithi169 ceph-mon[39949]: osdmap e134: 8 total, 8 up, 8 in 2024-03-12T03:37:35.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:35 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1136965607' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:37:35.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:35 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:37:35.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:36.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:36.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:36.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:36.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:36.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:36.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:36.281 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:36.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:36.383 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:36.435 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:36.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:36.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:36.570 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:36.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:36.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:36.580 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:36 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:37:36.580 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:36 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:37:36.580 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:36 smithi169 ceph-mon[29344]: osdmap e135: 8 total, 8 up, 8 in 2024-03-12T03:37:36.581 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:36 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:37:36.581 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:36 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:37:36.581 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:36 smithi169 ceph-mon[39949]: osdmap e135: 8 total, 8 up, 8 in 2024-03-12T03:37:36.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:36.700 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:36 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:37:36.700 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:36 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:37:36.700 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:36 smithi173 ceph-mon[33544]: osdmap e135: 8 total, 8 up, 8 in 2024-03-12T03:37:36.714 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:36.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:36.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-12T03:37:36.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:36.891 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:36.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:36.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:37.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:37.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:37.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:37.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:37.142 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:37:37.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:37.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:37.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:37.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:37.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:37.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:37.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:37.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:37.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:37.536 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:37 smithi173 ceph-mon[33544]: pgmap v163: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_wait+backfill_toofull, 149 active+clean, 3 active+remapped+backfilling; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 587 op/s; 33307/278877 objects misplaced (11.943%); 15 MiB/s, 235 objects/s recovering 2024-03-12T03:37:37.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:37.572 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:37.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:37.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:37.704 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:37:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:37:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:37.728 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:37.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:37.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:37.861 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:37.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:37.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:37 smithi169 ceph-mon[29344]: pgmap v163: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_wait+backfill_toofull, 149 active+clean, 3 active+remapped+backfilling; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 587 op/s; 33307/278877 objects misplaced (11.943%); 15 MiB/s, 235 objects/s recovering 2024-03-12T03:37:37.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:37 smithi169 ceph-mon[39949]: pgmap v163: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_wait+backfill_toofull, 149 active+clean, 3 active+remapped+backfilling; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 587 op/s; 33307/278877 objects misplaced (11.943%); 15 MiB/s, 235 objects/s recovering 2024-03-12T03:37:37.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:38.007 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:38.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:38.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:38.145 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:38.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:38.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:38.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:38.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:38.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:37:38.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:38.408 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:38.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:38.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:38.576 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:38.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:38.614 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:38 smithi173 ceph-mon[33544]: from='client.24773 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:37:38.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:38.702 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:38.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:38.804 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:38.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:38.844 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:38.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:38.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:38 smithi169 ceph-mon[29344]: from='client.24773 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:37:38.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:38 smithi169 ceph-mon[39949]: from='client.24773 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:37:38.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:38.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:38.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:39.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:39.103 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:37:39.103 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:37:39.109 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:39.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:39.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:39.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:39.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:39.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:39.385 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:39.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:39.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:39.528 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:39.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:39.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:39.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:39.673 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:39.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:39.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:39 smithi173 ceph-mon[33544]: pgmap v164: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 149 active+clean, 5 active+remapped+backfilling; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 882 op/s; 29228/271785 objects misplaced (10.754%); 29 MiB/s, 459 objects/s recovering 2024-03-12T03:37:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:39 smithi173 ceph-mon[33544]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:37:39.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:39.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:39.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:39.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:39.908 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:39.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:39 smithi169 ceph-mon[29344]: pgmap v164: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 149 active+clean, 5 active+remapped+backfilling; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 882 op/s; 29228/271785 objects misplaced (10.754%); 29 MiB/s, 459 objects/s recovering 2024-03-12T03:37:39.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:39 smithi169 ceph-mon[29344]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:37:39.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:39 smithi169 ceph-mon[39949]: pgmap v164: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 149 active+clean, 5 active+remapped+backfilling; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 882 op/s; 29228/271785 objects misplaced (10.754%); 29 MiB/s, 459 objects/s recovering 2024-03-12T03:37:39.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:39 smithi169 ceph-mon[39949]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:37:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:39.933 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:39.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:40.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:40.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:40.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:40.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:40.130 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:37:40.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:40.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:40.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:40.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-12T03:37:40.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:40.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:40.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:40.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:40.488 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:40.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:40.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:40.598 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:37:40.638 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:40.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:40.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:40.776 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:40.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:40.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:40.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:40.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:41.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:41.046 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:41.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:41.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:41.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:41.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:41.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:41.424 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:41.465 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:41.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:41.566 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:41.603 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:41.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:41.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:41.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:41.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:41.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:41.827 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:41 smithi169 ceph-mon[29344]: pgmap v165: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 149 active+clean, 5 active+remapped+backfilling; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 430 op/s; 29228/271785 objects misplaced (10.754%); 16 MiB/s, 257 objects/s recovering 2024-03-12T03:37:41.827 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:41 smithi169 ceph-mon[29344]: osdmap e136: 8 total, 8 up, 8 in 2024-03-12T03:37:41.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:41 smithi169 ceph-mon[39949]: pgmap v165: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 149 active+clean, 5 active+remapped+backfilling; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 430 op/s; 29228/271785 objects misplaced (10.754%); 16 MiB/s, 257 objects/s recovering 2024-03-12T03:37:41.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:41 smithi169 ceph-mon[39949]: osdmap e136: 8 total, 8 up, 8 in 2024-03-12T03:37:41.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:41.851 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:41 smithi173 ceph-mon[33544]: pgmap v165: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 149 active+clean, 5 active+remapped+backfilling; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 430 op/s; 29228/271785 objects misplaced (10.754%); 16 MiB/s, 257 objects/s recovering 2024-03-12T03:37:41.851 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:41 smithi173 ceph-mon[33544]: osdmap e136: 8 total, 8 up, 8 in 2024-03-12T03:37:41.877 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:41.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:41.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:41.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:42.001 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:42.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:42.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:42.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:42.151 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:42.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:42.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:37:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:37:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:37:42.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:42.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:42.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:42.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:42.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:42.433 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:42.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:42.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:42.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:37:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:37:42.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:42.545 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:37:42.545 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:37:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:42.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:42.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:42.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:42.712 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:42.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:42.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:42 smithi173 ceph-mon[33544]: osdmap e137: 8 total, 8 up, 8 in 2024-03-12T03:37:42.814 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:42.840 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:42.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:42.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:42 smithi169 ceph-mon[29344]: osdmap e137: 8 total, 8 up, 8 in 2024-03-12T03:37:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:42 smithi169 ceph-mon[39949]: osdmap e137: 8 total, 8 up, 8 in 2024-03-12T03:37:42.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:42.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:42.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:43.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:43.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:43.113 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:43.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:43.214 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:43.260 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:43.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:43.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:43.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-12T03:37:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:43.386 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:43.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:43.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:43.532 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:43.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:43.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:43.672 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:43.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:43.702 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:37:43.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:43.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:43.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:43.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:43 smithi169 ceph-mon[29344]: pgmap v168: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 149 active+clean, 5 active+remapped+backfilling; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.09k op/s; 24762/261429 objects misplaced (9.472%); 41 MiB/s, 659 objects/s recovering 2024-03-12T03:37:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:43 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:43.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:43 smithi169 ceph-mon[39949]: pgmap v168: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 149 active+clean, 5 active+remapped+backfilling; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.09k op/s; 24762/261429 objects misplaced (9.472%); 41 MiB/s, 659 objects/s recovering 2024-03-12T03:37:43.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:43 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:43.936 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:43.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:43.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:43.938 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:43 smithi173 ceph-mon[33544]: pgmap v168: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 149 active+clean, 5 active+remapped+backfilling; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.09k op/s; 24762/261429 objects misplaced (9.472%); 41 MiB/s, 659 objects/s recovering 2024-03-12T03:37:43.938 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:43 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:44.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:44.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:44.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:44.171 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:44.243 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:44.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:44.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:44.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:44.375 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:44.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:44.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:44.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:44.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:44.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:44.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:44.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:44.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:44.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:44.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:44.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:44.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:44.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:44.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:44.909 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:44.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:44.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:44 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:37:44.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:44 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:37:44.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:44 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:37:44.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:44 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:37:44.941 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:44 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:37:44.941 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:44 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:37:45.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:45.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:45.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:45.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:45.166 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:45.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:45.247 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-03-12T03:37:35.829262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6077618624,"num_objects":92959,"num_object_clones":0,"num_object_copies":278877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33307,"num_objects_unfound":0,"num_objects_dirty":92959,"num_whiteouts":0,"num_read":1472,"num_read_kb":1286,"num_write":139420,"num_write_kb":7291358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13167,"num_bytes_recovered":862914722,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163436,"ondisk_log_size":163436,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20116484,"kb_used_data":19111736,"kb_used_omap":12,"kb_used_meta":1004531,"kb_avail":729844732,"statfs":{"total":767960285184,"available":747361005568,"internally_reserved":0,"allocated":19570417664,"data_stored":19566045892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1028640304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-307298304,"num_objects":-4689,"num_object_clones":0,"num_object_copies":-14067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-5027,"num_objects_unfound":0,"num_objects_dirty":-4689,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4689,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1882,"num_bytes_recovered":123338752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002560"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":152,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.495773+0000","last_change":"2024-03-12T03:37:12.106676+0000","last_active":"2024-03-12T03:37:35.495773+0000","last_peered":"2024-03-12T03:37:35.495773+0000","last_clean":"2024-03-12T03:37:35.495773+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:37:35.495773+0000","last_undegraded":"2024-03-12T03:37:35.495773+0000","last_fullsized":"2024-03-12T03:37:35.495773+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:12.106550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:37:12.106550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:28.196420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":160,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:26.625427+0000","last_change":"2024-03-12T03:37:26.629588+0000","last_active":"2024-03-12T03:37:26.625427+0000","last_peered":"2024-03-12T03:37:26.625427+0000","last_clean":"2024-03-12T03:37:26.625427+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:37:26.625427+0000","last_undegraded":"2024-03-12T03:37:26.625427+0000","last_fullsized":"2024-03-12T03:37:26.625427+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:26.625326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:26.625326+0000","last_clean_scrub_stamp":"2024-03-12T03:37:26.625326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:41.473318+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":161,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.211921+0000","last_change":"2024-03-12T03:36:28.208156+0000","last_active":"2024-03-12T03:37:23.211921+0000","last_peered":"2024-03-12T03:37:23.211921+0000","last_clean":"2024-03-12T03:37:23.211921+0000","last_became_active":"2024-03-12T03:33:21.573080+0000","last_became_peered":"2024-03-12T03:33:21.573080+0000","last_unstale":"2024-03-12T03:37:23.211921+0000","last_undegraded":"2024-03-12T03:37:23.211921+0000","last_fullsized":"2024-03-12T03:37:23.211921+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:28.208044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:28.208044+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.208044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:40.375427+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":180,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.495507+0000","last_change":"2024-03-12T03:36:02.990432+0000","last_active":"2024-03-12T03:37:35.495507+0000","last_peered":"2024-03-12T03:37:35.495507+0000","last_clean":"2024-03-12T03:37:35.495507+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:37:35.495507+0000","last_undegraded":"2024-03-12T03:37:35.495507+0000","last_fullsized":"2024-03-12T03:37:35.495507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:36:02.990324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:02.990324+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":140,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.210009+0000","last_change":"2024-03-12T03:36:19.237362+0000","last_active":"2024-03-12T03:37:23.210009+0000","last_peered":"2024-03-12T03:37:23.210009+0000","last_clean":"2024-03-12T03:37:23.210009+0000","last_became_active":"2024-03-12T03:33:27.382050+0000","last_became_peered":"2024-03-12T03:33:27.382050+0000","last_unstale":"2024-03-12T03:37:23.210009+0000","last_undegraded":"2024-03-12T03:37:23.210009+0000","last_fullsized":"2024-03-12T03:37:23.210009+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.236997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.236997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:39.350923+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":156,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.495600+0000","last_change":"2024-03-12T03:37:18.123502+0000","last_active":"2024-03-12T03:37:35.495600+0000","last_peered":"2024-03-12T03:37:35.495600+0000","last_clean":"2024-03-12T03:37:35.495600+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:37:35.495600+0000","last_undegraded":"2024-03-12T03:37:35.495600+0000","last_fullsized":"2024-03-12T03:37:35.495600+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:37:18.123378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:35.084799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":153,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.213408+0000","last_change":"2024-03-12T03:34:41.143075+0000","last_active":"2024-03-12T03:37:23.213408+0000","last_peered":"2024-03-12T03:37:23.213408+0000","last_clean":"2024-03-12T03:37:23.213408+0000","last_became_active":"2024-03-12T03:33:21.572876+0000","last_became_peered":"2024-03-12T03:33:21.572876+0000","last_unstale":"2024-03-12T03:37:23.213408+0000","last_undegraded":"2024-03-12T03:37:23.213408+0000","last_fullsized":"2024-03-12T03:37:23.213408+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.142994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"94'50","reported_seq":232,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.210633+0000","last_change":"2024-03-12T03:35:05.042142+0000","last_active":"2024-03-12T03:37:23.210633+0000","last_peered":"2024-03-12T03:37:23.210633+0000","last_clean":"2024-03-12T03:37:23.210633+0000","last_became_active":"2024-03-12T03:33:23.161549+0000","last_became_peered":"2024-03-12T03:33:23.161549+0000","last_unstale":"2024-03-12T03:37:23.210633+0000","last_undegraded":"2024-03-12T03:37:23.210633+0000","last_fullsized":"2024-03-12T03:37:23.210633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.042030+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"135'7886","reported_seq":21248,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.759571+0000","last_change":"2024-03-12T03:37:35.528266+0000","last_active":"2024-03-12T03:37:35.759571+0000","last_peered":"2024-03-12T03:37:35.759571+0000","last_clean":"2024-03-12T03:37:35.759571+0000","last_became_active":"2024-03-12T03:37:35.518331+0000","last_became_peered":"2024-03-12T03:37:35.518331+0000","last_unstale":"2024-03-12T03:37:35.759571+0000","last_undegraded":"2024-03-12T03:37:35.759571+0000","last_fullsized":"2024-03-12T03:37:35.759571+0000","mapping_epoch":134,"log_start":"101'1100","ondisk_log_start":"101'1100","created":119,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6786,"log_dups_size":0,"ondisk_log_size":6786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186351616,"num_objects":2844,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4510,"num_write_kb":235296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3313,"num_bytes_recovered":217120768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":148,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.196529+0000","last_change":"2024-03-12T03:36:26.107553+0000","last_active":"2024-03-12T03:37:23.196529+0000","last_peered":"2024-03-12T03:37:23.196529+0000","last_clean":"2024-03-12T03:37:23.196529+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:37:23.196529+0000","last_undegraded":"2024-03-12T03:37:23.196529+0000","last_fullsized":"2024-03-12T03:37:23.196529+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.107439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.107439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:55.895456+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":156,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.498482+0000","last_change":"2024-03-12T03:36:16.187277+0000","last_active":"2024-03-12T03:37:35.498482+0000","last_peered":"2024-03-12T03:37:35.498482+0000","last_clean":"2024-03-12T03:37:35.498482+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:37:35.498482+0000","last_undegraded":"2024-03-12T03:37:35.498482+0000","last_fullsized":"2024-03-12T03:37:35.498482+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:16.187145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:16.187145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:45.580351+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":164,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.498910+0000","last_change":"2024-03-12T03:36:21.101794+0000","last_active":"2024-03-12T03:37:35.498910+0000","last_peered":"2024-03-12T03:37:35.498910+0000","last_clean":"2024-03-12T03:37:35.498910+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:37:35.498910+0000","last_undegraded":"2024-03-12T03:37:35.498910+0000","last_fullsized":"2024-03-12T03:37:35.498910+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:21.101693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:36:21.101693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:43.795511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'21","reported_seq":200,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.495888+0000","last_change":"2024-03-12T03:37:13.158914+0000","last_active":"2024-03-12T03:37:35.495888+0000","last_peered":"2024-03-12T03:37:35.495888+0000","last_clean":"2024-03-12T03:37:35.495888+0000","last_became_active":"2024-03-12T03:33:23.162749+0000","last_became_peered":"2024-03-12T03:33:23.162749+0000","last_unstale":"2024-03-12T03:37:35.495888+0000","last_undegraded":"2024-03-12T03:37:35.495888+0000","last_fullsized":"2024-03-12T03:37:35.495888+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:37:13.158814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:37:13.158814+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:37.959586+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"133'7699","reported_seq":10306,"reported_epoch":133,"state":"active+recovering+forced_recovery+undersized+remapped","last_fresh":"2024-03-12T03:37:31.052771+0000","last_change":"2024-03-12T03:37:23.049132+0000","last_active":"2024-03-12T03:37:31.052771+0000","last_peered":"2024-03-12T03:37:31.052771+0000","last_clean":"2024-03-12T03:36:50.232406+0000","last_became_active":"2024-03-12T03:36:52.548937+0000","last_became_peered":"2024-03-12T03:36:52.548937+0000","last_unstale":"2024-03-12T03:37:31.052771+0000","last_undegraded":"2024-03-12T03:37:31.052771+0000","last_fullsized":"2024-03-12T03:36:52.286167+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":7699,"log_dups_size":0,"ondisk_log_size":7699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191627264,"num_objects":2924,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5194,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4340,"num_write_kb":232448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":837,"num_bytes_recovered":54853632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[2,7],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":2597},{"shards":"2,7","objects":327}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":144,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.498027+0000","last_change":"2024-03-12T03:35:05.181569+0000","last_active":"2024-03-12T03:37:35.498027+0000","last_peered":"2024-03-12T03:37:35.498027+0000","last_clean":"2024-03-12T03:37:35.498027+0000","last_became_active":"2024-03-12T03:33:27.187943+0000","last_became_peered":"2024-03-12T03:33:27.187943+0000","last_unstale":"2024-03-12T03:37:35.498027+0000","last_undegraded":"2024-03-12T03:37:35.498027+0000","last_fullsized":"2024-03-12T03:37:35.498027+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:05.181454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.181454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":148,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.500662+0000","last_change":"2024-03-12T03:35:11.909632+0000","last_active":"2024-03-12T03:37:35.500662+0000","last_peered":"2024-03-12T03:37:35.500662+0000","last_clean":"2024-03-12T03:37:35.500662+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:37:35.500662+0000","last_undegraded":"2024-03-12T03:37:35.500662+0000","last_fullsized":"2024-03-12T03:37:35.500662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:11.909537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.909537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":164,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499769+0000","last_change":"2024-03-12T03:36:18.202204+0000","last_active":"2024-03-12T03:37:35.499769+0000","last_peered":"2024-03-12T03:37:35.499769+0000","last_clean":"2024-03-12T03:37:35.499769+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:37:35.499769+0000","last_undegraded":"2024-03-12T03:37:35.499769+0000","last_fullsized":"2024-03-12T03:37:35.499769+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.202082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.202082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"86'14","reported_seq":199,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.496292+0000","last_change":"2024-03-12T03:36:15.113589+0000","last_active":"2024-03-12T03:37:35.496292+0000","last_peered":"2024-03-12T03:37:35.496292+0000","last_clean":"2024-03-12T03:37:35.496292+0000","last_became_active":"2024-03-12T03:33:50.155908+0000","last_became_peered":"2024-03-12T03:33:50.155908+0000","last_unstale":"2024-03-12T03:37:35.496292+0000","last_undegraded":"2024-03-12T03:37:35.496292+0000","last_fullsized":"2024-03-12T03:37:35.496292+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:36:15.113450+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.113450+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":145,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.496131+0000","last_change":"2024-03-12T03:36:01.082667+0000","last_active":"2024-03-12T03:37:35.496131+0000","last_peered":"2024-03-12T03:37:35.496131+0000","last_clean":"2024-03-12T03:37:35.496131+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:37:35.496131+0000","last_undegraded":"2024-03-12T03:37:35.496131+0000","last_fullsized":"2024-03-12T03:37:35.496131+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:01.082531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:01.082531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":151,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.217412+0000","last_change":"2024-03-12T03:36:09.003751+0000","last_active":"2024-03-12T03:37:23.217412+0000","last_peered":"2024-03-12T03:37:23.217412+0000","last_clean":"2024-03-12T03:37:23.217412+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:37:23.217412+0000","last_undegraded":"2024-03-12T03:37:23.217412+0000","last_fullsized":"2024-03-12T03:37:23.217412+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:09.003476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:09.003476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":160,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.204396+0000","last_change":"2024-03-12T03:36:08.094593+0000","last_active":"2024-03-12T03:37:23.204396+0000","last_peered":"2024-03-12T03:37:23.204396+0000","last_clean":"2024-03-12T03:37:23.204396+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:37:23.204396+0000","last_undegraded":"2024-03-12T03:37:23.204396+0000","last_fullsized":"2024-03-12T03:37:23.204396+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:08.094475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:08.094475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":173,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.197802+0000","last_change":"2024-03-12T03:34:58.889062+0000","last_active":"2024-03-12T03:37:23.197802+0000","last_peered":"2024-03-12T03:37:23.197802+0000","last_clean":"2024-03-12T03:37:23.197802+0000","last_became_active":"2024-03-12T03:33:23.160702+0000","last_became_peered":"2024-03-12T03:33:23.160702+0000","last_unstale":"2024-03-12T03:37:23.197802+0000","last_undegraded":"2024-03-12T03:37:23.197802+0000","last_fullsized":"2024-03-12T03:37:23.197802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:34:58.888971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.888971+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":132,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.216186+0000","last_change":"2024-03-12T03:33:27.381440+0000","last_active":"2024-03-12T03:37:23.216186+0000","last_peered":"2024-03-12T03:37:23.216186+0000","last_clean":"2024-03-12T03:37:23.216186+0000","last_became_active":"2024-03-12T03:33:27.380880+0000","last_became_peered":"2024-03-12T03:33:27.380880+0000","last_unstale":"2024-03-12T03:37:23.216186+0000","last_undegraded":"2024-03-12T03:37:23.216186+0000","last_fullsized":"2024-03-12T03:37:23.216186+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":156,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.496393+0000","last_change":"2024-03-12T03:36:20.142515+0000","last_active":"2024-03-12T03:37:35.496393+0000","last_peered":"2024-03-12T03:37:35.496393+0000","last_clean":"2024-03-12T03:37:35.496393+0000","last_became_active":"2024-03-12T03:33:25.181087+0000","last_became_peered":"2024-03-12T03:33:25.181087+0000","last_unstale":"2024-03-12T03:37:35.496393+0000","last_undegraded":"2024-03-12T03:37:35.496393+0000","last_fullsized":"2024-03-12T03:37:35.496393+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:20.142437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.142437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:49.273040+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":164,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.495993+0000","last_change":"2024-03-12T03:36:18.123144+0000","last_active":"2024-03-12T03:37:35.495993+0000","last_peered":"2024-03-12T03:37:35.495993+0000","last_clean":"2024-03-12T03:37:35.495993+0000","last_became_active":"2024-03-12T03:33:21.573218+0000","last_became_peered":"2024-03-12T03:33:21.573218+0000","last_unstale":"2024-03-12T03:37:35.495993+0000","last_undegraded":"2024-03-12T03:37:35.495993+0000","last_fullsized":"2024-03-12T03:37:35.495993+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.123035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.123035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":183,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.213120+0000","last_change":"2024-03-12T03:36:28.133412+0000","last_active":"2024-03-12T03:37:23.213120+0000","last_peered":"2024-03-12T03:37:23.213120+0000","last_clean":"2024-03-12T03:37:23.213120+0000","last_became_active":"2024-03-12T03:33:23.162427+0000","last_became_peered":"2024-03-12T03:33:23.162427+0000","last_unstale":"2024-03-12T03:37:23.213120+0000","last_undegraded":"2024-03-12T03:37:23.213120+0000","last_fullsized":"2024-03-12T03:37:23.213120+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:36:28.133261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.133261+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:51.305356+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":153,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.501046+0000","last_change":"2024-03-12T03:35:57.051016+0000","last_active":"2024-03-12T03:37:35.501046+0000","last_peered":"2024-03-12T03:37:35.501046+0000","last_clean":"2024-03-12T03:37:35.501046+0000","last_became_active":"2024-03-12T03:33:27.189152+0000","last_became_peered":"2024-03-12T03:33:27.189152+0000","last_unstale":"2024-03-12T03:37:35.501046+0000","last_undegraded":"2024-03-12T03:37:35.501046+0000","last_fullsized":"2024-03-12T03:37:35.501046+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:35:57.050605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:57.050605+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":218,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499695+0000","last_change":"2024-03-12T03:36:00.063359+0000","last_active":"2024-03-12T03:37:35.499695+0000","last_peered":"2024-03-12T03:37:35.499695+0000","last_clean":"2024-03-12T03:37:35.499695+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:37:35.499695+0000","last_undegraded":"2024-03-12T03:37:35.499695+0000","last_fullsized":"2024-03-12T03:37:35.499695+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:00.063228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:00.063228+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":161,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.202937+0000","last_change":"2024-03-12T03:36:16.066119+0000","last_active":"2024-03-12T03:37:23.202937+0000","last_peered":"2024-03-12T03:37:23.202937+0000","last_clean":"2024-03-12T03:37:23.202937+0000","last_became_active":"2024-03-12T03:33:21.097218+0000","last_became_peered":"2024-03-12T03:33:21.097218+0000","last_unstale":"2024-03-12T03:37:23.202937+0000","last_undegraded":"2024-03-12T03:37:23.202937+0000","last_fullsized":"2024-03-12T03:37:23.202937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:16.066002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:16.066002+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:39.793679+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'17","reported_seq":185,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.204473+0000","last_change":"2024-03-12T03:36:12.216521+0000","last_active":"2024-03-12T03:37:23.204473+0000","last_peered":"2024-03-12T03:37:23.204473+0000","last_clean":"2024-03-12T03:37:23.204473+0000","last_became_active":"2024-03-12T03:33:23.552812+0000","last_became_peered":"2024-03-12T03:33:23.552812+0000","last_unstale":"2024-03-12T03:37:23.204473+0000","last_undegraded":"2024-03-12T03:37:23.204473+0000","last_fullsized":"2024-03-12T03:37:23.204473+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:36:12.216269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:12.216269+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:27.130146+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":147,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.217600+0000","last_change":"2024-03-12T03:36:22.163176+0000","last_active":"2024-03-12T03:37:23.217600+0000","last_peered":"2024-03-12T03:37:23.217600+0000","last_clean":"2024-03-12T03:37:23.217600+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:37:23.217600+0000","last_undegraded":"2024-03-12T03:37:23.217600+0000","last_fullsized":"2024-03-12T03:37:23.217600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:22.163050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:22.163050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:30.403724+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":153,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.204814+0000","last_change":"2024-03-12T03:36:24.163956+0000","last_active":"2024-03-12T03:37:23.204814+0000","last_peered":"2024-03-12T03:37:23.204814+0000","last_clean":"2024-03-12T03:37:23.204814+0000","last_became_active":"2024-03-12T03:33:25.178346+0000","last_became_peered":"2024-03-12T03:33:25.178346+0000","last_unstale":"2024-03-12T03:37:23.204814+0000","last_undegraded":"2024-03-12T03:37:23.204814+0000","last_fullsized":"2024-03-12T03:37:23.204814+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:24.162965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.162965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:35.015647+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":158,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.209906+0000","last_change":"2024-03-12T03:36:25.092455+0000","last_active":"2024-03-12T03:37:23.209906+0000","last_peered":"2024-03-12T03:37:23.209906+0000","last_clean":"2024-03-12T03:37:23.209906+0000","last_became_active":"2024-03-12T03:33:21.095556+0000","last_became_peered":"2024-03-12T03:33:21.095556+0000","last_unstale":"2024-03-12T03:37:23.209906+0000","last_undegraded":"2024-03-12T03:37:23.209906+0000","last_fullsized":"2024-03-12T03:37:23.209906+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:25.092313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.092313+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.705657+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"94'23","reported_seq":197,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499589+0000","last_change":"2024-03-12T03:36:20.132927+0000","last_active":"2024-03-12T03:37:35.499589+0000","last_peered":"2024-03-12T03:37:35.499589+0000","last_clean":"2024-03-12T03:37:35.499589+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:37:35.499589+0000","last_undegraded":"2024-03-12T03:37:35.499589+0000","last_fullsized":"2024-03-12T03:37:35.499589+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'23","last_scrub_stamp":"2024-03-12T03:36:20.132683+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:34:52.729983+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.132683+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:42.640904+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.13","version":"133'7842","reported_seq":7943,"reported_epoch":133,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-03-12T03:37:31.030346+0000","last_change":"2024-03-12T03:37:22.374972+0000","last_active":"2024-03-12T03:37:31.030346+0000","last_peered":"2024-03-12T03:37:31.030346+0000","last_clean":"2024-03-12T03:36:50.229510+0000","last_became_active":"2024-03-12T03:36:52.356636+0000","last_became_peered":"2024-03-12T03:36:52.356636+0000","last_unstale":"2024-03-12T03:37:31.030346+0000","last_undegraded":"2024-03-12T03:37:31.030346+0000","last_fullsized":"2024-03-12T03:37:31.030346+0000","mapping_epoch":122,"log_start":"101'2100","ondisk_log_start":"101'2100","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192348160,"num_objects":2935,"num_object_clones":0,"num_object_copies":8805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5870,"num_objects_unfound":0,"num_objects_dirty":2935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4467,"num_write_kb":236864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[2,3,4],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"2,3,4","objects":2935}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":144,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.500598+0000","last_change":"2024-03-12T03:35:15.855752+0000","last_active":"2024-03-12T03:37:35.500598+0000","last_peered":"2024-03-12T03:37:35.500598+0000","last_clean":"2024-03-12T03:37:35.500598+0000","last_became_active":"2024-03-12T03:33:27.189696+0000","last_became_peered":"2024-03-12T03:33:27.189696+0000","last_unstale":"2024-03-12T03:37:35.500598+0000","last_undegraded":"2024-03-12T03:37:35.500598+0000","last_fullsized":"2024-03-12T03:37:35.500598+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.855641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.855641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":155,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499582+0000","last_change":"2024-03-12T03:36:01.098836+0000","last_active":"2024-03-12T03:37:35.499582+0000","last_peered":"2024-03-12T03:37:35.499582+0000","last_clean":"2024-03-12T03:37:35.499582+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:37:35.499582+0000","last_undegraded":"2024-03-12T03:37:35.499582+0000","last_fullsized":"2024-03-12T03:37:35.499582+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:01.098726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:01.098726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":160,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.212023+0000","last_change":"2024-03-12T03:35:59.076304+0000","last_active":"2024-03-12T03:37:23.212023+0000","last_peered":"2024-03-12T03:37:23.212023+0000","last_clean":"2024-03-12T03:37:23.212023+0000","last_became_active":"2024-03-12T03:33:21.096560+0000","last_became_peered":"2024-03-12T03:33:21.096560+0000","last_unstale":"2024-03-12T03:37:23.212023+0000","last_undegraded":"2024-03-12T03:37:23.212023+0000","last_fullsized":"2024-03-12T03:37:23.212023+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.076203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.076203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"90'6","reported_seq":156,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.210040+0000","last_change":"2024-03-12T03:34:34.274478+0000","last_active":"2024-03-12T03:37:23.210040+0000","last_peered":"2024-03-12T03:37:23.210040+0000","last_clean":"2024-03-12T03:37:23.210040+0000","last_became_active":"2024-03-12T03:33:23.161801+0000","last_became_peered":"2024-03-12T03:33:23.161801+0000","last_unstale":"2024-03-12T03:37:23.210040+0000","last_undegraded":"2024-03-12T03:37:23.210040+0000","last_fullsized":"2024-03-12T03:37:23.210040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:34:34.274353+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"133'7790","reported_seq":7890,"reported_epoch":133,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-12T03:37:31.008858+0000","last_change":"2024-03-12T03:37:22.404161+0000","last_active":"2024-03-12T03:37:31.008858+0000","last_peered":"2024-03-12T03:37:31.008858+0000","last_clean":"2024-03-12T03:36:50.159748+0000","last_became_active":"2024-03-12T03:36:52.374031+0000","last_became_peered":"2024-03-12T03:36:52.374031+0000","last_unstale":"2024-03-12T03:37:31.008858+0000","last_undegraded":"2024-03-12T03:37:31.008858+0000","last_fullsized":"2024-03-12T03:37:31.008858+0000","mapping_epoch":122,"log_start":"101'2100","ondisk_log_start":"101'2100","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5690,"log_dups_size":0,"ondisk_log_size":5690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190185472,"num_objects":2902,"num_object_clones":0,"num_object_copies":8706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5804,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4428,"num_write_kb":234560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[1,2,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2902}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":140,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.210204+0000","last_change":"2024-03-12T03:34:40.191618+0000","last_active":"2024-03-12T03:37:23.210204+0000","last_peered":"2024-03-12T03:37:23.210204+0000","last_clean":"2024-03-12T03:37:23.210204+0000","last_became_active":"2024-03-12T03:33:27.381110+0000","last_became_peered":"2024-03-12T03:33:27.381110+0000","last_unstale":"2024-03-12T03:37:23.210204+0000","last_undegraded":"2024-03-12T03:37:23.210204+0000","last_fullsized":"2024-03-12T03:37:23.210204+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.191538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":148,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.500797+0000","last_change":"2024-03-12T03:34:53.878589+0000","last_active":"2024-03-12T03:37:35.500797+0000","last_peered":"2024-03-12T03:37:35.500797+0000","last_clean":"2024-03-12T03:37:35.500797+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:37:35.500797+0000","last_undegraded":"2024-03-12T03:37:35.500797+0000","last_fullsized":"2024-03-12T03:37:35.500797+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:53.878484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:53.878484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":164,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499475+0000","last_change":"2024-03-12T03:35:59.352393+0000","last_active":"2024-03-12T03:37:35.499475+0000","last_peered":"2024-03-12T03:37:35.499475+0000","last_clean":"2024-03-12T03:37:35.499475+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:37:35.499475+0000","last_undegraded":"2024-03-12T03:37:35.499475+0000","last_fullsized":"2024-03-12T03:37:35.499475+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.352272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.352272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'18","reported_seq":180,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.210500+0000","last_change":"2024-03-12T03:35:02.079317+0000","last_active":"2024-03-12T03:37:23.210500+0000","last_peered":"2024-03-12T03:37:23.210500+0000","last_clean":"2024-03-12T03:37:23.210500+0000","last_became_active":"2024-03-12T03:33:23.552210+0000","last_became_peered":"2024-03-12T03:33:23.552210+0000","last_unstale":"2024-03-12T03:37:23.210500+0000","last_undegraded":"2024-03-12T03:37:23.210500+0000","last_fullsized":"2024-03-12T03:37:23.210500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:35:02.079224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:02.079224+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"133'7687","reported_seq":7860,"reported_epoch":133,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-03-12T03:37:30.989785+0000","last_change":"2024-03-12T03:37:22.577477+0000","last_active":"2024-03-12T03:37:30.989785+0000","last_peered":"2024-03-12T03:37:30.989785+0000","last_clean":"2024-03-12T03:36:50.164308+0000","last_became_active":"2024-03-12T03:36:52.342472+0000","last_became_peered":"2024-03-12T03:36:52.342472+0000","last_unstale":"2024-03-12T03:37:30.989785+0000","last_undegraded":"2024-03-12T03:37:30.989785+0000","last_fullsized":"2024-03-12T03:37:30.989785+0000","mapping_epoch":122,"log_start":"101'1800","ondisk_log_start":"101'1800","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5887,"log_dups_size":0,"ondisk_log_size":5887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189792256,"num_objects":2896,"num_object_clones":0,"num_object_copies":8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2872,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":232192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":2424832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2896}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":148,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.196334+0000","last_change":"2024-03-12T03:36:18.113521+0000","last_active":"2024-03-12T03:37:23.196334+0000","last_peered":"2024-03-12T03:37:23.196334+0000","last_clean":"2024-03-12T03:37:23.196334+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:37:23.196334+0000","last_undegraded":"2024-03-12T03:37:23.196334+0000","last_fullsized":"2024-03-12T03:37:23.196334+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.113393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:18.113393+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.113393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:28.877769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":144,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.209923+0000","last_change":"2024-03-12T03:34:45.089928+0000","last_active":"2024-03-12T03:37:23.209923+0000","last_peered":"2024-03-12T03:37:23.209923+0000","last_clean":"2024-03-12T03:37:23.209923+0000","last_became_active":"2024-03-12T03:33:25.183725+0000","last_became_peered":"2024-03-12T03:33:25.183725+0000","last_unstale":"2024-03-12T03:37:23.209923+0000","last_undegraded":"2024-03-12T03:37:23.209923+0000","last_fullsized":"2024-03-12T03:37:23.209923+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:45.089762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:45.089762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":156,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.501400+0000","last_change":"2024-03-12T03:34:41.698397+0000","last_active":"2024-03-12T03:37:35.501400+0000","last_peered":"2024-03-12T03:37:35.501400+0000","last_clean":"2024-03-12T03:37:35.501400+0000","last_became_active":"2024-03-12T03:33:21.107037+0000","last_became_peered":"2024-03-12T03:33:21.107037+0000","last_unstale":"2024-03-12T03:37:35.501400+0000","last_undegraded":"2024-03-12T03:37:35.501400+0000","last_fullsized":"2024-03-12T03:37:35.501400+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.698268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.698268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":172,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.206155+0000","last_change":"2024-03-12T03:36:11.006658+0000","last_active":"2024-03-12T03:37:23.206155+0000","last_peered":"2024-03-12T03:37:23.206155+0000","last_clean":"2024-03-12T03:37:23.206155+0000","last_became_active":"2024-03-12T03:33:23.160818+0000","last_became_peered":"2024-03-12T03:33:23.160818+0000","last_unstale":"2024-03-12T03:37:23.206155+0000","last_undegraded":"2024-03-12T03:37:23.206155+0000","last_fullsized":"2024-03-12T03:37:23.206155+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-12T03:36:11.006154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:11.006154+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.382856+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.10","version":"133'7741","reported_seq":11521,"reported_epoch":133,"state":"active+remapped+backfilling","last_fresh":"2024-03-12T03:37:31.009055+0000","last_change":"2024-03-12T03:36:52.565904+0000","last_active":"2024-03-12T03:37:31.009055+0000","last_peered":"2024-03-12T03:37:31.009055+0000","last_clean":"2024-03-12T03:36:50.171417+0000","last_became_active":"2024-03-12T03:36:52.547003+0000","last_became_peered":"2024-03-12T03:36:52.547003+0000","last_unstale":"2024-03-12T03:37:31.009055+0000","last_undegraded":"2024-03-12T03:37:31.009055+0000","last_fullsized":"2024-03-12T03:37:31.009055+0000","mapping_epoch":122,"log_start":"100'800","ondisk_log_start":"100'800","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6941,"log_dups_size":0,"ondisk_log_size":6941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187203584,"num_objects":2857,"num_object_clones":0,"num_object_copies":8571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1224,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":232416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1842,"num_bytes_recovered":120717312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1,5],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":2857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":140,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.210193+0000","last_change":"2024-03-12T03:35:59.076156+0000","last_active":"2024-03-12T03:37:23.210193+0000","last_peered":"2024-03-12T03:37:23.210193+0000","last_clean":"2024-03-12T03:37:23.210193+0000","last_became_active":"2024-03-12T03:33:27.191710+0000","last_became_peered":"2024-03-12T03:33:27.191710+0000","last_unstale":"2024-03-12T03:37:23.210193+0000","last_undegraded":"2024-03-12T03:37:23.210193+0000","last_fullsized":"2024-03-12T03:37:23.210193+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.075897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.075897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.205936+0000","last_change":"2024-03-12T03:36:00.842130+0000","last_active":"2024-03-12T03:37:23.205936+0000","last_peered":"2024-03-12T03:37:23.205936+0000","last_clean":"2024-03-12T03:37:23.205936+0000","last_became_active":"2024-03-12T03:33:25.182004+0000","last_became_peered":"2024-03-12T03:33:25.182004+0000","last_unstale":"2024-03-12T03:37:23.205936+0000","last_undegraded":"2024-03-12T03:37:23.205936+0000","last_fullsized":"2024-03-12T03:37:23.205936+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:00.841986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:00.841986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":164,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.496159+0000","last_change":"2024-03-12T03:36:26.141166+0000","last_active":"2024-03-12T03:37:35.496159+0000","last_peered":"2024-03-12T03:37:35.496159+0000","last_clean":"2024-03-12T03:37:35.496159+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:37:35.496159+0000","last_undegraded":"2024-03-12T03:37:35.496159+0000","last_fullsized":"2024-03-12T03:37:35.496159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.141036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.141036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":179,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.496447+0000","last_change":"2024-03-12T03:37:02.194152+0000","last_active":"2024-03-12T03:37:35.496447+0000","last_peered":"2024-03-12T03:37:35.496447+0000","last_clean":"2024-03-12T03:37:35.496447+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:37:35.496447+0000","last_undegraded":"2024-03-12T03:37:35.496447+0000","last_fullsized":"2024-03-12T03:37:35.496447+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:37:02.193985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:37:02.193985+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:07.731309+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"133'7851","reported_seq":18499,"reported_epoch":133,"state":"active+remapped+backfilling","last_fresh":"2024-03-12T03:37:31.012166+0000","last_change":"2024-03-12T03:36:52.434219+0000","last_active":"2024-03-12T03:37:31.012166+0000","last_peered":"2024-03-12T03:37:31.012166+0000","last_clean":"2024-03-12T03:36:50.240631+0000","last_became_active":"2024-03-12T03:36:52.424496+0000","last_became_peered":"2024-03-12T03:36:52.424496+0000","last_unstale":"2024-03-12T03:37:31.012166+0000","last_undegraded":"2024-03-12T03:37:31.012166+0000","last_fullsized":"2024-03-12T03:37:31.012166+0000","mapping_epoch":122,"log_start":"100'500","ondisk_log_start":"100'500","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190447616,"num_objects":2906,"num_object_clones":0,"num_object_copies":8718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1764,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4456,"num_write_kb":235584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2638,"num_bytes_recovered":172883968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[0,3,6],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3,6","objects":2906}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":148,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.204057+0000","last_change":"2024-03-12T03:36:07.111872+0000","last_active":"2024-03-12T03:37:23.204057+0000","last_peered":"2024-03-12T03:37:23.204057+0000","last_clean":"2024-03-12T03:37:23.204057+0000","last_became_active":"2024-03-12T03:33:27.194094+0000","last_became_peered":"2024-03-12T03:33:27.194094+0000","last_unstale":"2024-03-12T03:37:23.204057+0000","last_undegraded":"2024-03-12T03:37:23.204057+0000","last_fullsized":"2024-03-12T03:37:23.204057+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:07.111722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.111722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:33.629653+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":215,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499067+0000","last_change":"2024-03-12T03:36:07.296952+0000","last_active":"2024-03-12T03:37:35.499067+0000","last_peered":"2024-03-12T03:37:35.499067+0000","last_clean":"2024-03-12T03:37:35.499067+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:37:35.499067+0000","last_undegraded":"2024-03-12T03:37:35.499067+0000","last_fullsized":"2024-03-12T03:37:35.499067+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:07.296796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.296796+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":170,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.501851+0000","last_change":"2024-03-12T03:36:27.955751+0000","last_active":"2024-03-12T03:37:35.501851+0000","last_peered":"2024-03-12T03:37:35.501851+0000","last_clean":"2024-03-12T03:37:35.501851+0000","last_became_active":"2024-03-12T03:33:21.573244+0000","last_became_peered":"2024-03-12T03:33:21.573244+0000","last_unstale":"2024-03-12T03:37:35.501851+0000","last_undegraded":"2024-03-12T03:37:35.501851+0000","last_fullsized":"2024-03-12T03:37:35.501851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:27.955624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:27.955624+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:42.177558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":187,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.213425+0000","last_change":"2024-03-12T03:35:58.079779+0000","last_active":"2024-03-12T03:37:23.213425+0000","last_peered":"2024-03-12T03:37:23.213425+0000","last_clean":"2024-03-12T03:37:23.213425+0000","last_became_active":"2024-03-12T03:33:23.551706+0000","last_became_peered":"2024-03-12T03:33:23.551706+0000","last_unstale":"2024-03-12T03:37:23.213425+0000","last_undegraded":"2024-03-12T03:37:23.213425+0000","last_fullsized":"2024-03-12T03:37:23.213425+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-03-12T03:35:58.079645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.079645+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.444037+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"133'7728","reported_seq":9756,"reported_epoch":133,"state":"active+remapped+backfilling","last_fresh":"2024-03-12T03:37:31.022339+0000","last_change":"2024-03-12T03:37:22.407262+0000","last_active":"2024-03-12T03:37:31.022339+0000","last_peered":"2024-03-12T03:37:31.022339+0000","last_clean":"2024-03-12T03:36:50.275996+0000","last_became_active":"2024-03-12T03:36:52.375928+0000","last_became_peered":"2024-03-12T03:36:52.375928+0000","last_unstale":"2024-03-12T03:37:31.022339+0000","last_undegraded":"2024-03-12T03:37:31.022339+0000","last_fullsized":"2024-03-12T03:37:31.022339+0000","mapping_epoch":122,"log_start":"101'1100","ondisk_log_start":"101'1100","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6628,"log_dups_size":0,"ondisk_log_size":6628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190644224,"num_objects":2909,"num_object_clones":0,"num_object_copies":8727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1972,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4379,"num_write_kb":233216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":63307776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":2909}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":152,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.497172+0000","last_change":"2024-03-12T03:37:10.111062+0000","last_active":"2024-03-12T03:37:35.497172+0000","last_peered":"2024-03-12T03:37:35.497172+0000","last_clean":"2024-03-12T03:37:35.497172+0000","last_became_active":"2024-03-12T03:33:27.184916+0000","last_became_peered":"2024-03-12T03:33:27.184916+0000","last_unstale":"2024-03-12T03:37:35.497172+0000","last_undegraded":"2024-03-12T03:37:35.497172+0000","last_fullsized":"2024-03-12T03:37:35.497172+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:10.110987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:37:10.110987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:13.190325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":205,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.213620+0000","last_change":"2024-03-12T03:35:08.006530+0000","last_active":"2024-03-12T03:37:23.213620+0000","last_peered":"2024-03-12T03:37:23.213620+0000","last_clean":"2024-03-12T03:37:23.213620+0000","last_became_active":"2024-03-12T03:33:25.179158+0000","last_became_peered":"2024-03-12T03:33:25.179158+0000","last_unstale":"2024-03-12T03:37:23.213620+0000","last_undegraded":"2024-03-12T03:37:23.213620+0000","last_fullsized":"2024-03-12T03:37:23.213620+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:35:08.006375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:08.006375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":160,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.202180+0000","last_change":"2024-03-12T03:36:14.197774+0000","last_active":"2024-03-12T03:37:23.202180+0000","last_peered":"2024-03-12T03:37:23.202180+0000","last_clean":"2024-03-12T03:37:23.202180+0000","last_became_active":"2024-03-12T03:33:21.101819+0000","last_became_peered":"2024-03-12T03:33:21.101819+0000","last_unstale":"2024-03-12T03:37:23.202180+0000","last_undegraded":"2024-03-12T03:37:23.202180+0000","last_fullsized":"2024-03-12T03:37:23.202180+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:14.197649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:36:14.197649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:41.087756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":188,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499222+0000","last_change":"2024-03-12T03:36:12.092317+0000","last_active":"2024-03-12T03:37:35.499222+0000","last_peered":"2024-03-12T03:37:35.499222+0000","last_clean":"2024-03-12T03:37:35.499222+0000","last_became_active":"2024-03-12T03:33:23.553480+0000","last_became_peered":"2024-03-12T03:33:23.553480+0000","last_unstale":"2024-03-12T03:37:35.499222+0000","last_undegraded":"2024-03-12T03:37:35.499222+0000","last_fullsized":"2024-03-12T03:37:35.499222+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:36:12.092175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:12.092175+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.15","version":"135'7862","reported_seq":14978,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.662774+0000","last_change":"2024-03-12T03:37:16.251138+0000","last_active":"2024-03-12T03:37:35.662774+0000","last_peered":"2024-03-12T03:37:35.662774+0000","last_clean":"2024-03-12T03:37:35.662774+0000","last_became_active":"2024-03-12T03:37:16.246657+0000","last_became_peered":"2024-03-12T03:37:16.246657+0000","last_unstale":"2024-03-12T03:37:35.662774+0000","last_undegraded":"2024-03-12T03:37:35.662774+0000","last_fullsized":"2024-03-12T03:37:35.662774+0000","mapping_epoch":128,"log_start":"101'1600","ondisk_log_start":"101'1600","created":119,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:37.290341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191102976,"num_objects":2916,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4468,"num_write_kb":236288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3504,"num_bytes_recovered":229638144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":148,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.200052+0000","last_change":"2024-03-12T03:36:02.155537+0000","last_active":"2024-03-12T03:37:23.200052+0000","last_peered":"2024-03-12T03:37:23.200052+0000","last_clean":"2024-03-12T03:37:23.200052+0000","last_became_active":"2024-03-12T03:33:27.187657+0000","last_became_peered":"2024-03-12T03:33:27.187657+0000","last_unstale":"2024-03-12T03:37:23.200052+0000","last_undegraded":"2024-03-12T03:37:23.200052+0000","last_fullsized":"2024-03-12T03:37:23.200052+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:02.155425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:02.155425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:24.813580+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":144,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.216034+0000","last_change":"2024-03-12T03:35:04.007338+0000","last_active":"2024-03-12T03:37:23.216034+0000","last_peered":"2024-03-12T03:37:23.216034+0000","last_clean":"2024-03-12T03:37:23.216034+0000","last_became_active":"2024-03-12T03:33:25.174953+0000","last_became_peered":"2024-03-12T03:33:25.174953+0000","last_unstale":"2024-03-12T03:37:23.216034+0000","last_undegraded":"2024-03-12T03:37:23.216034+0000","last_fullsized":"2024-03-12T03:37:23.216034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:35:04.007252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":156,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.500820+0000","last_change":"2024-03-12T03:34:52.848468+0000","last_active":"2024-03-12T03:37:35.500820+0000","last_peered":"2024-03-12T03:37:35.500820+0000","last_clean":"2024-03-12T03:37:35.500820+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:37:35.500820+0000","last_undegraded":"2024-03-12T03:37:35.500820+0000","last_fullsized":"2024-03-12T03:37:35.500820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:52.848366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:52.848366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":188,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.204617+0000","last_change":"2024-03-12T03:35:11.786139+0000","last_active":"2024-03-12T03:37:23.204617+0000","last_peered":"2024-03-12T03:37:23.204617+0000","last_clean":"2024-03-12T03:37:23.204617+0000","last_became_active":"2024-03-12T03:33:23.552164+0000","last_became_peered":"2024-03-12T03:33:23.552164+0000","last_unstale":"2024-03-12T03:37:23.204617+0000","last_undegraded":"2024-03-12T03:37:23.204617+0000","last_fullsized":"2024-03-12T03:37:23.204617+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:35:11.786040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.786040+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"133'7764","reported_seq":7864,"reported_epoch":133,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-03-12T03:37:30.900663+0000","last_change":"2024-03-12T03:37:22.414173+0000","last_active":"2024-03-12T03:37:30.900663+0000","last_peered":"2024-03-12T03:37:30.900663+0000","last_clean":"2024-03-12T03:36:50.262948+0000","last_became_active":"2024-03-12T03:36:52.402968+0000","last_became_peered":"2024-03-12T03:36:52.402968+0000","last_unstale":"2024-03-12T03:37:30.900663+0000","last_undegraded":"2024-03-12T03:37:30.900663+0000","last_fullsized":"2024-03-12T03:37:30.900663+0000","mapping_epoch":122,"log_start":"101'1100","ondisk_log_start":"101'1100","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6664,"log_dups_size":0,"ondisk_log_size":6664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187990016,"num_objects":2869,"num_object_clones":0,"num_object_copies":8607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8607,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":232800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[0,1,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":2869}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":152,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.501295+0000","last_change":"2024-03-12T03:36:24.959007+0000","last_active":"2024-03-12T03:37:35.501295+0000","last_peered":"2024-03-12T03:37:35.501295+0000","last_clean":"2024-03-12T03:37:35.501295+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:37:35.501295+0000","last_undegraded":"2024-03-12T03:37:35.501295+0000","last_fullsized":"2024-03-12T03:37:35.501295+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:24.958880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.958880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:47.117767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":156,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499556+0000","last_change":"2024-03-12T03:36:29.181191+0000","last_active":"2024-03-12T03:37:35.499556+0000","last_peered":"2024-03-12T03:37:35.499556+0000","last_clean":"2024-03-12T03:37:35.499556+0000","last_became_active":"2024-03-12T03:33:25.180752+0000","last_became_peered":"2024-03-12T03:33:25.180752+0000","last_unstale":"2024-03-12T03:37:35.499556+0000","last_undegraded":"2024-03-12T03:37:35.499556+0000","last_fullsized":"2024-03-12T03:37:35.499556+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:29.181010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:29.181010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":163,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.498040+0000","last_change":"2024-03-12T03:35:58.075845+0000","last_active":"2024-03-12T03:37:35.498040+0000","last_peered":"2024-03-12T03:37:35.498040+0000","last_clean":"2024-03-12T03:37:35.498040+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:37:35.498040+0000","last_undegraded":"2024-03-12T03:37:35.498040+0000","last_fullsized":"2024-03-12T03:37:35.498040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:58.075754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.075754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":169,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.216277+0000","last_change":"2024-03-12T03:35:00.079249+0000","last_active":"2024-03-12T03:37:23.216277+0000","last_peered":"2024-03-12T03:37:23.216277+0000","last_clean":"2024-03-12T03:37:23.216277+0000","last_became_active":"2024-03-12T03:33:23.551773+0000","last_became_peered":"2024-03-12T03:33:23.551773+0000","last_unstale":"2024-03-12T03:37:23.216277+0000","last_undegraded":"2024-03-12T03:37:23.216277+0000","last_fullsized":"2024-03-12T03:37:23.216277+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:35:00.079152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.079152+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"133'10557","reported_seq":10708,"reported_epoch":133,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-12T03:37:31.044260+0000","last_change":"2024-03-12T03:37:18.372743+0000","last_active":"2024-03-12T03:37:31.044260+0000","last_peered":"2024-03-12T03:37:31.044260+0000","last_clean":"2024-03-12T03:37:31.044260+0000","last_became_active":"2024-03-12T03:35:23.782112+0000","last_became_peered":"2024-03-12T03:35:23.782112+0000","last_unstale":"2024-03-12T03:37:31.044260+0000","last_undegraded":"2024-03-12T03:37:31.044260+0000","last_fullsized":"2024-03-12T03:37:31.044260+0000","mapping_epoch":97,"log_start":"113'5935","ondisk_log_start":"113'5935","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":2111,"log_size":4622,"log_dups_size":0,"ondisk_log_size":4622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 12s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381681664,"num_objects":5824,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8584,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":152,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499733+0000","last_change":"2024-03-12T03:36:20.147626+0000","last_active":"2024-03-12T03:37:35.499733+0000","last_peered":"2024-03-12T03:37:35.499733+0000","last_clean":"2024-03-12T03:37:35.499733+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:37:35.499733+0000","last_undegraded":"2024-03-12T03:37:35.499733+0000","last_fullsized":"2024-03-12T03:37:35.499733+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.147488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:44.684777+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":168,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.501822+0000","last_change":"2024-03-12T03:35:12.907083+0000","last_active":"2024-03-12T03:37:35.501822+0000","last_peered":"2024-03-12T03:37:35.501822+0000","last_clean":"2024-03-12T03:37:35.501822+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:37:35.501822+0000","last_undegraded":"2024-03-12T03:37:35.501822+0000","last_fullsized":"2024-03-12T03:37:35.501822+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:35:12.906988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:12.906988+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"90'36","reported_seq":218,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.498067+0000","last_change":"2024-03-12T03:36:15.181987+0000","last_active":"2024-03-12T03:37:35.498067+0000","last_peered":"2024-03-12T03:37:35.498067+0000","last_clean":"2024-03-12T03:37:35.498067+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:37:35.498067+0000","last_undegraded":"2024-03-12T03:37:35.498067+0000","last_fullsized":"2024-03-12T03:37:35.498067+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:36:15.181852+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.181852+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":207,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.501084+0000","last_change":"2024-03-12T03:34:39.659403+0000","last_active":"2024-03-12T03:37:35.501084+0000","last_peered":"2024-03-12T03:37:35.501084+0000","last_clean":"2024-03-12T03:37:35.501084+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:37:35.501084+0000","last_undegraded":"2024-03-12T03:37:35.501084+0000","last_fullsized":"2024-03-12T03:37:35.501084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:39.659277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.659277+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"133'8873","reported_seq":8959,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:31.008518+0000","last_change":"2024-03-12T03:35:23.777596+0000","last_active":"2024-03-12T03:37:31.008518+0000","last_peered":"2024-03-12T03:37:31.008518+0000","last_clean":"2024-03-12T03:37:31.008518+0000","last_became_active":"2024-03-12T03:35:23.777272+0000","last_became_peered":"2024-03-12T03:35:23.777272+0000","last_unstale":"2024-03-12T03:37:31.008518+0000","last_undegraded":"2024-03-12T03:37:31.008518+0000","last_fullsized":"2024-03-12T03:37:31.008518+0000","mapping_epoch":97,"log_start":"106'4622","ondisk_log_start":"106'4622","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4251,"log_dups_size":0,"ondisk_log_size":4251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386990080,"num_objects":5905,"num_object_clones":0,"num_object_copies":17715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8873,"num_write_kb":472896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":132,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.210099+0000","last_change":"2024-03-12T03:33:27.191091+0000","last_active":"2024-03-12T03:37:23.210099+0000","last_peered":"2024-03-12T03:37:23.210099+0000","last_clean":"2024-03-12T03:37:23.210099+0000","last_became_active":"2024-03-12T03:33:27.190753+0000","last_became_peered":"2024-03-12T03:33:27.190753+0000","last_unstale":"2024-03-12T03:37:23.210099+0000","last_undegraded":"2024-03-12T03:37:23.210099+0000","last_fullsized":"2024-03-12T03:37:23.210099+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:33:26.164901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":164,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.501256+0000","last_change":"2024-03-12T03:36:17.051625+0000","last_active":"2024-03-12T03:37:35.501256+0000","last_peered":"2024-03-12T03:37:35.501256+0000","last_clean":"2024-03-12T03:37:35.501256+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:37:35.501256+0000","last_undegraded":"2024-03-12T03:37:35.501256+0000","last_fullsized":"2024-03-12T03:37:35.501256+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:36:17.051487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":199,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.496630+0000","last_change":"2024-03-12T03:36:10.043243+0000","last_active":"2024-03-12T03:37:35.496630+0000","last_peered":"2024-03-12T03:37:35.496630+0000","last_clean":"2024-03-12T03:37:35.496630+0000","last_became_active":"2024-03-12T03:33:23.163351+0000","last_became_peered":"2024-03-12T03:33:23.163351+0000","last_unstale":"2024-03-12T03:37:35.496630+0000","last_undegraded":"2024-03-12T03:37:35.496630+0000","last_fullsized":"2024-03-12T03:37:35.496630+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-03-12T03:36:10.043108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:10.043108+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.376152+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.199018+0000","last_change":"2024-03-12T03:36:26.332254+0000","last_active":"2024-03-12T03:37:23.199018+0000","last_peered":"2024-03-12T03:37:23.199018+0000","last_clean":"2024-03-12T03:37:23.199018+0000","last_became_active":"2024-03-12T03:33:25.185596+0000","last_became_peered":"2024-03-12T03:33:25.185596+0000","last_unstale":"2024-03-12T03:37:23.199018+0000","last_undegraded":"2024-03-12T03:37:23.199018+0000","last_fullsized":"2024-03-12T03:37:23.199018+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.332124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.332124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:45.657747+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"133'7794","reported_seq":7886,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:30.956682+0000","last_change":"2024-03-12T03:36:49.287952+0000","last_active":"2024-03-12T03:37:30.956682+0000","last_peered":"2024-03-12T03:37:30.956682+0000","last_clean":"2024-03-12T03:37:30.956682+0000","last_became_active":"2024-03-12T03:36:49.278858+0000","last_became_peered":"2024-03-12T03:36:49.278858+0000","last_unstale":"2024-03-12T03:37:30.956682+0000","last_undegraded":"2024-03-12T03:37:30.956682+0000","last_fullsized":"2024-03-12T03:37:30.956682+0000","mapping_epoch":97,"log_start":"101'1100","ondisk_log_start":"101'1100","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:32.638904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195002368,"num_objects":2975,"num_object_clones":0,"num_object_copies":8925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4417,"num_write_kb":236576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":144,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.500453+0000","last_change":"2024-03-12T03:34:43.716123+0000","last_active":"2024-03-12T03:37:35.500453+0000","last_peered":"2024-03-12T03:37:35.500453+0000","last_clean":"2024-03-12T03:37:35.500453+0000","last_became_active":"2024-03-12T03:33:27.187955+0000","last_became_peered":"2024-03-12T03:33:27.187955+0000","last_unstale":"2024-03-12T03:37:35.500453+0000","last_undegraded":"2024-03-12T03:37:35.500453+0000","last_fullsized":"2024-03-12T03:37:35.500453+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:43.716021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.716021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.199438+0000","last_change":"2024-03-12T03:36:18.320863+0000","last_active":"2024-03-12T03:37:23.199438+0000","last_peered":"2024-03-12T03:37:23.199438+0000","last_clean":"2024-03-12T03:37:23.199438+0000","last_became_active":"2024-03-12T03:33:25.179593+0000","last_became_peered":"2024-03-12T03:33:25.179593+0000","last_unstale":"2024-03-12T03:37:23.199438+0000","last_undegraded":"2024-03-12T03:37:23.199438+0000","last_fullsized":"2024-03-12T03:37:23.199438+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.320728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.320728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.937998+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":164,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.501581+0000","last_change":"2024-03-12T03:36:09.026855+0000","last_active":"2024-03-12T03:37:35.501581+0000","last_peered":"2024-03-12T03:37:35.501581+0000","last_clean":"2024-03-12T03:37:35.501581+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:37:35.501581+0000","last_undegraded":"2024-03-12T03:37:35.501581+0000","last_fullsized":"2024-03-12T03:37:35.501581+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:09.026725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:09.026725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":202,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.498309+0000","last_change":"2024-03-12T03:36:27.151507+0000","last_active":"2024-03-12T03:37:35.498309+0000","last_peered":"2024-03-12T03:37:35.498309+0000","last_clean":"2024-03-12T03:37:35.498309+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:37:35.498309+0000","last_undegraded":"2024-03-12T03:37:35.498309+0000","last_fullsized":"2024-03-12T03:37:35.498309+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:36:27.151375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:27.151375+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"133'7801","reported_seq":7893,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:31.029593+0000","last_change":"2024-03-12T03:36:49.308754+0000","last_active":"2024-03-12T03:37:31.029593+0000","last_peered":"2024-03-12T03:37:31.029593+0000","last_clean":"2024-03-12T03:37:31.029593+0000","last_became_active":"2024-03-12T03:36:49.305916+0000","last_became_peered":"2024-03-12T03:36:49.305916+0000","last_unstale":"2024-03-12T03:37:31.029593+0000","last_undegraded":"2024-03-12T03:37:31.029593+0000","last_fullsized":"2024-03-12T03:37:31.029593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:31.736002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188350464,"num_objects":2874,"num_object_clones":0,"num_object_copies":8622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4442,"num_write_kb":234112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":148,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.205805+0000","last_change":"2024-03-12T03:36:10.178072+0000","last_active":"2024-03-12T03:37:23.205805+0000","last_peered":"2024-03-12T03:37:23.205805+0000","last_clean":"2024-03-12T03:37:23.205805+0000","last_became_active":"2024-03-12T03:33:27.189948+0000","last_became_peered":"2024-03-12T03:33:27.189948+0000","last_unstale":"2024-03-12T03:37:23.205805+0000","last_undegraded":"2024-03-12T03:37:23.205805+0000","last_fullsized":"2024-03-12T03:37:23.205805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:10.177808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:10.177808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:35.471819+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.198015+0000","last_change":"2024-03-12T03:36:11.345502+0000","last_active":"2024-03-12T03:37:23.198015+0000","last_peered":"2024-03-12T03:37:23.198015+0000","last_clean":"2024-03-12T03:37:23.198015+0000","last_became_active":"2024-03-12T03:33:25.184289+0000","last_became_peered":"2024-03-12T03:33:25.184289+0000","last_unstale":"2024-03-12T03:37:23.198015+0000","last_undegraded":"2024-03-12T03:37:23.198015+0000","last_fullsized":"2024-03-12T03:37:23.198015+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:11.345381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:11.345381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:32.654030+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":164,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499531+0000","last_change":"2024-03-12T03:36:26.134473+0000","last_active":"2024-03-12T03:37:35.499531+0000","last_peered":"2024-03-12T03:37:35.499531+0000","last_clean":"2024-03-12T03:37:35.499531+0000","last_became_active":"2024-03-12T03:33:21.102410+0000","last_became_peered":"2024-03-12T03:33:21.102410+0000","last_unstale":"2024-03-12T03:37:35.499531+0000","last_undegraded":"2024-03-12T03:37:35.499531+0000","last_fullsized":"2024-03-12T03:37:35.499531+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.134337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.134337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:45.916621+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"111'20","reported_seq":189,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.204159+0000","last_change":"2024-03-12T03:36:29.192473+0000","last_active":"2024-03-12T03:37:23.204159+0000","last_peered":"2024-03-12T03:37:23.204159+0000","last_clean":"2024-03-12T03:37:23.204159+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:37:23.204159+0000","last_undegraded":"2024-03-12T03:37:23.204159+0000","last_fullsized":"2024-03-12T03:37:23.204159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'20","last_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:36:29.192310+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:32.549710+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"135'7913","reported_seq":8006,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.798284+0000","last_change":"2024-03-12T03:36:49.295573+0000","last_active":"2024-03-12T03:37:35.798284+0000","last_peered":"2024-03-12T03:37:35.798284+0000","last_clean":"2024-03-12T03:37:35.798284+0000","last_became_active":"2024-03-12T03:36:49.292644+0000","last_became_peered":"2024-03-12T03:36:49.292644+0000","last_unstale":"2024-03-12T03:37:35.798284+0000","last_undegraded":"2024-03-12T03:37:35.798284+0000","last_fullsized":"2024-03-12T03:37:35.798284+0000","mapping_epoch":97,"log_start":"100'500","ondisk_log_start":"100'500","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:39.519431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189399040,"num_objects":2890,"num_object_clones":0,"num_object_copies":8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4518,"num_write_kb":237056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":140,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.213881+0000","last_change":"2024-03-12T03:34:49.088183+0000","last_active":"2024-03-12T03:37:23.213881+0000","last_peered":"2024-03-12T03:37:23.213881+0000","last_clean":"2024-03-12T03:37:23.213881+0000","last_became_active":"2024-03-12T03:33:27.185041+0000","last_became_peered":"2024-03-12T03:33:27.185041+0000","last_unstale":"2024-03-12T03:37:23.213881+0000","last_undegraded":"2024-03-12T03:37:23.213881+0000","last_fullsized":"2024-03-12T03:37:23.213881+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.088098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.088098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":176,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499939+0000","last_change":"2024-03-12T03:36:13.134209+0000","last_active":"2024-03-12T03:37:35.499939+0000","last_peered":"2024-03-12T03:37:35.499939+0000","last_clean":"2024-03-12T03:37:35.499939+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:37:35.499939+0000","last_undegraded":"2024-03-12T03:37:35.499939+0000","last_fullsized":"2024-03-12T03:37:35.499939+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:36:13.134087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.134087+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":165,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.496099+0000","last_change":"2024-03-12T03:37:00.206483+0000","last_active":"2024-03-12T03:37:35.496099+0000","last_peered":"2024-03-12T03:37:35.496099+0000","last_clean":"2024-03-12T03:37:35.496099+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:37:35.496099+0000","last_undegraded":"2024-03-12T03:37:35.496099+0000","last_fullsized":"2024-03-12T03:37:35.496099+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:00.206383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:37:00.206383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:20.230781+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'35","reported_seq":202,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.498557+0000","last_change":"2024-03-12T03:34:54.301841+0000","last_active":"2024-03-12T03:37:35.498557+0000","last_peered":"2024-03-12T03:37:35.498557+0000","last_clean":"2024-03-12T03:37:35.498557+0000","last_became_active":"2024-03-12T03:33:50.151664+0000","last_became_peered":"2024-03-12T03:33:50.151664+0000","last_unstale":"2024-03-12T03:37:35.498557+0000","last_undegraded":"2024-03-12T03:37:35.498557+0000","last_fullsized":"2024-03-12T03:37:35.498557+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'35","last_scrub_stamp":"2024-03-12T03:34:54.301725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:34:54.301725+0000","objects_scrubbed":11,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.207838+0000","last_change":"2024-03-12T03:36:15.170135+0000","last_active":"2024-03-12T03:37:23.207838+0000","last_peered":"2024-03-12T03:37:23.207838+0000","last_clean":"2024-03-12T03:37:23.207838+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:37:23.207838+0000","last_undegraded":"2024-03-12T03:37:23.207838+0000","last_fullsized":"2024-03-12T03:37:23.207838+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:15.170007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.170007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"133'7728","reported_seq":7821,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:30.948587+0000","last_change":"2024-03-12T03:36:49.289418+0000","last_active":"2024-03-12T03:37:30.948587+0000","last_peered":"2024-03-12T03:37:30.948587+0000","last_clean":"2024-03-12T03:37:30.948587+0000","last_became_active":"2024-03-12T03:36:49.281680+0000","last_became_peered":"2024-03-12T03:36:49.281680+0000","last_unstale":"2024-03-12T03:37:30.948587+0000","last_undegraded":"2024-03-12T03:37:30.948587+0000","last_fullsized":"2024-03-12T03:37:30.948587+0000","mapping_epoch":97,"log_start":"101'1100","ondisk_log_start":"101'1100","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6628,"log_dups_size":0,"ondisk_log_size":6628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:32.638852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193495040,"num_objects":2952,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4370,"num_write_kb":234336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":154,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.501742+0000","last_change":"2024-03-12T03:36:02.083741+0000","last_active":"2024-03-12T03:37:35.501742+0000","last_peered":"2024-03-12T03:37:35.501742+0000","last_clean":"2024-03-12T03:37:35.501742+0000","last_became_active":"2024-03-12T03:33:27.380389+0000","last_became_peered":"2024-03-12T03:33:27.380389+0000","last_unstale":"2024-03-12T03:37:35.501742+0000","last_undegraded":"2024-03-12T03:37:35.501742+0000","last_fullsized":"2024-03-12T03:37:35.501742+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:36:02.083611+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:36:02.083611+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":161,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.203319+0000","last_change":"2024-03-12T03:36:21.149936+0000","last_active":"2024-03-12T03:37:23.203319+0000","last_peered":"2024-03-12T03:37:23.203319+0000","last_clean":"2024-03-12T03:37:23.203319+0000","last_became_active":"2024-03-12T03:33:21.572911+0000","last_became_peered":"2024-03-12T03:33:21.572911+0000","last_unstale":"2024-03-12T03:37:23.203319+0000","last_undegraded":"2024-03-12T03:37:23.203319+0000","last_fullsized":"2024-03-12T03:37:23.203319+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:21.149686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:21.149686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:31.591082+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":170,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.198254+0000","last_change":"2024-03-12T03:36:15.348581+0000","last_active":"2024-03-12T03:37:23.198254+0000","last_peered":"2024-03-12T03:37:23.198254+0000","last_clean":"2024-03-12T03:37:23.198254+0000","last_became_active":"2024-03-12T03:33:20.093064+0000","last_became_peered":"2024-03-12T03:33:20.093064+0000","last_unstale":"2024-03-12T03:37:23.198254+0000","last_undegraded":"2024-03-12T03:37:23.198254+0000","last_fullsized":"2024-03-12T03:37:23.198254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:15.348471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.348471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.000516994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":195,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.207609+0000","last_change":"2024-03-12T03:36:19.124448+0000","last_active":"2024-03-12T03:37:23.207609+0000","last_peered":"2024-03-12T03:37:23.207609+0000","last_clean":"2024-03-12T03:37:23.207609+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:37:23.207609+0000","last_undegraded":"2024-03-12T03:37:23.207609+0000","last_fullsized":"2024-03-12T03:37:23.207609+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-12T03:36:19.124308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.124308+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:36.079349+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.200226+0000","last_change":"2024-03-12T03:35:58.091300+0000","last_active":"2024-03-12T03:37:23.200226+0000","last_peered":"2024-03-12T03:37:23.200226+0000","last_clean":"2024-03-12T03:37:23.200226+0000","last_became_active":"2024-03-12T03:33:25.179022+0000","last_became_peered":"2024-03-12T03:33:25.179022+0000","last_unstale":"2024-03-12T03:37:23.200226+0000","last_undegraded":"2024-03-12T03:37:23.200226+0000","last_fullsized":"2024-03-12T03:37:23.200226+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:58.091182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.091182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"133'7719","reported_seq":7813,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:31.000928+0000","last_change":"2024-03-12T03:36:49.322938+0000","last_active":"2024-03-12T03:37:31.000928+0000","last_peered":"2024-03-12T03:37:31.000928+0000","last_clean":"2024-03-12T03:37:31.000928+0000","last_became_active":"2024-03-12T03:36:49.303137+0000","last_became_peered":"2024-03-12T03:36:49.303137+0000","last_unstale":"2024-03-12T03:37:31.000928+0000","last_undegraded":"2024-03-12T03:37:31.000928+0000","last_fullsized":"2024-03-12T03:37:31.000928+0000","mapping_epoch":97,"log_start":"101'1100","ondisk_log_start":"101'1100","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6619,"log_dups_size":0,"ondisk_log_size":6619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:32.638880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191365120,"num_objects":2920,"num_object_clones":0,"num_object_copies":8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4370,"num_write_kb":233280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":152,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499909+0000","last_change":"2024-03-12T03:36:25.119508+0000","last_active":"2024-03-12T03:37:35.499909+0000","last_peered":"2024-03-12T03:37:35.499909+0000","last_clean":"2024-03-12T03:37:35.499909+0000","last_became_active":"2024-03-12T03:33:27.188551+0000","last_became_peered":"2024-03-12T03:33:27.188551+0000","last_unstale":"2024-03-12T03:37:35.499909+0000","last_undegraded":"2024-03-12T03:37:35.499909+0000","last_fullsized":"2024-03-12T03:37:35.499909+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:25.119358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.119358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:54.029889+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":412,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.500561+0000","last_change":"2024-03-12T03:34:58.903289+0000","last_active":"2024-03-12T03:37:35.500561+0000","last_peered":"2024-03-12T03:37:35.500561+0000","last_clean":"2024-03-12T03:37:35.500561+0000","last_became_active":"2024-03-12T03:33:11.319888+0000","last_became_peered":"2024-03-12T03:33:11.319888+0000","last_unstale":"2024-03-12T03:37:35.500561+0000","last_undegraded":"2024-03-12T03:37:35.500561+0000","last_fullsized":"2024-03-12T03:37:35.500561+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:34:58.903163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:34:58.903163+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.210028+0000","last_change":"2024-03-12T03:35:14.996451+0000","last_active":"2024-03-12T03:37:23.210028+0000","last_peered":"2024-03-12T03:37:23.210028+0000","last_clean":"2024-03-12T03:37:23.210028+0000","last_became_active":"2024-03-12T03:33:21.100900+0000","last_became_peered":"2024-03-12T03:33:21.100900+0000","last_unstale":"2024-03-12T03:37:23.210028+0000","last_undegraded":"2024-03-12T03:37:23.210028+0000","last_fullsized":"2024-03-12T03:37:23.210028+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:14.996357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:14.996357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":232,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.499065+0000","last_change":"2024-03-12T03:36:03.118128+0000","last_active":"2024-03-12T03:37:35.499065+0000","last_peered":"2024-03-12T03:37:35.499065+0000","last_clean":"2024-03-12T03:37:35.499065+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:37:35.499065+0000","last_undegraded":"2024-03-12T03:37:35.499065+0000","last_fullsized":"2024-03-12T03:37:35.499065+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:36:03.117990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:03.117990+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":148,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.501149+0000","last_change":"2024-03-12T03:34:47.796670+0000","last_active":"2024-03-12T03:37:35.501149+0000","last_peered":"2024-03-12T03:37:35.501149+0000","last_clean":"2024-03-12T03:37:35.501149+0000","last_became_active":"2024-03-12T03:33:25.177762+0000","last_became_peered":"2024-03-12T03:33:25.177762+0000","last_unstale":"2024-03-12T03:37:35.501149+0000","last_undegraded":"2024-03-12T03:37:35.501149+0000","last_fullsized":"2024-03-12T03:37:35.501149+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.796561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.796561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"135'8007","reported_seq":8104,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.713149+0000","last_change":"2024-03-12T03:36:49.285675+0000","last_active":"2024-03-12T03:37:35.713149+0000","last_peered":"2024-03-12T03:37:35.713149+0000","last_clean":"2024-03-12T03:37:35.713149+0000","last_became_active":"2024-03-12T03:36:49.284030+0000","last_became_peered":"2024-03-12T03:36:49.284030+0000","last_unstale":"2024-03-12T03:37:35.713149+0000","last_undegraded":"2024-03-12T03:37:35.713149+0000","last_fullsized":"2024-03-12T03:37:35.713149+0000","mapping_epoch":97,"log_start":"101'1600","ondisk_log_start":"101'1600","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:39.292231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185139200,"num_objects":2825,"num_object_clones":0,"num_object_copies":8475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4613,"num_write_kb":238016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":148,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.207244+0000","last_change":"2024-03-12T03:36:15.042978+0000","last_active":"2024-03-12T03:37:23.207244+0000","last_peered":"2024-03-12T03:37:23.207244+0000","last_clean":"2024-03-12T03:37:23.207244+0000","last_became_active":"2024-03-12T03:33:27.191878+0000","last_became_peered":"2024-03-12T03:33:27.191878+0000","last_unstale":"2024-03-12T03:37:23.207244+0000","last_undegraded":"2024-03-12T03:37:23.207244+0000","last_fullsized":"2024-03-12T03:37:23.207244+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:15.042610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.042610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":153,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.205653+0000","last_change":"2024-03-12T03:35:05.715931+0000","last_active":"2024-03-12T03:37:23.205653+0000","last_peered":"2024-03-12T03:37:23.205653+0000","last_clean":"2024-03-12T03:37:23.205653+0000","last_became_active":"2024-03-12T03:33:21.097086+0000","last_became_peered":"2024-03-12T03:33:21.097086+0000","last_unstale":"2024-03-12T03:37:23.205653+0000","last_undegraded":"2024-03-12T03:37:23.205653+0000","last_fullsized":"2024-03-12T03:37:23.205653+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:05.715849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.715849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":426,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.500542+0000","last_change":"2024-03-12T03:36:08.059941+0000","last_active":"2024-03-12T03:37:35.500542+0000","last_peered":"2024-03-12T03:37:35.500542+0000","last_clean":"2024-03-12T03:37:35.500542+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:37:35.500542+0000","last_undegraded":"2024-03-12T03:37:35.500542+0000","last_fullsized":"2024-03-12T03:37:35.500542+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:36:08.059814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:36:08.059814+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":249,"num_read_kb":244,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":174,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.204159+0000","last_change":"2024-03-12T03:36:03.015860+0000","last_active":"2024-03-12T03:37:23.204159+0000","last_peered":"2024-03-12T03:37:23.204159+0000","last_clean":"2024-03-12T03:37:23.204159+0000","last_became_active":"2024-03-12T03:33:23.168750+0000","last_became_peered":"2024-03-12T03:33:23.168750+0000","last_unstale":"2024-03-12T03:37:23.204159+0000","last_undegraded":"2024-03-12T03:37:23.204159+0000","last_fullsized":"2024-03-12T03:37:23.204159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:36:03.015565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:03.015565+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":156,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.500332+0000","last_change":"2024-03-12T03:36:24.109073+0000","last_active":"2024-03-12T03:37:35.500332+0000","last_peered":"2024-03-12T03:37:35.500332+0000","last_clean":"2024-03-12T03:37:35.500332+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:37:35.500332+0000","last_undegraded":"2024-03-12T03:37:35.500332+0000","last_fullsized":"2024-03-12T03:37:35.500332+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:24.108984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.108984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.677961+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"135'7833","reported_seq":7930,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.766652+0000","last_change":"2024-03-12T03:36:49.288153+0000","last_active":"2024-03-12T03:37:35.766652+0000","last_peered":"2024-03-12T03:37:35.766652+0000","last_clean":"2024-03-12T03:37:35.766652+0000","last_became_active":"2024-03-12T03:36:49.286111+0000","last_became_peered":"2024-03-12T03:36:49.286111+0000","last_unstale":"2024-03-12T03:37:35.766652+0000","last_undegraded":"2024-03-12T03:37:35.766652+0000","last_fullsized":"2024-03-12T03:37:35.766652+0000","mapping_epoch":97,"log_start":"100'800","ondisk_log_start":"100'800","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":7033,"log_dups_size":0,"ondisk_log_size":7033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:36.086898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178814976,"num_objects":2728,"num_object_clones":0,"num_object_copies":8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4498,"num_write_kb":231264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":148,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.203139+0000","last_change":"2024-03-12T03:36:18.066969+0000","last_active":"2024-03-12T03:37:23.203139+0000","last_peered":"2024-03-12T03:37:23.203139+0000","last_clean":"2024-03-12T03:37:23.203139+0000","last_became_active":"2024-03-12T03:33:27.380458+0000","last_became_peered":"2024-03-12T03:33:27.380458+0000","last_unstale":"2024-03-12T03:37:23.203139+0000","last_undegraded":"2024-03-12T03:37:23.203139+0000","last_fullsized":"2024-03-12T03:37:23.203139+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.066712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.066712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:47.296061+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":168,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.207443+0000","last_change":"2024-03-12T03:37:19.405077+0000","last_active":"2024-03-12T03:37:23.207443+0000","last_peered":"2024-03-12T03:37:23.207443+0000","last_clean":"2024-03-12T03:37:23.207443+0000","last_became_active":"2024-03-12T03:33:21.573003+0000","last_became_peered":"2024-03-12T03:33:21.573003+0000","last_unstale":"2024-03-12T03:37:23.207443+0000","last_undegraded":"2024-03-12T03:37:23.207443+0000","last_fullsized":"2024-03-12T03:37:23.207443+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:19.404884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:37:19.404884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:22.217330+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":204,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.204480+0000","last_change":"2024-03-12T03:36:22.118169+0000","last_active":"2024-03-12T03:37:23.204480+0000","last_peered":"2024-03-12T03:37:23.204480+0000","last_clean":"2024-03-12T03:37:23.204480+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:37:23.204480+0000","last_undegraded":"2024-03-12T03:37:23.204480+0000","last_fullsized":"2024-03-12T03:37:23.204480+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:36:22.118034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:22.118034+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":156,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.498443+0000","last_change":"2024-03-12T03:36:21.140906+0000","last_active":"2024-03-12T03:37:35.498443+0000","last_peered":"2024-03-12T03:37:35.498443+0000","last_clean":"2024-03-12T03:37:35.498443+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:37:35.498443+0000","last_undegraded":"2024-03-12T03:37:35.498443+0000","last_fullsized":"2024-03-12T03:37:35.498443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:21.140825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:21.140825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"135'7812","reported_seq":7950,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.640466+0000","last_change":"2024-03-12T03:36:58.116826+0000","last_active":"2024-03-12T03:37:35.640466+0000","last_peered":"2024-03-12T03:37:35.640466+0000","last_clean":"2024-03-12T03:37:35.640466+0000","last_became_active":"2024-03-12T03:36:49.303476+0000","last_became_peered":"2024-03-12T03:36:49.303476+0000","last_unstale":"2024-03-12T03:37:35.640466+0000","last_undegraded":"2024-03-12T03:37:35.640466+0000","last_fullsized":"2024-03-12T03:37:35.640466+0000","mapping_epoch":97,"log_start":"101'1800","ondisk_log_start":"101'1800","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'6867","last_scrub_stamp":"2024-03-12T03:36:58.116574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:36:58.116574+0000","objects_scrubbed":3562,"log_size":6012,"log_dups_size":0,"ondisk_log_size":6012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:04.811799+0000","scrub_duration":900,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183894016,"num_objects":2806,"num_object_clones":0,"num_object_copies":8418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":960,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":149,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.196415+0000","last_change":"2024-03-12T03:36:20.138118+0000","last_active":"2024-03-12T03:37:23.196415+0000","last_peered":"2024-03-12T03:37:23.196415+0000","last_clean":"2024-03-12T03:37:23.196415+0000","last_became_active":"2024-03-12T03:33:27.194266+0000","last_became_peered":"2024-03-12T03:33:27.194266+0000","last_unstale":"2024-03-12T03:37:23.196415+0000","last_undegraded":"2024-03-12T03:37:23.196415+0000","last_fullsized":"2024-03-12T03:37:23.196415+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:20.138004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.138004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:34.538433+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":165,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.498859+0000","last_change":"2024-03-12T03:36:17.238474+0000","last_active":"2024-03-12T03:37:35.498859+0000","last_peered":"2024-03-12T03:37:35.498859+0000","last_clean":"2024-03-12T03:37:35.498859+0000","last_became_active":"2024-03-12T03:33:21.573251+0000","last_became_peered":"2024-03-12T03:33:21.573251+0000","last_unstale":"2024-03-12T03:37:35.498859+0000","last_undegraded":"2024-03-12T03:37:35.498859+0000","last_fullsized":"2024-03-12T03:37:35.498859+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:17.238386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:17.238386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:42.056483+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":183,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.206028+0000","last_change":"2024-03-12T03:36:07.967637+0000","last_active":"2024-03-12T03:37:23.206028+0000","last_peered":"2024-03-12T03:37:23.206028+0000","last_clean":"2024-03-12T03:37:23.206028+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:37:23.206028+0000","last_undegraded":"2024-03-12T03:37:23.206028+0000","last_fullsized":"2024-03-12T03:37:23.206028+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:36:07.967492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.967492+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:29.190972+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":210,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.204040+0000","last_change":"2024-03-12T03:36:06.069994+0000","last_active":"2024-03-12T03:37:23.204040+0000","last_peered":"2024-03-12T03:37:23.204040+0000","last_clean":"2024-03-12T03:37:23.204040+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:37:23.204040+0000","last_undegraded":"2024-03-12T03:37:23.204040+0000","last_fullsized":"2024-03-12T03:37:23.204040+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:06.069862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.069862+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"133'7731","reported_seq":7827,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:31.009269+0000","last_change":"2024-03-12T03:36:49.340201+0000","last_active":"2024-03-12T03:37:31.009269+0000","last_peered":"2024-03-12T03:37:31.009269+0000","last_clean":"2024-03-12T03:37:31.009269+0000","last_became_active":"2024-03-12T03:36:49.321727+0000","last_became_peered":"2024-03-12T03:36:49.321727+0000","last_unstale":"2024-03-12T03:37:31.009269+0000","last_undegraded":"2024-03-12T03:37:31.009269+0000","last_fullsized":"2024-03-12T03:37:31.009269+0000","mapping_epoch":97,"log_start":"101'2100","ondisk_log_start":"101'2100","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5631,"log_dups_size":0,"ondisk_log_size":5631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:34.238992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191954944,"num_objects":2929,"num_object_clones":0,"num_object_copies":8787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4369,"num_write_kb":233536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":153,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.501384+0000","last_change":"2024-03-12T03:36:19.049860+0000","last_active":"2024-03-12T03:37:35.501384+0000","last_peered":"2024-03-12T03:37:35.501384+0000","last_clean":"2024-03-12T03:37:35.501384+0000","last_became_active":"2024-03-12T03:33:27.383249+0000","last_became_peered":"2024-03-12T03:33:27.383249+0000","last_unstale":"2024-03-12T03:37:35.501384+0000","last_undegraded":"2024-03-12T03:37:35.501384+0000","last_fullsized":"2024-03-12T03:37:35.501384+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.049740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.049740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:43.452600+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":161,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.204270+0000","last_change":"2024-03-12T03:36:11.217211+0000","last_active":"2024-03-12T03:37:23.204270+0000","last_peered":"2024-03-12T03:37:23.204270+0000","last_clean":"2024-03-12T03:37:23.204270+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:37:23.204270+0000","last_undegraded":"2024-03-12T03:37:23.204270+0000","last_fullsized":"2024-03-12T03:37:23.204270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:11.216960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:11.216960+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:38.390046+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":194,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.496543+0000","last_change":"2024-03-12T03:37:03.159318+0000","last_active":"2024-03-12T03:37:35.496543+0000","last_peered":"2024-03-12T03:37:35.496543+0000","last_clean":"2024-03-12T03:37:35.496543+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:37:35.496543+0000","last_undegraded":"2024-03-12T03:37:35.496543+0000","last_fullsized":"2024-03-12T03:37:35.496543+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-03-12T03:37:03.159190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:37:03.159190+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:28.146961+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":203,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:26.023371+0000","last_change":"2024-03-12T03:34:25.168128+0000","last_active":"2024-03-12T03:37:26.023371+0000","last_peered":"2024-03-12T03:37:26.023371+0000","last_clean":"2024-03-12T03:37:26.023371+0000","last_became_active":"2024-03-12T03:33:25.177311+0000","last_became_peered":"2024-03-12T03:33:25.177311+0000","last_unstale":"2024-03-12T03:37:26.023371+0000","last_undegraded":"2024-03-12T03:37:26.023371+0000","last_fullsized":"2024-03-12T03:37:26.023371+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:25.168002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.168002+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"133'7763","reported_seq":7855,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:31.017069+0000","last_change":"2024-03-12T03:36:49.332704+0000","last_active":"2024-03-12T03:37:31.017069+0000","last_peered":"2024-03-12T03:37:31.017069+0000","last_clean":"2024-03-12T03:37:31.017069+0000","last_became_active":"2024-03-12T03:36:49.327962+0000","last_became_peered":"2024-03-12T03:36:49.327962+0000","last_unstale":"2024-03-12T03:37:31.017069+0000","last_undegraded":"2024-03-12T03:37:31.017069+0000","last_fullsized":"2024-03-12T03:37:31.017069+0000","mapping_epoch":97,"log_start":"101'2100","ondisk_log_start":"101'2100","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5663,"log_dups_size":0,"ondisk_log_size":5663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:34.239033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191496192,"num_objects":2922,"num_object_clones":0,"num_object_copies":8766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4388,"num_write_kb":233920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":153,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.495832+0000","last_change":"2024-03-12T03:37:01.181373+0000","last_active":"2024-03-12T03:37:35.495832+0000","last_peered":"2024-03-12T03:37:35.495832+0000","last_clean":"2024-03-12T03:37:35.495832+0000","last_became_active":"2024-03-12T03:33:27.187864+0000","last_became_peered":"2024-03-12T03:33:27.187864+0000","last_unstale":"2024-03-12T03:37:35.495832+0000","last_undegraded":"2024-03-12T03:37:35.495832+0000","last_fullsized":"2024-03-12T03:37:35.495832+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:01.181261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:37:01.181261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:22.886398+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.213964+0000","last_change":"2024-03-12T03:34:47.155013+0000","last_active":"2024-03-12T03:37:23.213964+0000","last_peered":"2024-03-12T03:37:23.213964+0000","last_clean":"2024-03-12T03:37:23.213964+0000","last_became_active":"2024-03-12T03:33:21.104430+0000","last_became_peered":"2024-03-12T03:33:21.104430+0000","last_unstale":"2024-03-12T03:37:23.213964+0000","last_undegraded":"2024-03-12T03:37:23.213964+0000","last_fullsized":"2024-03-12T03:37:23.213964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.154925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.154925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":205,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.210472+0000","last_change":"2024-03-12T03:36:06.083361+0000","last_active":"2024-03-12T03:37:23.210472+0000","last_peered":"2024-03-12T03:37:23.210472+0000","last_clean":"2024-03-12T03:37:23.210472+0000","last_became_active":"2024-03-12T03:33:23.551623+0000","last_became_peered":"2024-03-12T03:33:23.551623+0000","last_unstale":"2024-03-12T03:37:23.210472+0000","last_undegraded":"2024-03-12T03:37:23.210472+0000","last_fullsized":"2024-03-12T03:37:23.210472+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-03-12T03:36:06.083052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.083052+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:36.041591+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":157,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.495821+0000","last_change":"2024-03-12T03:37:08.194374+0000","last_active":"2024-03-12T03:37:35.495821+0000","last_peered":"2024-03-12T03:37:35.495821+0000","last_clean":"2024-03-12T03:37:35.495821+0000","last_became_active":"2024-03-12T03:33:25.181020+0000","last_became_peered":"2024-03-12T03:33:25.181020+0000","last_unstale":"2024-03-12T03:37:35.495821+0000","last_undegraded":"2024-03-12T03:37:35.495821+0000","last_fullsized":"2024-03-12T03:37:35.495821+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:08.194302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:37:08.194302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:34.384971+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"135'9044","reported_seq":9133,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.803787+0000","last_change":"2024-03-12T03:35:23.785403+0000","last_active":"2024-03-12T03:37:35.803787+0000","last_peered":"2024-03-12T03:37:35.803787+0000","last_clean":"2024-03-12T03:37:35.803787+0000","last_became_active":"2024-03-12T03:35:23.785037+0000","last_became_peered":"2024-03-12T03:35:23.785037+0000","last_unstale":"2024-03-12T03:37:35.803787+0000","last_undegraded":"2024-03-12T03:37:35.803787+0000","last_fullsized":"2024-03-12T03:37:35.803787+0000","mapping_epoch":97,"log_start":"106'3377","ondisk_log_start":"106'3377","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375259136,"num_objects":5726,"num_object_clones":0,"num_object_copies":17178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":472640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":144,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.498533+0000","last_change":"2024-03-12T03:35:17.142169+0000","last_active":"2024-03-12T03:37:35.498533+0000","last_peered":"2024-03-12T03:37:35.498533+0000","last_clean":"2024-03-12T03:37:35.498533+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:37:35.498533+0000","last_undegraded":"2024-03-12T03:37:35.498533+0000","last_fullsized":"2024-03-12T03:37:35.498533+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:17.142086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:17.142086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":215,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.496918+0000","last_change":"2024-03-12T03:37:22.125851+0000","last_active":"2024-03-12T03:37:35.496918+0000","last_peered":"2024-03-12T03:37:35.496918+0000","last_clean":"2024-03-12T03:37:35.496918+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:37:35.496918+0000","last_undegraded":"2024-03-12T03:37:35.496918+0000","last_fullsized":"2024-03-12T03:37:35.496918+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:37:22.125684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:37:22.125684+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:33.045170+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.210346+0000","last_change":"2024-03-12T03:35:13.030518+0000","last_active":"2024-03-12T03:37:23.210346+0000","last_peered":"2024-03-12T03:37:23.210346+0000","last_clean":"2024-03-12T03:37:23.210346+0000","last_became_active":"2024-03-12T03:33:21.104293+0000","last_became_peered":"2024-03-12T03:33:21.104293+0000","last_unstale":"2024-03-12T03:37:23.210346+0000","last_undegraded":"2024-03-12T03:37:23.210346+0000","last_fullsized":"2024-03-12T03:37:23.210346+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:13.030426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:13.030426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.200441+0000","last_change":"2024-03-12T03:36:24.342489+0000","last_active":"2024-03-12T03:37:23.200441+0000","last_peered":"2024-03-12T03:37:23.200441+0000","last_clean":"2024-03-12T03:37:23.200441+0000","last_became_active":"2024-03-12T03:33:50.153240+0000","last_became_peered":"2024-03-12T03:33:50.153240+0000","last_unstale":"2024-03-12T03:37:23.200441+0000","last_undegraded":"2024-03-12T03:37:23.200441+0000","last_fullsized":"2024-03-12T03:37:23.200441+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:36:24.342356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.342356+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:26.628737+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"135'9073","reported_seq":9163,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.767872+0000","last_change":"2024-03-12T03:35:23.783266+0000","last_active":"2024-03-12T03:37:35.767872+0000","last_peered":"2024-03-12T03:37:35.767872+0000","last_clean":"2024-03-12T03:37:35.767872+0000","last_became_active":"2024-03-12T03:35:23.782945+0000","last_became_peered":"2024-03-12T03:35:23.782945+0000","last_unstale":"2024-03-12T03:37:35.767872+0000","last_undegraded":"2024-03-12T03:37:35.767872+0000","last_fullsized":"2024-03-12T03:37:35.767872+0000","mapping_epoch":97,"log_start":"106'3349","ondisk_log_start":"106'3349","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5724,"log_dups_size":0,"ondisk_log_size":5724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374210584,"num_objects":5711,"num_object_clones":0,"num_object_copies":17133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5711,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9073,"num_write_kb":473025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":140,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.213183+0000","last_change":"2024-03-12T03:36:13.114566+0000","last_active":"2024-03-12T03:37:23.213183+0000","last_peered":"2024-03-12T03:37:23.213183+0000","last_clean":"2024-03-12T03:37:23.213183+0000","last_became_active":"2024-03-12T03:33:27.380896+0000","last_became_peered":"2024-03-12T03:33:27.380896+0000","last_unstale":"2024-03-12T03:37:23.213183+0000","last_undegraded":"2024-03-12T03:37:23.213183+0000","last_fullsized":"2024-03-12T03:37:23.213183+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:13.114417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.114417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:37.138551+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":156,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.500286+0000","last_change":"2024-03-12T03:36:19.182812+0000","last_active":"2024-03-12T03:37:35.500286+0000","last_peered":"2024-03-12T03:37:35.500286+0000","last_clean":"2024-03-12T03:37:35.500286+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:37:35.500286+0000","last_undegraded":"2024-03-12T03:37:35.500286+0000","last_fullsized":"2024-03-12T03:37:35.500286+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.182685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.182685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":156,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.495869+0000","last_change":"2024-03-12T03:36:13.157358+0000","last_active":"2024-03-12T03:37:35.495869+0000","last_peered":"2024-03-12T03:37:35.495869+0000","last_clean":"2024-03-12T03:37:35.495869+0000","last_became_active":"2024-03-12T03:33:21.573257+0000","last_became_peered":"2024-03-12T03:33:21.573257+0000","last_unstale":"2024-03-12T03:37:35.495869+0000","last_undegraded":"2024-03-12T03:37:35.495869+0000","last_fullsized":"2024-03-12T03:37:35.495869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:13.157235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.157235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"95'23","reported_seq":197,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.496820+0000","last_change":"2024-03-12T03:37:05.133318+0000","last_active":"2024-03-12T03:37:35.496820+0000","last_peered":"2024-03-12T03:37:35.496820+0000","last_clean":"2024-03-12T03:37:35.496820+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:37:35.496820+0000","last_undegraded":"2024-03-12T03:37:35.496820+0000","last_fullsized":"2024-03-12T03:37:35.496820+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'23","last_scrub_stamp":"2024-03-12T03:37:05.133198+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:37:05.133198+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:19.791725+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"133'8948","reported_seq":9034,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:31.061798+0000","last_change":"2024-03-12T03:35:23.777241+0000","last_active":"2024-03-12T03:37:31.061798+0000","last_peered":"2024-03-12T03:37:31.061798+0000","last_clean":"2024-03-12T03:37:31.061798+0000","last_became_active":"2024-03-12T03:35:23.776932+0000","last_became_peered":"2024-03-12T03:35:23.776932+0000","last_unstale":"2024-03-12T03:37:31.061798+0000","last_undegraded":"2024-03-12T03:37:31.061798+0000","last_fullsized":"2024-03-12T03:37:31.061798+0000","mapping_epoch":97,"log_start":"105'3128","ondisk_log_start":"105'3128","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5820,"log_dups_size":0,"ondisk_log_size":5820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386531328,"num_objects":5898,"num_object_clones":0,"num_object_copies":17694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8948,"num_write_kb":475072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":148,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.200807+0000","last_change":"2024-03-12T03:35:56.071430+0000","last_active":"2024-03-12T03:37:23.200807+0000","last_peered":"2024-03-12T03:37:23.200807+0000","last_clean":"2024-03-12T03:37:23.200807+0000","last_became_active":"2024-03-12T03:33:27.187815+0000","last_became_peered":"2024-03-12T03:33:27.187815+0000","last_unstale":"2024-03-12T03:37:23.200807+0000","last_undegraded":"2024-03-12T03:37:23.200807+0000","last_fullsized":"2024-03-12T03:37:23.200807+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:56.071324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:56.071324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.207577+0000","last_change":"2024-03-12T03:36:04.907090+0000","last_active":"2024-03-12T03:37:23.207577+0000","last_peered":"2024-03-12T03:37:23.207577+0000","last_clean":"2024-03-12T03:37:23.207577+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:37:23.207577+0000","last_undegraded":"2024-03-12T03:37:23.207577+0000","last_fullsized":"2024-03-12T03:37:23.207577+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:04.906798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:04.906798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:31.071522+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":163,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.498158+0000","last_change":"2024-03-12T03:35:59.016620+0000","last_active":"2024-03-12T03:37:35.498158+0000","last_peered":"2024-03-12T03:37:35.498158+0000","last_clean":"2024-03-12T03:37:35.498158+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:37:35.498158+0000","last_undegraded":"2024-03-12T03:37:35.498158+0000","last_fullsized":"2024-03-12T03:37:35.498158+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.016489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.016489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":180,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.496861+0000","last_change":"2024-03-12T03:35:59.053600+0000","last_active":"2024-03-12T03:37:35.496861+0000","last_peered":"2024-03-12T03:37:35.496861+0000","last_clean":"2024-03-12T03:37:35.496861+0000","last_became_active":"2024-03-12T03:33:23.163871+0000","last_became_peered":"2024-03-12T03:33:23.163871+0000","last_unstale":"2024-03-12T03:37:35.496861+0000","last_undegraded":"2024-03-12T03:37:35.496861+0000","last_fullsized":"2024-03-12T03:37:35.496861+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:35:59.053486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.053486+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"133'8924","reported_seq":9010,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:30.992006+0000","last_change":"2024-03-12T03:35:23.784927+0000","last_active":"2024-03-12T03:37:30.992006+0000","last_peered":"2024-03-12T03:37:30.992006+0000","last_clean":"2024-03-12T03:37:30.992006+0000","last_became_active":"2024-03-12T03:35:23.784627+0000","last_became_peered":"2024-03-12T03:35:23.784627+0000","last_unstale":"2024-03-12T03:37:30.992006+0000","last_undegraded":"2024-03-12T03:37:30.992006+0000","last_fullsized":"2024-03-12T03:37:30.992006+0000","mapping_epoch":97,"log_start":"106'3961","ondisk_log_start":"106'3961","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4963,"log_dups_size":0,"ondisk_log_size":4963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385744896,"num_objects":5886,"num_object_clones":0,"num_object_copies":17658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8924,"num_write_kb":473920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":144,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.497016+0000","last_change":"2024-03-12T03:36:07.101180+0000","last_active":"2024-03-12T03:37:35.497016+0000","last_peered":"2024-03-12T03:37:35.497016+0000","last_clean":"2024-03-12T03:37:35.497016+0000","last_became_active":"2024-03-12T03:33:27.184405+0000","last_became_peered":"2024-03-12T03:33:27.184405+0000","last_unstale":"2024-03-12T03:37:35.497016+0000","last_undegraded":"2024-03-12T03:37:35.497016+0000","last_fullsized":"2024-03-12T03:37:35.497016+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:07.101098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.101098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":210,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.199501+0000","last_change":"2024-03-12T03:36:25.346270+0000","last_active":"2024-03-12T03:37:23.199501+0000","last_peered":"2024-03-12T03:37:23.199501+0000","last_clean":"2024-03-12T03:37:23.199501+0000","last_became_active":"2024-03-12T03:33:25.177613+0000","last_became_peered":"2024-03-12T03:33:25.177613+0000","last_unstale":"2024-03-12T03:37:23.199501+0000","last_undegraded":"2024-03-12T03:37:23.199501+0000","last_fullsized":"2024-03-12T03:37:23.199501+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:25.346151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.346151+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:37:44.762442+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":152,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-12T03:37:23.209828+0000","last_change":"2024-03-12T03:35:15.981464+0000","last_active":"2024-03-12T03:37:23.209828+0000","last_peered":"2024-03-12T03:37:23.209828+0000","last_clean":"2024-03-12T03:37:23.209828+0000","last_became_active":"2024-03-12T03:33:21.097304+0000","last_became_peered":"2024-03-12T03:33:21.097304+0000","last_unstale":"2024-03-12T03:37:23.209828+0000","last_undegraded":"2024-03-12T03:37:23.209828+0000","last_fullsized":"2024-03-12T03:37:23.209828+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.981378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.981378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":167,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-12T03:37:35.496300+0000","last_change":"2024-03-12T03:35:58.089544+0000","last_active":"2024-03-12T03:37:35.496300+0000","last_peered":"2024-03-12T03:37:35.496300+0000","last_clean":"2024-03-12T03:37:35.496300+0000","last_became_active":"2024-03-12T03:33:23.162657+0000","last_became_peered":"2024-03-12T03:33:23.162657+0000","last_unstale":"2024-03-12T03:37:35.496300+0000","last_undegraded":"2024-03-12T03:37:35.496300+0000","last_fullsized":"2024-03-12T03:37:35.496300+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-12T03:35:58.089405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.089405+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":6077022232,"num_objects":92729,"num_object_clones":0,"num_object_copies":278187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33307,"num_objects_unfound":0,"num_objects_dirty":92729,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138613,"num_write_kb":7288897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13140,"num_bytes_recovered":861143040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19527184384,"data_stored":19527172168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162598,"ondisk_log_size":162598,"up":78,"acting":77,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":249,"num_read_kb":244,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":630,"ondisk_log_size":630,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070520,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2322888,"kb_used_data":2175104,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":91422264,"statfs":{"total":95995035648,"available":93616398336,"internally_reserved":0,"allocated":2227306496,"data_stored":2226755466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":151321034},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332157,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2160376,"kb_used_data":2026476,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":91584776,"statfs":{"total":95995035648,"available":93782810624,"internally_reserved":0,"allocated":2075111424,"data_stored":2074578935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":137099718},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2575756,"kb_used_data":2436292,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91169396,"statfs":{"total":95995035648,"available":93357461504,"internally_reserved":0,"allocated":2494763008,"data_stored":2494199916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2986240,"kb_used_data":2856376,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":90758912,"statfs":{"total":95995035648,"available":92937125888,"internally_reserved":0,"allocated":2924929024,"data_stored":2924378351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2818276,"kb_used_data":2682972,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":90926876,"statfs":{"total":95995035648,"available":93109121024,"internally_reserved":0,"allocated":2747363328,"data_stored":2746840775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117066,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3509640,"kb_used_data":3388224,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":90235512,"statfs":{"total":95995035648,"available":92401164288,"internally_reserved":0,"allocated":3469541376,"data_stored":3468982053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124320199},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953544,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2029772,"kb_used_data":1936128,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":91715380,"statfs":{"total":95995035648,"available":93916549120,"internally_reserved":0,"allocated":1982595072,"data_stored":1982043038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659202,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1713536,"kb_used_data":1610164,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92031616,"statfs":{"total":95995035648,"available":94240374784,"internally_reserved":0,"allocated":1648807936,"data_stored":1648267358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2916290560,"data_stored":2916286488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2488012800,"data_stored":2488008728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2743140352,"data_stored":2743140352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3449815040,"data_stored":3449815040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1982201856,"data_stored":1982201856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1650851840,"data_stored":1650851840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2069237760,"data_stored":2069233688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2227634176,"data_stored":2227634176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:37:45.254 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-12T03:37:45.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-12T03:37:45.268 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:45.339 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:45.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:45.440 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:45.558 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:45.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:45.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:45.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:45.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:45.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:45.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:45.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:45.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:45.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:45 smithi169 ceph-mon[29344]: pgmap v169: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 149 active+clean, 5 active+remapped+backfilling; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 727 op/s; 24762/261429 objects misplaced (9.472%); 31 MiB/s, 488 objects/s recovering 2024-03-12T03:37:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:45 smithi169 ceph-mon[29344]: osdmap e138: 8 total, 8 up, 8 in 2024-03-12T03:37:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:45 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:45 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:45.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:45 smithi169 ceph-mon[39949]: pgmap v169: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 149 active+clean, 5 active+remapped+backfilling; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 727 op/s; 24762/261429 objects misplaced (9.472%); 31 MiB/s, 488 objects/s recovering 2024-03-12T03:37:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:45 smithi169 ceph-mon[39949]: osdmap e138: 8 total, 8 up, 8 in 2024-03-12T03:37:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:45 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:45 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:45.944 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:45 smithi173 ceph-mon[33544]: pgmap v169: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 149 active+clean, 5 active+remapped+backfilling; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 727 op/s; 24762/261429 objects misplaced (9.472%); 31 MiB/s, 488 objects/s recovering 2024-03-12T03:37:45.945 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:45 smithi173 ceph-mon[33544]: osdmap e138: 8 total, 8 up, 8 in 2024-03-12T03:37:45.945 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:45 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:37:45.945 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:45 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:37:45.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:45.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:46.019 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:37:46.019 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:37:46.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:46.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:46.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:46.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:46.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:46.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:46.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:46.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:46.344 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:46.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:46.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:46.479 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:46.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:46.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:46.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:46.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:46.710 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:46.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:46.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:46.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-12T03:37:46.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:46.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:46.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:46.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:46 smithi173 ceph-mon[33544]: osdmap e139: 8 total, 8 up, 8 in 2024-03-12T03:37:46.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:47.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:47.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:47.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:47.131 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:47.155 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:47.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:47.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:47.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:46 smithi169 ceph-mon[29344]: osdmap e139: 8 total, 8 up, 8 in 2024-03-12T03:37:47.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:46 smithi169 ceph-mon[39949]: osdmap e139: 8 total, 8 up, 8 in 2024-03-12T03:37:47.203 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:37:47.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:47.290 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:47.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:47.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:47.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:47.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:47.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:47.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:47.550 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:47.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:47.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:47.701 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:47.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:47.803 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:47.829 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:47.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:47.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:47.976 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:47.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:47.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:47 smithi173 ceph-mon[33544]: pgmap v172: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 150 active+clean, 4 active+remapped+backfilling; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 880 op/s; 24202/257883 objects misplaced (9.385%); 31 MiB/s, 501 objects/s recovering 2024-03-12T03:37:47.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:47 smithi173 ceph-mon[33544]: osdmap e140: 8 total, 8 up, 8 in 2024-03-12T03:37:48.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:48.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:48.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:48.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:48.172 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:47 smithi169 ceph-mon[39949]: pgmap v172: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 150 active+clean, 4 active+remapped+backfilling; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 880 op/s; 24202/257883 objects misplaced (9.385%); 31 MiB/s, 501 objects/s recovering 2024-03-12T03:37:48.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:47 smithi169 ceph-mon[39949]: osdmap e140: 8 total, 8 up, 8 in 2024-03-12T03:37:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:47 smithi169 ceph-mon[29344]: pgmap v172: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 150 active+clean, 4 active+remapped+backfilling; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 880 op/s; 24202/257883 objects misplaced (9.385%); 31 MiB/s, 501 objects/s recovering 2024-03-12T03:37:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:47 smithi169 ceph-mon[29344]: osdmap e140: 8 total, 8 up, 8 in 2024-03-12T03:37:48.212 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:48.242 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:48.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:48.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:48.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:48.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:48.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:48.472 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:48.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:48.517 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:48.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:48.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:48.652 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:48.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:48.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:48.791 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:48.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:48.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:48.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:48.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:48.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:48.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:48 smithi173 ceph-mon[33544]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:37:48.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:48 smithi173 ceph-mon[33544]: osdmap e141: 8 total, 8 up, 8 in 2024-03-12T03:37:49.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:49.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:49.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:49.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:49.067 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:48 smithi169 ceph-mon[29344]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:37:49.067 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:48 smithi169 ceph-mon[29344]: osdmap e141: 8 total, 8 up, 8 in 2024-03-12T03:37:49.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:48 smithi169 ceph-mon[39949]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:37:49.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:48 smithi169 ceph-mon[39949]: osdmap e141: 8 total, 8 up, 8 in 2024-03-12T03:37:49.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:49.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:49.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:49.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:49.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:49.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:49.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:49.378 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:37:49.378 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:37:49.386 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:37:49.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:49.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:49.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:49.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:49.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:49.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:49.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:49.721 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:49.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:49.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:49.730 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:49 smithi169 ceph-mon[29344]: pgmap v174: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 150 active+clean, 4 active+remapped+backfilling; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 543 op/s; 20290/251655 objects misplaced (8.063%); 26 MiB/s, 411 objects/s recovering 2024-03-12T03:37:49.730 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:49 smithi169 ceph-mon[39949]: pgmap v174: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 150 active+clean, 4 active+remapped+backfilling; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 543 op/s; 20290/251655 objects misplaced (8.063%); 26 MiB/s, 411 objects/s recovering 2024-03-12T03:37:49.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:49.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:49.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:49.954 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:49.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:49.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:49.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:49.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:49 smithi173 ceph-mon[33544]: pgmap v174: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 150 active+clean, 4 active+remapped+backfilling; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 543 op/s; 20290/251655 objects misplaced (8.063%); 26 MiB/s, 411 objects/s recovering 2024-03-12T03:37:50.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:50.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:50.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:50.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:50.269 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:50.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:50.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:50.430 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:50.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:50.531 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:50.577 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:50.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:50.679 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:50.757 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:37:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:50.759 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:50.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:50.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:50.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:50.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:50.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:50.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:51.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:51.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:51.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:51.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:51.159 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:51.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:51.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:51.307 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:51.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:51.408 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:51.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:51.447 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:51.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:51.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:51.575 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:51.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:51.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:51.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:51.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:51.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:51.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:51.857 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:51.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:51.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:51.977 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:51 smithi173 ceph-mon[33544]: pgmap v176: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 150 active+clean, 4 active+remapped+backfilling; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 990 B/s rd, 0 B/s wr, 631 op/s; 20290/251655 objects misplaced (8.063%); 30 MiB/s, 478 objects/s recovering 2024-03-12T03:37:51.977 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:51 smithi173 ceph-mon[33544]: osdmap e142: 8 total, 8 up, 8 in 2024-03-12T03:37:51.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:51.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:51.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:52.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-12T03:37:52.028 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:37:52.030 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:51 smithi169 ceph-mon[29344]: pgmap v176: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 150 active+clean, 4 active+remapped+backfilling; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 990 B/s rd, 0 B/s wr, 631 op/s; 20290/251655 objects misplaced (8.063%); 30 MiB/s, 478 objects/s recovering 2024-03-12T03:37:52.030 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:51 smithi169 ceph-mon[29344]: osdmap e142: 8 total, 8 up, 8 in 2024-03-12T03:37:52.030 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:37:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:37:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:37:52.030 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:51 smithi169 ceph-mon[39949]: pgmap v176: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 150 active+clean, 4 active+remapped+backfilling; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 990 B/s rd, 0 B/s wr, 631 op/s; 20290/251655 objects misplaced (8.063%); 30 MiB/s, 478 objects/s recovering 2024-03-12T03:37:52.030 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:51 smithi169 ceph-mon[39949]: osdmap e142: 8 total, 8 up, 8 in 2024-03-12T03:37:52.069 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:37:52.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:52.112 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:37:52.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:52.137 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:52.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:52.155 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:37:52.197 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:37:52.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:52.251 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:37:52.299 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:37:52.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:52.299 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:52.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:52.340 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:37:52.345 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:37:52.345 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:37:52.345 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:37:52.346 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:37:52.346 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:37:52.346 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:37:52.346 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:37:52.346 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:37:52.385 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:37:52.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:52.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:52.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:52.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:52.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:52.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:52.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:52.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:52.682 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:37:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:37:52.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:52.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:52.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:52.868 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:52.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:52.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:52.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:52 smithi173 ceph-mon[33544]: osdmap e143: 8 total, 8 up, 8 in 2024-03-12T03:37:52.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:52 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1639318319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:37:53.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:53.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:53.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:53.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:53.145 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:53.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:53.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:52 smithi169 ceph-mon[29344]: osdmap e143: 8 total, 8 up, 8 in 2024-03-12T03:37:53.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:52 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1639318319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:37:53.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:52 smithi169 ceph-mon[39949]: osdmap e143: 8 total, 8 up, 8 in 2024-03-12T03:37:53.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:52 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1639318319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:37:53.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:53.276 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:53.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:53.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:53.408 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:53.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:53.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:53.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:53.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:53.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:53.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:53.685 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:53.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:53.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:53.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:53.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:53.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:53.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:53.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:53 smithi169 ceph-mon[29344]: pgmap v179: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 151 active+clean, 3 active+remapped+backfilling; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 994 B/s rd, 0 B/s wr, 649 op/s; 15906/241632 objects misplaced (6.583%); 28 MiB/s, 451 objects/s recovering 2024-03-12T03:37:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:53 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:53 smithi169 ceph-mon[29344]: 7.b deep-scrub ok 2024-03-12T03:37:53.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:53 smithi169 ceph-mon[39949]: pgmap v179: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 151 active+clean, 3 active+remapped+backfilling; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 994 B/s rd, 0 B/s wr, 649 op/s; 15906/241632 objects misplaced (6.583%); 28 MiB/s, 451 objects/s recovering 2024-03-12T03:37:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:53 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:53 smithi169 ceph-mon[39949]: 7.b deep-scrub ok 2024-03-12T03:37:53.929 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:53 smithi173 ceph-mon[33544]: pgmap v179: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 151 active+clean, 3 active+remapped+backfilling; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 994 B/s rd, 0 B/s wr, 649 op/s; 15906/241632 objects misplaced (6.583%); 28 MiB/s, 451 objects/s recovering 2024-03-12T03:37:53.929 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:53 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:53.929 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:53 smithi173 ceph-mon[33544]: 7.b deep-scrub ok 2024-03-12T03:37:53.949 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:53.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:54.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:54.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:54.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:54.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:54.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:54.228 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:54.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:54.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:54.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:54.363 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:54.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:54.396 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:37:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:37:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:37:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:37:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24359.0:1322 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94067641432704 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-03-12T03:37:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:33:49.138906+0000", 2024-03-12T03:37:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 245.25243332900001, 2024-03-12T03:37:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0179765549999999, 2024-03-12T03:37:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:37:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24359", 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 1322 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.138906+0000", 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.138906+0000", 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.138908+0000", 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6869999999999999e-06 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.138916+0000", 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8509999999999999e-06 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.138933+0000", 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7326999999999999e-05 2024-03-12T03:37:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.138943+0000", 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0159e-05 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.143672+0000", 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047292829999999999 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.143684+0000", 2024-03-12T03:37:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1768e-05 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.151966+0000", 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.008281574 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for missing object", 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.152130+0000", 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000164096 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.153140+0000", 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001010431 2024-03-12T03:37:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.153217+0000", 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6908000000000003e-05 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.153476+0000", 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025934299999999999 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.153537+0000", 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0319000000000003e-05 2024-03-12T03:37:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.156883+0000", 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033458089999999999 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:24244 7.0 7:0be9df2a:::benchmark_data_smithi169_79822_object24243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:40.731735+0000", 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 133.65960418099999, 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078767513999999997, 2024-03-12T03:37:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24244 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.731735+0000", 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.731735+0000", 2024-03-12T03:37:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.731736+0000", 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0890000000000001e-06 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.731811+0000", 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4771000000000006e-05 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.731833+0000", 2024-03-12T03:37:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2421000000000001e-05 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.731844+0000", 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0127e-05 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.731892+0000", 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8026999999999999e-05 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.731996+0000", 2024-03-12T03:37:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104492 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.732097+0000", 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010040899999999999 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.732926+0000", 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00082975200000000001 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.809824+0000", 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.076897451000000006 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.810397+0000", 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000573589 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.810449+0000", 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1214999999999997e-05 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:40.810503+0000", 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4171e-05 2024-03-12T03:37:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:25347 7.0 7:050e0962:::benchmark_data_smithi169_79822_object25346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:37:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:41.348191+0000", 2024-03-12T03:37:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 133.04314838100001, 2024-03-12T03:37:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.066860790000000003, 2024-03-12T03:37:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25347 2024-03-12T03:37:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348191+0000", 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348191+0000", 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348192+0000", 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4950000000000001e-06 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348238+0000", 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5850999999999999e-05 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348252+0000", 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3781000000000001e-05 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348260+0000", 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8979999999999996e-06 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348317+0000", 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7031999999999999e-05 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348434+0000", 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011733799999999999 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348573+0000", 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013812399999999999 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.350639+0000", 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020669199999999999 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.351985+0000", 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00134581 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.414957+0000", 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.062972034999999996 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.415010+0000", 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3164000000000001e-05 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.415052+0000", 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1341999999999998e-05 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:25349 7.0 7:0e482728:::benchmark_data_smithi169_79822_object25348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:41.348384+0000", 2024-03-12T03:37:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 133.04295562999999, 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.066790672999999995, 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25349 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348384+0000", 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348384+0000", 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348385+0000", 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0200000000000001e-07 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348492+0000", 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010715199999999999 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348502+0000", 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0713999999999999e-05 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.348510+0000", 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9740000000000002e-06 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.349556+0000", 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010456549999999999 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.349673+0000", 2024-03-12T03:37:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011664599999999999 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.349812+0000", 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000139037 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.350717+0000", 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090548400000000002 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.352763+0000", 2024-03-12T03:37:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020455769999999998 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.415098+0000", 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.062335138999999998 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.415134+0000", 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6338999999999999e-05 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.415174+0000", 2024-03-12T03:37:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0154000000000002e-05 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:25352 7.0 7:0032bd4c:::benchmark_data_smithi169_79822_object25351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:41.351026+0000", 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 133.04031374300001, 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.064356654999999999, 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25352 2024-03-12T03:37:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.351026+0000", 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.351026+0000", 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.351027+0000", 2024-03-12T03:37:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8299999999999995e-07 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.351108+0000", 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1317999999999996e-05 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.351118+0000", 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0307e-05 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.351126+0000", 2024-03-12T03:37:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2500000000000006e-06 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.351188+0000", 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1487000000000003e-05 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.351267+0000", 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8998000000000005e-05 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.351380+0000", 2024-03-12T03:37:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011304299999999999 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.352127+0000", 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00074667500000000003 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.352859+0000", 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073233100000000002 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.415226+0000", 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.062367365000000001 2024-03-12T03:37:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.415285+0000", 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8610000000000003e-05 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:41.415382+0000", 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7387999999999997e-05 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:29236 7.0 7:065c5898:::benchmark_data_smithi169_79822_object29235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:37:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:43.518470+0000", 2024-03-12T03:37:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 130.87286891799999, 2024-03-12T03:37:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13041738999999999, 2024-03-12T03:37:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29236 2024-03-12T03:37:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518470+0000", 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518470+0000", 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518480+0000", 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4930000000000004e-06 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518535+0000", 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4787999999999998e-05 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518551+0000", 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5985e-05 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518558+0000", 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7389999999999999e-06 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518614+0000", 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5525000000000001e-05 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518718+0000", 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103692 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518806+0000", 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7848000000000003e-05 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.520467+0000", 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016614310000000001 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.523282+0000", 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028148890000000001 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.648772+0000", 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12549065300000001 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.648840+0000", 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7738999999999996e-05 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.648888+0000", 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7608000000000002e-05 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:29239 7.0 7:0bde76fb:::benchmark_data_smithi169_79822_object29238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:37:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:43.518868+0000", 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 130.872471107, 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.130465672, 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29239 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518868+0000", 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518868+0000", 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518869+0000", 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3900000000000004e-07 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518953+0000", 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3907999999999997e-05 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518971+0000", 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8162999999999999e-05 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518979+0000", 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8050000000000007e-06 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.519707+0000", 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00072848100000000001 2024-03-12T03:37:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.519786+0000", 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8532e-05 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.519881+0000", 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4783000000000004e-05 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.520596+0000", 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000715432 2024-03-12T03:37:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.523409+0000", 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028132949999999999 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649240+0000", 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12583079 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649295+0000", 2024-03-12T03:37:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4503999999999999e-05 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649334+0000", 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9339999999999999e-05 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:29673 7.0 7:0e6d0047:::benchmark_data_smithi169_79822_object29672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:43.913876+0000", 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 130.477462944, 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.086770059999999996, 2024-03-12T03:37:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29673 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.913876+0000", 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.913876+0000", 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.913879+0000", 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.091e-06 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.913941+0000", 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2853000000000003e-05 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.913954+0000", 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2853e-05 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.913967+0000", 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2826999999999999e-05 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.914025+0000", 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8037e-05 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.914158+0000", 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013267199999999999 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.914293+0000", 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000135115 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.915229+0000", 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093657799999999998 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.915674+0000", 2024-03-12T03:37:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044414299999999998 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.000552+0000", 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.084878466 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.000608+0000", 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6007000000000001e-05 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.000646+0000", 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8417999999999997e-05 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:29676 7.0 7:0e365ff4:::benchmark_data_smithi169_79822_object29675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:43.916872+0000", 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 130.474467858, 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.083873319000000002, 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29676 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.916872+0000", 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.916872+0000", 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.916873+0000", 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0860000000000001e-06 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.916969+0000", 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6889e-05 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.916981+0000", 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1551e-05 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.916989+0000", 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8320000000000006e-06 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.917038+0000", 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9484999999999998e-05 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.917126+0000", 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7152999999999997e-05 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.917214+0000", 2024-03-12T03:37:54.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8602000000000003e-05 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.918071+0000", 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000857082 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.918688+0000", 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00061719800000000001 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.000689+0000", 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.082000455 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.000719+0000", 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9663999999999999e-05 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.000745+0000", 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6322e-05 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:30008 7.0 7:011962bf:::benchmark_data_smithi169_79822_object30007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:44.169406+0000", 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 130.221933479, 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.061323997999999998, 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 30008 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.169406+0000", 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.169406+0000", 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.169407+0000", 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6050000000000001e-06 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.169452+0000", 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4935000000000003e-05 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.169462+0000", 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0030999999999999e-05 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.169470+0000", 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6760000000000007e-06 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.169514+0000", 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4215999999999998e-05 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.169605+0000", 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0321999999999993e-05 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.169692+0000", 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7650000000000003e-05 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.170547+0000", 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00085439900000000004 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.172547+0000", 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002000422 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.230609+0000", 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.058061990000000001 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.230679+0000", 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9391e-05 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:44.230730+0000", 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1360999999999998e-05 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:63350 7.0 7:0b5751c3:::benchmark_data_smithi169_79822_object63349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:06.818001+0000", 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 107.5733388, 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.067948934000000002, 2024-03-12T03:37:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 63350 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.818001+0000", 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.818001+0000", 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.818002+0000", 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2759999999999999e-06 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.818055+0000", 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3653000000000003e-05 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.818066+0000", 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0637e-05 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.818073+0000", 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4919999999999998e-06 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.818119+0000", 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6118000000000001e-05 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.818220+0000", 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010105700000000001 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.818373+0000", 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000153156 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.820115+0000", 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00174193 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.821305+0000", 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001190095 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.885884+0000", 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.064578917 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.885921+0000", 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6956999999999999e-05 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.885950+0000", 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8646e-05 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:63363 7.0 7:03d12f89:::benchmark_data_smithi169_79822_object63362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:06.822459+0000", 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 107.568880542, 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.067555557000000002, 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 63363 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.822459+0000", 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.822459+0000", 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.822461+0000", 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.79e-06 2024-03-12T03:37:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.822502+0000", 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1866000000000002e-05 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.822514+0000", 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1545e-05 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.822524+0000", 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9860000000000006e-06 2024-03-12T03:37:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.822575+0000", 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0951000000000001e-05 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.822705+0000", 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000130264 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.822847+0000", 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000142006 2024-03-12T03:37:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.824853+0000", 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020054529999999999 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.827362+0000", 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025097169999999999 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.889900+0000", 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.062537774000000004 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.889959+0000", 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9027999999999998e-05 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:06.890014+0000", 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5176999999999997e-05 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:65719 7.0 7:079bf592:::benchmark_data_smithi169_79822_object65718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:08.779495+0000", 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 105.611844387, 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.053409548000000001, 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 65719 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.779495+0000", 2024-03-12T03:37:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.779495+0000", 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.779496+0000", 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5199999999999996e-07 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.779562+0000", 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6266999999999994e-05 2024-03-12T03:37:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.779572+0000", 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0504e-05 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.779581+0000", 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1929999999999993e-06 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.779638+0000", 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7312000000000003e-05 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.779757+0000", 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011947699999999999 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.779896+0000", 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013860200000000001 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.781707+0000", 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001810779 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.782443+0000", 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073609200000000004 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.832818+0000", 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.050374901 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.832860+0000", 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2719000000000002e-05 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:08.832905+0000", 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.405e-05 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:66565 7.0 7:0ae2d32e:::benchmark_data_smithi169_79822_object66564:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:09.490942+0000", 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 104.900397047, 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.055831313, 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 66565 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.490942+0000", 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.490942+0000", 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.490943+0000", 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.018e-06 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.490989+0000", 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5908000000000002e-05 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.490999+0000", 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8509999999999994e-06 2024-03-12T03:37:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.491005+0000", 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0109999999999997e-06 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.491048+0000", 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2437e-05 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.491144+0000", 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6637000000000003e-05 2024-03-12T03:37:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.491234+0000", 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9881000000000002e-05 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.493129+0000", 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018951840000000001 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.494121+0000", 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00099200500000000001 2024-03-12T03:37:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.546697+0000", 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.052575470999999999 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.546736+0000", 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9104999999999999e-05 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:09.546774+0000", 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7805000000000001e-05 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107734 7.10 7:09752c19:::benchmark_data_smithi169_79822_object107733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.406825+0000", 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 65.984513938000006, 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90505219699999995, 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107734 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406825+0000", 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406825+0000", 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406827+0000", 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6139999999999999e-06 2024-03-12T03:37:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406856+0000", 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9417999999999999e-05 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406870+0000", 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3414999999999999e-05 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406881+0000", 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1232e-05 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406940+0000", 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8978999999999999e-05 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406951+0000", 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0831e-05 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.301975+0000", 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89502379700000001 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.302104+0000", 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012923399999999999 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.302398+0000", 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029388400000000001 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.303969+0000", 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001571476 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.309180+0000", 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0052102030000000001 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.311767+0000", 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025875160000000002 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.311830+0000", 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2638000000000006e-05 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.311878+0000", 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7960000000000002e-05 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:121768 7.0 7:019d2c4e:::benchmark_data_smithi169_79822_object4351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:06.420440+0000", 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 47.970899226999997, 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.075440241000000005, 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 121768 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.420440+0000", 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.420440+0000", 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.420442+0000", 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.46e-06 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.420448+0000", 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9569999999999999e-06 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.420458+0000", 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0883000000000001e-05 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.420471+0000", 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2266000000000001e-05 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.420539+0000", 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8773e-05 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.420663+0000", 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123323 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.420969+0000", 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000306664 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.422234+0000", 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001264238 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.423145+0000", 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091104800000000002 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.495792+0000", 2024-03-12T03:37:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.072647195999999997 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.495837+0000", 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5482999999999998e-05 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.495880+0000", 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.295e-05 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:123455 7.0 7:04f681fe:::benchmark_data_smithi169_79822_object6038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:37:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:08.405591+0000", 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 45.985748714000003, 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15875210000000001, 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 123455 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405591+0000", 2024-03-12T03:37:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405591+0000", 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405592+0000", 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.44e-06 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405599+0000", 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6950000000000003e-06 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405610+0000", 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1488000000000001e-05 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405619+0000", 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2619999999999998e-06 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405664+0000", 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5083999999999997e-05 2024-03-12T03:37:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405765+0000", 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010178099999999999 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.406019+0000", 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025336500000000002 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.407608+0000", 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015897179999999999 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.408103+0000", 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00049444199999999999 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.564247+0000", 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.156143808 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.564293+0000", 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6502999999999998e-05 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.564343+0000", 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9514000000000001e-05 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:123489 7.0 7:0587e091:::benchmark_data_smithi169_79822_object6072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:08.594113+0000", 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 45.797225910000002, 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.065969904999999995, 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 123489 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594113+0000", 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594113+0000", 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594115+0000", 2024-03-12T03:37:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5460000000000001e-06 2024-03-12T03:37:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594122+0000", 2024-03-12T03:37:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8399999999999997e-06 2024-03-12T03:37:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594133+0000", 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1423000000000001e-05 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594140+0000", 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2419999999999997e-06 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594198+0000", 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8721999999999998e-05 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594325+0000", 2024-03-12T03:37:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012722800000000001 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594655+0000", 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032969999999999999 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.597954+0000", 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032984020000000002 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.600142+0000", 2024-03-12T03:37:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021884769999999999 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.660010+0000", 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.059868017000000003 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.660048+0000", 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7771000000000001e-05 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.660083+0000", 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5537000000000002e-05 2024-03-12T03:37:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:123490 7.0 7:03a71b0b:::benchmark_data_smithi169_79822_object6073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:08.594154+0000", 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 45.797185216999999, 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.066455159, 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 123490 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594154+0000", 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594154+0000", 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594155+0000", 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3799999999999998e-07 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594166+0000", 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0910000000000001e-05 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594171+0000", 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8730000000000004e-06 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.594177+0000", 2024-03-12T03:37:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.169e-06 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.596232+0000", 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020555590000000002 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.596344+0000", 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112185 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.596566+0000", 2024-03-12T03:37:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022208199999999999 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.598080+0000", 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001513222 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.600278+0000", 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021987019999999999 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.660510+0000", 2024-03-12T03:37:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.060231675999999998 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.660564+0000", 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4308000000000002e-05 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.660609+0000", 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5034999999999998e-05 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:132013 7.0 7:0297fb2b:::benchmark_data_smithi169_79822_object14596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-12T03:37:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:19.009353+0000", 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.381985880999999, 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.058052716999999997, 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 132013 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.009353+0000", 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.009353+0000", 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.009355+0000", 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.24e-06 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.009360+0000", 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1540000000000001e-06 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.009371+0000", 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0891e-05 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.009377+0000", 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1870000000000002e-06 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.009419+0000", 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1658e-05 2024-03-12T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.009508+0000", 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9586000000000006e-05 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.009789+0000", 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028031699999999998 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.011182+0000", 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013931589999999999 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.012130+0000", 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00094865599999999998 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.067301+0000", 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.055170782000000002 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.067358+0000", 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6471999999999997e-05 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:37:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:37:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:37:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:19.067406+0000", 2024-03-12T03:37:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8615000000000001e-05 2024-03-12T03:37:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:37:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:37:54.513 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:37:54.543 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:54.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:54.645 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:54.673 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:54.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:54.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:54.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:54.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:54.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:54.906 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:54.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:54.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:55.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:55.113 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:55.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:55.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:54 smithi169 ceph-mon[29344]: pgmap v180: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 151 active+clean, 3 active+remapped+backfilling; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 557 op/s; 15906/241632 objects misplaced (6.583%); 24 MiB/s, 387 objects/s recovering 2024-03-12T03:37:55.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:54 smithi169 ceph-mon[29344]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-12T03:37:55.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:54 smithi169 ceph-mon[29344]: osdmap e144: 8 total, 8 up, 8 in 2024-03-12T03:37:55.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:54 smithi169 ceph-mon[39949]: pgmap v180: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 151 active+clean, 3 active+remapped+backfilling; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 557 op/s; 15906/241632 objects misplaced (6.583%); 24 MiB/s, 387 objects/s recovering 2024-03-12T03:37:55.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:54 smithi169 ceph-mon[39949]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-12T03:37:55.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:54 smithi169 ceph-mon[39949]: osdmap e144: 8 total, 8 up, 8 in 2024-03-12T03:37:55.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:55.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:54 smithi173 ceph-mon[33544]: pgmap v180: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 151 active+clean, 3 active+remapped+backfilling; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 557 op/s; 15906/241632 objects misplaced (6.583%); 24 MiB/s, 387 objects/s recovering 2024-03-12T03:37:55.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:54 smithi173 ceph-mon[33544]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-12T03:37:55.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:54 smithi173 ceph-mon[33544]: osdmap e144: 8 total, 8 up, 8 in 2024-03-12T03:37:55.245 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:55.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:55.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:55.382 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:55.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:55.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:55.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:55.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:55.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:55.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:55.646 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:55.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:55.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:55.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:55.784 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:55.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:55.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:55.886 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:37:55.933 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:55.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:37:56.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:56.081 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:56.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:56.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:56.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:56.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:56.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:37:56.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:56.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:37:56.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:56.472 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:56.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:56.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:56.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:56.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:56.703 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:37:56.746 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:56.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:56.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:37:56.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:56.887 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:56.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:56.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:56.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:37:56.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:57.029 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:37:57.031 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:56 smithi169 ceph-mon[29344]: osdmap e145: 8 total, 8 up, 8 in 2024-03-12T03:37:57.031 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:56 smithi169 ceph-mon[29344]: pgmap v183: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 152 active+clean, 2 active+remapped+backfilling; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 812 op/s; 15323/239205 objects misplaced (6.406%); 35 MiB/s, 559 objects/s recovering 2024-03-12T03:37:57.031 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:56 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1171802539' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:37:57.031 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:56 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:37:57.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:56 smithi169 ceph-mon[39949]: osdmap e145: 8 total, 8 up, 8 in 2024-03-12T03:37:57.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:56 smithi169 ceph-mon[39949]: pgmap v183: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 152 active+clean, 2 active+remapped+backfilling; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 812 op/s; 15323/239205 objects misplaced (6.406%); 35 MiB/s, 559 objects/s recovering 2024-03-12T03:37:57.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:56 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1171802539' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:37:57.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:56 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:37:57.041 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:57.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:57.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:57.144 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:57.193 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:57.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:57.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:56 smithi173 ceph-mon[33544]: osdmap e145: 8 total, 8 up, 8 in 2024-03-12T03:37:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:56 smithi173 ceph-mon[33544]: pgmap v183: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing+deep, 152 active+clean, 2 active+remapped+backfilling; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 812 op/s; 15323/239205 objects misplaced (6.406%); 35 MiB/s, 559 objects/s recovering 2024-03-12T03:37:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:56 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1171802539' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:37:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:56 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:37:57.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:57.318 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:57.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:57.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:57.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:57.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:57.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:57.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:57.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:57.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:57.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-12T03:37:57.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:57.724 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:57.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:57.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:57.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:57.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:57.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:57.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:57.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:57 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:37:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:57 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:37:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:57 smithi173 ceph-mon[33544]: osdmap e146: 8 total, 8 up, 8 in 2024-03-12T03:37:58.000 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:58.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:58.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:58.002 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:37:58.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:58.142 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:58.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:58.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:58.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:57 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:37:58.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:57 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:37:58.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:57 smithi169 ceph-mon[29344]: osdmap e146: 8 total, 8 up, 8 in 2024-03-12T03:37:58.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:57 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:37:58.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:57 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:37:58.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:57 smithi169 ceph-mon[39949]: osdmap e146: 8 total, 8 up, 8 in 2024-03-12T03:37:58.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:58.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:58.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:58.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:58.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:58.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:58.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:58.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:58.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:37:58.544 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:58.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:58.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:37:58.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:37:58.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:58.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:58.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:37:58.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:58.808 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:58.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:58.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:58.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:58.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:59.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:59.082 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:59.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:59.094 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:58 smithi169 ceph-mon[29344]: pgmap v185: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 3 active+remapped+backfilling; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 610 op/s; 11221/230649 objects misplaced (4.865%); 22 MiB/s, 358 objects/s recovering 2024-03-12T03:37:59.095 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:58 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:37:59.095 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:58 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:37:59.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:58 smithi169 ceph-mon[39949]: pgmap v185: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 3 active+remapped+backfilling; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 610 op/s; 11221/230649 objects misplaced (4.865%); 22 MiB/s, 358 objects/s recovering 2024-03-12T03:37:59.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:58 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:37:59.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:58 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:37:59.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:59.192 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:58 smithi173 ceph-mon[33544]: pgmap v185: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 3 active+remapped+backfilling; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 610 op/s; 11221/230649 objects misplaced (4.865%); 22 MiB/s, 358 objects/s recovering 2024-03-12T03:37:59.192 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:58 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:37:59.192 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:58 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:37:59.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:59.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:59.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:59.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:59.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:59.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:37:59.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:59.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:37:59.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:37:59.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:37:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:59.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:37:59.709 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:59.751 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:59.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:59.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:37:59.893 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:37:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:37:59.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:37:59.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:00.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:00.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:00.039 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:59 smithi173 ceph-mon[33544]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-12T03:38:00.039 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:37:59 smithi173 ceph-mon[33544]: osdmap e147: 8 total, 8 up, 8 in 2024-03-12T03:38:00.042 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:38:00.042 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:38:00.042 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-03-12T03:38:00.042 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:38:00.097 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:59 smithi169 ceph-mon[29344]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-12T03:38:00.097 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:37:59 smithi169 ceph-mon[29344]: osdmap e147: 8 total, 8 up, 8 in 2024-03-12T03:38:00.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:59 smithi169 ceph-mon[39949]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-12T03:38:00.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:37:59 smithi169 ceph-mon[39949]: osdmap e147: 8 total, 8 up, 8 in 2024-03-12T03:38:00.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:00.167 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:00.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:00.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:00.269 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:00.313 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:00.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:00.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:00.474 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:00.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:00.575 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:00.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:00.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:00.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:00.702 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:00.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:00.736 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:00.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:00.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:00.838 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:00.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-12T03:38:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:00.900 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:00.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:01.002 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:01.055 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:01.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:01.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:38:01.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:01.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:01 smithi169 ceph-mon[29344]: pgmap v187: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 3 active+remapped+backfilling; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1006 B/s rd, 0 B/s wr, 720 op/s; 11221/230649 objects misplaced (4.865%); 26 MiB/s, 422 objects/s recovering 2024-03-12T03:38:01.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:01 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:01.159 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:01 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:01.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:01.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:01.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:01.203 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:38:01.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:01.320 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:01.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:01.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:01.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:01.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:01 smithi169 ceph-mon[39949]: pgmap v187: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 3 active+remapped+backfilling; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1006 B/s rd, 0 B/s wr, 720 op/s; 11221/230649 objects misplaced (4.865%); 26 MiB/s, 422 objects/s recovering 2024-03-12T03:38:01.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:01 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:01 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:01 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/703299928' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-12T03:38:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:01 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-12T03:38:01.426 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:01 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/703299928' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-12T03:38:01.426 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:01 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-12T03:38:01.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:01.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:01 smithi173 ceph-mon[33544]: pgmap v187: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 3 active+remapped+backfilling; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1006 B/s rd, 0 B/s wr, 720 op/s; 11221/230649 objects misplaced (4.865%); 26 MiB/s, 422 objects/s recovering 2024-03-12T03:38:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:01 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:01 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:01 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/703299928' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-12T03:38:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:01 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-12T03:38:01.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:01.586 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:01.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:01.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:01.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:01.713 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:01.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:01.804 INFO:teuthology.orchestra.run.smithi169.stderr:set pool 1 pg_num to 11 2024-03-12T03:38:01.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:01.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:01.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:01.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:01.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:01.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:01.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:01.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:02.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:02.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:02.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:02.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:02.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:38:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:38:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:38:02.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:02.240 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:02.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:02.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:02.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:02.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:02.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:02.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:38:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:38:02.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:02.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:02.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:02.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:02.645 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:02.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:02.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:02.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:02.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:02.874 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:02.910 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:02 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-12T03:38:02.911 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:02 smithi169 ceph-mon[29344]: osdmap e148: 8 total, 8 up, 8 in 2024-03-12T03:38:02.911 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:02 smithi169 ceph-mon[29344]: pgmap v189: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 1 active+clean+remapped, 2 active+remapped+backfilling; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 8054/217347 objects misplaced (3.706%); 30 MiB/s, 473 objects/s recovering 2024-03-12T03:38:02.911 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:02 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-12T03:38:02.911 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:02 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-12T03:38:02.911 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:02 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:02.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:02 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-12T03:38:02.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:02 smithi169 ceph-mon[39949]: osdmap e148: 8 total, 8 up, 8 in 2024-03-12T03:38:02.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:02 smithi169 ceph-mon[39949]: pgmap v189: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 1 active+clean+remapped, 2 active+remapped+backfilling; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 8054/217347 objects misplaced (3.706%); 30 MiB/s, 473 objects/s recovering 2024-03-12T03:38:02.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:02 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-12T03:38:02.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:02 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-12T03:38:02.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:02 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:02.919 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:02.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:02.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:03.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:03.061 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:03.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:03.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:03.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:03.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:03.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:03.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:02 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-12T03:38:03.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:02 smithi173 ceph-mon[33544]: osdmap e148: 8 total, 8 up, 8 in 2024-03-12T03:38:03.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:02 smithi173 ceph-mon[33544]: pgmap v189: 158 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 1 active+clean+remapped, 2 active+remapped+backfilling; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 8054/217347 objects misplaced (3.706%); 30 MiB/s, 473 objects/s recovering 2024-03-12T03:38:03.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:02 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-12T03:38:03.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:02 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-12T03:38:03.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:02 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:03.296 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:03.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:03.339 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:03.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:03.440 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:03.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:03.486 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:03.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:03.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:03.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:03.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:03.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:03.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:03.783 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:03.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:03.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:03.898 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:38:03.898 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:38:03.898 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:38:03.898 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:38:03.898 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:38:03.898 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:03 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-12T03:38:03.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:03 smithi173 ceph-mon[33544]: osdmap e149: 8 total, 8 up, 8 in 2024-03-12T03:38:03.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:03 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:03.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:03 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:03.913 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:03.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:04.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:04.043 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:04.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:04.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:04.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:03 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-12T03:38:04.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:03 smithi169 ceph-mon[29344]: osdmap e149: 8 total, 8 up, 8 in 2024-03-12T03:38:04.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:03 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:04.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:03 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:04.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:04.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:04.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:04.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:03 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-12T03:38:04.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:03 smithi169 ceph-mon[39949]: osdmap e149: 8 total, 8 up, 8 in 2024-03-12T03:38:04.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:03 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:04.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:03 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:04.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:04.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:04.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:04.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:04.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:04.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:04.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:04.580 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:04.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:04.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-12T03:38:04.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:04.759 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:04.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:04.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:04.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:04.906 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:04.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:04 smithi173 ceph-mon[33544]: osdmap e150: 8 total, 8 up, 8 in 2024-03-12T03:38:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:04 smithi173 ceph-mon[33544]: pgmap v192: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 10 unknown, 1 active+clean+remapped, 2 active+remapped+backfilling; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1012 B/s rd, 0 B/s wr, 877 op/s; 8054/217347 objects misplaced (3.706%); 15 MiB/s, 243 objects/s recovering 2024-03-12T03:38:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:04 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-12T03:38:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:04 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-12T03:38:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:04 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:38:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:04 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:38:04.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:04 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:05.004 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:38:05.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:05.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:05.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:05.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[29344]: osdmap e150: 8 total, 8 up, 8 in 2024-03-12T03:38:05.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[29344]: pgmap v192: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 10 unknown, 1 active+clean+remapped, 2 active+remapped+backfilling; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1012 B/s rd, 0 B/s wr, 877 op/s; 8054/217347 objects misplaced (3.706%); 15 MiB/s, 243 objects/s recovering 2024-03-12T03:38:05.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-12T03:38:05.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-12T03:38:05.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:38:05.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:38:05.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:05.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[39949]: osdmap e150: 8 total, 8 up, 8 in 2024-03-12T03:38:05.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[39949]: pgmap v192: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 10 unknown, 1 active+clean+remapped, 2 active+remapped+backfilling; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1012 B/s rd, 0 B/s wr, 877 op/s; 8054/217347 objects misplaced (3.706%); 15 MiB/s, 243 objects/s recovering 2024-03-12T03:38:05.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-12T03:38:05.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-12T03:38:05.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:38:05.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:38:05.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:04 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:05.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:05.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:05.283 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:05.325 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:05.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:05.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:05.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:05.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:05.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:05.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:05.602 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:05.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:05.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:05.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:05.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:05.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:05.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:05.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:05.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:05.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:05.993 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:05.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:05.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:06.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:06.135 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:06.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:06.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-12T03:38:06.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:05 smithi169 ceph-mon[29344]: osdmap e151: 8 total, 8 up, 8 in 2024-03-12T03:38:06.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:05 smithi169 ceph-mon[29344]: osdmap e152: 8 total, 8 up, 8 in 2024-03-12T03:38:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-12T03:38:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:05 smithi169 ceph-mon[39949]: osdmap e151: 8 total, 8 up, 8 in 2024-03-12T03:38:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:05 smithi169 ceph-mon[39949]: osdmap e152: 8 total, 8 up, 8 in 2024-03-12T03:38:06.211 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-12T03:38:06.211 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:05 smithi173 ceph-mon[33544]: osdmap e151: 8 total, 8 up, 8 in 2024-03-12T03:38:06.211 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:05 smithi173 ceph-mon[33544]: osdmap e152: 8 total, 8 up, 8 in 2024-03-12T03:38:06.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:06.261 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:06.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:06.362 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:06.406 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:06.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:06.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:06.544 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:06.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:06.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:06.645 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:06.709 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:06.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:06.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:06.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:06.846 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:06.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:06 smithi169 ceph-mon[29344]: pgmap v195: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 10 unknown, 1 active+clean+remapped, 2 active+remapped+backfilling; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 206 op/s; 8054/214473 objects misplaced (3.755%) 2024-03-12T03:38:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:06 smithi169 ceph-mon[39949]: pgmap v195: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 10 unknown, 1 active+clean+remapped, 2 active+remapped+backfilling; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 206 op/s; 8054/214473 objects misplaced (3.755%) 2024-03-12T03:38:06.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:06.991 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:06.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:06.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:07.047 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:38:07.048 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:38:07.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:38:07.048 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:38:07.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.048 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:29231 7.d 7:b9e4e021:::benchmark_data_smithi169_79822_object29230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:38:07.048 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:43.516459+0000", 2024-03-12T03:38:07.048 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 143.527286492, 2024-03-12T03:38:07.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13259663399999999, 2024-03-12T03:38:07.048 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.048 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.048 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.049 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.049 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.049 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29231 2024-03-12T03:38:07.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.049 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.516459+0000", 2024-03-12T03:38:07.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.516459+0000", 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.516460+0000", 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4099999999999998e-07 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.516548+0000", 2024-03-12T03:38:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7938999999999999e-05 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.516557+0000", 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4810000000000006e-06 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.516565+0000", 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3590000000000008e-06 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.516611+0000", 2024-03-12T03:38:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6139e-05 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.516735+0000", 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000124122 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.516881+0000", 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000145384 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518559+0000", 2024-03-12T03:38:07.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001678833 2024-03-12T03:38:07.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.519052+0000", 2024-03-12T03:38:07.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00049271799999999995 2024-03-12T03:38:07.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.648957+0000", 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.129904569 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649023+0000", 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6303999999999994e-05 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649056+0000", 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3045e-05 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.054 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:29233 7.d 7:b91d8b07:::benchmark_data_smithi169_79822_object29232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:43.516991+0000", 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 143.526754942, 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13218660900000001, 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29233 2024-03-12T03:38:07.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.056 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.516991+0000", 2024-03-12T03:38:07.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.516991+0000", 2024-03-12T03:38:07.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.516992+0000", 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6899999999999997e-07 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.517047+0000", 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5133999999999998e-05 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.517058+0000", 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1086e-05 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.517064+0000", 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8880000000000002e-06 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.517727+0000", 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00066359100000000005 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.517835+0000", 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107477 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.517923+0000", 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7927e-05 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518658+0000", 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073484799999999997 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521958+0000", 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033009070000000001 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649100+0000", 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12714135500000001 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649135+0000", 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5376000000000001e-05 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649178+0000", 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2450999999999997e-05 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.061 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.061 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:29242 7.c 7:30bfa0a5:::benchmark_data_smithi169_79822_object29241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:38:07.061 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:43.521038+0000", 2024-03-12T03:38:07.061 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 143.522707655, 2024-03-12T03:38:07.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12830894900000001, 2024-03-12T03:38:07.061 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.061 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.061 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.061 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.061 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.062 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29242 2024-03-12T03:38:07.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.062 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521038+0000", 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521038+0000", 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521040+0000", 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.342e-06 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521128+0000", 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8150000000000001e-05 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521141+0000", 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2857000000000001e-05 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521148+0000", 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5830000000000001e-06 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521191+0000", 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3285999999999999e-05 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521283+0000", 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1267999999999994e-05 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521371+0000", 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8505999999999996e-05 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.522526+0000", 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001154757 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.522787+0000", 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000261227 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649253+0000", 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12646587500000001 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649302+0000", 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8995000000000001e-05 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649347+0000", 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5102999999999998e-05 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:29243 7.7 7:e4d7534a:::benchmark_data_smithi169_79822_object29242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:38:07.067 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:43.521247+0000", 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 143.52249941900001, 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13067451099999999, 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29243 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521247+0000", 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521247+0000", 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521247+0000", 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1099999999999995e-07 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521350+0000", 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010240599999999999 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521357+0000", 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0670000000000003e-06 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521362+0000", 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7559999999999996e-06 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521404+0000", 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1217999999999997e-05 2024-03-12T03:38:07.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521490+0000", 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6509999999999997e-05 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.521587+0000", 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6453999999999995e-05 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.522788+0000", 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001201517 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649370+0000", 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12658229300000001 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.651818+0000", 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024471889999999998 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.651875+0000", 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7045e-05 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.651921+0000", 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6445e-05 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107727 7.7 7:ea575e37:::benchmark_data_smithi169_79822_object107726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.181526+0000", 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 78.862219871999997, 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.110791575, 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.073 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107727 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.181526+0000", 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.181526+0000", 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.181528+0000", 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6610000000000001e-06 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.181580+0000", 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2420000000000003e-05 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.181597+0000", 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7e-05 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.181606+0000", 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2299999999999997e-06 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.181684+0000", 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8016999999999995e-05 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.181875+0000", 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019051699999999999 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.182060+0000", 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018557400000000001 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.185419+0000", 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033589750000000002 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.195858+0000", 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010438912999999999 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.292307+0000", 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.096448544999999997 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.292318+0000", 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0723000000000001e-05 2024-03-12T03:38:07.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107727 7.17 7:ea575e37:::benchmark_data_smithi169_79822_object107726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.269042+0000", 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 78.774703787999997, 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0759451440000001, 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107727 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269042+0000", 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269042+0000", 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269044+0000", 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0250000000000001e-06 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269080+0000", 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5604e-05 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269135+0000", 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5294000000000003e-05 2024-03-12T03:38:07.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269165+0000", 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0079000000000001e-05 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.292245+0000", 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.023079873000000001 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.292257+0000", 2024-03-12T03:38:07.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1508999999999999e-05 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.327768+0000", 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.035511074 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for degraded object", 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.328777+0000", 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001009111 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.344893+0000", 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.016116164999999998 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.344987+0000", 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4409999999999999e-05 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:122006 7.d 7:b4556b5a:::benchmark_data_smithi169_79822_object4589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:06.753649+0000", 2024-03-12T03:38:07.084 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 60.290097287999998, 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13027840700000001, 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 122006 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.753649+0000", 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.753649+0000", 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.753650+0000", 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4759999999999999e-06 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.753656+0000", 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6890000000000001e-06 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.753667+0000", 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.131e-05 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.753676+0000", 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5450000000000003e-06 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.753738+0000", 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2584999999999998e-05 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.753880+0000", 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014186400000000001 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.754009+0000", 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000129023 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.756294+0000", 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022844789999999999 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.760658+0000", 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043648810000000001 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.883850+0000", 2024-03-12T03:38:07.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.123191327 2024-03-12T03:38:07.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.883885+0000", 2024-03-12T03:38:07.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5497999999999998e-05 2024-03-12T03:38:07.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.883927+0000", 2024-03-12T03:38:07.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1730000000000002e-05 2024-03-12T03:38:07.090 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.090 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:122010 7.d 7:b1950f11:::benchmark_data_smithi169_79822_object4593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:06.757582+0000", 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 60.286163795, 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.126452178, 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 122010 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.091 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.757582+0000", 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.757582+0000", 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.757583+0000", 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6400000000000001e-07 2024-03-12T03:38:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.757587+0000", 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2490000000000003e-06 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.757597+0000", 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2960000000000004e-06 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.757602+0000", 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2449999999999998e-06 2024-03-12T03:38:07.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.757647+0000", 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5516999999999998e-05 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.757756+0000", 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108491 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.757965+0000", 2024-03-12T03:38:07.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020947199999999999 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.760725+0000", 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027595459999999999 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.761565+0000", 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083993100000000003 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.883971+0000", 2024-03-12T03:38:07.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.122406608 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.884000+0000", 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8464999999999999e-05 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:06.884034+0000", 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4493999999999999e-05 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:123448 7.d 7:b6d9185f:::benchmark_data_smithi169_79822_object6031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:08.399418+0000", 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.644328074999997, 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16357666500000001, 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 123448 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.399418+0000", 2024-03-12T03:38:07.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.399418+0000", 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.399419+0000", 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5899999999999995e-07 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.399423+0000", 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1690000000000002e-06 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.399431+0000", 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8290000000000008e-06 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.399436+0000", 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4430000000000002e-06 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.399477+0000", 2024-03-12T03:38:07.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0969e-05 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.399598+0000", 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012065099999999999 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.399708+0000", 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110596 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.403181+0000", 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034723739999999999 2024-03-12T03:38:07.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.403720+0000", 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053882400000000003 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.562901+0000", 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15918117600000001 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.562942+0000", 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1115999999999997e-05 2024-03-12T03:38:07.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.562995+0000", 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2658999999999998e-05 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:124526 7.7 7:e0dfe3ae:::benchmark_data_smithi169_79822_object7109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:09.788068+0000", 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 57.255677517999999, 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14941812900000001, 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.102 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124526 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.788068+0000", 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.788068+0000", 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.788070+0000", 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.708e-06 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.788075+0000", 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9930000000000003e-06 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.788088+0000", 2024-03-12T03:38:07.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2557e-05 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.788095+0000", 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7589999999999997e-06 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.788150+0000", 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4483e-05 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.788268+0000", 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117713 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.788584+0000", 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031587899999999998 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.790299+0000", 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001714995 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.802795+0000", 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012496484 2024-03-12T03:38:07.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.937390+0000", 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.134594993 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.937452+0000", 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2353000000000004e-05 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.937487+0000", 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4211999999999997e-05 2024-03-12T03:38:07.107 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:124534 7.d 7:b5781af2:::benchmark_data_smithi169_79822_object7117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:09.793354+0000", 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 57.25039177, 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.143486897, 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.108 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124534 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.793354+0000", 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.793354+0000", 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.793355+0000", 2024-03-12T03:38:07.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.082e-06 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.793360+0000", 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8300000000000003e-06 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.793372+0000", 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1833e-05 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.793380+0000", 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9279999999999993e-06 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.793439+0000", 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8768999999999999e-05 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.793570+0000", 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000131858 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:38:07.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.793694+0000", 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123504 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.796154+0000", 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024598799999999998 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.799452+0000", 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032977739999999998 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.936741+0000", 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.137289877 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.936798+0000", 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6069000000000001e-05 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.936841+0000", 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3492999999999998e-05 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.114 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:124537 7.7 7:e1542afb:::benchmark_data_smithi169_79822_object7120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:38:07.114 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:09.797272+0000", 2024-03-12T03:38:07.114 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 57.246474288999998, 2024-03-12T03:38:07.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14713242800000001, 2024-03-12T03:38:07.114 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.114 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.114 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.114 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:07.116 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.116 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124537 2024-03-12T03:38:07.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.116 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.797272+0000", 2024-03-12T03:38:07.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.797272+0000", 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.797273+0000", 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0699999999999999e-06 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.797280+0000", 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1999999999999997e-06 2024-03-12T03:38:07.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.797289+0000", 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7590000000000003e-06 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.797294+0000", 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7069999999999997e-06 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.797342+0000", 2024-03-12T03:38:07.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7794999999999999e-05 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.797468+0000", 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000125778 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.797798+0000", 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00033029100000000003 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.799568+0000", 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017696840000000001 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.802849+0000", 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032809359999999999 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.944307+0000", 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14145781599999999 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.944359+0000", 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2472999999999997e-05 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.944404+0000", 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4919000000000001e-05 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:124546 7.d 7:b772d2a7:::benchmark_data_smithi169_79822_object7129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:38:07.121 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:09.805853+0000", 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 57.237893270000001, 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13109089199999999, 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124546 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.805853+0000", 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.805853+0000", 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.805854+0000", 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3820000000000001e-06 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.805859+0000", 2024-03-12T03:38:07.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2310000000000004e-06 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.805871+0000", 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.129e-05 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.805879+0000", 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8289999999999997e-06 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.805940+0000", 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0775e-05 2024-03-12T03:38:07.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.806076+0000", 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013566300000000001 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.806199+0000", 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123662 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.808710+0000", 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025103880000000001 2024-03-12T03:38:07.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.808824+0000", 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114288 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.936875+0000", 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.128051208 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.936907+0000", 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1291000000000002e-05 2024-03-12T03:38:07.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:09.936944+0000", 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6884999999999997e-05 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:124692 7.7 7:e139262b:::benchmark_data_smithi169_79822_object7275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:10.070709+0000", 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 56.973036862000001, 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.109315272, 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.127 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124692 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.070709+0000", 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.070709+0000", 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.070711+0000", 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5719999999999999e-06 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.070718+0000", 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9530000000000002e-06 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.070732+0000", 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.467e-05 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.070741+0000", 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7479999999999999e-06 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.070787+0000", 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5868000000000002e-05 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.070914+0000", 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000127176 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.071245+0000", 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00033108600000000002 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.072653+0000", 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00140741 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.073158+0000", 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00050535900000000004 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.179915+0000", 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10675727 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.179972+0000", 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6814999999999997e-05 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.180024+0000", 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2345000000000001e-05 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:124815 7.d 7:b0c53038:::benchmark_data_smithi169_79822_object7398:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:10.286641+0000", 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 56.757105109999998, 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14361850300000001, 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124815 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.286641+0000", 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.286641+0000", 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.286642+0000", 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2359999999999999e-06 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.286648+0000", 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7429999999999999e-06 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.286657+0000", 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8079999999999994e-06 2024-03-12T03:38:07.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.286662+0000", 2024-03-12T03:38:07.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5609999999999998e-06 2024-03-12T03:38:07.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.286716+0000", 2024-03-12T03:38:07.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3467999999999997e-05 2024-03-12T03:38:07.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.286830+0000", 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114039 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.286981+0000", 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015091900000000001 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.292099+0000", 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00511831 2024-03-12T03:38:07.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.299910+0000", 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0078113569999999997 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.430196+0000", 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.130285647 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.430230+0000", 2024-03-12T03:38:07.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4442e-05 2024-03-12T03:38:07.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.430259+0000", 2024-03-12T03:38:07.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8972999999999999e-05 2024-03-12T03:38:07.142 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.142 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.142 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.142 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:124820 7.d 7:b85f3c29:::benchmark_data_smithi169_79822_object7403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:38:07.142 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:10.287374+0000", 2024-03-12T03:38:07.142 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 56.756371807000001, 2024-03-12T03:38:07.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.145667185, 2024-03-12T03:38:07.143 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.143 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.143 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.143 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.143 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.143 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124820 2024-03-12T03:38:07.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.143 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.287374+0000", 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.287374+0000", 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.287375+0000", 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.540000000000001e-07 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.287389+0000", 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3634e-05 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.287402+0000", 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3675999999999999e-05 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.287411+0000", 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2279999999999998e-06 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.289472+0000", 2024-03-12T03:38:07.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020609370000000001 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.289549+0000", 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7078999999999998e-05 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.289618+0000", 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8986000000000007e-05 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.295465+0000", 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058474599999999996 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.301669+0000", 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0062038420000000002 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.432972+0000", 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.131302746 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.433017+0000", 2024-03-12T03:38:07.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4870999999999998e-05 2024-03-12T03:38:07.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.433041+0000", 2024-03-12T03:38:07.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4771999999999999e-05 2024-03-12T03:38:07.148 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.148 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.148 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:124822 7.d 7:b2abafc2:::benchmark_data_smithi169_79822_object7405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:10.287451+0000", 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 56.756294619999998, 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14568046000000001, 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124822 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.287451+0000", 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.287451+0000", 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.287452+0000", 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4300000000000002e-07 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.287456+0000", 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.262e-06 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.287468+0000", 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1477e-05 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.287475+0000", 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0280000000000003e-06 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.292130+0000", 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046549260000000002 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.292258+0000", 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000128343 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.292371+0000", 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011246799999999999 2024-03-12T03:38:07.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.297861+0000", 2024-03-12T03:38:07.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.005490796 2024-03-12T03:38:07.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.301742+0000", 2024-03-12T03:38:07.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003880465 2024-03-12T03:38:07.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.433072+0000", 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.131329952 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.433108+0000", 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6019000000000002e-05 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.433132+0000", 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3981e-05 2024-03-12T03:38:07.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:124827 7.d 7:bc46813a:::benchmark_data_smithi169_79822_object7410:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:10.293014+0000", 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 56.750732309999997, 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14316733500000001, 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.155 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124827 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.293014+0000", 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.293014+0000", 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.293015+0000", 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2669999999999999e-06 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.293022+0000", 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1359999999999999e-06 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.293040+0000", 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7510999999999999e-05 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.293046+0000", 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6440000000000003e-06 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.295492+0000", 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002446227 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.295609+0000", 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000116359 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:38:07.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.295730+0000", 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012157799999999999 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.297985+0000", 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022549219999999999 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.301768+0000", 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00378248 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.436098+0000", 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.134330485 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.436150+0000", 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1898999999999999e-05 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:10.436181+0000", 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0827000000000001e-05 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:130349 7.d 7:ba9923a4:::benchmark_data_smithi169_79822_object12932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:16.919777+0000", 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 50.123969047000003, 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10936976299999999, 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 130349 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.919777+0000", 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.919777+0000", 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.919778+0000", 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4900000000000005e-07 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.919785+0000", 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0910000000000004e-06 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.919796+0000", 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1138000000000001e-05 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.919803+0000", 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2259999999999999e-06 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.921289+0000", 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001485451 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.921360+0000", 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0961e-05 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.921434+0000", 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4078000000000005e-05 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.922814+0000", 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001380439 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.923121+0000", 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030733699999999997 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:17.029074+0000", 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105952318 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:17.029112+0000", 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7796000000000002e-05 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:17.029147+0000", 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5079000000000001e-05 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.166 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:158463 7.7 7:e4778d0c:::benchmark_data_smithi169_79822_object41046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:56.316333+0000", 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.727413002, 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11439097300000001, 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 158463 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.316333+0000", 2024-03-12T03:38:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.316333+0000", 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.316334+0000", 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2109999999999999e-06 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.316338+0000", 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5880000000000002e-06 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.316347+0000", 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7849999999999997e-06 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.316351+0000", 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8919999999999999e-06 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.316405+0000", 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3362000000000002e-05 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:38:07.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.316533+0000", 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012818700000000001 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.316920+0000", 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038734400000000002 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.319065+0000", 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002144675 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.319387+0000", 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032214599999999999 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.430652+0000", 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11126530699999999 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.430686+0000", 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3129000000000001e-05 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:38:07.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.430724+0000", 2024-03-12T03:38:07.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8346999999999997e-05 2024-03-12T03:38:07.172 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.172 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.172 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.172 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:38:07.172 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:38:07.172 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:38:07.172 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:06 smithi173 ceph-mon[33544]: pgmap v195: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 154 active+clean, 10 unknown, 1 active+clean+remapped, 2 active+remapped+backfilling; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 206 op/s; 8054/214473 objects misplaced (3.755%) 2024-03-12T03:38:07.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:07.174 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:07.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:07.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:07.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:07.319 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:07.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:07.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:07.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:07.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:07.558 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:07.604 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:07.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:07.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:07.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:07.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:07.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:07.803 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:07.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:07.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:07.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:07.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:07.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:07.983 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:07 smithi169 ceph-mon[29344]: osdmap e153: 8 total, 8 up, 8 in 2024-03-12T03:38:07.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:07 smithi169 ceph-mon[39949]: osdmap e153: 8 total, 8 up, 8 in 2024-03-12T03:38:07.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:07.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:08.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-12T03:38:08.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:08.142 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:08.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:08.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:07 smithi173 ceph-mon[33544]: osdmap e153: 8 total, 8 up, 8 in 2024-03-12T03:38:08.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:08.291 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:08.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:08.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:08.400 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:38:08.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:08.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:08.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:08.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:08.578 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:08.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:08.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:08.721 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:08.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:08.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:08.745 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:38:08.791 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:38:08.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:08.842 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:38:08.862 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:08.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:08.886 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:38:08.923 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:38:08.961 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:38:08.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:09.005 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:09.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:09.013 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:38:09.013 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:08 smithi169 ceph-mon[29344]: pgmap v197: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 8 remapped+peering, 155 active+clean, 2 unknown, 2 active+remapped+backfilling; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 5227/204135 objects misplaced (2.561%); 18 MiB/s, 282 objects/s recovering 2024-03-12T03:38:09.013 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:08 smithi169 ceph-mon[29344]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-03-12T03:38:09.013 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:08 smithi169 ceph-mon[29344]: osdmap e154: 8 total, 8 up, 8 in 2024-03-12T03:38:09.013 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:08 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3741566634' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:38:09.013 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:08 smithi169 ceph-mon[39949]: pgmap v197: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 8 remapped+peering, 155 active+clean, 2 unknown, 2 active+remapped+backfilling; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 5227/204135 objects misplaced (2.561%); 18 MiB/s, 282 objects/s recovering 2024-03-12T03:38:09.013 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:08 smithi169 ceph-mon[39949]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-03-12T03:38:09.013 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:08 smithi169 ceph-mon[39949]: osdmap e154: 8 total, 8 up, 8 in 2024-03-12T03:38:09.014 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:08 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3741566634' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:38:09.064 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:38:09.076 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:38:09.076 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:38:09.076 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:38:09.076 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:38:09.076 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:38:09.076 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:38:09.076 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:38:09.077 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:38:09.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:09.157 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:09.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:08 smithi173 ceph-mon[33544]: pgmap v197: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 8 remapped+peering, 155 active+clean, 2 unknown, 2 active+remapped+backfilling; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 5227/204135 objects misplaced (2.561%); 18 MiB/s, 282 objects/s recovering 2024-03-12T03:38:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:08 smithi173 ceph-mon[33544]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-03-12T03:38:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:08 smithi173 ceph-mon[33544]: osdmap e154: 8 total, 8 up, 8 in 2024-03-12T03:38:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:08 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3741566634' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:38:09.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:09.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:09.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:09.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:09.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:09.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:09.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:09.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:09.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:09.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:09.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:09.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:09.699 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:09.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:09.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:09.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:09.845 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:09.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:09.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:09.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:09.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:09.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:10.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:10.229 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:09 smithi169 ceph-mon[29344]: osdmap e155: 8 total, 8 up, 8 in 2024-03-12T03:38:10.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:09 smithi173 ceph-mon[33544]: osdmap e155: 8 total, 8 up, 8 in 2024-03-12T03:38:10.230 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:09 smithi169 ceph-mon[39949]: osdmap e155: 8 total, 8 up, 8 in 2024-03-12T03:38:10.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:10.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:10.359 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:10.393 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:10.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:10.397 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:38:10.397 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:38:10.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:10.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:10.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:10.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:10.650 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:10.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:10.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:10.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:10.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:10.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:10.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:10.915 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:10.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:10.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:11.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:11.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:11.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:11.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:11.143 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:10 smithi173 ceph-mon[33544]: pgmap v200: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 8 remapped+peering, 155 active+clean, 2 unknown, 2 active+remapped+backfilling; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 54 KiB/s rd, 438 KiB/s wr, 928 op/s; 5227/204135 objects misplaced (2.561%); 18 MiB/s, 283 objects/s recovering 2024-03-12T03:38:11.143 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:10 smithi173 ceph-mon[33544]: osdmap e156: 8 total, 8 up, 8 in 2024-03-12T03:38:11.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:11.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:10 smithi169 ceph-mon[29344]: pgmap v200: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 8 remapped+peering, 155 active+clean, 2 unknown, 2 active+remapped+backfilling; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 54 KiB/s rd, 438 KiB/s wr, 928 op/s; 5227/204135 objects misplaced (2.561%); 18 MiB/s, 283 objects/s recovering 2024-03-12T03:38:11.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:10 smithi169 ceph-mon[29344]: osdmap e156: 8 total, 8 up, 8 in 2024-03-12T03:38:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:10 smithi169 ceph-mon[39949]: pgmap v200: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 8 remapped+peering, 155 active+clean, 2 unknown, 2 active+remapped+backfilling; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 54 KiB/s rd, 438 KiB/s wr, 928 op/s; 5227/204135 objects misplaced (2.561%); 18 MiB/s, 283 objects/s recovering 2024-03-12T03:38:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:10 smithi169 ceph-mon[39949]: osdmap e156: 8 total, 8 up, 8 in 2024-03-12T03:38:11.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:11.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:11.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-12T03:38:11.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:11.317 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:11.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:11.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:11.419 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:11.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:11.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:11.546 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:38:11.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:11.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:11.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:11.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:11.690 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:38:11.690 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:38:11.690 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:38:11.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd out 2 2024-03-12T03:38:11.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:11.785 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:11.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:11.886 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:11.911 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:11.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:11.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:38:11.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:11 smithi173 ceph-mon[33544]: osdmap e157: 8 total, 8 up, 8 in 2024-03-12T03:38:11.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:11 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/433959745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:38:11.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:11 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:38:11.971 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:38:11.991 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:11 smithi169 ceph-mon[29344]: osdmap e157: 8 total, 8 up, 8 in 2024-03-12T03:38:11.992 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:11 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/433959745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:38:11.992 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:11 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:38:11.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:11 smithi169 ceph-mon[39949]: osdmap e157: 8 total, 8 up, 8 in 2024-03-12T03:38:11.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:11 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/433959745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:38:11.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:11 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:38:11.993 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:38:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:38:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:38:12.013 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:12.083 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:12.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:12.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:12.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:12.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:12.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:12.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:12.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:12.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:12.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:12.486 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:12.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:12.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:12.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:12.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:12.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:12.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:12.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:38:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:38:12.759 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:12.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:12.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:12.900 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:12.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:12.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:13.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:13.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:13.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:13.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:13.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:13.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:12 smithi169 ceph-mon[29344]: pgmap v203: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 166 active+clean, 1 active+remapped+backfilling; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 2974/191010 objects misplaced (1.557%); 17 MiB/s, 267 objects/s recovering 2024-03-12T03:38:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:12 smithi169 ceph-mon[29344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs peering) 2024-03-12T03:38:13.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:12 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:38:13.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:12 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:38:13.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:12 smithi169 ceph-mon[29344]: osdmap e158: 8 total, 8 up, 8 in 2024-03-12T03:38:13.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:12 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:12 smithi169 ceph-mon[39949]: pgmap v203: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 166 active+clean, 1 active+remapped+backfilling; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 2974/191010 objects misplaced (1.557%); 17 MiB/s, 267 objects/s recovering 2024-03-12T03:38:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:12 smithi169 ceph-mon[39949]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs peering) 2024-03-12T03:38:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:12 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:38:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:12 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:38:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:12 smithi169 ceph-mon[39949]: osdmap e158: 8 total, 8 up, 8 in 2024-03-12T03:38:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:12 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:13.187 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:12 smithi173 ceph-mon[33544]: pgmap v203: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 166 active+clean, 1 active+remapped+backfilling; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 2974/191010 objects misplaced (1.557%); 17 MiB/s, 267 objects/s recovering 2024-03-12T03:38:13.188 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:12 smithi173 ceph-mon[33544]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs peering) 2024-03-12T03:38:13.188 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:12 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:38:13.188 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:12 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:38:13.188 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:12 smithi173 ceph-mon[33544]: osdmap e158: 8 total, 8 up, 8 in 2024-03-12T03:38:13.188 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:12 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:13.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:13.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:13.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:13.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:13.370 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:13.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:13.455 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:13.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:13.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:13.558 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:13.597 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:13.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:13.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:13.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:13.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:13.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:13.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:13.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:13.873 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:38:13.873 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:38:13.877 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:13.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:13.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:13.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:14.009 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:14.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:14.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:13 smithi169 ceph-mon[39949]: 7.2 scrub starts 2024-03-12T03:38:14.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:13 smithi169 ceph-mon[39949]: osdmap e159: 8 total, 8 up, 8 in 2024-03-12T03:38:14.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:13 smithi169 ceph-mon[39949]: osdmap e160: 8 total, 8 up, 8 in 2024-03-12T03:38:14.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:13 smithi169 ceph-mon[39949]: 4.d scrub starts 2024-03-12T03:38:14.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:13 smithi169 ceph-mon[39949]: 4.d scrub ok 2024-03-12T03:38:14.050 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:13 smithi169 ceph-mon[29344]: 7.2 scrub starts 2024-03-12T03:38:14.050 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:13 smithi169 ceph-mon[29344]: osdmap e159: 8 total, 8 up, 8 in 2024-03-12T03:38:14.050 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:13 smithi169 ceph-mon[29344]: osdmap e160: 8 total, 8 up, 8 in 2024-03-12T03:38:14.050 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:13 smithi169 ceph-mon[29344]: 4.d scrub starts 2024-03-12T03:38:14.050 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:13 smithi169 ceph-mon[29344]: 4.d scrub ok 2024-03-12T03:38:14.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:14.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:14.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:14.205 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:13 smithi173 ceph-mon[33544]: 7.2 scrub starts 2024-03-12T03:38:14.206 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:13 smithi173 ceph-mon[33544]: osdmap e159: 8 total, 8 up, 8 in 2024-03-12T03:38:14.206 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:13 smithi173 ceph-mon[33544]: osdmap e160: 8 total, 8 up, 8 in 2024-03-12T03:38:14.206 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:13 smithi173 ceph-mon[33544]: 4.d scrub starts 2024-03-12T03:38:14.206 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:13 smithi173 ceph-mon[33544]: 4.d scrub ok 2024-03-12T03:38:14.241 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:14.283 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:14.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:14.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:14.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:14.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:14.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:14.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:14.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:14.561 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:14.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:14.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-12T03:38:14.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:14.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:14.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:14.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:14.882 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:14.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:14.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:15.006 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:38:15.023 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:15.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:15.125 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:15.164 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:15.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:15.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:14 smithi173 ceph-mon[33544]: 7.2 scrub ok 2024-03-12T03:38:15.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:14 smithi173 ceph-mon[33544]: 7.0 scrub starts 2024-03-12T03:38:15.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:14 smithi173 ceph-mon[33544]: 7.3 scrub starts 2024-03-12T03:38:15.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:14 smithi173 ceph-mon[33544]: pgmap v207: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 166 active+clean, 1 active+remapped+backfilling; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.11k op/s; 2974/191010 objects misplaced (1.557%); 17 MiB/s, 270 objects/s recovering 2024-03-12T03:38:15.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:14 smithi173 ceph-mon[33544]: 7.0 scrub ok 2024-03-12T03:38:15.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:14 smithi173 ceph-mon[33544]: 7.16 scrub starts 2024-03-12T03:38:15.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:14 smithi173 ceph-mon[33544]: 1.3 scrub starts 2024-03-12T03:38:15.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:14 smithi173 ceph-mon[33544]: 1.3 scrub ok 2024-03-12T03:38:15.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:14 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:38:15.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:14 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:38:15.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:15.296 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:38:15.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:15.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:15.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:15.338 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[39949]: 7.2 scrub ok 2024-03-12T03:38:15.338 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[39949]: 7.0 scrub starts 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[39949]: 7.3 scrub starts 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[39949]: pgmap v207: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 166 active+clean, 1 active+remapped+backfilling; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.11k op/s; 2974/191010 objects misplaced (1.557%); 17 MiB/s, 270 objects/s recovering 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[39949]: 7.0 scrub ok 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[39949]: 7.16 scrub starts 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[39949]: 1.3 scrub starts 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[39949]: 1.3 scrub ok 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[29344]: 7.2 scrub ok 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[29344]: 7.0 scrub starts 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[29344]: 7.3 scrub starts 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[29344]: pgmap v207: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 166 active+clean, 1 active+remapped+backfilling; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.11k op/s; 2974/191010 objects misplaced (1.557%); 17 MiB/s, 270 objects/s recovering 2024-03-12T03:38:15.339 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[29344]: 7.0 scrub ok 2024-03-12T03:38:15.340 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[29344]: 7.16 scrub starts 2024-03-12T03:38:15.340 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[29344]: 1.3 scrub starts 2024-03-12T03:38:15.340 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[29344]: 1.3 scrub ok 2024-03-12T03:38:15.340 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:38:15.340 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:14 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:38:15.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:15.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:15.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:15.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:15.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:15.578 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:15.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:15.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:15.725 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:15.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:15.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:15.857 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:15.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:15.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:15.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:15.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:16.088 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:16.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[29344]: 7.3 scrub ok 2024-03-12T03:38:16.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[29344]: 7.10 scrub starts 2024-03-12T03:38:16.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[29344]: 7.16 scrub ok 2024-03-12T03:38:16.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:16.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[29344]: osdmap e161: 8 total, 8 up, 8 in 2024-03-12T03:38:16.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[29344]: 5.1d scrub starts 2024-03-12T03:38:16.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[29344]: 5.1d scrub ok 2024-03-12T03:38:16.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[29344]: 1.a scrub starts 2024-03-12T03:38:16.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[29344]: 1.a scrub ok 2024-03-12T03:38:16.118 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[29344]: 1.5 scrub starts 2024-03-12T03:38:16.118 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[29344]: 1.5 scrub ok 2024-03-12T03:38:16.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[39949]: 7.3 scrub ok 2024-03-12T03:38:16.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[39949]: 7.10 scrub starts 2024-03-12T03:38:16.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[39949]: 7.16 scrub ok 2024-03-12T03:38:16.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:16.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[39949]: osdmap e161: 8 total, 8 up, 8 in 2024-03-12T03:38:16.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[39949]: 5.1d scrub starts 2024-03-12T03:38:16.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[39949]: 5.1d scrub ok 2024-03-12T03:38:16.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[39949]: 1.a scrub starts 2024-03-12T03:38:16.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[39949]: 1.a scrub ok 2024-03-12T03:38:16.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[39949]: 1.5 scrub starts 2024-03-12T03:38:16.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:15 smithi169 ceph-mon[39949]: 1.5 scrub ok 2024-03-12T03:38:16.138 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:16.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:16.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:15 smithi173 ceph-mon[33544]: 7.3 scrub ok 2024-03-12T03:38:16.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:15 smithi173 ceph-mon[33544]: 7.10 scrub starts 2024-03-12T03:38:16.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:15 smithi173 ceph-mon[33544]: 7.16 scrub ok 2024-03-12T03:38:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:15 smithi173 ceph-mon[33544]: osdmap e161: 8 total, 8 up, 8 in 2024-03-12T03:38:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:15 smithi173 ceph-mon[33544]: 5.1d scrub starts 2024-03-12T03:38:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:15 smithi173 ceph-mon[33544]: 5.1d scrub ok 2024-03-12T03:38:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:15 smithi173 ceph-mon[33544]: 1.a scrub starts 2024-03-12T03:38:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:15 smithi173 ceph-mon[33544]: 1.a scrub ok 2024-03-12T03:38:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:15 smithi173 ceph-mon[33544]: 1.5 scrub starts 2024-03-12T03:38:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:15 smithi173 ceph-mon[33544]: 1.5 scrub ok 2024-03-12T03:38:16.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:16.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:16.289 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:16.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:16.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:16.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:16.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:16.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:16.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:16.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:16.740 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:16.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:16.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:16.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:16.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:16.882 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:16.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:16.903 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:16.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:17.031 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:17.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:17.091 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:38:17.092 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:38:17.132 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:17.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[29344]: 7.10 scrub ok 2024-03-12T03:38:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[29344]: 1.4 scrub starts 2024-03-12T03:38:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[29344]: 1.4 scrub ok 2024-03-12T03:38:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[29344]: pgmap v209: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 166 active+clean, 1 active+remapped+backfilling; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 2974/187914 objects misplaced (1.583%) 2024-03-12T03:38:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[29344]: 6.4 scrub starts 2024-03-12T03:38:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[29344]: 6.4 scrub ok 2024-03-12T03:38:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[29344]: 1.1 scrub starts 2024-03-12T03:38:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[29344]: 1.1 scrub ok 2024-03-12T03:38:17.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[39949]: 7.10 scrub ok 2024-03-12T03:38:17.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[39949]: 1.4 scrub starts 2024-03-12T03:38:17.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[39949]: 1.4 scrub ok 2024-03-12T03:38:17.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[39949]: pgmap v209: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 166 active+clean, 1 active+remapped+backfilling; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 2974/187914 objects misplaced (1.583%) 2024-03-12T03:38:17.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[39949]: 6.4 scrub starts 2024-03-12T03:38:17.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[39949]: 6.4 scrub ok 2024-03-12T03:38:17.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[39949]: 1.1 scrub starts 2024-03-12T03:38:17.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:16 smithi169 ceph-mon[39949]: 1.1 scrub ok 2024-03-12T03:38:17.177 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:17.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:17.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:17.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:16 smithi173 ceph-mon[33544]: 7.10 scrub ok 2024-03-12T03:38:17.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:16 smithi173 ceph-mon[33544]: 1.4 scrub starts 2024-03-12T03:38:17.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:16 smithi173 ceph-mon[33544]: 1.4 scrub ok 2024-03-12T03:38:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:16 smithi173 ceph-mon[33544]: pgmap v209: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 166 active+clean, 1 active+remapped+backfilling; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 2974/187914 objects misplaced (1.583%) 2024-03-12T03:38:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:16 smithi173 ceph-mon[33544]: 6.4 scrub starts 2024-03-12T03:38:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:16 smithi173 ceph-mon[33544]: 6.4 scrub ok 2024-03-12T03:38:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:16 smithi173 ceph-mon[33544]: 1.1 scrub starts 2024-03-12T03:38:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:16 smithi173 ceph-mon[33544]: 1.1 scrub ok 2024-03-12T03:38:17.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:17.307 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:17.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:17.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:17.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:17.440 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:17.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:17.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:17.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:17.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:17.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:17.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:17.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a[29340]: 2024-03-12T03:38:17.305+0000 7f7050ca9700 -1 mon.a@0(leader).osd e161 definitely_dead 0 2024-03-12T03:38:17.716 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:17.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:17.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:17.859 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:17.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:17.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:17.993 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:17.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:18.005 INFO:teuthology.orchestra.run.smithi169.stderr:marked out osd.2. 2024-03-12T03:38:18.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:18.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:18.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:18.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:18.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:17 smithi173 ceph-mon[33544]: 6.1f scrub starts 2024-03-12T03:38:18.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:17 smithi173 ceph-mon[33544]: 6.1f scrub ok 2024-03-12T03:38:18.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:17 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1240334941' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-12T03:38:18.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:17 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-12T03:38:18.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:17 smithi173 ceph-mon[33544]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-12T03:38:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:17 smithi173 ceph-mon[33544]: 1.8 scrub starts 2024-03-12T03:38:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:17 smithi173 ceph-mon[33544]: 1.8 scrub ok 2024-03-12T03:38:18.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:18.251 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[39949]: 6.1f scrub starts 2024-03-12T03:38:18.252 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[39949]: 6.1f scrub ok 2024-03-12T03:38:18.252 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1240334941' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-12T03:38:18.252 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-12T03:38:18.252 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[39949]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-12T03:38:18.252 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[39949]: 1.8 scrub starts 2024-03-12T03:38:18.252 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[39949]: 1.8 scrub ok 2024-03-12T03:38:18.252 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[29344]: 6.1f scrub starts 2024-03-12T03:38:18.252 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[29344]: 6.1f scrub ok 2024-03-12T03:38:18.252 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1240334941' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-12T03:38:18.253 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-12T03:38:18.253 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[29344]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-12T03:38:18.253 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[29344]: 1.8 scrub starts 2024-03-12T03:38:18.253 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:17 smithi169 ceph-mon[29344]: 1.8 scrub ok 2024-03-12T03:38:18.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:18.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:18.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:18.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:18.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:18.401 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:38:18.401 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:38:18.402 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:38:18.402 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:38:18.402 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:38:18.402 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:38:18.402 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:38:18.402 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:38:18.402 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:38:18.402 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:38:18.402 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:38:18.402 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:38:18.403 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:38:18.403 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:38:18.403 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:38:18.403 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:38:18.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:18.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:18.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:18.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:18.629 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:18.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:18.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:18.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:18.755 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:18.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:18.857 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:18.882 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:18.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:18.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:19.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:19.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:19.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:19.132 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:19.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:19.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:19.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:18 smithi173 ceph-mon[33544]: 7.14 scrub starts 2024-03-12T03:38:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:18 smithi173 ceph-mon[33544]: pgmap v210: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing, 166 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 2398/175428 objects misplaced (1.367%) 2024-03-12T03:38:19.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:18 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-12T03:38:19.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:18 smithi173 ceph-mon[33544]: osdmap e162: 8 total, 8 up, 7 in 2024-03-12T03:38:19.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:18 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2441663709' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:38:19.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:18 smithi173 ceph-mon[33544]: osdmap e163: 8 total, 8 up, 7 in 2024-03-12T03:38:19.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:19.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:19.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:19.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-12T03:38:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:19.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:19.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:18 smithi169 ceph-mon[29344]: 7.14 scrub starts 2024-03-12T03:38:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: pgmap v210: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing, 166 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 2398/175428 objects misplaced (1.367%) 2024-03-12T03:38:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-12T03:38:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: osdmap e162: 8 total, 8 up, 7 in 2024-03-12T03:38:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2441663709' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:38:19.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: osdmap e163: 8 total, 8 up, 7 in 2024-03-12T03:38:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:18 smithi169 ceph-mon[39949]: 7.14 scrub starts 2024-03-12T03:38:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: pgmap v210: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing, 166 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 2398/175428 objects misplaced (1.367%) 2024-03-12T03:38:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-12T03:38:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: osdmap e162: 8 total, 8 up, 7 in 2024-03-12T03:38:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2441663709' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:38:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: osdmap e163: 8 total, 8 up, 7 in 2024-03-12T03:38:19.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:19.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:19.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:19.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:19.745 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:19.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:19.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:19.899 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:19.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:20.000 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:20.038 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:20.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:20.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:20.140 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:20.183 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:20.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:19 smithi173 ceph-mon[33544]: 6.b scrub starts 2024-03-12T03:38:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:19 smithi173 ceph-mon[33544]: 6.b scrub ok 2024-03-12T03:38:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:19 smithi173 ceph-mon[33544]: 7.12 starting backfill to osd.7 from (0'0,0'0] MAX to 161'8944 2024-03-12T03:38:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:19 smithi173 ceph-mon[33544]: osdmap e164: 8 total, 8 up, 7 in 2024-03-12T03:38:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:19 smithi173 ceph-mon[33544]: 7.17 starting backfill to osd.1 from (0'0,0'0] MAX to 161'9023 2024-03-12T03:38:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:19 smithi173 ceph-mon[33544]: 7.8 starting backfill to osd.3 from (0'0,0'0] MAX to 161'8968 2024-03-12T03:38:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:19 smithi173 ceph-mon[33544]: 7.15 starting backfill to osd.5 from (0'0,0'0] MAX to 161'8958 2024-03-12T03:38:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:19 smithi173 ceph-mon[33544]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 161'11354 2024-03-12T03:38:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:19 smithi173 ceph-mon[33544]: 7.19 starting backfill to osd.5 from (0'0,0'0] MAX to 161'8954 2024-03-12T03:38:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:19 smithi173 ceph-mon[33544]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 161'9080 2024-03-12T03:38:20.276 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: 6.b scrub starts 2024-03-12T03:38:20.276 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: 6.b scrub ok 2024-03-12T03:38:20.276 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: 7.12 starting backfill to osd.7 from (0'0,0'0] MAX to 161'8944 2024-03-12T03:38:20.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: osdmap e164: 8 total, 8 up, 7 in 2024-03-12T03:38:20.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: 7.17 starting backfill to osd.1 from (0'0,0'0] MAX to 161'9023 2024-03-12T03:38:20.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: 7.8 starting backfill to osd.3 from (0'0,0'0] MAX to 161'8968 2024-03-12T03:38:20.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: 7.15 starting backfill to osd.5 from (0'0,0'0] MAX to 161'8958 2024-03-12T03:38:20.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 161'11354 2024-03-12T03:38:20.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: 7.19 starting backfill to osd.5 from (0'0,0'0] MAX to 161'8954 2024-03-12T03:38:20.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[29344]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 161'9080 2024-03-12T03:38:20.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: 6.b scrub starts 2024-03-12T03:38:20.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: 6.b scrub ok 2024-03-12T03:38:20.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: 7.12 starting backfill to osd.7 from (0'0,0'0] MAX to 161'8944 2024-03-12T03:38:20.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: osdmap e164: 8 total, 8 up, 7 in 2024-03-12T03:38:20.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: 7.17 starting backfill to osd.1 from (0'0,0'0] MAX to 161'9023 2024-03-12T03:38:20.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: 7.8 starting backfill to osd.3 from (0'0,0'0] MAX to 161'8968 2024-03-12T03:38:20.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: 7.15 starting backfill to osd.5 from (0'0,0'0] MAX to 161'8958 2024-03-12T03:38:20.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 161'11354 2024-03-12T03:38:20.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: 7.19 starting backfill to osd.5 from (0'0,0'0] MAX to 161'8954 2024-03-12T03:38:20.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:19 smithi169 ceph-mon[39949]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 161'9080 2024-03-12T03:38:20.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:20.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:20.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:20.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:20.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:20.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:20.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:20.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:20.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:20.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:20.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:20.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:20.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:20.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:20.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:20.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:20.865 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:20.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:20.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:21.006 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:21.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:21.091 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:21.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:21.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:21.165 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:21.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:21.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[29344]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 161'8867 2024-03-12T03:38:21.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[29344]: 7.3 starting backfill to osd.6 from (0'0,0'0] MAX to 161'8947 2024-03-12T03:38:21.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[29344]: 7.10 starting backfill to osd.7 from (0'0,0'0] MAX to 161'8810 2024-03-12T03:38:21.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[29344]: 7.14 starting backfill to osd.0 from (0'0,0'0] MAX to 161'8977 2024-03-12T03:38:21.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[29344]: 7.13 starting backfill to osd.5 from (0'0,0'0] MAX to 161'9028 2024-03-12T03:38:21.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[29344]: 7.a starting backfill to osd.1 from (0'0,0'0] MAX to 161'11252 2024-03-12T03:38:21.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[29344]: pgmap v214: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing, 166 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2398/175428 objects misplaced (1.367%) 2024-03-12T03:38:21.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[29344]: osdmap e165: 8 total, 8 up, 7 in 2024-03-12T03:38:21.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[39949]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 161'8867 2024-03-12T03:38:21.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[39949]: 7.3 starting backfill to osd.6 from (0'0,0'0] MAX to 161'8947 2024-03-12T03:38:21.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[39949]: 7.10 starting backfill to osd.7 from (0'0,0'0] MAX to 161'8810 2024-03-12T03:38:21.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[39949]: 7.14 starting backfill to osd.0 from (0'0,0'0] MAX to 161'8977 2024-03-12T03:38:21.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[39949]: 7.13 starting backfill to osd.5 from (0'0,0'0] MAX to 161'9028 2024-03-12T03:38:21.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[39949]: 7.a starting backfill to osd.1 from (0'0,0'0] MAX to 161'11252 2024-03-12T03:38:21.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[39949]: pgmap v214: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing, 166 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2398/175428 objects misplaced (1.367%) 2024-03-12T03:38:21.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:20 smithi169 ceph-mon[39949]: osdmap e165: 8 total, 8 up, 7 in 2024-03-12T03:38:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:20 smithi173 ceph-mon[33544]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 161'8867 2024-03-12T03:38:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:20 smithi173 ceph-mon[33544]: 7.3 starting backfill to osd.6 from (0'0,0'0] MAX to 161'8947 2024-03-12T03:38:21.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:20 smithi173 ceph-mon[33544]: 7.10 starting backfill to osd.7 from (0'0,0'0] MAX to 161'8810 2024-03-12T03:38:21.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:20 smithi173 ceph-mon[33544]: 7.14 starting backfill to osd.0 from (0'0,0'0] MAX to 161'8977 2024-03-12T03:38:21.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:20 smithi173 ceph-mon[33544]: 7.13 starting backfill to osd.5 from (0'0,0'0] MAX to 161'9028 2024-03-12T03:38:21.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:20 smithi173 ceph-mon[33544]: 7.a starting backfill to osd.1 from (0'0,0'0] MAX to 161'11252 2024-03-12T03:38:21.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:20 smithi173 ceph-mon[33544]: pgmap v214: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+scrubbing, 166 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2398/175428 objects misplaced (1.367%) 2024-03-12T03:38:21.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:20 smithi173 ceph-mon[33544]: osdmap e165: 8 total, 8 up, 7 in 2024-03-12T03:38:21.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:21.301 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:21.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:21.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:21.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:21.436 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:21.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:21.539 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:21.580 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:21.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:21.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:21.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:21.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:21.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:21.860 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:21.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:21.962 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:22.009 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:22.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:22.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:22.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:22.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:22.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:38:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:38:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:38:22.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:22.278 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:22.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:22.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:22.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:38:22.379 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:22.444 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:22.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:22.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:38:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:38:22.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:22 smithi173 ceph-mon[33544]: osdmap e166: 8 total, 8 up, 7 in 2024-03-12T03:38:22.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:22 smithi173 ceph-mon[33544]: 4.1c scrub starts 2024-03-12T03:38:22.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:22 smithi173 ceph-mon[33544]: 6.1e scrub starts 2024-03-12T03:38:22.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:22 smithi173 ceph-mon[33544]: 6.1e scrub ok 2024-03-12T03:38:22.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:22 smithi173 ceph-mon[33544]: 4.1c scrub ok 2024-03-12T03:38:22.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:22.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:22 smithi169 ceph-mon[29344]: osdmap e166: 8 total, 8 up, 7 in 2024-03-12T03:38:22.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:22 smithi169 ceph-mon[29344]: 4.1c scrub starts 2024-03-12T03:38:22.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:22 smithi169 ceph-mon[29344]: 6.1e scrub starts 2024-03-12T03:38:22.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:22 smithi169 ceph-mon[29344]: 6.1e scrub ok 2024-03-12T03:38:22.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:22 smithi169 ceph-mon[29344]: 4.1c scrub ok 2024-03-12T03:38:22.553 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:22 smithi169 ceph-mon[39949]: osdmap e166: 8 total, 8 up, 7 in 2024-03-12T03:38:22.553 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:22 smithi169 ceph-mon[39949]: 4.1c scrub starts 2024-03-12T03:38:22.553 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:22 smithi169 ceph-mon[39949]: 6.1e scrub starts 2024-03-12T03:38:22.553 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:22 smithi169 ceph-mon[39949]: 6.1e scrub ok 2024-03-12T03:38:22.553 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:22 smithi169 ceph-mon[39949]: 4.1c scrub ok 2024-03-12T03:38:22.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:22.609 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:22.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:22.710 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:22.758 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:22.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:22.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:22.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:22.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:22.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:22.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:23.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:23.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:23.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:23.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:23.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:23.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:23.222 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:23 smithi169 ceph-mon[29344]: pgmap v217: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 5 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 5 peering, 149 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 976 op/s; 26070/163992 objects misplaced (15.897%); 23 MiB/s, 378 objects/s recovering 2024-03-12T03:38:23.223 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:23 smithi169 ceph-mon[29344]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:38:23.223 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:23 smithi169 ceph-mon[29344]: 5.12 scrub starts 2024-03-12T03:38:23.223 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:23 smithi169 ceph-mon[29344]: 5.12 scrub ok 2024-03-12T03:38:23.223 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:23 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:23.271 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:23.294 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:23.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:23.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:23.433 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:23.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:23 smithi173 ceph-mon[33544]: pgmap v217: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 5 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 5 peering, 149 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 976 op/s; 26070/163992 objects misplaced (15.897%); 23 MiB/s, 378 objects/s recovering 2024-03-12T03:38:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:23 smithi173 ceph-mon[33544]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:38:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:23 smithi173 ceph-mon[33544]: 5.12 scrub starts 2024-03-12T03:38:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:23 smithi173 ceph-mon[33544]: 5.12 scrub ok 2024-03-12T03:38:23.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:23 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:23.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:23 smithi169 ceph-mon[39949]: pgmap v217: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 5 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 5 peering, 149 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 976 op/s; 26070/163992 objects misplaced (15.897%); 23 MiB/s, 378 objects/s recovering 2024-03-12T03:38:23.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:23 smithi169 ceph-mon[39949]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:38:23.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:23 smithi169 ceph-mon[39949]: 5.12 scrub starts 2024-03-12T03:38:23.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:23 smithi169 ceph-mon[39949]: 5.12 scrub ok 2024-03-12T03:38:23.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:23 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:23.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:23.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:23.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:23.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:23.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:23.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:23.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:23.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:23.842 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:23.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:23.943 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:23.982 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:23.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:23.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:24.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:24.116 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:24.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:24.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:24.260 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:24.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:24.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:24.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:24.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:24 smithi169 ceph-mon[29344]: 4.12 scrub starts 2024-03-12T03:38:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:24 smithi169 ceph-mon[29344]: 4.12 scrub ok 2024-03-12T03:38:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:24 smithi169 ceph-mon[39949]: 4.12 scrub starts 2024-03-12T03:38:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:24 smithi169 ceph-mon[39949]: 4.12 scrub ok 2024-03-12T03:38:24.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:24 smithi173 ceph-mon[33544]: 4.12 scrub starts 2024-03-12T03:38:24.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:24 smithi173 ceph-mon[33544]: 4.12 scrub ok 2024-03-12T03:38:24.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:24.528 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:24.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:24.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:24.647 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:38:24.647 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:38:24.648 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-12T03:38:24.648 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-12T03:38:24.648 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-12T03:38:24.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-12T03:38:24.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:24.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:24.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:24.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:24.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:24.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:24.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:24.952 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:24.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:25.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:25.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:25.100 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:25.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:25.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:25.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:25.246 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:25.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:25.268 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:38:25.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:25.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:25.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:25.468 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:38:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:38:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:184904 7.9 7:92807e5e:::benchmark_data_smithi169_79822_object67487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:25.460602+0000", 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0044171890000000002, 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004451076, 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 184904 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.460602+0000", 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.460602+0000", 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.460604+0000", 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4610000000000001e-06 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.460618+0000", 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4609e-05 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.460630+0000", 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1496e-05 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.460637+0000", 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4540000000000001e-06 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:25.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.461910+0000", 2024-03-12T03:38:25.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012732450000000001 2024-03-12T03:38:25.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:25.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.462018+0000", 2024-03-12T03:38:25.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000107372 2024-03-12T03:38:25.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,4", 2024-03-12T03:38:25.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.462363+0000", 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00034472499999999999 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.464858+0000", 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002495822 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.464870+0000", 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1409999999999999e-05 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:25.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:184906 7.4 7:2669b6a6:::benchmark_data_smithi169_79822_object67489:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:25.461688+0000", 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0033311320000000001, 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034709390000000001, 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 184906 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:25.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.461688+0000", 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.461688+0000", 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.461689+0000", 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7699999999999992e-07 2024-03-12T03:38:25.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.461694+0000", 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9849999999999997e-06 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.461709+0000", 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4978e-05 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:25.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.461717+0000", 2024-03-12T03:38:25.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.525e-06 2024-03-12T03:38:25.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:25.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.461797+0000", 2024-03-12T03:38:25.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0552999999999999e-05 2024-03-12T03:38:25.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.461928+0000", 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000130328 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,4", 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.462324+0000", 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00039632900000000001 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.464855+0000", 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025310879999999999 2024-03-12T03:38:25.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.464879+0000", 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4331000000000001e-05 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(7.14 167/163 [PushOp(7:2a1dbf65:::benchmark_data_smithi169_79822_object75746:head, version: 107'4789, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2a1dbf65:::benchmark_data_smithi169_79822_object75746:head@107'4789, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:25.464995+0000", 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 2.4362000000000002e-05, 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000284987, 2024-03-12T03:38:25.489 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:25.490 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "queued for pg", 2024-03-12T03:38:25.490 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:25.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:25.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.464995+0000", 2024-03-12T03:38:25.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:25.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:25.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.464995+0000", 2024-03-12T03:38:25.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.464995+0000", 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.84e-07 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.465029+0000", 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3735999999999997e-05 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.465045+0000", 2024-03-12T03:38:25.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5485000000000002e-05 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.465059+0000", 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4246e-05 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:184900 7.4 7:20f8ae4e:::benchmark_data_smithi169_79822_object67483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:25.455588+0000", 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0094316039999999997, 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0097499619999999992, 2024-03-12T03:38:25.492 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 184900 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.455588+0000", 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.455588+0000", 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:25.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:25.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.455589+0000", 2024-03-12T03:38:25.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.37e-06 2024-03-12T03:38:25.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:25.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.455595+0000", 2024-03-12T03:38:25.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4489999999999998e-06 2024-03-12T03:38:25.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.455609+0000", 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4493e-05 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.455617+0000", 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8590000000000005e-06 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.455678+0000", 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0875999999999998e-05 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.455792+0000", 2024-03-12T03:38:25.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000114607 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,4", 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.456170+0000", 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000377606 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.459583+0000", 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003413134 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:25.461452+0000", 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001868563 2024-03-12T03:38:25.496 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:25.497 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:25.497 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:25.497 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:25.497 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:38:25.497 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 4 2024-03-12T03:38:25.497 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:38:25.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:25.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:25.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:25.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:25.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:25.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:25.661 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:25.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:25.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:25 smithi169 ceph-mon[29344]: pgmap v218: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 5 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 5 peering, 149 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 669 op/s; 26070/163992 objects misplaced (15.897%); 15 MiB/s, 259 objects/s recovering 2024-03-12T03:38:25.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:25 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3156361837' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:38:25.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:25 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:38:25.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:25 smithi169 ceph-mon[39949]: pgmap v218: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 5 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 5 peering, 149 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 669 op/s; 26070/163992 objects misplaced (15.897%); 15 MiB/s, 259 objects/s recovering 2024-03-12T03:38:25.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:25 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3156361837' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:38:25.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:25 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:38:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:25 smithi173 ceph-mon[33544]: pgmap v218: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 5 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 5 peering, 149 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 669 op/s; 26070/163992 objects misplaced (15.897%); 15 MiB/s, 259 objects/s recovering 2024-03-12T03:38:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:25 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3156361837' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:38:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:25 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:38:25.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:25.795 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:25.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:25.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:25.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:25.930 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:25.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:25.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:26.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:26.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:26.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:26.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:26.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:26.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:26.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:26.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:26.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:26.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:38:26.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:38:26.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[29344]: osdmap e167: 8 total, 8 up, 7 in 2024-03-12T03:38:26.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[29344]: 4.2 scrub starts 2024-03-12T03:38:26.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[29344]: 4.2 scrub ok 2024-03-12T03:38:26.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[29344]: 6.e scrub starts 2024-03-12T03:38:26.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[29344]: 6.e scrub ok 2024-03-12T03:38:26.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:38:26.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:38:26.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[39949]: osdmap e167: 8 total, 8 up, 7 in 2024-03-12T03:38:26.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[39949]: 4.2 scrub starts 2024-03-12T03:38:26.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[39949]: 4.2 scrub ok 2024-03-12T03:38:26.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[39949]: 6.e scrub starts 2024-03-12T03:38:26.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:26 smithi169 ceph-mon[39949]: 6.e scrub ok 2024-03-12T03:38:26.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:26.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:26.459 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:26.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:26.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:26.592 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:26.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:26.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:26 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:38:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:26 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:38:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:26 smithi173 ceph-mon[33544]: osdmap e167: 8 total, 8 up, 7 in 2024-03-12T03:38:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:26 smithi173 ceph-mon[33544]: 4.2 scrub starts 2024-03-12T03:38:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:26 smithi173 ceph-mon[33544]: 4.2 scrub ok 2024-03-12T03:38:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:26 smithi173 ceph-mon[33544]: 6.e scrub starts 2024-03-12T03:38:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:26 smithi173 ceph-mon[33544]: 6.e scrub ok 2024-03-12T03:38:26.737 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:26.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:26.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:26.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:26.879 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:26.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:26.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:27.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:27.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:27.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:27.156 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:27.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:27.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:27.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:27.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:27.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:27.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:27.426 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:27.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:27.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:27.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:27.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:27.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:27.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:27.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:27.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:27 smithi169 ceph-mon[29344]: pgmap v220: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 5 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 154 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 883 op/s; 24277/159429 objects misplaced (15.227%); 24 MiB/s, 397 objects/s recovering 2024-03-12T03:38:27.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:27 smithi169 ceph-mon[29344]: osdmap e168: 8 total, 8 up, 7 in 2024-03-12T03:38:27.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:27 smithi169 ceph-mon[29344]: 3.9 scrub starts 2024-03-12T03:38:27.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:27 smithi169 ceph-mon[29344]: 3.9 scrub ok 2024-03-12T03:38:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:27 smithi169 ceph-mon[39949]: pgmap v220: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 5 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 154 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 883 op/s; 24277/159429 objects misplaced (15.227%); 24 MiB/s, 397 objects/s recovering 2024-03-12T03:38:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:27 smithi169 ceph-mon[39949]: osdmap e168: 8 total, 8 up, 7 in 2024-03-12T03:38:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:27 smithi169 ceph-mon[39949]: 3.9 scrub starts 2024-03-12T03:38:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:27 smithi169 ceph-mon[39949]: 3.9 scrub ok 2024-03-12T03:38:27.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:27.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:27 smithi173 ceph-mon[33544]: pgmap v220: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 5 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 154 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 883 op/s; 24277/159429 objects misplaced (15.227%); 24 MiB/s, 397 objects/s recovering 2024-03-12T03:38:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:27 smithi173 ceph-mon[33544]: osdmap e168: 8 total, 8 up, 7 in 2024-03-12T03:38:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:27 smithi173 ceph-mon[33544]: 3.9 scrub starts 2024-03-12T03:38:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:27 smithi173 ceph-mon[33544]: 3.9 scrub ok 2024-03-12T03:38:27.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:27.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:27.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:27.932 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:27.975 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:27.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:27.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:28.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:28.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:38:28.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:28.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:28.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:28.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:28.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:28.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:28.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:28.399 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:28.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:28.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:28.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:28 smithi169 ceph-mon[29344]: 4.18 scrub starts 2024-03-12T03:38:28.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:28 smithi169 ceph-mon[29344]: 4.18 scrub ok 2024-03-12T03:38:28.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:28 smithi169 ceph-mon[29344]: 6.1 scrub starts 2024-03-12T03:38:28.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:28 smithi169 ceph-mon[29344]: 6.1 scrub ok 2024-03-12T03:38:28.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:28 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:38:28.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:28 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:38:28.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:28 smithi169 ceph-mon[39949]: 4.18 scrub starts 2024-03-12T03:38:28.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:28 smithi169 ceph-mon[39949]: 4.18 scrub ok 2024-03-12T03:38:28.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:28 smithi169 ceph-mon[39949]: 6.1 scrub starts 2024-03-12T03:38:28.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:28 smithi169 ceph-mon[39949]: 6.1 scrub ok 2024-03-12T03:38:28.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:28 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:38:28.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:28 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:38:28.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:28.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:28.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:28.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:28.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:28.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:28.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:28.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:28 smithi173 ceph-mon[33544]: 4.18 scrub starts 2024-03-12T03:38:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:28 smithi173 ceph-mon[33544]: 4.18 scrub ok 2024-03-12T03:38:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:28 smithi173 ceph-mon[33544]: 6.1 scrub starts 2024-03-12T03:38:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:28 smithi173 ceph-mon[33544]: 6.1 scrub ok 2024-03-12T03:38:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:28 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:38:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:28 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:38:28.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:28.802 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:28.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:28.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:28.946 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:28.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:29.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:29.091 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:29.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:29.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:29.226 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:29.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:29.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:29.366 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:29.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:29.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:29.507 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:29.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:29.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:29.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:29.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:29 smithi173 ceph-mon[33544]: pgmap v222: 168 pgs: 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 617 B/s rd, 0 B/s wr, 750 op/s; 20297/149793 objects misplaced (13.550%); 33 MiB/s, 533 objects/s recovering 2024-03-12T03:38:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:29 smithi173 ceph-mon[33544]: osdmap e169: 8 total, 8 up, 7 in 2024-03-12T03:38:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:29 smithi173 ceph-mon[33544]: 4.1b scrub starts 2024-03-12T03:38:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:29 smithi173 ceph-mon[33544]: 4.1b scrub ok 2024-03-12T03:38:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:29 smithi173 ceph-mon[33544]: 6.1d scrub starts 2024-03-12T03:38:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:29 smithi173 ceph-mon[33544]: 6.1d scrub ok 2024-03-12T03:38:29.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:29 smithi169 ceph-mon[29344]: pgmap v222: 168 pgs: 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 617 B/s rd, 0 B/s wr, 750 op/s; 20297/149793 objects misplaced (13.550%); 33 MiB/s, 533 objects/s recovering 2024-03-12T03:38:29.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:29 smithi169 ceph-mon[29344]: osdmap e169: 8 total, 8 up, 7 in 2024-03-12T03:38:29.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:29 smithi169 ceph-mon[29344]: 4.1b scrub starts 2024-03-12T03:38:29.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:29 smithi169 ceph-mon[29344]: 4.1b scrub ok 2024-03-12T03:38:29.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:29 smithi169 ceph-mon[29344]: 6.1d scrub starts 2024-03-12T03:38:29.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:29 smithi169 ceph-mon[29344]: 6.1d scrub ok 2024-03-12T03:38:29.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:29 smithi169 ceph-mon[39949]: pgmap v222: 168 pgs: 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 617 B/s rd, 0 B/s wr, 750 op/s; 20297/149793 objects misplaced (13.550%); 33 MiB/s, 533 objects/s recovering 2024-03-12T03:38:29.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:29 smithi169 ceph-mon[39949]: osdmap e169: 8 total, 8 up, 7 in 2024-03-12T03:38:29.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:29 smithi169 ceph-mon[39949]: 4.1b scrub starts 2024-03-12T03:38:29.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:29 smithi169 ceph-mon[39949]: 4.1b scrub ok 2024-03-12T03:38:29.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:29 smithi169 ceph-mon[39949]: 6.1d scrub starts 2024-03-12T03:38:29.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:29 smithi169 ceph-mon[39949]: 6.1d scrub ok 2024-03-12T03:38:29.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:29.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:29.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:29.876 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:29.909 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:29.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:30.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:30.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:30.050 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:30.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:30.151 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:30.193 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:30.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:30.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:30.318 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:30.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:30.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:30.460 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:30.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:30.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:30.600 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:30.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:30.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:30.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:30 smithi169 ceph-mon[29344]: 3.1a scrub starts 2024-03-12T03:38:30.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:30 smithi169 ceph-mon[29344]: 3.1a scrub ok 2024-03-12T03:38:30.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:30 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:30.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:30 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:30.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:30 smithi169 ceph-mon[39949]: 3.1a scrub starts 2024-03-12T03:38:30.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:30 smithi169 ceph-mon[39949]: 3.1a scrub ok 2024-03-12T03:38:30.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:30 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:30.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:30 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:30.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:30.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:30.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:30.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:30 smithi173 ceph-mon[33544]: 3.1a scrub starts 2024-03-12T03:38:30.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:30 smithi173 ceph-mon[33544]: 3.1a scrub ok 2024-03-12T03:38:30.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:30 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:30.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:30 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:30.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:30.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:30.866 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:30.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:30.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:30.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:30.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:30.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:31.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:31.134 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:31.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:31.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:31.276 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:31.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:31.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:31.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:31.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:31.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:31.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:31.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:31.543 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:31.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:31.645 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:31.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:31 smithi169 ceph-mon[29344]: pgmap v224: 168 pgs: 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 789 op/s; 20297/149793 objects misplaced (13.550%); 37 MiB/s, 586 objects/s recovering 2024-03-12T03:38:31.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:31 smithi169 ceph-mon[29344]: 5.1c scrub starts 2024-03-12T03:38:31.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:31 smithi169 ceph-mon[29344]: 5.1c scrub ok 2024-03-12T03:38:31.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:31 smithi169 ceph-mon[29344]: 5.c scrub starts 2024-03-12T03:38:31.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:31 smithi169 ceph-mon[29344]: 5.c scrub ok 2024-03-12T03:38:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:31 smithi169 ceph-mon[39949]: pgmap v224: 168 pgs: 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 789 op/s; 20297/149793 objects misplaced (13.550%); 37 MiB/s, 586 objects/s recovering 2024-03-12T03:38:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:31 smithi169 ceph-mon[39949]: 5.1c scrub starts 2024-03-12T03:38:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:31 smithi169 ceph-mon[39949]: 5.1c scrub ok 2024-03-12T03:38:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:31 smithi169 ceph-mon[39949]: 5.c scrub starts 2024-03-12T03:38:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:31 smithi169 ceph-mon[39949]: 5.c scrub ok 2024-03-12T03:38:31.675 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:31.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:31.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:31 smithi173 ceph-mon[33544]: pgmap v224: 168 pgs: 1 peering, 1 active+recovering+forced_recovery+undersized+remapped, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 789 op/s; 20297/149793 objects misplaced (13.550%); 37 MiB/s, 586 objects/s recovering 2024-03-12T03:38:31.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:31 smithi173 ceph-mon[33544]: 5.1c scrub starts 2024-03-12T03:38:31.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:31 smithi173 ceph-mon[33544]: 5.1c scrub ok 2024-03-12T03:38:31.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:31 smithi173 ceph-mon[33544]: 5.c scrub starts 2024-03-12T03:38:31.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:31 smithi173 ceph-mon[33544]: 5.c scrub ok 2024-03-12T03:38:31.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:31.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:31.815 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:31.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:31.917 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:31.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:31.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:32.050 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:32.096 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:32.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:32.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:38:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:38:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:38:32.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:32.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:32.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:32.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:32.362 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:32.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:32.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:32.501 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:32.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:32.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:32.603 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:32.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:32.627 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:32.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:32.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:32 smithi169 ceph-mon[29344]: 4.7 scrub starts 2024-03-12T03:38:32.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:32 smithi169 ceph-mon[29344]: 4.7 scrub ok 2024-03-12T03:38:32.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:32 smithi169 ceph-mon[39949]: 4.7 scrub starts 2024-03-12T03:38:32.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:32 smithi169 ceph-mon[39949]: 4.7 scrub ok 2024-03-12T03:38:32.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:32.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:38:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:38:32.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:32 smithi173 ceph-mon[33544]: 4.7 scrub starts 2024-03-12T03:38:32.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:32 smithi173 ceph-mon[33544]: 4.7 scrub ok 2024-03-12T03:38:32.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:32.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:32.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:32.902 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:32.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:33.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:33.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:33.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:33.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:33.172 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:33.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:33.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:33.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:33.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:33.313 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:33.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:33.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:33.452 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:33.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:33.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:33.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-12T03:38:33.587 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:33.604 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:33.622 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:33.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:33.678 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:33 smithi169 ceph-mon[29344]: pgmap v225: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.39k op/s; 16349/136560 objects misplaced (11.972%); 53 MiB/s, 855 objects/s recovering 2024-03-12T03:38:33.678 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:33 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:33.678 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:33 smithi169 ceph-mon[29344]: osdmap e170: 8 total, 8 up, 7 in 2024-03-12T03:38:33.679 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:33 smithi169 ceph-mon[39949]: pgmap v225: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.39k op/s; 16349/136560 objects misplaced (11.972%); 53 MiB/s, 855 objects/s recovering 2024-03-12T03:38:33.679 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:33 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:33.679 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:33 smithi169 ceph-mon[39949]: osdmap e170: 8 total, 8 up, 7 in 2024-03-12T03:38:33.723 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:33.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:33 smithi173 ceph-mon[33544]: pgmap v225: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.39k op/s; 16349/136560 objects misplaced (11.972%); 53 MiB/s, 855 objects/s recovering 2024-03-12T03:38:33.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:33 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:33.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:33 smithi173 ceph-mon[33544]: osdmap e170: 8 total, 8 up, 7 in 2024-03-12T03:38:33.795 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:33.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:33.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:33.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:33.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:33.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:34.031 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:34.054 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:34.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:34.156 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:34.198 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:34.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:34.213 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:34.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:34.323 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:34.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:34.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:34.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:34.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:34.452 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:34.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:34.553 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:34.588 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:34.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:34.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:34 smithi169 ceph-mon[29344]: 4.c scrub starts 2024-03-12T03:38:34.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:34 smithi169 ceph-mon[29344]: 4.c scrub ok 2024-03-12T03:38:34.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:34 smithi169 ceph-mon[29344]: 4.f scrub starts 2024-03-12T03:38:34.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:34 smithi169 ceph-mon[29344]: 4.f scrub ok 2024-03-12T03:38:34.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:34 smithi169 ceph-mon[39949]: 4.c scrub starts 2024-03-12T03:38:34.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:34 smithi169 ceph-mon[39949]: 4.c scrub ok 2024-03-12T03:38:34.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:34 smithi169 ceph-mon[39949]: 4.f scrub starts 2024-03-12T03:38:34.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:34 smithi169 ceph-mon[39949]: 4.f scrub ok 2024-03-12T03:38:34.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:34.734 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:34.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:34.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:34 smithi173 ceph-mon[33544]: 4.c scrub starts 2024-03-12T03:38:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:34 smithi173 ceph-mon[33544]: 4.c scrub ok 2024-03-12T03:38:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:34 smithi173 ceph-mon[33544]: 4.f scrub starts 2024-03-12T03:38:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:34 smithi173 ceph-mon[33544]: 4.f scrub ok 2024-03-12T03:38:34.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:34.876 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:34.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:34.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:35.023 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:35.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:35.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:35.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:35.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:35.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:35.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:35.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:35.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:35.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:35.436 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:35.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:35.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:35.574 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:35.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:35.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:35.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:35 smithi169 ceph-mon[29344]: pgmap v227: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 674 B/s rd, 0 B/s wr, 621 op/s; 16349/136560 objects misplaced (11.972%); 17 MiB/s, 279 objects/s recovering 2024-03-12T03:38:35.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:35 smithi169 ceph-mon[29344]: 3.14 deep-scrub starts 2024-03-12T03:38:35.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:35 smithi169 ceph-mon[29344]: 3.14 deep-scrub ok 2024-03-12T03:38:35.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:35 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:38:35.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:35 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:38:35.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:35 smithi169 ceph-mon[39949]: pgmap v227: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 674 B/s rd, 0 B/s wr, 621 op/s; 16349/136560 objects misplaced (11.972%); 17 MiB/s, 279 objects/s recovering 2024-03-12T03:38:35.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:35 smithi169 ceph-mon[39949]: 3.14 deep-scrub starts 2024-03-12T03:38:35.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:35 smithi169 ceph-mon[39949]: 3.14 deep-scrub ok 2024-03-12T03:38:35.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:35 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:38:35.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:35 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:38:35.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:35.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:35.708 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:35.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:35 smithi173 ceph-mon[33544]: pgmap v227: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 674 B/s rd, 0 B/s wr, 621 op/s; 16349/136560 objects misplaced (11.972%); 17 MiB/s, 279 objects/s recovering 2024-03-12T03:38:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:35 smithi173 ceph-mon[33544]: 3.14 deep-scrub starts 2024-03-12T03:38:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:35 smithi173 ceph-mon[33544]: 3.14 deep-scrub ok 2024-03-12T03:38:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:35 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:38:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:35 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:38:35.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:35.836 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:35.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:35.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:35.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:35.962 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:35.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:36.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:36.091 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:36.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:36.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:36.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:36.218 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:36.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:36.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:36.342 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:36.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:36.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:36.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:36.476 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:36.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:36.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:36.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:36.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:36.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:38:36.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:38:36.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[29344]: 6.14 scrub starts 2024-03-12T03:38:36.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[29344]: 6.14 scrub ok 2024-03-12T03:38:36.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[29344]: osdmap e171: 8 total, 8 up, 7 in 2024-03-12T03:38:36.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[29344]: 3.19 scrub starts 2024-03-12T03:38:36.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[29344]: 3.19 scrub ok 2024-03-12T03:38:36.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[29344]: osdmap e172: 8 total, 8 up, 7 in 2024-03-12T03:38:36.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:38:36.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:38:36.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[39949]: 6.14 scrub starts 2024-03-12T03:38:36.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[39949]: 6.14 scrub ok 2024-03-12T03:38:36.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[39949]: osdmap e171: 8 total, 8 up, 7 in 2024-03-12T03:38:36.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[39949]: 3.19 scrub starts 2024-03-12T03:38:36.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[39949]: 3.19 scrub ok 2024-03-12T03:38:36.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:36 smithi169 ceph-mon[39949]: osdmap e172: 8 total, 8 up, 7 in 2024-03-12T03:38:36.716 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:36 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:38:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:36 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:38:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:36 smithi173 ceph-mon[33544]: 6.14 scrub starts 2024-03-12T03:38:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:36 smithi173 ceph-mon[33544]: 6.14 scrub ok 2024-03-12T03:38:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:36 smithi173 ceph-mon[33544]: osdmap e171: 8 total, 8 up, 7 in 2024-03-12T03:38:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:36 smithi173 ceph-mon[33544]: 3.19 scrub starts 2024-03-12T03:38:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:36 smithi173 ceph-mon[33544]: 3.19 scrub ok 2024-03-12T03:38:36.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:36 smithi173 ceph-mon[33544]: osdmap e172: 8 total, 8 up, 7 in 2024-03-12T03:38:36.745 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:36.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:36.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:36.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:36.872 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:36.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:36.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:36.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:36.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:37.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:37.124 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:37.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:37.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:37.252 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:37.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:37.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:37.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:37.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:37.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:37.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:37.480 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:37.521 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:37.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:37.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:37.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:37.662 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:37.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:37.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: pgmap v229: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 776 op/s; 14778/132255 objects misplaced (11.174%); 25 MiB/s, 401 objects/s recovering 2024-03-12T03:38:37.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:38:37.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:38:37.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: 5.a scrub starts 2024-03-12T03:38:37.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: 5.a scrub ok 2024-03-12T03:38:37.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: 1.2 scrub starts 2024-03-12T03:38:37.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: 1.2 scrub ok 2024-03-12T03:38:37.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: 7.7 scrub starts 2024-03-12T03:38:37.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: 3.0 deep-scrub starts 2024-03-12T03:38:37.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: 3.0 deep-scrub ok 2024-03-12T03:38:37.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: 7.7 scrub ok 2024-03-12T03:38:37.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: 7.18 scrub starts 2024-03-12T03:38:37.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: 7.18 scrub ok 2024-03-12T03:38:37.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[39949]: osdmap e173: 8 total, 8 up, 7 in 2024-03-12T03:38:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: pgmap v229: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 776 op/s; 14778/132255 objects misplaced (11.174%); 25 MiB/s, 401 objects/s recovering 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: 5.a scrub starts 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: 5.a scrub ok 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: 1.2 scrub starts 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: 1.2 scrub ok 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: 7.7 scrub starts 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: 3.0 deep-scrub starts 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: 3.0 deep-scrub ok 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: 7.7 scrub ok 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: 7.18 scrub starts 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: 7.18 scrub ok 2024-03-12T03:38:37.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:37 smithi169 ceph-mon[29344]: osdmap e173: 8 total, 8 up, 7 in 2024-03-12T03:38:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: pgmap v229: 168 pgs: 1 active+recovering+forced_recovery+undersized+remapped, 1 active+clean+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 776 op/s; 14778/132255 objects misplaced (11.174%); 25 MiB/s, 401 objects/s recovering 2024-03-12T03:38:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:38:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:38:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: 5.a scrub starts 2024-03-12T03:38:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: 5.a scrub ok 2024-03-12T03:38:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: 1.2 scrub starts 2024-03-12T03:38:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: 1.2 scrub ok 2024-03-12T03:38:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: 7.7 scrub starts 2024-03-12T03:38:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: 3.0 deep-scrub starts 2024-03-12T03:38:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: 3.0 deep-scrub ok 2024-03-12T03:38:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: 7.7 scrub ok 2024-03-12T03:38:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: 7.18 scrub starts 2024-03-12T03:38:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: 7.18 scrub ok 2024-03-12T03:38:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:37 smithi173 ceph-mon[33544]: osdmap e173: 8 total, 8 up, 7 in 2024-03-12T03:38:37.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:37.797 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:37.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:37.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:37.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:37.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:37.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:38.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:38.061 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:38.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:38.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:38.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:38.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:38.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:38.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:38.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:38.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:38.486 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:38.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:38.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:38.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:38.614 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:38 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:38:38.614 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:38 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:38:38.614 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:38 smithi169 ceph-mon[29344]: 1.6 scrub starts 2024-03-12T03:38:38.614 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:38 smithi169 ceph-mon[29344]: 1.6 scrub ok 2024-03-12T03:38:38.614 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:38 smithi169 ceph-mon[29344]: 7.4 scrub starts 2024-03-12T03:38:38.614 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:38 smithi169 ceph-mon[29344]: osdmap e174: 8 total, 8 up, 7 in 2024-03-12T03:38:38.614 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:38 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:38:38.615 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:38 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:38:38.615 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:38 smithi169 ceph-mon[39949]: 1.6 scrub starts 2024-03-12T03:38:38.615 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:38 smithi169 ceph-mon[39949]: 1.6 scrub ok 2024-03-12T03:38:38.615 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:38 smithi169 ceph-mon[39949]: 7.4 scrub starts 2024-03-12T03:38:38.615 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:38 smithi169 ceph-mon[39949]: osdmap e174: 8 total, 8 up, 7 in 2024-03-12T03:38:38.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:38.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:38.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:38 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:38:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:38 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:38:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:38 smithi173 ceph-mon[33544]: 1.6 scrub starts 2024-03-12T03:38:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:38 smithi173 ceph-mon[33544]: 1.6 scrub ok 2024-03-12T03:38:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:38 smithi173 ceph-mon[33544]: 7.4 scrub starts 2024-03-12T03:38:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:38 smithi173 ceph-mon[33544]: osdmap e174: 8 total, 8 up, 7 in 2024-03-12T03:38:38.745 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:38:38.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:38.765 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:38.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:38.794 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:38:38.834 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:38:38.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:38.893 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:38:38.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:38.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:38.941 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:38:38.979 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:38:38.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:39.018 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:38:39.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:39.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:39.062 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:38:39.072 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:38:39.072 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:38:39.072 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:38:39.072 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:38:39.073 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:38:39.073 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:38:39.073 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:38:39.073 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:38:39.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:39.169 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:39.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:39.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:39.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:39.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:39.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:39.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:39.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:39.528 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:39.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:39.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:39.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:39.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: pgmap v232: 168 pgs: 1 creating+peering, 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.5 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 911 op/s; 12710/123882 objects misplaced (10.260%); 26 MiB/s, 415 objects/s recovering 2024-03-12T03:38:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: 7.4 scrub ok 2024-03-12T03:38:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-12T03:38:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-12T03:38:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: 1.0 scrub starts 2024-03-12T03:38:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: 1.0 scrub ok 2024-03-12T03:38:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: 1.7 scrub starts 2024-03-12T03:38:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: 1.7 scrub ok 2024-03-12T03:38:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1782192715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:38:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: 7.6 scrub starts 2024-03-12T03:38:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: 4.10 scrub starts 2024-03-12T03:38:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: 4.10 scrub ok 2024-03-12T03:38:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[29344]: osdmap e175: 8 total, 8 up, 7 in 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: pgmap v232: 168 pgs: 1 creating+peering, 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.5 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 911 op/s; 12710/123882 objects misplaced (10.260%); 26 MiB/s, 415 objects/s recovering 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: 7.4 scrub ok 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: 1.0 scrub starts 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: 1.0 scrub ok 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: 1.7 scrub starts 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: 1.7 scrub ok 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1782192715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: 7.6 scrub starts 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: 4.10 scrub starts 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: 4.10 scrub ok 2024-03-12T03:38:39.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:39 smithi169 ceph-mon[39949]: osdmap e175: 8 total, 8 up, 7 in 2024-03-12T03:38:39.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:39.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: pgmap v232: 168 pgs: 1 creating+peering, 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.5 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 911 op/s; 12710/123882 objects misplaced (10.260%); 26 MiB/s, 415 objects/s recovering 2024-03-12T03:38:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: 7.4 scrub ok 2024-03-12T03:38:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-12T03:38:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-12T03:38:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: 1.0 scrub starts 2024-03-12T03:38:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: 1.0 scrub ok 2024-03-12T03:38:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: 1.7 scrub starts 2024-03-12T03:38:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: 1.7 scrub ok 2024-03-12T03:38:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1782192715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:38:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: 7.6 scrub starts 2024-03-12T03:38:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: 4.10 scrub starts 2024-03-12T03:38:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: 4.10 scrub ok 2024-03-12T03:38:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:39 smithi173 ceph-mon[33544]: osdmap e175: 8 total, 8 up, 7 in 2024-03-12T03:38:39.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:39.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:39.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:39.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:39.905 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:39.929 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:39.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:39.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:40.031 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:40.055 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:40.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:40.091 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:38:40.091 INFO:teuthology.orchestra.run.smithi169.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-12T03:38:40.091 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:38:40.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:40.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:40.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:40.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:40.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:40.320 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:40.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:40.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:40.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:40.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:40.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:40.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:40.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:40.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[29344]: 7.6 scrub ok 2024-03-12T03:38:40.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[29344]: 7.19 scrub starts 2024-03-12T03:38:40.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[29344]: 7.11 scrub starts 2024-03-12T03:38:40.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[29344]: 4.4 scrub starts 2024-03-12T03:38:40.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[29344]: 4.4 scrub ok 2024-03-12T03:38:40.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[29344]: 7.11 scrub ok 2024-03-12T03:38:40.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[29344]: 7.19 scrub ok 2024-03-12T03:38:40.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[29344]: 7.9 scrub starts 2024-03-12T03:38:40.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[29344]: 7.9 scrub ok 2024-03-12T03:38:40.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[29344]: osdmap e176: 8 total, 8 up, 7 in 2024-03-12T03:38:40.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[39949]: 7.6 scrub ok 2024-03-12T03:38:40.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[39949]: 7.19 scrub starts 2024-03-12T03:38:40.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[39949]: 7.11 scrub starts 2024-03-12T03:38:40.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[39949]: 4.4 scrub starts 2024-03-12T03:38:40.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[39949]: 4.4 scrub ok 2024-03-12T03:38:40.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[39949]: 7.11 scrub ok 2024-03-12T03:38:40.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[39949]: 7.19 scrub ok 2024-03-12T03:38:40.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[39949]: 7.9 scrub starts 2024-03-12T03:38:40.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[39949]: 7.9 scrub ok 2024-03-12T03:38:40.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:40 smithi169 ceph-mon[39949]: osdmap e176: 8 total, 8 up, 7 in 2024-03-12T03:38:40.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:40.731 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:40.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:40.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:40 smithi173 ceph-mon[33544]: 7.6 scrub ok 2024-03-12T03:38:40.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:40 smithi173 ceph-mon[33544]: 7.19 scrub starts 2024-03-12T03:38:40.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:40 smithi173 ceph-mon[33544]: 7.11 scrub starts 2024-03-12T03:38:40.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:40 smithi173 ceph-mon[33544]: 4.4 scrub starts 2024-03-12T03:38:40.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:40 smithi173 ceph-mon[33544]: 4.4 scrub ok 2024-03-12T03:38:40.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:40 smithi173 ceph-mon[33544]: 7.11 scrub ok 2024-03-12T03:38:40.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:40 smithi173 ceph-mon[33544]: 7.19 scrub ok 2024-03-12T03:38:40.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:40 smithi173 ceph-mon[33544]: 7.9 scrub starts 2024-03-12T03:38:40.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:40 smithi173 ceph-mon[33544]: 7.9 scrub ok 2024-03-12T03:38:40.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:40 smithi173 ceph-mon[33544]: osdmap e176: 8 total, 8 up, 7 in 2024-03-12T03:38:40.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:40.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:40.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:40.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:41.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:41.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:41.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:41.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:41.172 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:41.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:41.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:41.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:41.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:41.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:41.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:41.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:41.458 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:41.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:41.559 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:41.605 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:41.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:41.691 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:41 smithi169 ceph-mon[29344]: pgmap v235: 168 pgs: 1 creating+peering, 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.5 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 697 op/s; 12710/123882 objects misplaced (10.260%); 16 MiB/s, 256 objects/s recovering 2024-03-12T03:38:41.691 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:41 smithi169 ceph-mon[29344]: 1.9 scrub starts 2024-03-12T03:38:41.692 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:41 smithi169 ceph-mon[29344]: 1.9 scrub ok 2024-03-12T03:38:41.692 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:41 smithi169 ceph-mon[39949]: pgmap v235: 168 pgs: 1 creating+peering, 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.5 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 697 op/s; 12710/123882 objects misplaced (10.260%); 16 MiB/s, 256 objects/s recovering 2024-03-12T03:38:41.692 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:41 smithi169 ceph-mon[39949]: 1.9 scrub starts 2024-03-12T03:38:41.692 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:41 smithi169 ceph-mon[39949]: 1.9 scrub ok 2024-03-12T03:38:41.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:41.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:41 smithi173 ceph-mon[33544]: pgmap v235: 168 pgs: 1 creating+peering, 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 155 active+clean; 2.5 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 697 op/s; 12710/123882 objects misplaced (10.260%); 16 MiB/s, 256 objects/s recovering 2024-03-12T03:38:41.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:41 smithi173 ceph-mon[33544]: 1.9 scrub starts 2024-03-12T03:38:41.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:41 smithi173 ceph-mon[33544]: 1.9 scrub ok 2024-03-12T03:38:41.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:41.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:41.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:41.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:41.874 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:41.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:41.884 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:41.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:41.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:42.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:42.017 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:38:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:38:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:38:42.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:42.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:42.119 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:42.171 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:42.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:42.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:42.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:42.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:42.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:42.404 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:42.444 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:42.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:42.462 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:38:42.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:42.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:42.579 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:42.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:42.610 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[29344]: 3.4 scrub starts 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[29344]: 3.4 scrub ok 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[29344]: 7.8 scrub starts 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[29344]: 6.19 scrub starts 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[29344]: 6.19 scrub ok 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[29344]: 3.7 scrub starts 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[29344]: 3.7 scrub ok 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3982821636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[39949]: 3.4 scrub starts 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[39949]: 3.4 scrub ok 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[39949]: 7.8 scrub starts 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[39949]: 6.19 scrub starts 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[39949]: 6.19 scrub ok 2024-03-12T03:38:42.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[39949]: 3.7 scrub starts 2024-03-12T03:38:42.612 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[39949]: 3.7 scrub ok 2024-03-12T03:38:42.612 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:42 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3982821636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:38:42.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:42.718 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:42.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:42.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:38:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:38:42.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:42 smithi173 ceph-mon[33544]: 3.4 scrub starts 2024-03-12T03:38:42.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:42 smithi173 ceph-mon[33544]: 3.4 scrub ok 2024-03-12T03:38:42.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:42 smithi173 ceph-mon[33544]: 7.8 scrub starts 2024-03-12T03:38:42.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:42 smithi173 ceph-mon[33544]: 6.19 scrub starts 2024-03-12T03:38:42.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:42 smithi173 ceph-mon[33544]: 6.19 scrub ok 2024-03-12T03:38:42.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:42 smithi173 ceph-mon[33544]: 3.7 scrub starts 2024-03-12T03:38:42.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:42 smithi173 ceph-mon[33544]: 3.7 scrub ok 2024-03-12T03:38:42.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:42 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3982821636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:38:42.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:42.859 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:42.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:42.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:42.993 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:42.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:42.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:43.095 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:43.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:43.139 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:43.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:43.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:38:43.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:43.310 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:43.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:43.412 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:43.458 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:43.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:43.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:43.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:43.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:43.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:43.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[29344]: 5.19 scrub starts 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[29344]: 5.19 scrub ok 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[29344]: 7.8 scrub ok 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[29344]: pgmap v237: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 2.3 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 10647/112836 objects misplaced (9.436%) 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[29344]: 4.9 scrub starts 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[29344]: 4.9 scrub ok 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[29344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3982821636' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[29344]: osdmap e177: 8 total, 8 up, 7 in 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[29344]: osdmap e178: 8 total, 8 up, 7 in 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[39949]: 5.19 scrub starts 2024-03-12T03:38:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[39949]: 5.19 scrub ok 2024-03-12T03:38:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[39949]: 7.8 scrub ok 2024-03-12T03:38:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[39949]: pgmap v237: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 2.3 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 10647/112836 objects misplaced (9.436%) 2024-03-12T03:38:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[39949]: 4.9 scrub starts 2024-03-12T03:38:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[39949]: 4.9 scrub ok 2024-03-12T03:38:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[39949]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-12T03:38:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:38:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3982821636' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:38:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[39949]: osdmap e177: 8 total, 8 up, 7 in 2024-03-12T03:38:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:43 smithi169 ceph-mon[39949]: osdmap e178: 8 total, 8 up, 7 in 2024-03-12T03:38:43.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:43.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:43.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:43.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:43 smithi173 ceph-mon[33544]: 5.19 scrub starts 2024-03-12T03:38:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:43 smithi173 ceph-mon[33544]: 5.19 scrub ok 2024-03-12T03:38:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:43 smithi173 ceph-mon[33544]: 7.8 scrub ok 2024-03-12T03:38:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:43 smithi173 ceph-mon[33544]: pgmap v237: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 2.3 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 10647/112836 objects misplaced (9.436%) 2024-03-12T03:38:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:43 smithi173 ceph-mon[33544]: 4.9 scrub starts 2024-03-12T03:38:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:43 smithi173 ceph-mon[33544]: 4.9 scrub ok 2024-03-12T03:38:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:43 smithi173 ceph-mon[33544]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-12T03:38:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:43 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:38:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:43 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3982821636' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:38:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:43 smithi173 ceph-mon[33544]: osdmap e177: 8 total, 8 up, 7 in 2024-03-12T03:38:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:43 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:43 smithi173 ceph-mon[33544]: osdmap e178: 8 total, 8 up, 7 in 2024-03-12T03:38:43.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:43.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:43.870 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:43.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:43.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:44.039 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:44.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:44.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:44.188 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:44.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:44.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:44.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:44.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:44.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:44.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:44.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:44.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:44.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:44.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:44.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:44.632 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:44.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:44.707 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:38:44.707 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:38:44.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:44.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:44.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:44.871 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:44.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:44.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:45.015 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:45.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:45.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:45.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:45.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:45.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:45.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:45.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:45.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:45.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:45.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:45.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:45.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:45.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:38:45.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:45.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:45.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:45.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:45.688 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:45.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:45 smithi173 ceph-mon[33544]: pgmap v240: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 2.3 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 10647/112836 objects misplaced (9.436%) 2024-03-12T03:38:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:45 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:45 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:45.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:45.813 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:45.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:45.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:45 smithi169 ceph-mon[29344]: pgmap v240: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 2.3 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 10647/112836 objects misplaced (9.436%) 2024-03-12T03:38:45.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:45 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:45.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:45 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:45 smithi169 ceph-mon[39949]: pgmap v240: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 2.3 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 10647/112836 objects misplaced (9.436%) 2024-03-12T03:38:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:45 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:38:45.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:45 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:38:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:45.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:46.052 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:46.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:46.096 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:46.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:46.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:46.224 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:46.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:46.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:46.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:46.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:46.465 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:46.503 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:46.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:46.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:46.622 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:38:46.623 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:38:46.623 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-12T03:38:46.623 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:38:46.642 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:46.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:46.745 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:46.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:46.799 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:46.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:46.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-12T03:38:46.900 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:46.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:46.972 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:46.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:47.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:47.122 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:47.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:47.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:47.270 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:47.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:47.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:47.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:47.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:47.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:47.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:47.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:47.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:47.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:47.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:47.672 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:47.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:47 smithi169 ceph-mon[29344]: pgmap v241: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 2.2 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 10043/106881 objects misplaced (9.396%) 2024-03-12T03:38:47.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:47 smithi169 ceph-mon[39949]: pgmap v241: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 2.2 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 10043/106881 objects misplaced (9.396%) 2024-03-12T03:38:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:47.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:47 smithi173 ceph-mon[33544]: pgmap v241: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 2.2 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 10043/106881 objects misplaced (9.396%) 2024-03-12T03:38:47.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:47.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:47.884 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:47.903 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:47.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:47.930 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:47.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:47.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:48.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:48.088 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:48.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:48.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:48.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:48.244 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:48.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:48.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:48.380 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:48.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:48.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:48.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:48.530 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:48.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:48.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:48.632 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:48.676 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:48.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:48.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:48.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:48.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:48.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:48.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:48.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:48.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:48.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:49.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:49.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:49.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:49.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:49.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:49.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:49.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:49.353 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:49.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:49.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:49.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:49.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:49.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:49.633 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:49.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:49.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:49 smithi169 ceph-mon[29344]: pgmap v242: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 1.9 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 8843/94056 objects misplaced (9.402%) 2024-03-12T03:38:49.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:49 smithi169 ceph-mon[39949]: pgmap v242: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 1.9 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 8843/94056 objects misplaced (9.402%) 2024-03-12T03:38:49.735 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:49.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:49 smithi173 ceph-mon[33544]: pgmap v242: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 1.9 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 8843/94056 objects misplaced (9.402%) 2024-03-12T03:38:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:49.782 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:49.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:49.806 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:38:49.860 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:38:49.883 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:49.911 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:38:49.930 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:49.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:49.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:49.956 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:38:49.993 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:38:50.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:50.038 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:38:50.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:50.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:50.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:50.080 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:38:50.122 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:38:50.130 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:38:50.130 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:38:50.130 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:38:50.130 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:38:50.131 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:38:50.131 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:38:50.131 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:38:50.131 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:38:50.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:50.212 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:50.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:50.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:50.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:50.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:50.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:50.448 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:50.491 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:50.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:50.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:50.593 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:50.646 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:50.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:50.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:50.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:50.752 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:50 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2323327066' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:38:50.752 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:50 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2323327066' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:38:50.792 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:50.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:50.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:50.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:50.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:50.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:50 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2323327066' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:38:51.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:51.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:51.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:51.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:51.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:51.218 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:51.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:51.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:51.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:51.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:51.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:51.456 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:51.494 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:51.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:51.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:51.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:51.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:51.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:51.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:51 smithi169 ceph-mon[29344]: pgmap v243: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 1.9 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1 op/s; 8843/94056 objects misplaced (9.402%) 2024-03-12T03:38:51.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:51 smithi169 ceph-mon[39949]: pgmap v243: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 1.9 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1 op/s; 8843/94056 objects misplaced (9.402%) 2024-03-12T03:38:51.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:51.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:51.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:51.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:51.904 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:51.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:51 smithi173 ceph-mon[33544]: pgmap v243: 168 pgs: 8 active+remapped+backfill_toofull, 160 active+clean; 1.9 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1 op/s; 8843/94056 objects misplaced (9.402%) 2024-03-12T03:38:52.007 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:52.050 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:52.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:52.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:52.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:38:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:38:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:38:52.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:52.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:52.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:52.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:52.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:52.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:52.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:52.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:52.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:52.586 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:52.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:52.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:52.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:52.729 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:52.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:52.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:52.868 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:52 smithi169 ceph-mon[29344]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:38:52.868 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:52 smithi169 ceph-mon[29344]: osdmap e179: 8 total, 8 up, 7 in 2024-03-12T03:38:52.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:52 smithi169 ceph-mon[39949]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:38:52.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:52 smithi169 ceph-mon[39949]: osdmap e179: 8 total, 8 up, 7 in 2024-03-12T03:38:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:52.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:52.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:52.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:52.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:52 smithi173 ceph-mon[33544]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:38:52.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:52 smithi173 ceph-mon[33544]: osdmap e179: 8 total, 8 up, 7 in 2024-03-12T03:38:52.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:38:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:38:53.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:53.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:53.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:53.142 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:53.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:53.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:53.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:53.286 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:53.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:53.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:53.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:53.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:53.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:53.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:53.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:53.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:53.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:53.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:53.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:53.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:53.856 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:53.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:53.876 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:53 smithi169 ceph-mon[29344]: pgmap v244: 168 pgs: 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 1.5 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 678 op/s; 6102/74982 objects misplaced (8.138%) 2024-03-12T03:38:53.876 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:53 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:38:53.876 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:53 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:38:53.876 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:53 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:53.876 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:53 smithi169 ceph-mon[29344]: osdmap e180: 8 total, 8 up, 7 in 2024-03-12T03:38:53.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:53 smithi169 ceph-mon[39949]: pgmap v244: 168 pgs: 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 1.5 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 678 op/s; 6102/74982 objects misplaced (8.138%) 2024-03-12T03:38:53.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:53 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:38:53.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:53 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:38:53.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:53 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:53.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:53 smithi169 ceph-mon[39949]: osdmap e180: 8 total, 8 up, 7 in 2024-03-12T03:38:53.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:53 smithi173 ceph-mon[33544]: pgmap v244: 168 pgs: 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 1.5 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 678 op/s; 6102/74982 objects misplaced (8.138%) 2024-03-12T03:38:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:53 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:38:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:53 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:38:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:53 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:53 smithi173 ceph-mon[33544]: osdmap e180: 8 total, 8 up, 7 in 2024-03-12T03:38:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:54.010 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:54.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:54.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:54.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:54.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:54.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:54.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:54.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:54.376 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:54.418 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:54.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:54.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:54.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:54.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:54.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:54.693 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:54.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:54.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:54.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:54.820 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:54.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:54.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:54.945 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:54.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:55.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:55.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:55.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:55.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:55.174 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:55.199 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:55.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:55.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:55.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:55.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:55.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:55.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:55.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:55.452 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:55.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:55.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:55.583 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:55.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:55.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:55.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:55.713 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:55.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:55.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:55.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:55.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:55.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:55.860 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:55 smithi169 ceph-mon[29344]: pgmap v247: 168 pgs: 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 1.5 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.33k op/s; 6102/74982 objects misplaced (8.138%); 7.5 MiB/s, 120 objects/s recovering 2024-03-12T03:38:55.860 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:55 smithi169 ceph-mon[29344]: osdmap e181: 8 total, 8 up, 7 in 2024-03-12T03:38:55.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:55 smithi169 ceph-mon[39949]: pgmap v247: 168 pgs: 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 1.5 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.33k op/s; 6102/74982 objects misplaced (8.138%); 7.5 MiB/s, 120 objects/s recovering 2024-03-12T03:38:55.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:55 smithi169 ceph-mon[39949]: osdmap e181: 8 total, 8 up, 7 in 2024-03-12T03:38:55.947 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:55.981 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:55.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:55 smithi173 ceph-mon[33544]: pgmap v247: 168 pgs: 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 1.5 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.33k op/s; 6102/74982 objects misplaced (8.138%); 7.5 MiB/s, 120 objects/s recovering 2024-03-12T03:38:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:55 smithi173 ceph-mon[33544]: osdmap e181: 8 total, 8 up, 7 in 2024-03-12T03:38:56.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:56.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:38:56.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:56.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:56.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:56.212 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:56.240 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:56.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:56.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:56.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:56.367 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:56.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:56.377 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:107725 7.16 7:6f9b276b:::benchmark_data_smithi169_79822_object107724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:48.269004+0000", 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 127.58595878200001, 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0593569410000001, 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 107725 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.269004+0000", 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.269004+0000", 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.269006+0000", 2024-03-12T03:38:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3489999999999999e-06 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.269070+0000", 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3579999999999998e-05 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.269095+0000", 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4842999999999999e-05 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.269106+0000", 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1090999999999999e-05 2024-03-12T03:38:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.477048+0000", 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.207942302 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.477056+0000", 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5159999999999997e-06 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.328289+0000", 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.85123356100000003 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.328361+0000", 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1699000000000006e-05 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:107730 7.6 7:653605d5:::benchmark_data_smithi169_79822_object107729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:48.269129+0000", 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 127.58583415, 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0509726800000001, 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:38:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 107730 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.269129+0000", 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.269129+0000", 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.269129+0000", 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8299999999999997e-07 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.269153+0000", 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3615999999999999e-05 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.269164+0000", 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1756000000000001e-05 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.269172+0000", 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0430000000000002e-06 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.476514+0000", 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20734276400000001 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.476540+0000", 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5942e-05 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.306552+0000", 2024-03-12T03:38:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.83001225700000003 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for degraded object", 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.309820+0000", 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032673239999999998 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.320009+0000", 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010189432 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.320101+0000", 2024-03-12T03:38:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1963e-05 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:107736 7.16 7:6c0678ae:::benchmark_data_smithi169_79822_object107735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:48.407127+0000", 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 127.447836059, 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.93801350299999997, 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 107736 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.407127+0000", 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.407127+0000", 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.407129+0000", 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8449999999999999e-06 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.407155+0000", 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5298000000000001e-05 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.407178+0000", 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3351999999999999e-05 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.407190+0000", 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2019e-05 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.477078+0000", 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.069887947000000006 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.477084+0000", 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8869999999999997e-06 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.328597+0000", 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.85151252600000005 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.328726+0000", 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012972600000000001 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.329010+0000", 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00028357700000000003 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.334610+0000", 2024-03-12T03:38:56.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056006789999999999 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.338139+0000", 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0035288089999999999 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.345045+0000", 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069053409999999997 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.345101+0000", 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6617999999999999e-05 2024-03-12T03:38:56.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.345140+0000", 2024-03-12T03:38:56.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8878999999999998e-05 2024-03-12T03:38:56.393 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.393 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.393 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.393 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:107737 7.16 7:6c187044:::benchmark_data_smithi169_79822_object107736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:38:56.393 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:48.418265+0000", 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 127.436697231, 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.92702620300000005, 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 107737 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.418265+0000", 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.418265+0000", 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.418268+0000", 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4119999999999999e-06 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.418304+0000", 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5976000000000002e-05 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.418331+0000", 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7012000000000002e-05 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.418344+0000", 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3407000000000001e-05 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.477104+0000", 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.058759604999999999 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.477110+0000", 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9580000000000004e-06 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.330226+0000", 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.85311662200000005 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.330350+0000", 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012306099999999999 2024-03-12T03:38:56.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:38:56.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.332477+0000", 2024-03-12T03:38:56.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021278550000000001 2024-03-12T03:38:56.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.337009+0000", 2024-03-12T03:38:56.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045319399999999999 2024-03-12T03:38:56.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.338145+0000", 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001135654 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.345186+0000", 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0070408939999999998 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.345236+0000", 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0451999999999997e-05 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.345292+0000", 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5355000000000001e-05 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:163201 7.13 7:ca3fc84c:::benchmark_data_smithi169_79822_object45784:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:01.806813+0000", 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 54.048150016999998, 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0158324750000001, 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:38:56.400 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 163201 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.806813+0000", 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.806813+0000", 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.806814+0000", 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.505e-06 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.806818+0000", 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3120000000000004e-06 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.806827+0000", 2024-03-12T03:38:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9910000000000008e-06 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.806840+0000", 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2704999999999999e-05 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.811729+0000", 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048891150000000003 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.811739+0000", 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7680000000000003e-06 2024-03-12T03:38:56.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:02.822565+0000", 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0108257730000001 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:02.822645+0000", 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0306e-05 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.404 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:163330 7.13 7:c8ae95c3:::benchmark_data_smithi169_79822_object45913:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:01.928659+0000", 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 53.926303519000001, 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.909432358, 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 163330 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.928659+0000", 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.928659+0000", 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.928660+0000", 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1289999999999999e-06 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.928665+0000", 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2009999999999998e-06 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.928680+0000", 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4045e-05 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.928687+0000", 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6569999999999997e-06 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.928990+0000", 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030290200000000002 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:01.928999+0000", 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7110000000000001e-06 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:02.823052+0000", 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89405319900000002 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:02.823180+0000", 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012810499999999999 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2,7", 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:02.823537+0000", 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00035658599999999999 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:02.827820+0000", 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004282915 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:02.827879+0000", 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8907999999999999e-05 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:02.837901+0000", 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010022457 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:02.838046+0000", 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014463099999999999 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:02.838092+0000", 2024-03-12T03:38:56.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5911999999999997e-05 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:193438 7.18 7:1a771b38:::benchmark_data_smithi169_79822_object76021:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e171)", 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:35.394950+0000", 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.460012368000001, 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0202605199999999, 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 193438 2024-03-12T03:38:56.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.394950+0000", 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.394950+0000", 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.394952+0000", 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5039999999999999e-06 2024-03-12T03:38:56.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.394958+0000", 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1129999999999997e-06 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.394973+0000", 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5007999999999999e-05 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.394982+0000", 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0720000000000006e-06 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.398085+0000", 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0031027049999999999 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.398095+0000", 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.842e-06 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.415119+0000", 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0170240319999999 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.415211+0000", 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2244000000000007e-05 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:193498 7.18 7:1aab6e34:::benchmark_data_smithi169_79822_object76081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:35.448645+0000", 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.406317910999999, 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.98526466999999995, 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.415 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 193498 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.448645+0000", 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.448645+0000", 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.448646+0000", 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0100000000000003e-07 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.448650+0000", 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9779999999999999e-06 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.448706+0000", 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6227000000000002e-05 2024-03-12T03:38:56.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.448715+0000", 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5699999999999999e-06 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.448779+0000", 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3955999999999995e-05 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.448789+0000", 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2979999999999997e-06 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.415727+0000", 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96693837699999996 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.415851+0000", 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012441000000000001 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:38:56.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.416239+0000", 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038707200000000002 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.431938+0000", 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.015699305 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.431994+0000", 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5838999999999998e-05 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.433816+0000", 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001821825 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.433873+0000", 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7098000000000001e-05 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.433909+0000", 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6813999999999997e-05 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.421 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:193556 7.18 7:1e8b2268:::benchmark_data_smithi169_79822_object76139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:35.498131+0000", 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.356831458999999, 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.94005487200000004, 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 193556 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.498131+0000", 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.498131+0000", 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.498133+0000", 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3430000000000001e-06 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.498138+0000", 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.643e-06 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.498153+0000", 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4688999999999999e-05 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.498171+0000", 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8278999999999999e-05 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.498259+0000", 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8265999999999998e-05 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.498315+0000", 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5390999999999999e-05 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.429712+0000", 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.931396954 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.429826+0000", 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000113825 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.430187+0000", 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000361782 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.435936+0000", 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0057489630000000002 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.438047+0000", 2024-03-12T03:38:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021110109999999999 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.438106+0000", 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8363999999999999e-05 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.438143+0000", 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7336000000000003e-05 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.438186+0000", 2024-03-12T03:38:56.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3025999999999998e-05 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:193605 7.18 7:19f7da69:::benchmark_data_smithi169_79822_object76188:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:35.533541+0000", 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.321421416, 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.91104316399999996, 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.428 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 193605 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.533541+0000", 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.533541+0000", 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.533543+0000", 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2440000000000001e-06 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.533547+0000", 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4259999999999996e-06 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.533556+0000", 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4900000000000006e-06 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.533565+0000", 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8850000000000005e-06 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.533715+0000", 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000150025 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:35.533728+0000", 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2366e-05 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.432031+0000", 2024-03-12T03:38:56.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89830354199999995 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.432142+0000", 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000110981 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.432378+0000", 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000235338 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.437992+0000", 2024-03-12T03:38:56.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056146729999999997 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.442655+0000", 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046627229999999997 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.444487+0000", 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018322729999999999 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.444539+0000", 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1595999999999999e-05 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:36.444584+0000", 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5602000000000001e-05 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:213972 7.13 7:c930d1e8:::benchmark_data_smithi169_79822_object96555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:54.563069+0000", 2024-03-12T03:38:56.434 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.2918934120000001, 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.143688834, 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 213972 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.563069+0000", 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.563069+0000", 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.563070+0000", 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2139999999999999e-06 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.563075+0000", 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5889999999999996e-06 2024-03-12T03:38:56.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.563091+0000", 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5747e-05 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.563100+0000", 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4329999999999993e-06 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.563164+0000", 2024-03-12T03:38:56.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3664999999999994e-05 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.563303+0000", 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013893099999999999 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2,5,7", 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.563807+0000", 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00050464300000000004 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.568440+0000", 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046325769999999997 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.578416+0000", 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0099760939999999996 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.578429+0000", 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3054000000000001e-05 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.706680+0000", 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1282506889999999 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.706758+0000", 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8197999999999999e-05 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:214007 7.13 7:cfb4b50f:::benchmark_data_smithi169_79822_object96590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:54.600158+0000", 2024-03-12T03:38:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.25480419, 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1122426409999999, 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 214007 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.600158+0000", 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.600158+0000", 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.600159+0000", 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6300000000000005e-07 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.600162+0000", 2024-03-12T03:38:56.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2150000000000001e-06 2024-03-12T03:38:56.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.600172+0000", 2024-03-12T03:38:56.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0047999999999999e-05 2024-03-12T03:38:56.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.600179+0000", 2024-03-12T03:38:56.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9360000000000002e-06 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.600536+0000", 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00035682699999999999 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.600544+0000", 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8820000000000001e-06 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.707186+0000", 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.106642267 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.707285+0000", 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8824000000000001e-05 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.707734+0000", 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00044879400000000003 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.712082+0000", 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043481550000000002 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.712125+0000", 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3016999999999998e-05 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.712260+0000", 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013526 2024-03-12T03:38:56.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.712362+0000", 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101593 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.712401+0000", 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9159999999999998e-05 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:214011 7.13 7:cd867578:::benchmark_data_smithi169_79822_object96594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:54.604133+0000", 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.2508298259999999, 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1135616429999999, 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 214011 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.604133+0000", 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.604133+0000", 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.604134+0000", 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.136e-06 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.604139+0000", 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1599999999999997e-06 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.604150+0000", 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0570999999999999e-05 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.604157+0000", 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6329999999999996e-06 2024-03-12T03:38:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.604214+0000", 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6492000000000001e-05 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.604222+0000", 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.49e-06 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.709899+0000", 2024-03-12T03:38:56.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1056767270000001 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.710007+0000", 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000107938 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.710334+0000", 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00032748199999999998 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.717533+0000", 2024-03-12T03:38:56.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071980799999999999 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.717570+0000", 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7870000000000002e-05 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.717613+0000", 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2187000000000001e-05 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.717657+0000", 2024-03-12T03:38:56.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3977999999999998e-05 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.717695+0000", 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7898999999999997e-05 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:214072 7.13 7:c8c55696:::benchmark_data_smithi169_79822_object96655:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:54.652856+0000", 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.2021069209999999, 2024-03-12T03:38:56.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.065040784, 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 214072 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.652856+0000", 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.652856+0000", 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.652857+0000", 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3799999999999999e-06 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.652862+0000", 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0610000000000003e-06 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.652877+0000", 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5043e-05 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.652886+0000", 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3459999999999994e-06 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.652948+0000", 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2529999999999999e-05 2024-03-12T03:38:56.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.652957+0000", 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4479999999999994e-06 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.712436+0000", 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.059479367 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.712539+0000", 2024-03-12T03:38:56.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000102876 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.712949+0000", 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00041006899999999999 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.717581+0000", 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046322830000000001 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.717729+0000", 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014768600000000001 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.717791+0000", 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2291000000000004e-05 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.717856+0000", 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4389999999999998e-05 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.717897+0000", 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1013999999999997e-05 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:214080 7.13 7:cb5088f9:::benchmark_data_smithi169_79822_object96663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:54.660432+0000", 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.19453108, 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0579112239999999, 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.461 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 214080 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.660432+0000", 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.660432+0000", 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.660433+0000", 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3820000000000001e-06 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.660439+0000", 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8429999999999998e-06 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.660454+0000", 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4953999999999999e-05 2024-03-12T03:38:56.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.660462+0000", 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5250000000000005e-06 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.660523+0000", 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0866999999999998e-05 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.660532+0000", 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0089999999999996e-06 2024-03-12T03:38:56.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.715075+0000", 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0545432299999999 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.715214+0000", 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000138338 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.715557+0000", 2024-03-12T03:38:56.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00034351999999999999 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.718224+0000", 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026668030000000001 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.718235+0000", 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0917999999999999e-05 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.718263+0000", 2024-03-12T03:38:56.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7926e-05 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.718307+0000", 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4245000000000001e-05 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.718343+0000", 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5664000000000003e-05 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:214090 7.13 7:c86be81e:::benchmark_data_smithi169_79822_object96673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:54.668371+0000", 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.186591376, 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.059907836, 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 214090 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.668371+0000", 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.668371+0000", 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.668372+0000", 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5000000000000001e-07 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.668377+0000", 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.578e-06 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.668388+0000", 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1269e-05 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.668397+0000", 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5029999999999997e-06 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.668613+0000", 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00021646400000000001 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.668622+0000", 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3200000000000006e-06 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.718368+0000", 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0497459039999999 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.718462+0000", 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3622999999999995e-05 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.719406+0000", 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00094465200000000004 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.728070+0000", 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0086631999999999994 2024-03-12T03:38:56.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.728113+0000", 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3489999999999999e-05 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.728184+0000", 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0926000000000005e-05 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.728234+0000", 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9786000000000001e-05 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.728279+0000", 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5271000000000001e-05 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:214100 7.13 7:cc21261e:::benchmark_data_smithi169_79822_object96683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:54.678577+0000", 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.1763861179999999, 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0537102970000001, 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.474 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 214100 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.678577+0000", 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.678577+0000", 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.678578+0000", 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1599999999999999e-06 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.678583+0000", 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.434e-06 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.678596+0000", 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.289e-05 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.678603+0000", 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3849999999999996e-06 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.678661+0000", 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7663e-05 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.678670+0000", 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6109999999999994e-06 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.721295+0000", 2024-03-12T03:38:56.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0426249620000001 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.722411+0000", 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001116267 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.722751+0000", 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00034015500000000003 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.728129+0000", 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053775139999999999 2024-03-12T03:38:56.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.732118+0000", 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003989444 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.732174+0000", 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6375999999999997e-05 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.732241+0000", 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6255999999999997e-05 2024-03-12T03:38:56.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.732287+0000", 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6180000000000002e-05 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:214104 7.13 7:cb3db3a6:::benchmark_data_smithi169_79822_object96687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:54.685396+0000", 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.169567102, 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.051501515, 2024-03-12T03:38:56.480 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 214104 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.685396+0000", 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.685396+0000", 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.685396+0000", 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1999999999999999e-07 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.685401+0000", 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6879999999999998e-06 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.685410+0000", 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0750000000000004e-06 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.685418+0000", 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7700000000000001e-06 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.685476+0000", 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7734e-05 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.685483+0000", 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.255e-06 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.725757+0000", 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0402746620000001 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.725866+0000", 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010854299999999999 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.726171+0000", 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030493300000000002 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.732130+0000", 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0059595259999999997 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.736752+0000", 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046211510000000004 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.736794+0000", 2024-03-12T03:38:56.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2240000000000002e-05 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.736849+0000", 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4985000000000003e-05 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.736897+0000", 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8232999999999997e-05 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:214120 7.13 7:cce4fa55:::benchmark_data_smithi169_79822_object96703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:54.707324+0000", 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.147638634, 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0346030639999999, 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 214120 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.707324+0000", 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.707324+0000", 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.707325+0000", 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0890000000000001e-06 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.707330+0000", 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0640000000000001e-06 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.707345+0000", 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4796e-05 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.707353+0000", 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1009999999999992e-06 2024-03-12T03:38:56.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.707412+0000", 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8508000000000003e-05 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.707420+0000", 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6880000000000005e-06 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.728315+0000", 2024-03-12T03:38:56.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.020894908 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.728426+0000", 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000110627 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.728703+0000", 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00027723299999999999 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.732136+0000", 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034328639999999999 2024-03-12T03:38:56.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.736935+0000", 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0047988789999999998 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.741811+0000", 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048757269999999998 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.741870+0000", 2024-03-12T03:38:56.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.91e-05 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.741927+0000", 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7479999999999999e-05 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:214164 7.13 7:ce88851f:::benchmark_data_smithi169_79822_object96747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:54.778113+0000", 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.076849355, 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96633553299999997, 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:38:56.493 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 214164 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.778113+0000", 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:38:56.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.778113+0000", 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.778115+0000", 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.353e-06 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.778120+0000", 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4430000000000002e-06 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.778135+0000", 2024-03-12T03:38:56.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5174000000000001e-05 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.778144+0000", 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9989999999999997e-06 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.778211+0000", 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6884000000000005e-05 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:54.778222+0000", 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1172000000000001e-05 2024-03-12T03:38:56.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.730308+0000", 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.952085293 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.730410+0000", 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000102278 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.730675+0000", 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00026476000000000001 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:38:56.498 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.736712+0000", 2024-03-12T03:38:56.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060370340000000002 2024-03-12T03:38:56.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.736969+0000", 2024-03-12T03:38:56.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00025700400000000002 2024-03-12T03:38:56.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.500 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:38:56.500 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.742018+0000", 2024-03-12T03:38:56.500 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050493209999999998 2024-03-12T03:38:56.500 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.500 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.500 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:38:56.500 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.742183+0000", 2024-03-12T03:38:56.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000164995 2024-03-12T03:38:56.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:38:56.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:38:56.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:38:56.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:55.744449+0000", 2024-03-12T03:38:56.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022658230000000001 2024-03-12T03:38:56.501 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.501 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.501 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.501 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:38:56.502 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:38:56.502 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:38:56.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:56.527 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:56.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:56.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:56.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:56.671 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:56.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:56.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:56 smithi169 ceph-mon[29344]: osdmap e182: 8 total, 8 up, 7 in 2024-03-12T03:38:56.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:56 smithi169 ceph-mon[39949]: osdmap e182: 8 total, 8 up, 7 in 2024-03-12T03:38:56.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:56.812 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:56.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:56.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:56.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:56.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:56.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:56 smithi173 ceph-mon[33544]: osdmap e182: 8 total, 8 up, 7 in 2024-03-12T03:38:57.050 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:38:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:57.104 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:57.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:38:57.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:57.246 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:57.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:57.348 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:38:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:57.394 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:57.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:38:57.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:57.527 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:57.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:57.606 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:38:57.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:57.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:57.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:57.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-12T03:38:57.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:57.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:57.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:57.896 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:57 smithi169 ceph-mon[29344]: pgmap v250: 168 pgs: 1 peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 1.4 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 352 op/s; 5195/70767 objects misplaced (7.341%); 9.3 MiB/s, 148 objects/s recovering 2024-03-12T03:38:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:57 smithi169 ceph-mon[29344]: osdmap e183: 8 total, 8 up, 7 in 2024-03-12T03:38:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:57 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1203060797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:38:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:57 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:38:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:57 smithi169 ceph-mon[39949]: pgmap v250: 168 pgs: 1 peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 1.4 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 352 op/s; 5195/70767 objects misplaced (7.341%); 9.3 MiB/s, 148 objects/s recovering 2024-03-12T03:38:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:57 smithi169 ceph-mon[39949]: osdmap e183: 8 total, 8 up, 7 in 2024-03-12T03:38:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:57 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1203060797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:38:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:57 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:38:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:57.945 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:57.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:57 smithi173 ceph-mon[33544]: pgmap v250: 168 pgs: 1 peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 1.4 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 352 op/s; 5195/70767 objects misplaced (7.341%); 9.3 MiB/s, 148 objects/s recovering 2024-03-12T03:38:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:57 smithi173 ceph-mon[33544]: osdmap e183: 8 total, 8 up, 7 in 2024-03-12T03:38:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:57 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1203060797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:38:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:57 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:38:58.046 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:38:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:58.096 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:58.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:38:58.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:58.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:58.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:58.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:58.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:58.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:58.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:58.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:58.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:58.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:58.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:58.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:58.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:58.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:58.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:58.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:58 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:38:58.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:58 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:38:58.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:58 smithi169 ceph-mon[29344]: osdmap e184: 8 total, 8 up, 7 in 2024-03-12T03:38:58.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:58 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:38:58.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:58 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:38:58.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:58 smithi169 ceph-mon[39949]: osdmap e184: 8 total, 8 up, 7 in 2024-03-12T03:38:58.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:58.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:58.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:58.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:58.874 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:58.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:58.923 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:58.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:58 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:38:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:58 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:38:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:58 smithi173 ceph-mon[33544]: osdmap e184: 8 total, 8 up, 7 in 2024-03-12T03:38:59.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:38:59.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:59.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:59.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:38:59.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:59.194 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:59.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:59.237 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:59.296 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:59.325 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:38:59.347 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:59.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:59.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:59.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:38:59.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:59.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:59.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:38:59.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:38:59.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:59.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:38:59.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:59 smithi169 ceph-mon[29344]: pgmap v253: 168 pgs: 1 peering, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 162 active+clean; 1.2 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 2562/61176 objects misplaced (4.188%); 41 MiB/s, 660 objects/s recovering 2024-03-12T03:38:59.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:38:59 smithi169 ceph-mon[29344]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:38:59.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:59 smithi169 ceph-mon[39949]: pgmap v253: 168 pgs: 1 peering, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 162 active+clean; 1.2 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 2562/61176 objects misplaced (4.188%); 41 MiB/s, 660 objects/s recovering 2024-03-12T03:38:59.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:38:59 smithi169 ceph-mon[39949]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:38:59.690 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:38:59.694 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:38:59.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:38:59.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:38:59.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:59.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:38:59.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:38:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:38:59.880 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:38:59.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:38:59.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:59 smithi173 ceph-mon[33544]: pgmap v253: 168 pgs: 1 peering, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 162 active+clean; 1.2 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 2562/61176 objects misplaced (4.188%); 41 MiB/s, 660 objects/s recovering 2024-03-12T03:38:59.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:38:59 smithi173 ceph-mon[33544]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:38:59.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:00.005 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:00.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:00.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:00.131 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:00.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:00.232 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:00.256 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:00.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:00.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:00.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:00.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:00.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:39:00.488 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:00.512 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:00.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:00.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:00.634 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:00 smithi173 ceph-mon[33544]: osdmap e185: 8 total, 8 up, 7 in 2024-03-12T03:39:00.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:00 smithi173 ceph-mon[33544]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:39:00.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:00.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:00.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:00.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:00.781 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:00.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:00.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:00 smithi169 ceph-mon[29344]: osdmap e185: 8 total, 8 up, 7 in 2024-03-12T03:39:00.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:00 smithi169 ceph-mon[29344]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:39:00.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:00.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:00 smithi169 ceph-mon[39949]: osdmap e185: 8 total, 8 up, 7 in 2024-03-12T03:39:00.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:00 smithi169 ceph-mon[39949]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:39:00.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:00.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:00.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:00.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:01.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:01.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:01.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:01.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:01.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:01.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:01.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:01.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:01.336 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:01.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:01.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:01.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:01.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:01.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:01.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:01.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:01.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:01.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:01.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:01.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:01.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:01.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:01.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:01 smithi169 ceph-mon[29344]: pgmap v255: 168 pgs: 1 peering, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 162 active+clean; 1.2 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 2562/61176 objects misplaced (4.188%); 39 MiB/s, 616 objects/s recovering 2024-03-12T03:39:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:01 smithi169 ceph-mon[29344]: osdmap e186: 8 total, 8 up, 7 in 2024-03-12T03:39:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:01 smithi169 ceph-mon[39949]: pgmap v255: 168 pgs: 1 peering, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 162 active+clean; 1.2 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 2562/61176 objects misplaced (4.188%); 39 MiB/s, 616 objects/s recovering 2024-03-12T03:39:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:01 smithi169 ceph-mon[39949]: osdmap e186: 8 total, 8 up, 7 in 2024-03-12T03:39:01.925 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:39:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:39:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:39:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:01 smithi173 ceph-mon[33544]: pgmap v255: 168 pgs: 1 peering, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 162 active+clean; 1.2 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 2562/61176 objects misplaced (4.188%); 39 MiB/s, 616 objects/s recovering 2024-03-12T03:39:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:01 smithi173 ceph-mon[33544]: osdmap e186: 8 total, 8 up, 7 in 2024-03-12T03:39:01.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:02.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:02.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:02.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:02.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:02.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:02.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:02.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:02.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:02.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:02.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:02.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:02.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:02.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:02.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:02.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:02.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:02.723 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:02.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:02.825 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:02.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:02.864 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:02.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:02.965 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:02.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:39:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:39:03.001 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:03.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:03.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:03.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:03.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:03.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:03.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:03.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:03.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:03.271 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:03.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:03.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:03.415 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:03.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:03.438 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-03-12T03:38:59.603728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1321834944,"num_objects":20392,"num_object_clones":0,"num_object_copies":61176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2562,"num_objects_unfound":0,"num_objects_dirty":20392,"num_whiteouts":0,"num_read":1339,"num_read_kb":1059,"num_write":135858,"num_write_kb":4249294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23313,"num_bytes_recovered":1522404701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122613,"ondisk_log_size":122613,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":515,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6056204,"kb_used_data":4755036,"kb_used_omap":10,"kb_used_meta":1301045,"kb_avail":650159860,"statfs":{"total":671965249536,"available":665763696640,"internally_reserved":0,"allocated":4869156864,"data_stored":4865269815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1332270187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,5,2,1,4,1,1,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-301596672,"num_objects":-4602,"num_object_clones":0,"num_object_copies":-13806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3540,"num_objects_unfound":0,"num_objects_dirty":-4602,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4602,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2643,"num_bytes_recovered":173146112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.036046"},"pg_stats":[{"pgid":"1.9","version":"83'128","reported_seq":540,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577400+0000","last_change":"2024-03-12T03:38:40.534427+0000","last_active":"2024-03-12T03:38:55.577400+0000","last_peered":"2024-03-12T03:38:55.577400+0000","last_clean":"2024-03-12T03:38:55.577400+0000","last_became_active":"2024-03-12T03:38:21.231758+0000","last_became_peered":"2024-03-12T03:38:21.231758+0000","last_unstale":"2024-03-12T03:38:55.577400+0000","last_undegraded":"2024-03-12T03:38:55.577400+0000","last_fullsized":"2024-03-12T03:38:55.577400+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:38:40.534294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:38:40.534294+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:40:00.328681+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":270,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575807+0000","last_change":"2024-03-12T03:38:33.278147+0000","last_active":"2024-03-12T03:38:55.575807+0000","last_peered":"2024-03-12T03:38:55.575807+0000","last_clean":"2024-03-12T03:38:55.575807+0000","last_became_active":"2024-03-12T03:33:23.162657+0000","last_became_peered":"2024-03-12T03:33:23.162657+0000","last_unstale":"2024-03-12T03:38:55.575807+0000","last_undegraded":"2024-03-12T03:38:55.575807+0000","last_fullsized":"2024-03-12T03:38:55.575807+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-12T03:38:33.277987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:33.277987+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:47.933778+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574647+0000","last_change":"2024-03-12T03:35:15.981464+0000","last_active":"2024-03-12T03:38:55.574647+0000","last_peered":"2024-03-12T03:38:55.574647+0000","last_clean":"2024-03-12T03:38:55.574647+0000","last_became_active":"2024-03-12T03:33:21.097304+0000","last_became_peered":"2024-03-12T03:33:21.097304+0000","last_unstale":"2024-03-12T03:38:55.574647+0000","last_undegraded":"2024-03-12T03:38:55.574647+0000","last_fullsized":"2024-03-12T03:38:55.574647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.981378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.981378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"166'9","reported_seq":86,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.579505+0000","last_change":"2024-03-12T03:38:18.277462+0000","last_active":"2024-03-12T03:38:55.579505+0000","last_peered":"2024-03-12T03:38:55.579505+0000","last_clean":"2024-03-12T03:38:55.579505+0000","last_became_active":"2024-03-12T03:38:18.243880+0000","last_became_peered":"2024-03-12T03:38:18.243880+0000","last_unstale":"2024-03-12T03:38:55.579505+0000","last_undegraded":"2024-03-12T03:38:55.579505+0000","last_fullsized":"2024-03-12T03:38:55.579505+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:25.346151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.346151+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":253,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.576474+0000","last_change":"2024-03-12T03:38:25.369277+0000","last_active":"2024-03-12T03:38:55.576474+0000","last_peered":"2024-03-12T03:38:55.576474+0000","last_clean":"2024-03-12T03:38:55.576474+0000","last_became_active":"2024-03-12T03:38:18.214845+0000","last_became_peered":"2024-03-12T03:38:18.214845+0000","last_unstale":"2024-03-12T03:38:55.576474+0000","last_undegraded":"2024-03-12T03:38:55.576474+0000","last_fullsized":"2024-03-12T03:38:55.576474+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:25.369168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:25.369168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:42.593992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"182'13509","reported_seq":13694,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:56.166016+0000","last_change":"2024-03-12T03:35:23.784927+0000","last_active":"2024-03-12T03:38:56.166016+0000","last_peered":"2024-03-12T03:38:56.166016+0000","last_clean":"2024-03-12T03:38:56.166016+0000","last_became_active":"2024-03-12T03:35:23.784627+0000","last_became_peered":"2024-03-12T03:35:23.784627+0000","last_unstale":"2024-03-12T03:38:56.166016+0000","last_undegraded":"2024-03-12T03:38:56.166016+0000","last_fullsized":"2024-03-12T03:38:56.166016+0000","mapping_epoch":97,"log_start":"135'9254","ondisk_log_start":"135'9254","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4255,"log_dups_size":0,"ondisk_log_size":4255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85262336,"num_objects":1301,"num_object_clones":0,"num_object_copies":3903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13509,"num_write_kb":473920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":307,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.576184+0000","last_change":"2024-03-12T03:38:18.233239+0000","last_active":"2024-03-12T03:38:55.576184+0000","last_peered":"2024-03-12T03:38:55.576184+0000","last_clean":"2024-03-12T03:38:55.576184+0000","last_became_active":"2024-03-12T03:38:18.215622+0000","last_became_peered":"2024-03-12T03:38:18.215622+0000","last_unstale":"2024-03-12T03:38:55.576184+0000","last_undegraded":"2024-03-12T03:38:55.576184+0000","last_fullsized":"2024-03-12T03:38:55.576184+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:38:13.406302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:13.406302+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:33.935354+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":258,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577319+0000","last_change":"2024-03-12T03:35:59.016620+0000","last_active":"2024-03-12T03:38:55.577319+0000","last_peered":"2024-03-12T03:38:55.577319+0000","last_clean":"2024-03-12T03:38:55.577319+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:38:55.577319+0000","last_undegraded":"2024-03-12T03:38:55.577319+0000","last_fullsized":"2024-03-12T03:38:55.577319+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.016489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.016489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.679847+0000","last_change":"2024-03-12T03:38:30.459904+0000","last_active":"2024-03-12T03:38:55.679847+0000","last_peered":"2024-03-12T03:38:55.679847+0000","last_clean":"2024-03-12T03:38:55.679847+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:38:55.679847+0000","last_undegraded":"2024-03-12T03:38:55.679847+0000","last_fullsized":"2024-03-12T03:38:55.679847+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:30.459619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:38:30.459619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:30.559493+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":50,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575073+0000","last_change":"2024-03-12T03:38:18.222174+0000","last_active":"2024-03-12T03:38:55.575073+0000","last_peered":"2024-03-12T03:38:55.575073+0000","last_clean":"2024-03-12T03:38:55.575073+0000","last_became_active":"2024-03-12T03:38:18.221803+0000","last_became_peered":"2024-03-12T03:38:18.221803+0000","last_unstale":"2024-03-12T03:38:55.575073+0000","last_undegraded":"2024-03-12T03:38:55.575073+0000","last_fullsized":"2024-03-12T03:38:55.575073+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:56.071324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:56.071324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"182'13576","reported_seq":19262,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.808194+0000","last_change":"2024-03-12T03:38:40.240604+0000","last_active":"2024-03-12T03:38:55.808194+0000","last_peered":"2024-03-12T03:38:55.808194+0000","last_clean":"2024-03-12T03:38:55.808194+0000","last_became_active":"2024-03-12T03:38:40.216851+0000","last_became_peered":"2024-03-12T03:38:40.216851+0000","last_unstale":"2024-03-12T03:38:55.808194+0000","last_undegraded":"2024-03-12T03:38:55.808194+0000","last_fullsized":"2024-03-12T03:38:55.808194+0000","mapping_epoch":175,"log_start":"135'9143","ondisk_log_start":"135'9143","created":97,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4433,"log_dups_size":0,"ondisk_log_size":4433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83230720,"num_objects":1270,"num_object_clones":0,"num_object_copies":3810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13576,"num_write_kb":475072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2743,"num_bytes_recovered":179765248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":386,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.573735+0000","last_change":"2024-03-12T03:38:21.237758+0000","last_active":"2024-03-12T03:38:55.573735+0000","last_peered":"2024-03-12T03:38:55.573735+0000","last_clean":"2024-03-12T03:38:55.573735+0000","last_became_active":"2024-03-12T03:38:21.237336+0000","last_became_peered":"2024-03-12T03:38:21.237336+0000","last_unstale":"2024-03-12T03:38:55.573735+0000","last_undegraded":"2024-03-12T03:38:55.573735+0000","last_fullsized":"2024-03-12T03:38:55.573735+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:38:17.432354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:38:17.432354+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:41.354520+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"95'23","reported_seq":292,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.576374+0000","last_change":"2024-03-12T03:37:05.133318+0000","last_active":"2024-03-12T03:38:55.576374+0000","last_peered":"2024-03-12T03:38:55.576374+0000","last_clean":"2024-03-12T03:38:55.576374+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:38:55.576374+0000","last_undegraded":"2024-03-12T03:38:55.576374+0000","last_fullsized":"2024-03-12T03:38:55.576374+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'23","last_scrub_stamp":"2024-03-12T03:37:05.133198+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:37:05.133198+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.084355+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":265,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574528+0000","last_change":"2024-03-12T03:38:26.342203+0000","last_active":"2024-03-12T03:38:55.574528+0000","last_peered":"2024-03-12T03:38:55.574528+0000","last_clean":"2024-03-12T03:38:55.574528+0000","last_became_active":"2024-03-12T03:38:18.213092+0000","last_became_peered":"2024-03-12T03:38:18.213092+0000","last_unstale":"2024-03-12T03:38:55.574528+0000","last_undegraded":"2024-03-12T03:38:55.574528+0000","last_fullsized":"2024-03-12T03:38:55.574528+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:26.342088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:26.342088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:26.833612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575094+0000","last_change":"2024-03-12T03:36:19.182812+0000","last_active":"2024-03-12T03:38:55.575094+0000","last_peered":"2024-03-12T03:38:55.575094+0000","last_clean":"2024-03-12T03:38:55.575094+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:38:55.575094+0000","last_undegraded":"2024-03-12T03:38:55.575094+0000","last_fullsized":"2024-03-12T03:38:55.575094+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.182685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.182685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574592+0000","last_change":"2024-03-12T03:36:13.114566+0000","last_active":"2024-03-12T03:38:55.574592+0000","last_peered":"2024-03-12T03:38:55.574592+0000","last_clean":"2024-03-12T03:38:55.574592+0000","last_became_active":"2024-03-12T03:33:27.380896+0000","last_became_peered":"2024-03-12T03:33:27.380896+0000","last_unstale":"2024-03-12T03:38:55.574592+0000","last_undegraded":"2024-03-12T03:38:55.574592+0000","last_fullsized":"2024-03-12T03:38:55.574592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:13.114417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.114417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"182'13446","reported_seq":13631,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.927930+0000","last_change":"2024-03-12T03:35:23.783266+0000","last_active":"2024-03-12T03:38:55.927930+0000","last_peered":"2024-03-12T03:38:55.927930+0000","last_clean":"2024-03-12T03:38:55.927930+0000","last_became_active":"2024-03-12T03:35:23.782945+0000","last_became_peered":"2024-03-12T03:35:23.782945+0000","last_unstale":"2024-03-12T03:38:55.927930+0000","last_undegraded":"2024-03-12T03:38:55.927930+0000","last_fullsized":"2024-03-12T03:38:55.927930+0000","mapping_epoch":97,"log_start":"133'8970","ondisk_log_start":"133'8970","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4476,"log_dups_size":0,"ondisk_log_size":4476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87621656,"num_objects":1338,"num_object_clones":0,"num_object_copies":4014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13446,"num_write_kb":473025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":382,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.573882+0000","last_change":"2024-03-12T03:38:21.221611+0000","last_active":"2024-03-12T03:38:55.573882+0000","last_peered":"2024-03-12T03:38:55.573882+0000","last_clean":"2024-03-12T03:38:55.573882+0000","last_became_active":"2024-03-12T03:38:21.221235+0000","last_became_peered":"2024-03-12T03:38:21.221235+0000","last_unstale":"2024-03-12T03:38:55.573882+0000","last_undegraded":"2024-03-12T03:38:55.573882+0000","last_fullsized":"2024-03-12T03:38:55.573882+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:38:15.400800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:38:15.400800+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:40.518096+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":73,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.680617+0000","last_change":"2024-03-12T03:38:33.556041+0000","last_active":"2024-03-12T03:38:55.680617+0000","last_peered":"2024-03-12T03:38:55.680617+0000","last_clean":"2024-03-12T03:38:55.680617+0000","last_became_active":"2024-03-12T03:38:18.223630+0000","last_became_peered":"2024-03-12T03:38:18.223630+0000","last_unstale":"2024-03-12T03:38:55.680617+0000","last_undegraded":"2024-03-12T03:38:55.680617+0000","last_fullsized":"2024-03-12T03:38:55.680617+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:38:33.555893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:33.555893+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:54.581443+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.573902+0000","last_change":"2024-03-12T03:35:13.030518+0000","last_active":"2024-03-12T03:38:55.573902+0000","last_peered":"2024-03-12T03:38:55.573902+0000","last_clean":"2024-03-12T03:38:55.573902+0000","last_became_active":"2024-03-12T03:33:21.104293+0000","last_became_peered":"2024-03-12T03:33:21.104293+0000","last_unstale":"2024-03-12T03:38:55.573902+0000","last_undegraded":"2024-03-12T03:38:55.573902+0000","last_fullsized":"2024-03-12T03:38:55.573902+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:13.030426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:13.030426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":326,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.576394+0000","last_change":"2024-03-12T03:37:22.125851+0000","last_active":"2024-03-12T03:38:55.576394+0000","last_peered":"2024-03-12T03:38:55.576394+0000","last_clean":"2024-03-12T03:38:55.576394+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:38:55.576394+0000","last_undegraded":"2024-03-12T03:38:55.576394+0000","last_fullsized":"2024-03-12T03:38:55.576394+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:37:22.125684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:37:22.125684+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.084449+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574433+0000","last_change":"2024-03-12T03:35:17.142169+0000","last_active":"2024-03-12T03:38:55.574433+0000","last_peered":"2024-03-12T03:38:55.574433+0000","last_clean":"2024-03-12T03:38:55.574433+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:38:55.574433+0000","last_undegraded":"2024-03-12T03:38:55.574433+0000","last_fullsized":"2024-03-12T03:38:55.574433+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:17.142086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:17.142086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"182'13495","reported_seq":13679,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.878262+0000","last_change":"2024-03-12T03:35:23.785403+0000","last_active":"2024-03-12T03:38:55.878262+0000","last_peered":"2024-03-12T03:38:55.878262+0000","last_clean":"2024-03-12T03:38:55.878262+0000","last_became_active":"2024-03-12T03:35:23.785037+0000","last_became_peered":"2024-03-12T03:35:23.785037+0000","last_unstale":"2024-03-12T03:38:55.878262+0000","last_undegraded":"2024-03-12T03:38:55.878262+0000","last_fullsized":"2024-03-12T03:38:55.878262+0000","mapping_epoch":97,"log_start":"133'8997","ondisk_log_start":"133'8997","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4498,"log_dups_size":0,"ondisk_log_size":4498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83558400,"num_objects":1275,"num_object_clones":0,"num_object_copies":3825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13495,"num_write_kb":472640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574435+0000","last_change":"2024-03-12T03:37:08.194374+0000","last_active":"2024-03-12T03:38:55.574435+0000","last_peered":"2024-03-12T03:38:55.574435+0000","last_clean":"2024-03-12T03:38:55.574435+0000","last_became_active":"2024-03-12T03:33:25.181020+0000","last_became_peered":"2024-03-12T03:33:25.181020+0000","last_unstale":"2024-03-12T03:38:55.574435+0000","last_undegraded":"2024-03-12T03:38:55.574435+0000","last_fullsized":"2024-03-12T03:38:55.574435+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:08.194302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:37:08.194302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.084464+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":303,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.573947+0000","last_change":"2024-03-12T03:36:06.083361+0000","last_active":"2024-03-12T03:38:55.573947+0000","last_peered":"2024-03-12T03:38:55.573947+0000","last_clean":"2024-03-12T03:38:55.573947+0000","last_became_active":"2024-03-12T03:33:23.551623+0000","last_became_peered":"2024-03-12T03:33:23.551623+0000","last_unstale":"2024-03-12T03:38:55.573947+0000","last_undegraded":"2024-03-12T03:38:55.573947+0000","last_fullsized":"2024-03-12T03:38:55.573947+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-03-12T03:36:06.083052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.083052+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575240+0000","last_change":"2024-03-12T03:38:41.918686+0000","last_active":"2024-03-12T03:38:55.575240+0000","last_peered":"2024-03-12T03:38:55.575240+0000","last_clean":"2024-03-12T03:38:55.575240+0000","last_became_active":"2024-03-12T03:33:21.104430+0000","last_became_peered":"2024-03-12T03:33:21.104430+0000","last_unstale":"2024-03-12T03:38:55.575240+0000","last_undegraded":"2024-03-12T03:38:55.575240+0000","last_fullsized":"2024-03-12T03:38:55.575240+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:41.918538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:41.918538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:40:06.319501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":253,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575373+0000","last_change":"2024-03-12T03:38:18.215214+0000","last_active":"2024-03-12T03:38:55.575373+0000","last_peered":"2024-03-12T03:38:55.575373+0000","last_clean":"2024-03-12T03:38:55.575373+0000","last_became_active":"2024-03-12T03:38:18.214835+0000","last_became_peered":"2024-03-12T03:38:18.214835+0000","last_unstale":"2024-03-12T03:38:55.575373+0000","last_undegraded":"2024-03-12T03:38:55.575373+0000","last_fullsized":"2024-03-12T03:38:55.575373+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:01.181261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:37:01.181261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.084401+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"182'10087","reported_seq":11795,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:56.082427+0000","last_change":"2024-03-12T03:38:55.692698+0000","last_active":"2024-03-12T03:38:56.082427+0000","last_peered":"2024-03-12T03:38:56.082427+0000","last_clean":"2024-03-12T03:38:56.082427+0000","last_became_active":"2024-03-12T03:38:55.664474+0000","last_became_peered":"2024-03-12T03:38:55.664474+0000","last_unstale":"2024-03-12T03:38:56.082427+0000","last_undegraded":"2024-03-12T03:38:56.082427+0000","last_fullsized":"2024-03-12T03:38:56.082427+0000","mapping_epoch":181,"log_start":"114'6153","ondisk_log_start":"114'6153","created":97,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'8813","last_scrub_stamp":"2024-03-12T03:38:14.165448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:14.165448+0000","objects_scrubbed":1919,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:20.341786+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41222144,"num_objects":629,"num_object_clones":0,"num_object_copies":1887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":735,"num_bytes_recovered":48103424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":382,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575160+0000","last_change":"2024-03-12T03:38:21.217211+0000","last_active":"2024-03-12T03:38:55.575160+0000","last_peered":"2024-03-12T03:38:55.575160+0000","last_clean":"2024-03-12T03:38:55.575160+0000","last_became_active":"2024-03-12T03:38:21.216833+0000","last_became_peered":"2024-03-12T03:38:21.216833+0000","last_unstale":"2024-03-12T03:38:55.575160+0000","last_undegraded":"2024-03-12T03:38:55.575160+0000","last_fullsized":"2024-03-12T03:38:55.575160+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:38:15.405215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:38:15.405215+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:29.499277+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":318,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574300+0000","last_change":"2024-03-12T03:34:25.168128+0000","last_active":"2024-03-12T03:38:55.574300+0000","last_peered":"2024-03-12T03:38:55.574300+0000","last_clean":"2024-03-12T03:38:55.574300+0000","last_became_active":"2024-03-12T03:33:25.177311+0000","last_became_peered":"2024-03-12T03:33:25.177311+0000","last_unstale":"2024-03-12T03:38:55.574300+0000","last_undegraded":"2024-03-12T03:38:55.574300+0000","last_fullsized":"2024-03-12T03:38:55.574300+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:25.168002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:25.168002+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":288,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574789+0000","last_change":"2024-03-12T03:37:03.159318+0000","last_active":"2024-03-12T03:38:55.574789+0000","last_peered":"2024-03-12T03:38:55.574789+0000","last_clean":"2024-03-12T03:38:55.574789+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:38:55.574789+0000","last_undegraded":"2024-03-12T03:38:55.574789+0000","last_fullsized":"2024-03-12T03:38:55.574789+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-03-12T03:37:03.159190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:37:03.159190+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.084418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":260,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.655939+0000","last_change":"2024-03-12T03:36:11.217211+0000","last_active":"2024-03-12T03:38:55.655939+0000","last_peered":"2024-03-12T03:38:55.655939+0000","last_clean":"2024-03-12T03:38:55.655939+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:38:55.655939+0000","last_undegraded":"2024-03-12T03:38:55.655939+0000","last_fullsized":"2024-03-12T03:38:55.655939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:11.216960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:11.216960+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.797047+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":253,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577684+0000","last_change":"2024-03-12T03:38:18.215355+0000","last_active":"2024-03-12T03:38:55.577684+0000","last_peered":"2024-03-12T03:38:55.577684+0000","last_clean":"2024-03-12T03:38:55.577684+0000","last_became_active":"2024-03-12T03:38:18.214250+0000","last_became_peered":"2024-03-12T03:38:18.214250+0000","last_unstale":"2024-03-12T03:38:55.577684+0000","last_undegraded":"2024-03-12T03:38:55.577684+0000","last_fullsized":"2024-03-12T03:38:55.577684+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.049740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.049740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"182'10010","reported_seq":11132,"reported_epoch":182,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-12T03:38:56.031410+0000","last_change":"2024-03-12T03:38:54.300825+0000","last_active":"2024-03-12T03:38:56.031410+0000","last_peered":"2024-03-12T03:38:56.031410+0000","last_clean":"2024-03-12T03:38:17.908915+0000","last_became_active":"2024-03-12T03:38:19.350448+0000","last_became_peered":"2024-03-12T03:38:19.350448+0000","last_unstale":"2024-03-12T03:38:56.031410+0000","last_undegraded":"2024-03-12T03:38:56.031410+0000","last_fullsized":"2024-03-12T03:38:56.031410+0000","mapping_epoch":163,"log_start":"113'5963","ondisk_log_start":"113'5963","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'8702","last_scrub_stamp":"2024-03-12T03:38:13.250858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:13.250858+0000","objects_scrubbed":1922,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39845888,"num_objects":608,"num_object_clones":0,"num_object_copies":1824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":252,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1314,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":29032448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,2],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":374,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574293+0000","last_change":"2024-03-12T03:38:15.587792+0000","last_active":"2024-03-12T03:38:55.574293+0000","last_peered":"2024-03-12T03:38:55.574293+0000","last_clean":"2024-03-12T03:38:55.574293+0000","last_became_active":"2024-03-12T03:38:08.909304+0000","last_became_peered":"2024-03-12T03:38:08.909304+0000","last_unstale":"2024-03-12T03:38:55.574293+0000","last_undegraded":"2024-03-12T03:38:55.574293+0000","last_fullsized":"2024-03-12T03:38:55.574293+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:38:15.587624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:38:15.587624+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:40.903171+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":326,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577808+0000","last_change":"2024-03-12T03:36:06.069994+0000","last_active":"2024-03-12T03:38:55.577808+0000","last_peered":"2024-03-12T03:38:55.577808+0000","last_clean":"2024-03-12T03:38:55.577808+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:38:55.577808+0000","last_undegraded":"2024-03-12T03:38:55.577808+0000","last_fullsized":"2024-03-12T03:38:55.577808+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:06.069862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.069862+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796674+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":290,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.679559+0000","last_change":"2024-03-12T03:38:25.351897+0000","last_active":"2024-03-12T03:38:55.679559+0000","last_peered":"2024-03-12T03:38:55.679559+0000","last_clean":"2024-03-12T03:38:55.679559+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:38:55.679559+0000","last_undegraded":"2024-03-12T03:38:55.679559+0000","last_fullsized":"2024-03-12T03:38:55.679559+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:38:25.351758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:25.351758+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:26.676757+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":265,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574439+0000","last_change":"2024-03-12T03:38:18.221931+0000","last_active":"2024-03-12T03:38:55.574439+0000","last_peered":"2024-03-12T03:38:55.574439+0000","last_clean":"2024-03-12T03:38:55.574439+0000","last_became_active":"2024-03-12T03:38:18.221243+0000","last_became_peered":"2024-03-12T03:38:18.221243+0000","last_unstale":"2024-03-12T03:38:55.574439+0000","last_undegraded":"2024-03-12T03:38:55.574439+0000","last_fullsized":"2024-03-12T03:38:55.574439+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:17.238386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:17.238386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":253,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.576892+0000","last_change":"2024-03-12T03:38:18.236746+0000","last_active":"2024-03-12T03:38:55.576892+0000","last_peered":"2024-03-12T03:38:55.576892+0000","last_clean":"2024-03-12T03:38:55.576892+0000","last_became_active":"2024-03-12T03:38:18.236362+0000","last_became_peered":"2024-03-12T03:38:18.236362+0000","last_unstale":"2024-03-12T03:38:55.576892+0000","last_undegraded":"2024-03-12T03:38:55.576892+0000","last_fullsized":"2024-03-12T03:38:55.576892+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:20.138004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.138004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796993+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"182'9976","reported_seq":10208,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.576466+0000","last_change":"2024-03-12T03:36:58.116826+0000","last_active":"2024-03-12T03:38:55.576466+0000","last_peered":"2024-03-12T03:38:55.576466+0000","last_clean":"2024-03-12T03:38:55.576466+0000","last_became_active":"2024-03-12T03:36:49.303476+0000","last_became_peered":"2024-03-12T03:36:49.303476+0000","last_unstale":"2024-03-12T03:38:55.576466+0000","last_undegraded":"2024-03-12T03:38:55.576466+0000","last_fullsized":"2024-03-12T03:38:55.576466+0000","mapping_epoch":97,"log_start":"110'5299","ondisk_log_start":"110'5299","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'6867","last_scrub_stamp":"2024-03-12T03:36:58.116574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:36:58.116574+0000","objects_scrubbed":3562,"log_size":4677,"log_dups_size":0,"ondisk_log_size":4677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.084304+0000","scrub_duration":900,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42074112,"num_objects":642,"num_object_clones":0,"num_object_copies":1926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3124,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":379,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578431+0000","last_change":"2024-03-12T03:38:38.526599+0000","last_active":"2024-03-12T03:38:55.578431+0000","last_peered":"2024-03-12T03:38:55.578431+0000","last_clean":"2024-03-12T03:38:55.578431+0000","last_became_active":"2024-03-12T03:38:08.887553+0000","last_became_peered":"2024-03-12T03:38:08.887553+0000","last_unstale":"2024-03-12T03:38:55.578431+0000","last_undegraded":"2024-03-12T03:38:55.578431+0000","last_fullsized":"2024-03-12T03:38:55.578431+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:38:38.526478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:38:38.526478+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:41.195998+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":250,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577576+0000","last_change":"2024-03-12T03:36:21.140906+0000","last_active":"2024-03-12T03:38:55.577576+0000","last_peered":"2024-03-12T03:38:55.577576+0000","last_clean":"2024-03-12T03:38:55.577576+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:38:55.577576+0000","last_undegraded":"2024-03-12T03:38:55.577576+0000","last_fullsized":"2024-03-12T03:38:55.577576+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:21.140825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:21.140825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":303,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578037+0000","last_change":"2024-03-12T03:36:22.118169+0000","last_active":"2024-03-12T03:38:55.578037+0000","last_peered":"2024-03-12T03:38:55.578037+0000","last_clean":"2024-03-12T03:38:55.578037+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:38:55.578037+0000","last_undegraded":"2024-03-12T03:38:55.578037+0000","last_fullsized":"2024-03-12T03:38:55.578037+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:36:22.118034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:22.118034+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796754+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":281,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.679683+0000","last_change":"2024-03-12T03:38:40.708651+0000","last_active":"2024-03-12T03:38:55.679683+0000","last_peered":"2024-03-12T03:38:55.679683+0000","last_clean":"2024-03-12T03:38:55.679683+0000","last_became_active":"2024-03-12T03:38:18.237880+0000","last_became_peered":"2024-03-12T03:38:18.237880+0000","last_unstale":"2024-03-12T03:38:55.679683+0000","last_undegraded":"2024-03-12T03:38:55.679683+0000","last_fullsized":"2024-03-12T03:38:55.679683+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:40.708062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:40.708062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:44.240629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":261,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.579017+0000","last_change":"2024-03-12T03:38:27.367381+0000","last_active":"2024-03-12T03:38:55.579017+0000","last_peered":"2024-03-12T03:38:55.579017+0000","last_clean":"2024-03-12T03:38:55.579017+0000","last_became_active":"2024-03-12T03:38:18.211980+0000","last_became_peered":"2024-03-12T03:38:18.211980+0000","last_unstale":"2024-03-12T03:38:55.579017+0000","last_undegraded":"2024-03-12T03:38:55.579017+0000","last_fullsized":"2024-03-12T03:38:55.579017+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:27.367266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:27.367266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:39.875679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"182'10002","reported_seq":10221,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.769139+0000","last_change":"2024-03-12T03:38:13.872123+0000","last_active":"2024-03-12T03:38:55.769139+0000","last_peered":"2024-03-12T03:38:55.769139+0000","last_clean":"2024-03-12T03:38:55.769139+0000","last_became_active":"2024-03-12T03:36:49.286111+0000","last_became_peered":"2024-03-12T03:36:49.286111+0000","last_unstale":"2024-03-12T03:38:55.769139+0000","last_undegraded":"2024-03-12T03:38:55.769139+0000","last_fullsized":"2024-03-12T03:38:55.769139+0000","mapping_epoch":97,"log_start":"106'4537","ondisk_log_start":"106'4537","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'8731","last_scrub_stamp":"2024-03-12T03:38:13.871955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:13.871955+0000","objects_scrubbed":1887,"log_size":5465,"log_dups_size":0,"ondisk_log_size":5465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:14.998338+0000","scrub_duration":472,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39845888,"num_objects":608,"num_object_clones":0,"num_object_copies":1824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1279,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":379,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577753+0000","last_change":"2024-03-12T03:38:37.490116+0000","last_active":"2024-03-12T03:38:55.577753+0000","last_peered":"2024-03-12T03:38:55.577753+0000","last_clean":"2024-03-12T03:38:55.577753+0000","last_became_active":"2024-03-12T03:38:08.885195+0000","last_became_peered":"2024-03-12T03:38:08.885195+0000","last_unstale":"2024-03-12T03:38:55.577753+0000","last_undegraded":"2024-03-12T03:38:55.577753+0000","last_fullsized":"2024-03-12T03:38:55.577753+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:38:37.490004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:38:37.490004+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:40:06.727828+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575257+0000","last_change":"2024-03-12T03:36:24.109073+0000","last_active":"2024-03-12T03:38:55.575257+0000","last_peered":"2024-03-12T03:38:55.575257+0000","last_clean":"2024-03-12T03:38:55.575257+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:38:55.575257+0000","last_undegraded":"2024-03-12T03:38:55.575257+0000","last_fullsized":"2024-03-12T03:38:55.575257+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:24.108984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.108984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":293,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578268+0000","last_change":"2024-03-12T03:38:18.276826+0000","last_active":"2024-03-12T03:38:55.578268+0000","last_peered":"2024-03-12T03:38:55.578268+0000","last_clean":"2024-03-12T03:38:55.578268+0000","last_became_active":"2024-03-12T03:38:18.233580+0000","last_became_peered":"2024-03-12T03:38:18.233580+0000","last_unstale":"2024-03-12T03:38:55.578268+0000","last_undegraded":"2024-03-12T03:38:55.578268+0000","last_fullsized":"2024-03-12T03:38:55.578268+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:36:03.015565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:03.015565+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796910+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":600,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.579287+0000","last_change":"2024-03-12T03:36:08.059941+0000","last_active":"2024-03-12T03:38:55.579287+0000","last_peered":"2024-03-12T03:38:55.579287+0000","last_clean":"2024-03-12T03:38:55.579287+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:38:55.579287+0000","last_undegraded":"2024-03-12T03:38:55.579287+0000","last_fullsized":"2024-03-12T03:38:55.579287+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:36:08.059814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:36:08.059814+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":329,"num_read_kb":324,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577277+0000","last_change":"2024-03-12T03:35:05.715931+0000","last_active":"2024-03-12T03:38:55.577277+0000","last_peered":"2024-03-12T03:38:55.577277+0000","last_clean":"2024-03-12T03:38:55.577277+0000","last_became_active":"2024-03-12T03:33:21.097086+0000","last_became_peered":"2024-03-12T03:33:21.097086+0000","last_unstale":"2024-03-12T03:38:55.577277+0000","last_undegraded":"2024-03-12T03:38:55.577277+0000","last_fullsized":"2024-03-12T03:38:55.577277+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:05.715849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.715849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.678247+0000","last_change":"2024-03-12T03:38:16.322511+0000","last_active":"2024-03-12T03:38:55.678247+0000","last_peered":"2024-03-12T03:38:55.678247+0000","last_clean":"2024-03-12T03:38:55.678247+0000","last_became_active":"2024-03-12T03:33:27.191878+0000","last_became_peered":"2024-03-12T03:33:27.191878+0000","last_unstale":"2024-03-12T03:38:55.678247+0000","last_undegraded":"2024-03-12T03:38:55.678247+0000","last_fullsized":"2024-03-12T03:38:55.678247+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:16.322249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:16.322249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:28.274564+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"181'10190","reported_seq":11925,"reported_epoch":181,"state":"peering","last_fresh":"2024-03-12T03:38:54.972507+0000","last_change":"2024-03-12T03:38:55.579301+0000","last_active":"2024-03-12T03:38:55.579301+0000","last_peered":"2024-03-12T03:38:54.972507+0000","last_clean":"2024-03-12T03:38:54.972507+0000","last_became_active":"2024-03-12T03:38:19.348574+0000","last_became_peered":"2024-03-12T03:38:19.348574+0000","last_unstale":"2024-03-12T03:38:54.972507+0000","last_undegraded":"2024-03-12T03:38:54.972507+0000","last_fullsized":"2024-03-12T03:38:54.972507+0000","mapping_epoch":182,"log_start":"112'5667","ondisk_log_start":"112'5667","created":97,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42074112,"num_objects":642,"num_object_clones":0,"num_object_copies":1926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6796,"num_write_kb":238016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":764,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":374,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578945+0000","last_change":"2024-03-12T03:38:14.385677+0000","last_active":"2024-03-12T03:38:55.578945+0000","last_peered":"2024-03-12T03:38:55.578945+0000","last_clean":"2024-03-12T03:38:55.578945+0000","last_became_active":"2024-03-12T03:38:08.891939+0000","last_became_peered":"2024-03-12T03:38:08.891939+0000","last_unstale":"2024-03-12T03:38:55.578945+0000","last_undegraded":"2024-03-12T03:38:55.578945+0000","last_fullsized":"2024-03-12T03:38:55.578945+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:38:14.385537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:38:14.385537+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:31.933064+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":249,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578481+0000","last_change":"2024-03-12T03:38:18.226197+0000","last_active":"2024-03-12T03:38:55.578481+0000","last_peered":"2024-03-12T03:38:55.578481+0000","last_clean":"2024-03-12T03:38:55.578481+0000","last_became_active":"2024-03-12T03:38:18.225853+0000","last_became_peered":"2024-03-12T03:38:18.225853+0000","last_unstale":"2024-03-12T03:38:55.578481+0000","last_undegraded":"2024-03-12T03:38:55.578481+0000","last_fullsized":"2024-03-12T03:38:55.578481+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:47.796561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:47.796561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":343,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578383+0000","last_change":"2024-03-12T03:36:03.118128+0000","last_active":"2024-03-12T03:38:55.578383+0000","last_peered":"2024-03-12T03:38:55.578383+0000","last_clean":"2024-03-12T03:38:55.578383+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:38:55.578383+0000","last_undegraded":"2024-03-12T03:38:55.578383+0000","last_fullsized":"2024-03-12T03:38:55.578383+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:36:03.117990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:03.117990+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574431+0000","last_change":"2024-03-12T03:35:14.996451+0000","last_active":"2024-03-12T03:38:55.574431+0000","last_peered":"2024-03-12T03:38:55.574431+0000","last_clean":"2024-03-12T03:38:55.574431+0000","last_became_active":"2024-03-12T03:33:21.100900+0000","last_became_peered":"2024-03-12T03:33:21.100900+0000","last_unstale":"2024-03-12T03:38:55.574431+0000","last_undegraded":"2024-03-12T03:38:55.574431+0000","last_fullsized":"2024-03-12T03:38:55.574431+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:14.996357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:14.996357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":523,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.579352+0000","last_change":"2024-03-12T03:38:38.459466+0000","last_active":"2024-03-12T03:38:55.579352+0000","last_peered":"2024-03-12T03:38:55.579352+0000","last_clean":"2024-03-12T03:38:55.579352+0000","last_became_active":"2024-03-12T03:38:03.852040+0000","last_became_peered":"2024-03-12T03:38:03.852040+0000","last_unstale":"2024-03-12T03:38:55.579352+0000","last_undegraded":"2024-03-12T03:38:55.579352+0000","last_fullsized":"2024-03-12T03:38:55.579352+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:38:38.459333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:38:38.459333+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:47.705139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574806+0000","last_change":"2024-03-12T03:36:25.119508+0000","last_active":"2024-03-12T03:38:55.574806+0000","last_peered":"2024-03-12T03:38:55.574806+0000","last_clean":"2024-03-12T03:38:55.574806+0000","last_became_active":"2024-03-12T03:33:27.188551+0000","last_became_peered":"2024-03-12T03:33:27.188551+0000","last_unstale":"2024-03-12T03:38:55.574806+0000","last_undegraded":"2024-03-12T03:38:55.574806+0000","last_fullsized":"2024-03-12T03:38:55.574806+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:25.119358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.119358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"182'10026","reported_seq":10239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:56.141995+0000","last_change":"2024-03-12T03:38:39.307742+0000","last_active":"2024-03-12T03:38:56.141995+0000","last_peered":"2024-03-12T03:38:56.141995+0000","last_clean":"2024-03-12T03:38:56.141995+0000","last_became_active":"2024-03-12T03:36:49.303137+0000","last_became_peered":"2024-03-12T03:36:49.303137+0000","last_unstale":"2024-03-12T03:38:56.141995+0000","last_undegraded":"2024-03-12T03:38:56.141995+0000","last_fullsized":"2024-03-12T03:38:56.141995+0000","mapping_epoch":97,"log_start":"109'5005","ondisk_log_start":"109'5005","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9434","last_scrub_stamp":"2024-03-12T03:38:39.307498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:39.307498+0000","objects_scrubbed":1226,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:48.934938+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41222144,"num_objects":629,"num_object_clones":0,"num_object_copies":1887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":597,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":51,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575122+0000","last_change":"2024-03-12T03:38:18.217813+0000","last_active":"2024-03-12T03:38:55.575122+0000","last_peered":"2024-03-12T03:38:55.575122+0000","last_clean":"2024-03-12T03:38:55.575122+0000","last_became_active":"2024-03-12T03:38:18.217500+0000","last_became_peered":"2024-03-12T03:38:18.217500+0000","last_unstale":"2024-03-12T03:38:55.575122+0000","last_undegraded":"2024-03-12T03:38:55.575122+0000","last_fullsized":"2024-03-12T03:38:55.575122+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:58.091182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.091182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":302,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.680375+0000","last_change":"2024-03-12T03:38:31.467414+0000","last_active":"2024-03-12T03:38:55.680375+0000","last_peered":"2024-03-12T03:38:55.680375+0000","last_clean":"2024-03-12T03:38:55.680375+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:38:55.680375+0000","last_undegraded":"2024-03-12T03:38:55.680375+0000","last_fullsized":"2024-03-12T03:38:55.680375+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-12T03:38:31.467274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:31.467274+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:40.000860+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.573932+0000","last_change":"2024-03-12T03:38:18.239687+0000","last_active":"2024-03-12T03:38:55.573932+0000","last_peered":"2024-03-12T03:38:55.573932+0000","last_clean":"2024-03-12T03:38:55.573932+0000","last_became_active":"2024-03-12T03:38:18.239005+0000","last_became_peered":"2024-03-12T03:38:18.239005+0000","last_unstale":"2024-03-12T03:38:55.573932+0000","last_undegraded":"2024-03-12T03:38:55.573932+0000","last_fullsized":"2024-03-12T03:38:55.573932+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:15.348471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.348471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":273,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578946+0000","last_change":"2024-03-12T03:38:36.598583+0000","last_active":"2024-03-12T03:38:55.578946+0000","last_peered":"2024-03-12T03:38:55.578946+0000","last_clean":"2024-03-12T03:38:55.578946+0000","last_became_active":"2024-03-12T03:38:18.227446+0000","last_became_peered":"2024-03-12T03:38:18.227446+0000","last_unstale":"2024-03-12T03:38:55.578946+0000","last_undegraded":"2024-03-12T03:38:55.578946+0000","last_fullsized":"2024-03-12T03:38:55.578946+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:36.598179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:38:36.598179+0000","last_clean_scrub_stamp":"2024-03-12T03:38:36.598179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:51.808839+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":249,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.580123+0000","last_change":"2024-03-12T03:36:02.083741+0000","last_active":"2024-03-12T03:38:55.580123+0000","last_peered":"2024-03-12T03:38:55.580123+0000","last_clean":"2024-03-12T03:38:55.580123+0000","last_became_active":"2024-03-12T03:33:27.380389+0000","last_became_peered":"2024-03-12T03:33:27.380389+0000","last_unstale":"2024-03-12T03:38:55.580123+0000","last_undegraded":"2024-03-12T03:38:55.580123+0000","last_fullsized":"2024-03-12T03:38:55.580123+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:36:02.083611+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:36:02.083611+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"182'10031","reported_seq":10242,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:56.143404+0000","last_change":"2024-03-12T03:38:38.373717+0000","last_active":"2024-03-12T03:38:56.143404+0000","last_peered":"2024-03-12T03:38:56.143404+0000","last_clean":"2024-03-12T03:38:56.143404+0000","last_became_active":"2024-03-12T03:36:49.281680+0000","last_became_peered":"2024-03-12T03:36:49.281680+0000","last_unstale":"2024-03-12T03:38:56.143404+0000","last_undegraded":"2024-03-12T03:38:56.143404+0000","last_fullsized":"2024-03-12T03:38:56.143404+0000","mapping_epoch":97,"log_start":"108'4993","ondisk_log_start":"108'4993","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'9442","last_scrub_stamp":"2024-03-12T03:38:38.373531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:38.373531+0000","objects_scrubbed":1190,"log_size":5038,"log_dups_size":0,"ondisk_log_size":5038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:40:02.873676+0000","scrub_duration":486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38797312,"num_objects":592,"num_object_clones":0,"num_object_copies":1776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":598,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":531,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578275+0000","last_change":"2024-03-12T03:38:36.431889+0000","last_active":"2024-03-12T03:38:55.578275+0000","last_peered":"2024-03-12T03:38:55.578275+0000","last_clean":"2024-03-12T03:38:55.578275+0000","last_became_active":"2024-03-12T03:38:08.893327+0000","last_became_peered":"2024-03-12T03:38:08.893327+0000","last_unstale":"2024-03-12T03:38:55.578275+0000","last_undegraded":"2024-03-12T03:38:55.578275+0000","last_fullsized":"2024-03-12T03:38:55.578275+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:38:36.431750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:38:36.431750+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:40:05.257091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578120+0000","last_change":"2024-03-12T03:36:15.170135+0000","last_active":"2024-03-12T03:38:55.578120+0000","last_peered":"2024-03-12T03:38:55.578120+0000","last_clean":"2024-03-12T03:38:55.578120+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:38:55.578120+0000","last_undegraded":"2024-03-12T03:38:55.578120+0000","last_fullsized":"2024-03-12T03:38:55.578120+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:15.170007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.170007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796619+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"166'36","reported_seq":351,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574258+0000","last_change":"2024-03-12T03:38:39.557767+0000","last_active":"2024-03-12T03:38:55.574258+0000","last_peered":"2024-03-12T03:38:55.574258+0000","last_clean":"2024-03-12T03:38:55.574258+0000","last_became_active":"2024-03-12T03:38:18.221593+0000","last_became_peered":"2024-03-12T03:38:18.221593+0000","last_unstale":"2024-03-12T03:38:55.574258+0000","last_undegraded":"2024-03-12T03:38:55.574258+0000","last_fullsized":"2024-03-12T03:38:55.574258+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'36","last_scrub_stamp":"2024-03-12T03:38:39.557592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:39.557592+0000","objects_scrubbed":11,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:49.172277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575616+0000","last_change":"2024-03-12T03:37:00.206483+0000","last_active":"2024-03-12T03:38:55.575616+0000","last_peered":"2024-03-12T03:38:55.575616+0000","last_clean":"2024-03-12T03:38:55.575616+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:38:55.575616+0000","last_undegraded":"2024-03-12T03:38:55.575616+0000","last_fullsized":"2024-03-12T03:38:55.575616+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:00.206383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:37:00.206383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.084370+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":270,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574572+0000","last_change":"2024-03-12T03:36:13.134209+0000","last_active":"2024-03-12T03:38:55.574572+0000","last_peered":"2024-03-12T03:38:55.574572+0000","last_clean":"2024-03-12T03:38:55.574572+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:38:55.574572+0000","last_undegraded":"2024-03-12T03:38:55.574572+0000","last_fullsized":"2024-03-12T03:38:55.574572+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:36:13.134087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.134087+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575427+0000","last_change":"2024-03-12T03:34:49.088183+0000","last_active":"2024-03-12T03:38:55.575427+0000","last_peered":"2024-03-12T03:38:55.575427+0000","last_clean":"2024-03-12T03:38:55.575427+0000","last_became_active":"2024-03-12T03:33:27.185041+0000","last_became_peered":"2024-03-12T03:33:27.185041+0000","last_unstale":"2024-03-12T03:38:55.575427+0000","last_undegraded":"2024-03-12T03:38:55.575427+0000","last_fullsized":"2024-03-12T03:38:55.575427+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:49.088098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:49.088098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"182'10102","reported_seq":10312,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.930684+0000","last_change":"2024-03-12T03:38:36.696071+0000","last_active":"2024-03-12T03:38:55.930684+0000","last_peered":"2024-03-12T03:38:55.930684+0000","last_clean":"2024-03-12T03:38:55.930684+0000","last_became_active":"2024-03-12T03:36:49.292644+0000","last_became_peered":"2024-03-12T03:36:49.292644+0000","last_unstale":"2024-03-12T03:38:55.930684+0000","last_undegraded":"2024-03-12T03:38:55.930684+0000","last_fullsized":"2024-03-12T03:38:55.930684+0000","mapping_epoch":97,"log_start":"106'4577","ondisk_log_start":"106'4577","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'9492","last_scrub_stamp":"2024-03-12T03:38:36.695889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:36.695889+0000","objects_scrubbed":1273,"log_size":5525,"log_dups_size":0,"ondisk_log_size":5525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:47.468450+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43319296,"num_objects":661,"num_object_clones":0,"num_object_copies":1983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":612,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":382,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.573787+0000","last_change":"2024-03-12T03:38:21.225431+0000","last_active":"2024-03-12T03:38:55.573787+0000","last_peered":"2024-03-12T03:38:55.573787+0000","last_clean":"2024-03-12T03:38:55.573787+0000","last_became_active":"2024-03-12T03:38:21.224389+0000","last_became_peered":"2024-03-12T03:38:21.224389+0000","last_unstale":"2024-03-12T03:38:55.573787+0000","last_undegraded":"2024-03-12T03:38:55.573787+0000","last_fullsized":"2024-03-12T03:38:55.573787+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:38:16.424028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:38:16.424028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:42.134032+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"111'20","reported_seq":288,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577911+0000","last_change":"2024-03-12T03:36:29.192473+0000","last_active":"2024-03-12T03:38:55.577911+0000","last_peered":"2024-03-12T03:38:55.577911+0000","last_clean":"2024-03-12T03:38:55.577911+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:38:55.577911+0000","last_undegraded":"2024-03-12T03:38:55.577911+0000","last_fullsized":"2024-03-12T03:38:55.577911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'20","last_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:36:29.192310+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796942+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574643+0000","last_change":"2024-03-12T03:36:26.134473+0000","last_active":"2024-03-12T03:38:55.574643+0000","last_peered":"2024-03-12T03:38:55.574643+0000","last_clean":"2024-03-12T03:38:55.574643+0000","last_became_active":"2024-03-12T03:33:21.102410+0000","last_became_peered":"2024-03-12T03:33:21.102410+0000","last_unstale":"2024-03-12T03:38:55.574643+0000","last_undegraded":"2024-03-12T03:38:55.574643+0000","last_fullsized":"2024-03-12T03:38:55.574643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.134337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.134337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":59,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575666+0000","last_change":"2024-03-12T03:38:36.241931+0000","last_active":"2024-03-12T03:38:55.575666+0000","last_peered":"2024-03-12T03:38:55.575666+0000","last_clean":"2024-03-12T03:38:55.575666+0000","last_became_active":"2024-03-12T03:38:18.222389+0000","last_became_peered":"2024-03-12T03:38:18.222389+0000","last_unstale":"2024-03-12T03:38:55.575666+0000","last_undegraded":"2024-03-12T03:38:55.575666+0000","last_fullsized":"2024-03-12T03:38:55.575666+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:36.241848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:38:36.241848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:40:04.062046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":253,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577507+0000","last_change":"2024-03-12T03:38:18.235036+0000","last_active":"2024-03-12T03:38:55.577507+0000","last_peered":"2024-03-12T03:38:55.577507+0000","last_clean":"2024-03-12T03:38:55.577507+0000","last_became_active":"2024-03-12T03:38:18.234209+0000","last_became_peered":"2024-03-12T03:38:18.234209+0000","last_unstale":"2024-03-12T03:38:55.577507+0000","last_undegraded":"2024-03-12T03:38:55.577507+0000","last_fullsized":"2024-03-12T03:38:55.577507+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:10.177808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:10.177808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796810+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"182'10026","reported_seq":13070,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.831817+0000","last_change":"2024-03-12T03:38:41.823265+0000","last_active":"2024-03-12T03:38:55.831817+0000","last_peered":"2024-03-12T03:38:55.831817+0000","last_clean":"2024-03-12T03:38:55.831817+0000","last_became_active":"2024-03-12T03:38:37.401255+0000","last_became_peered":"2024-03-12T03:38:37.401255+0000","last_unstale":"2024-03-12T03:38:55.831817+0000","last_undegraded":"2024-03-12T03:38:55.831817+0000","last_fullsized":"2024-03-12T03:38:55.831817+0000","mapping_epoch":172,"log_start":"110'5331","ondisk_log_start":"110'5331","created":97,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9566","last_scrub_stamp":"2024-03-12T03:38:41.823079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:41.823079+0000","objects_scrubbed":1066,"log_size":4695,"log_dups_size":0,"ondisk_log_size":4695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:52.594000+0000","scrub_duration":258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39583744,"num_objects":604,"num_object_clones":0,"num_object_copies":1812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":462,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":297,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577480+0000","last_change":"2024-03-12T03:36:27.151507+0000","last_active":"2024-03-12T03:38:55.577480+0000","last_peered":"2024-03-12T03:38:55.577480+0000","last_clean":"2024-03-12T03:38:55.577480+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:38:55.577480+0000","last_undegraded":"2024-03-12T03:38:55.577480+0000","last_fullsized":"2024-03-12T03:38:55.577480+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:36:27.151375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:27.151375+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.579517+0000","last_change":"2024-03-12T03:36:09.026855+0000","last_active":"2024-03-12T03:38:55.579517+0000","last_peered":"2024-03-12T03:38:55.579517+0000","last_clean":"2024-03-12T03:38:55.579517+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:38:55.579517+0000","last_undegraded":"2024-03-12T03:38:55.579517+0000","last_fullsized":"2024-03-12T03:38:55.579517+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:09.026725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:09.026725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":51,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577722+0000","last_change":"2024-03-12T03:38:18.231158+0000","last_active":"2024-03-12T03:38:55.577722+0000","last_peered":"2024-03-12T03:38:55.577722+0000","last_clean":"2024-03-12T03:38:55.577722+0000","last_became_active":"2024-03-12T03:38:18.230798+0000","last_became_peered":"2024-03-12T03:38:18.230798+0000","last_unstale":"2024-03-12T03:38:55.577722+0000","last_undegraded":"2024-03-12T03:38:55.577722+0000","last_fullsized":"2024-03-12T03:38:55.577722+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.320728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.320728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":245,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.579122+0000","last_change":"2024-03-12T03:38:18.224874+0000","last_active":"2024-03-12T03:38:55.579122+0000","last_peered":"2024-03-12T03:38:55.579122+0000","last_clean":"2024-03-12T03:38:55.579122+0000","last_became_active":"2024-03-12T03:38:18.223903+0000","last_became_peered":"2024-03-12T03:38:18.223903+0000","last_unstale":"2024-03-12T03:38:55.579122+0000","last_undegraded":"2024-03-12T03:38:55.579122+0000","last_fullsized":"2024-03-12T03:38:55.579122+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:43.716021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:34:43.716021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"182'10160","reported_seq":10371,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:56.192890+0000","last_change":"2024-03-12T03:38:39.981497+0000","last_active":"2024-03-12T03:38:56.192890+0000","last_peered":"2024-03-12T03:38:56.192890+0000","last_clean":"2024-03-12T03:38:56.192890+0000","last_became_active":"2024-03-12T03:36:49.278858+0000","last_became_peered":"2024-03-12T03:36:49.278858+0000","last_unstale":"2024-03-12T03:38:56.192890+0000","last_undegraded":"2024-03-12T03:38:56.192890+0000","last_fullsized":"2024-03-12T03:38:56.192890+0000","mapping_epoch":97,"log_start":"110'5233","ondisk_log_start":"110'5233","created":97,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9554","last_scrub_stamp":"2024-03-12T03:38:39.981389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:39.981389+0000","objects_scrubbed":1208,"log_size":4927,"log_dups_size":0,"ondisk_log_size":4927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:46.374860+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39452672,"num_objects":602,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":606,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":51,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577014+0000","last_change":"2024-03-12T03:38:18.235891+0000","last_active":"2024-03-12T03:38:55.577014+0000","last_peered":"2024-03-12T03:38:55.577014+0000","last_clean":"2024-03-12T03:38:55.577014+0000","last_became_active":"2024-03-12T03:38:18.233967+0000","last_became_peered":"2024-03-12T03:38:18.233967+0000","last_unstale":"2024-03-12T03:38:55.577014+0000","last_undegraded":"2024-03-12T03:38:55.577014+0000","last_fullsized":"2024-03-12T03:38:55.577014+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.332124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.332124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.797099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":302,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575879+0000","last_change":"2024-03-12T03:38:42.238946+0000","last_active":"2024-03-12T03:38:55.575879+0000","last_peered":"2024-03-12T03:38:55.575879+0000","last_clean":"2024-03-12T03:38:55.575879+0000","last_became_active":"2024-03-12T03:33:23.163351+0000","last_became_peered":"2024-03-12T03:33:23.163351+0000","last_unstale":"2024-03-12T03:38:55.575879+0000","last_undegraded":"2024-03-12T03:38:55.575879+0000","last_fullsized":"2024-03-12T03:38:55.575879+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-03-12T03:38:42.238782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:42.238782+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:56.219707+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578866+0000","last_change":"2024-03-12T03:36:17.051625+0000","last_active":"2024-03-12T03:38:55.578866+0000","last_peered":"2024-03-12T03:38:55.578866+0000","last_clean":"2024-03-12T03:38:55.578866+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:38:55.578866+0000","last_undegraded":"2024-03-12T03:38:55.578866+0000","last_fullsized":"2024-03-12T03:38:55.578866+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:36:17.051487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574928+0000","last_change":"2024-03-12T03:38:18.651533+0000","last_active":"2024-03-12T03:38:55.574928+0000","last_peered":"2024-03-12T03:38:55.574928+0000","last_clean":"2024-03-12T03:38:55.574928+0000","last_became_active":"2024-03-12T03:33:27.190753+0000","last_became_peered":"2024-03-12T03:33:27.190753+0000","last_unstale":"2024-03-12T03:38:55.574928+0000","last_undegraded":"2024-03-12T03:38:55.574928+0000","last_fullsized":"2024-03-12T03:38:55.574928+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:18.651418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:18.651418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:38.722772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"182'13495","reported_seq":15582,"reported_epoch":182,"state":"active+remapped+backfilling","last_fresh":"2024-03-12T03:38:56.305264+0000","last_change":"2024-03-12T03:38:49.378972+0000","last_active":"2024-03-12T03:38:56.305264+0000","last_peered":"2024-03-12T03:38:56.305264+0000","last_clean":"2024-03-12T03:38:17.951704+0000","last_became_active":"2024-03-12T03:38:19.307393+0000","last_became_peered":"2024-03-12T03:38:19.307393+0000","last_unstale":"2024-03-12T03:38:56.305264+0000","last_undegraded":"2024-03-12T03:38:56.305264+0000","last_fullsized":"2024-03-12T03:38:56.305264+0000","mapping_epoch":163,"log_start":"143'9794","ondisk_log_start":"143'9794","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":3701,"log_dups_size":0,"ondisk_log_size":3701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84082688,"num_objects":1283,"num_object_clones":0,"num_object_copies":3849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":398,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13495,"num_write_kb":472896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":946,"num_bytes_recovered":61997056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,2],"avail_no_missing":["3","2","5"],"object_location_counts":[{"shards":"2,3,5","objects":1283}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":318,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577845+0000","last_change":"2024-03-12T03:34:39.659403+0000","last_active":"2024-03-12T03:38:55.577845+0000","last_peered":"2024-03-12T03:38:55.577845+0000","last_clean":"2024-03-12T03:38:55.577845+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:38:55.577845+0000","last_undegraded":"2024-03-12T03:38:55.577845+0000","last_fullsized":"2024-03-12T03:38:55.577845+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:34:39.659277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:39.659277+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"90'36","reported_seq":313,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.573491+0000","last_change":"2024-03-12T03:36:15.181987+0000","last_active":"2024-03-12T03:38:55.573491+0000","last_peered":"2024-03-12T03:38:55.573491+0000","last_clean":"2024-03-12T03:38:55.573491+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:38:55.573491+0000","last_undegraded":"2024-03-12T03:38:55.573491+0000","last_fullsized":"2024-03-12T03:38:55.573491+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:36:15.181852+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:36:15.181852+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":263,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.579208+0000","last_change":"2024-03-12T03:35:12.907083+0000","last_active":"2024-03-12T03:38:55.579208+0000","last_peered":"2024-03-12T03:38:55.579208+0000","last_clean":"2024-03-12T03:38:55.579208+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:38:55.579208+0000","last_undegraded":"2024-03-12T03:38:55.579208+0000","last_fullsized":"2024-03-12T03:38:55.579208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:35:12.906988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:12.906988+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574040+0000","last_change":"2024-03-12T03:36:20.147626+0000","last_active":"2024-03-12T03:38:55.574040+0000","last_peered":"2024-03-12T03:38:55.574040+0000","last_clean":"2024-03-12T03:38:55.574040+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:38:55.574040+0000","last_undegraded":"2024-03-12T03:38:55.574040+0000","last_fullsized":"2024-03-12T03:38:55.574040+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.147488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"182'18617","reported_seq":19037,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:56.177966+0000","last_change":"2024-03-12T03:37:53.185726+0000","last_active":"2024-03-12T03:38:56.177966+0000","last_peered":"2024-03-12T03:38:56.177966+0000","last_clean":"2024-03-12T03:38:56.177966+0000","last_became_active":"2024-03-12T03:35:23.782112+0000","last_became_peered":"2024-03-12T03:35:23.782112+0000","last_unstale":"2024-03-12T03:38:56.177966+0000","last_undegraded":"2024-03-12T03:38:56.177966+0000","last_fullsized":"2024-03-12T03:38:56.177966+0000","mapping_epoch":97,"log_start":"142'14521","ondisk_log_start":"142'14521","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'14972","last_scrub_stamp":"2024-03-12T03:37:53.185533+0000","last_deep_scrub":"143'14972","last_deep_scrub_stamp":"2024-03-12T03:37:53.185533+0000","last_clean_scrub_stamp":"2024-03-12T03:37:53.185533+0000","objects_scrubbed":5605,"log_size":4096,"log_dups_size":0,"ondisk_log_size":4096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":35,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:21.227504+0000","scrub_duration":34812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82378752,"num_objects":1257,"num_object_clones":0,"num_object_copies":3771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13151,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":268,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575979+0000","last_change":"2024-03-12T03:35:00.079249+0000","last_active":"2024-03-12T03:38:55.575979+0000","last_peered":"2024-03-12T03:38:55.575979+0000","last_clean":"2024-03-12T03:38:55.575979+0000","last_became_active":"2024-03-12T03:33:23.551773+0000","last_became_peered":"2024-03-12T03:33:23.551773+0000","last_unstale":"2024-03-12T03:38:55.575979+0000","last_undegraded":"2024-03-12T03:38:55.575979+0000","last_fullsized":"2024-03-12T03:38:55.575979+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:35:00.079152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.079152+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":258,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577000+0000","last_change":"2024-03-12T03:35:58.075845+0000","last_active":"2024-03-12T03:38:55.577000+0000","last_peered":"2024-03-12T03:38:55.577000+0000","last_clean":"2024-03-12T03:38:55.577000+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:38:55.577000+0000","last_undegraded":"2024-03-12T03:38:55.577000+0000","last_fullsized":"2024-03-12T03:38:55.577000+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:58.075754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.075754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575021+0000","last_change":"2024-03-12T03:36:29.181191+0000","last_active":"2024-03-12T03:38:55.575021+0000","last_peered":"2024-03-12T03:38:55.575021+0000","last_clean":"2024-03-12T03:38:55.575021+0000","last_became_active":"2024-03-12T03:33:25.180752+0000","last_became_peered":"2024-03-12T03:33:25.180752+0000","last_unstale":"2024-03-12T03:38:55.575021+0000","last_undegraded":"2024-03-12T03:38:55.575021+0000","last_fullsized":"2024-03-12T03:38:55.575021+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:29.181010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:29.181010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578494+0000","last_change":"2024-03-12T03:36:24.959007+0000","last_active":"2024-03-12T03:38:55.578494+0000","last_peered":"2024-03-12T03:38:55.578494+0000","last_clean":"2024-03-12T03:38:55.578494+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:38:55.578494+0000","last_undegraded":"2024-03-12T03:38:55.578494+0000","last_fullsized":"2024-03-12T03:38:55.578494+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:24.958880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.958880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"182'10102","reported_seq":23015,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:56.107900+0000","last_change":"2024-03-12T03:38:37.420699+0000","last_active":"2024-03-12T03:38:56.107900+0000","last_peered":"2024-03-12T03:38:56.107900+0000","last_clean":"2024-03-12T03:38:56.107900+0000","last_became_active":"2024-03-12T03:38:37.402964+0000","last_became_peered":"2024-03-12T03:38:37.402964+0000","last_unstale":"2024-03-12T03:38:56.107900+0000","last_undegraded":"2024-03-12T03:38:56.107900+0000","last_fullsized":"2024-03-12T03:38:56.107900+0000","mapping_epoch":172,"log_start":"114'6196","ondisk_log_start":"114'6196","created":119,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":1049,"log_size":3906,"log_dups_size":0,"ondisk_log_size":3906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.819938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34766848,"num_objects":531,"num_object_clones":0,"num_object_copies":1593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6745,"num_write_kb":232800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3886,"num_bytes_recovered":254672896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":287,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.656061+0000","last_change":"2024-03-12T03:35:11.786139+0000","last_active":"2024-03-12T03:38:55.656061+0000","last_peered":"2024-03-12T03:38:55.656061+0000","last_clean":"2024-03-12T03:38:55.656061+0000","last_became_active":"2024-03-12T03:33:23.552164+0000","last_became_peered":"2024-03-12T03:33:23.552164+0000","last_unstale":"2024-03-12T03:38:55.656061+0000","last_undegraded":"2024-03-12T03:38:55.656061+0000","last_fullsized":"2024-03-12T03:38:55.656061+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:35:11.786040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.786040+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796556+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577739+0000","last_change":"2024-03-12T03:34:52.848468+0000","last_active":"2024-03-12T03:38:55.577739+0000","last_peered":"2024-03-12T03:38:55.577739+0000","last_clean":"2024-03-12T03:38:55.577739+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:38:55.577739+0000","last_undegraded":"2024-03-12T03:38:55.577739+0000","last_fullsized":"2024-03-12T03:38:55.577739+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:52.848366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:52.848366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575466+0000","last_change":"2024-03-12T03:35:04.007338+0000","last_active":"2024-03-12T03:38:55.575466+0000","last_peered":"2024-03-12T03:38:55.575466+0000","last_clean":"2024-03-12T03:38:55.575466+0000","last_became_active":"2024-03-12T03:33:25.174953+0000","last_became_peered":"2024-03-12T03:33:25.174953+0000","last_unstale":"2024-03-12T03:38:55.575466+0000","last_undegraded":"2024-03-12T03:38:55.575466+0000","last_fullsized":"2024-03-12T03:38:55.575466+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:35:04.007252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":58,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.576134+0000","last_change":"2024-03-12T03:38:35.255743+0000","last_active":"2024-03-12T03:38:55.576134+0000","last_peered":"2024-03-12T03:38:55.576134+0000","last_clean":"2024-03-12T03:38:55.576134+0000","last_became_active":"2024-03-12T03:38:18.219875+0000","last_became_peered":"2024-03-12T03:38:18.219875+0000","last_unstale":"2024-03-12T03:38:55.576134+0000","last_undegraded":"2024-03-12T03:38:55.576134+0000","last_fullsized":"2024-03-12T03:38:55.576134+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:35.255629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:35.255629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:44.223068+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.15","version":"182'10062","reported_seq":17283,"reported_epoch":182,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-12T03:38:55.800630+0000","last_change":"2024-03-12T03:38:49.369862+0000","last_active":"2024-03-12T03:38:55.800630+0000","last_peered":"2024-03-12T03:38:55.800630+0000","last_clean":"2024-03-12T03:38:17.921102+0000","last_became_active":"2024-03-12T03:38:19.326603+0000","last_became_peered":"2024-03-12T03:38:19.326603+0000","last_unstale":"2024-03-12T03:38:55.800630+0000","last_undegraded":"2024-03-12T03:38:55.800630+0000","last_fullsized":"2024-03-12T03:38:55.800630+0000","mapping_epoch":163,"log_start":"110'5478","ondisk_log_start":"110'5478","created":119,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4584,"log_dups_size":0,"ondisk_log_size":4584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46923776,"num_objects":716,"num_object_clones":0,"num_object_copies":2148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":716,"num_objects_unfound":0,"num_objects_dirty":716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6668,"num_write_kb":236288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3504,"num_bytes_recovered":229638144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,2],"avail_no_missing":["7","1","2"],"object_location_counts":[{"shards":"1,2,7","objects":716}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":283,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574777+0000","last_change":"2024-03-12T03:36:12.092317+0000","last_active":"2024-03-12T03:38:55.574777+0000","last_peered":"2024-03-12T03:38:55.574777+0000","last_clean":"2024-03-12T03:38:55.574777+0000","last_became_active":"2024-03-12T03:33:23.553480+0000","last_became_peered":"2024-03-12T03:33:23.553480+0000","last_unstale":"2024-03-12T03:38:55.574777+0000","last_undegraded":"2024-03-12T03:38:55.574777+0000","last_fullsized":"2024-03-12T03:38:55.574777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:36:12.092175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:12.092175+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577455+0000","last_change":"2024-03-12T03:36:14.197774+0000","last_active":"2024-03-12T03:38:55.577455+0000","last_peered":"2024-03-12T03:38:55.577455+0000","last_clean":"2024-03-12T03:38:55.577455+0000","last_became_active":"2024-03-12T03:33:21.101819+0000","last_became_peered":"2024-03-12T03:33:21.101819+0000","last_unstale":"2024-03-12T03:38:55.577455+0000","last_undegraded":"2024-03-12T03:38:55.577455+0000","last_fullsized":"2024-03-12T03:38:55.577455+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:14.197649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:36:14.197649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.797074+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"166'9","reported_seq":331,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575050+0000","last_change":"2024-03-12T03:38:18.235751+0000","last_active":"2024-03-12T03:38:55.575050+0000","last_peered":"2024-03-12T03:38:55.575050+0000","last_clean":"2024-03-12T03:38:55.575050+0000","last_became_active":"2024-03-12T03:38:18.221313+0000","last_became_peered":"2024-03-12T03:38:18.221313+0000","last_unstale":"2024-03-12T03:38:55.575050+0000","last_undegraded":"2024-03-12T03:38:55.575050+0000","last_fullsized":"2024-03-12T03:38:55.575050+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:35:08.006375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:08.006375+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":253,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.576251+0000","last_change":"2024-03-12T03:38:18.220179+0000","last_active":"2024-03-12T03:38:55.576251+0000","last_peered":"2024-03-12T03:38:55.576251+0000","last_clean":"2024-03-12T03:38:55.576251+0000","last_became_active":"2024-03-12T03:38:18.219841+0000","last_became_peered":"2024-03-12T03:38:18.219841+0000","last_unstale":"2024-03-12T03:38:55.576251+0000","last_undegraded":"2024-03-12T03:38:55.576251+0000","last_fullsized":"2024-03-12T03:38:55.576251+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:10.110987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:37:10.110987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.084385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"182'10000","reported_seq":15851,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.802698+0000","last_change":"2024-03-12T03:38:14.754754+0000","last_active":"2024-03-12T03:38:55.802698+0000","last_peered":"2024-03-12T03:38:55.802698+0000","last_clean":"2024-03-12T03:38:55.802698+0000","last_became_active":"2024-03-12T03:37:46.698100+0000","last_became_peered":"2024-03-12T03:37:46.698100+0000","last_unstale":"2024-03-12T03:38:55.802698+0000","last_undegraded":"2024-03-12T03:38:55.802698+0000","last_fullsized":"2024-03-12T03:38:55.802698+0000","mapping_epoch":139,"log_start":"110'5301","ondisk_log_start":"110'5301","created":119,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"160'8785","last_scrub_stamp":"2024-03-12T03:38:14.754432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:14.754432+0000","objects_scrubbed":1845,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:34.639547+0000","scrub_duration":397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40697856,"num_objects":621,"num_object_clones":0,"num_object_copies":1863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1224,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":286,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574894+0000","last_change":"2024-03-12T03:35:58.079779+0000","last_active":"2024-03-12T03:38:55.574894+0000","last_peered":"2024-03-12T03:38:55.574894+0000","last_clean":"2024-03-12T03:38:55.574894+0000","last_became_active":"2024-03-12T03:33:23.551706+0000","last_became_peered":"2024-03-12T03:33:23.551706+0000","last_unstale":"2024-03-12T03:38:55.574894+0000","last_undegraded":"2024-03-12T03:38:55.574894+0000","last_fullsized":"2024-03-12T03:38:55.574894+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-03-12T03:35:58.079645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.079645+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":275,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577797+0000","last_change":"2024-03-12T03:38:18.226777+0000","last_active":"2024-03-12T03:38:55.577797+0000","last_peered":"2024-03-12T03:38:55.577797+0000","last_clean":"2024-03-12T03:38:55.577797+0000","last_became_active":"2024-03-12T03:38:18.217470+0000","last_became_peered":"2024-03-12T03:38:18.217470+0000","last_unstale":"2024-03-12T03:38:55.577797+0000","last_undegraded":"2024-03-12T03:38:55.577797+0000","last_fullsized":"2024-03-12T03:38:55.577797+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:27.955624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:27.955624+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":326,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574625+0000","last_change":"2024-03-12T03:36:07.296952+0000","last_active":"2024-03-12T03:38:55.574625+0000","last_peered":"2024-03-12T03:38:55.574625+0000","last_clean":"2024-03-12T03:38:55.574625+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:38:55.574625+0000","last_undegraded":"2024-03-12T03:38:55.574625+0000","last_fullsized":"2024-03-12T03:38:55.574625+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:07.296796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.296796+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578273+0000","last_change":"2024-03-12T03:36:07.111872+0000","last_active":"2024-03-12T03:38:55.578273+0000","last_peered":"2024-03-12T03:38:55.578273+0000","last_clean":"2024-03-12T03:38:55.578273+0000","last_became_active":"2024-03-12T03:33:27.194094+0000","last_became_peered":"2024-03-12T03:33:27.194094+0000","last_unstale":"2024-03-12T03:38:55.578273+0000","last_undegraded":"2024-03-12T03:38:55.578273+0000","last_fullsized":"2024-03-12T03:38:55.578273+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:07.111722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:07.111722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796968+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"182'10170","reported_seq":23374,"reported_epoch":182,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-12T03:38:55.831205+0000","last_change":"2024-03-12T03:38:49.358748+0000","last_active":"2024-03-12T03:38:55.831205+0000","last_peered":"2024-03-12T03:38:55.831205+0000","last_clean":"2024-03-12T03:38:17.986599+0000","last_became_active":"2024-03-12T03:38:19.316197+0000","last_became_peered":"2024-03-12T03:38:19.316197+0000","last_unstale":"2024-03-12T03:38:55.831205+0000","last_undegraded":"2024-03-12T03:38:55.831205+0000","last_fullsized":"2024-03-12T03:38:55.831205+0000","mapping_epoch":163,"log_start":"112'5644","ondisk_log_start":"112'5644","created":119,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":4526,"log_dups_size":0,"ondisk_log_size":4526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38469632,"num_objects":587,"num_object_clones":0,"num_object_copies":1761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":587,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6775,"num_write_kb":235584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3248,"num_bytes_recovered":212795392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,2],"avail_no_missing":["7","2","3","5","6"],"object_location_counts":[{"shards":"2,5,7","objects":587}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":274,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575904+0000","last_change":"2024-03-12T03:37:02.194152+0000","last_active":"2024-03-12T03:38:55.575904+0000","last_peered":"2024-03-12T03:38:55.575904+0000","last_clean":"2024-03-12T03:38:55.575904+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:38:55.575904+0000","last_undegraded":"2024-03-12T03:38:55.575904+0000","last_fullsized":"2024-03-12T03:38:55.575904+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:37:02.193985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:37:02.193985+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.084339+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":267,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575766+0000","last_change":"2024-03-12T03:38:34.264824+0000","last_active":"2024-03-12T03:38:55.575766+0000","last_peered":"2024-03-12T03:38:55.575766+0000","last_clean":"2024-03-12T03:38:55.575766+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:38:55.575766+0000","last_undegraded":"2024-03-12T03:38:55.575766+0000","last_fullsized":"2024-03-12T03:38:55.575766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:34.264717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:38:34.264717+0000","last_clean_scrub_stamp":"2024-03-12T03:38:34.264717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:40:02.274429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.679455+0000","last_change":"2024-03-12T03:38:22.343862+0000","last_active":"2024-03-12T03:38:55.679455+0000","last_peered":"2024-03-12T03:38:55.679455+0000","last_clean":"2024-03-12T03:38:55.679455+0000","last_became_active":"2024-03-12T03:33:25.182004+0000","last_became_peered":"2024-03-12T03:33:25.182004+0000","last_unstale":"2024-03-12T03:38:55.679455+0000","last_undegraded":"2024-03-12T03:38:55.679455+0000","last_fullsized":"2024-03-12T03:38:55.679455+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:22.343725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:38:22.343725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:35.045542+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.573820+0000","last_change":"2024-03-12T03:35:59.076156+0000","last_active":"2024-03-12T03:38:55.573820+0000","last_peered":"2024-03-12T03:38:55.573820+0000","last_clean":"2024-03-12T03:38:55.573820+0000","last_became_active":"2024-03-12T03:33:27.191710+0000","last_became_peered":"2024-03-12T03:33:27.191710+0000","last_unstale":"2024-03-12T03:38:55.573820+0000","last_undegraded":"2024-03-12T03:38:55.573820+0000","last_fullsized":"2024-03-12T03:38:55.573820+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.075897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.075897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"182'9904","reported_seq":19021,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:56.076655+0000","last_change":"2024-03-12T03:38:36.415536+0000","last_active":"2024-03-12T03:38:56.076655+0000","last_peered":"2024-03-12T03:38:56.076655+0000","last_clean":"2024-03-12T03:38:56.076655+0000","last_became_active":"2024-03-12T03:38:36.397989+0000","last_became_peered":"2024-03-12T03:38:36.397989+0000","last_unstale":"2024-03-12T03:38:56.076655+0000","last_undegraded":"2024-03-12T03:38:56.076655+0000","last_fullsized":"2024-03-12T03:38:56.076655+0000","mapping_epoch":171,"log_start":"113'5853","ondisk_log_start":"113'5853","created":119,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":5,"last_scrub":"161'8743","last_scrub_stamp":"2024-03-12T03:38:15.361698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:15.361698+0000","objects_scrubbed":1829,"log_size":4051,"log_dups_size":0,"ondisk_log_size":4051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:23.274581+0000","scrub_duration":792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42270720,"num_objects":645,"num_object_clones":0,"num_object_copies":1935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1184,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1509,"num_bytes_recovered":98893824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":295,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.679735+0000","last_change":"2024-03-12T03:38:23.363276+0000","last_active":"2024-03-12T03:38:55.679735+0000","last_peered":"2024-03-12T03:38:55.679735+0000","last_clean":"2024-03-12T03:38:55.679735+0000","last_became_active":"2024-03-12T03:38:18.236536+0000","last_became_peered":"2024-03-12T03:38:18.236536+0000","last_unstale":"2024-03-12T03:38:55.679735+0000","last_undegraded":"2024-03-12T03:38:55.679735+0000","last_fullsized":"2024-03-12T03:38:55.679735+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-12T03:38:23.362600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:23.362600+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:46.239985+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578580+0000","last_change":"2024-03-12T03:34:41.698397+0000","last_active":"2024-03-12T03:38:55.578580+0000","last_peered":"2024-03-12T03:38:55.578580+0000","last_clean":"2024-03-12T03:38:55.578580+0000","last_became_active":"2024-03-12T03:33:21.107037+0000","last_became_peered":"2024-03-12T03:33:21.107037+0000","last_unstale":"2024-03-12T03:38:55.578580+0000","last_undegraded":"2024-03-12T03:38:55.578580+0000","last_fullsized":"2024-03-12T03:38:55.578580+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.698268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.698268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574735+0000","last_change":"2024-03-12T03:34:45.089928+0000","last_active":"2024-03-12T03:38:55.574735+0000","last_peered":"2024-03-12T03:38:55.574735+0000","last_clean":"2024-03-12T03:38:55.574735+0000","last_became_active":"2024-03-12T03:33:25.183725+0000","last_became_peered":"2024-03-12T03:33:25.183725+0000","last_unstale":"2024-03-12T03:38:55.574735+0000","last_undegraded":"2024-03-12T03:38:55.574735+0000","last_fullsized":"2024-03-12T03:38:55.574735+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:45.089762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:45.089762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.576828+0000","last_change":"2024-03-12T03:36:18.113521+0000","last_active":"2024-03-12T03:38:55.576828+0000","last_peered":"2024-03-12T03:38:55.576828+0000","last_clean":"2024-03-12T03:38:55.576828+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:38:55.576828+0000","last_undegraded":"2024-03-12T03:38:55.576828+0000","last_fullsized":"2024-03-12T03:38:55.576828+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.113393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:18.113393+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.113393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796853+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"182'9911","reported_seq":15204,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.824122+0000","last_change":"2024-03-12T03:38:39.628659+0000","last_active":"2024-03-12T03:38:55.824122+0000","last_peered":"2024-03-12T03:38:55.824122+0000","last_clean":"2024-03-12T03:38:55.824122+0000","last_became_active":"2024-03-12T03:37:55.756190+0000","last_became_peered":"2024-03-12T03:37:55.756190+0000","last_unstale":"2024-03-12T03:38:55.824122+0000","last_undegraded":"2024-03-12T03:38:55.824122+0000","last_fullsized":"2024-03-12T03:38:55.824122+0000","mapping_epoch":144,"log_start":"110'5264","ondisk_log_start":"110'5264","created":119,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":5,"last_scrub":"175'9379","last_scrub_stamp":"2024-03-12T03:38:39.628512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:39.628512+0000","objects_scrubbed":1167,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:51.445423+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41615360,"num_objects":635,"num_object_clones":0,"num_object_copies":1905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":532,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"74'18","reported_seq":279,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574453+0000","last_change":"2024-03-12T03:35:02.079317+0000","last_active":"2024-03-12T03:38:55.574453+0000","last_peered":"2024-03-12T03:38:55.574453+0000","last_clean":"2024-03-12T03:38:55.574453+0000","last_became_active":"2024-03-12T03:33:23.552210+0000","last_became_peered":"2024-03-12T03:33:23.552210+0000","last_unstale":"2024-03-12T03:38:55.574453+0000","last_undegraded":"2024-03-12T03:38:55.574453+0000","last_fullsized":"2024-03-12T03:38:55.574453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:35:02.079224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:02.079224+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575014+0000","last_change":"2024-03-12T03:35:59.352393+0000","last_active":"2024-03-12T03:38:55.575014+0000","last_peered":"2024-03-12T03:38:55.575014+0000","last_clean":"2024-03-12T03:38:55.575014+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:38:55.575014+0000","last_undegraded":"2024-03-12T03:38:55.575014+0000","last_fullsized":"2024-03-12T03:38:55.575014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.352272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.352272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.579512+0000","last_change":"2024-03-12T03:34:53.878589+0000","last_active":"2024-03-12T03:38:55.579512+0000","last_peered":"2024-03-12T03:38:55.579512+0000","last_clean":"2024-03-12T03:38:55.579512+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:38:55.579512+0000","last_undegraded":"2024-03-12T03:38:55.579512+0000","last_fullsized":"2024-03-12T03:38:55.579512+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:53.878484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:34:53.878484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575608+0000","last_change":"2024-03-12T03:34:40.191618+0000","last_active":"2024-03-12T03:38:55.575608+0000","last_peered":"2024-03-12T03:38:55.575608+0000","last_clean":"2024-03-12T03:38:55.575608+0000","last_became_active":"2024-03-12T03:33:27.381110+0000","last_became_peered":"2024-03-12T03:33:27.381110+0000","last_unstale":"2024-03-12T03:38:55.575608+0000","last_undegraded":"2024-03-12T03:38:55.575608+0000","last_fullsized":"2024-03-12T03:38:55.575608+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:34:40.191538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"182'10083","reported_seq":16924,"reported_epoch":182,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-12T03:38:55.924848+0000","last_change":"2024-03-12T03:38:49.363222+0000","last_active":"2024-03-12T03:38:55.924848+0000","last_peered":"2024-03-12T03:38:55.924848+0000","last_clean":"2024-03-12T03:38:17.880414+0000","last_became_active":"2024-03-12T03:38:19.333585+0000","last_became_peered":"2024-03-12T03:38:19.333585+0000","last_unstale":"2024-03-12T03:38:55.924848+0000","last_undegraded":"2024-03-12T03:38:55.924848+0000","last_fullsized":"2024-03-12T03:38:55.924848+0000","mapping_epoch":163,"log_start":"107'4632","ondisk_log_start":"107'4632","created":119,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5451,"log_dups_size":0,"ondisk_log_size":5451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39911424,"num_objects":609,"num_object_clones":0,"num_object_copies":1827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":609,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6721,"num_write_kb":234560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2207,"num_bytes_recovered":144637952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,2],"avail_no_missing":["6","0","2","3"],"object_location_counts":[{"shards":"0,2,6","objects":609}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"90'6","reported_seq":263,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.573842+0000","last_change":"2024-03-12T03:38:38.943722+0000","last_active":"2024-03-12T03:38:55.573842+0000","last_peered":"2024-03-12T03:38:55.573842+0000","last_clean":"2024-03-12T03:38:55.573842+0000","last_became_active":"2024-03-12T03:33:23.161801+0000","last_became_peered":"2024-03-12T03:33:23.161801+0000","last_unstale":"2024-03-12T03:38:55.573842+0000","last_undegraded":"2024-03-12T03:38:55.573842+0000","last_fullsized":"2024-03-12T03:38:55.573842+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:38:38.943094+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:38:38.943094+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:40:05.413319+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.656113+0000","last_change":"2024-03-12T03:35:59.076304+0000","last_active":"2024-03-12T03:38:55.656113+0000","last_peered":"2024-03-12T03:38:55.656113+0000","last_clean":"2024-03-12T03:38:55.656113+0000","last_became_active":"2024-03-12T03:33:21.096560+0000","last_became_peered":"2024-03-12T03:33:21.096560+0000","last_unstale":"2024-03-12T03:38:55.656113+0000","last_undegraded":"2024-03-12T03:38:55.656113+0000","last_fullsized":"2024-03-12T03:38:55.656113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.076203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.076203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796645+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":250,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577822+0000","last_change":"2024-03-12T03:36:01.098836+0000","last_active":"2024-03-12T03:38:55.577822+0000","last_peered":"2024-03-12T03:38:55.577822+0000","last_clean":"2024-03-12T03:38:55.577822+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:38:55.577822+0000","last_undegraded":"2024-03-12T03:38:55.577822+0000","last_fullsized":"2024-03-12T03:38:55.577822+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:01.098726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:01.098726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":245,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578371+0000","last_change":"2024-03-12T03:38:18.221471+0000","last_active":"2024-03-12T03:38:55.578371+0000","last_peered":"2024-03-12T03:38:55.578371+0000","last_clean":"2024-03-12T03:38:55.578371+0000","last_became_active":"2024-03-12T03:38:18.221111+0000","last_became_peered":"2024-03-12T03:38:18.221111+0000","last_unstale":"2024-03-12T03:38:55.578371+0000","last_undegraded":"2024-03-12T03:38:55.578371+0000","last_fullsized":"2024-03-12T03:38:55.578371+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.855641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.855641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"182'10119","reported_seq":19354,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:56.180996+0000","last_change":"2024-03-12T03:38:55.707035+0000","last_active":"2024-03-12T03:38:56.180996+0000","last_peered":"2024-03-12T03:38:56.180996+0000","last_clean":"2024-03-12T03:38:56.180996+0000","last_became_active":"2024-03-12T03:38:55.656452+0000","last_became_peered":"2024-03-12T03:38:55.656452+0000","last_unstale":"2024-03-12T03:38:56.180996+0000","last_undegraded":"2024-03-12T03:38:56.180996+0000","last_fullsized":"2024-03-12T03:38:56.180996+0000","mapping_epoch":181,"log_start":"109'5103","ondisk_log_start":"109'5103","created":119,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:35:22.761495+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.754146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43122688,"num_objects":658,"num_object_clones":0,"num_object_copies":1974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6744,"num_write_kb":236864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3245,"num_bytes_recovered":212664320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"94'23","reported_seq":291,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578613+0000","last_change":"2024-03-12T03:36:20.132927+0000","last_active":"2024-03-12T03:38:55.578613+0000","last_peered":"2024-03-12T03:38:55.578613+0000","last_clean":"2024-03-12T03:38:55.578613+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:38:55.578613+0000","last_undegraded":"2024-03-12T03:38:55.578613+0000","last_fullsized":"2024-03-12T03:38:55.578613+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'23","last_scrub_stamp":"2024-03-12T03:36:20.132683+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:34:52.729983+0000","last_clean_scrub_stamp":"2024-03-12T03:36:20.132683+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":257,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.573792+0000","last_change":"2024-03-12T03:36:25.092455+0000","last_active":"2024-03-12T03:38:55.573792+0000","last_peered":"2024-03-12T03:38:55.573792+0000","last_clean":"2024-03-12T03:38:55.573792+0000","last_became_active":"2024-03-12T03:33:21.095556+0000","last_became_peered":"2024-03-12T03:33:21.095556+0000","last_unstale":"2024-03-12T03:38:55.573792+0000","last_undegraded":"2024-03-12T03:38:55.573792+0000","last_fullsized":"2024-03-12T03:38:55.573792+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:25.092313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.092313+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":257,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.656200+0000","last_change":"2024-03-12T03:38:18.235604+0000","last_active":"2024-03-12T03:38:55.656200+0000","last_peered":"2024-03-12T03:38:55.656200+0000","last_clean":"2024-03-12T03:38:55.656200+0000","last_became_active":"2024-03-12T03:38:18.235308+0000","last_became_peered":"2024-03-12T03:38:18.235308+0000","last_unstale":"2024-03-12T03:38:55.656200+0000","last_undegraded":"2024-03-12T03:38:55.656200+0000","last_fullsized":"2024-03-12T03:38:55.656200+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:24.162965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:24.162965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.797018+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":254,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.680499+0000","last_change":"2024-03-12T03:38:28.378219+0000","last_active":"2024-03-12T03:38:55.680499+0000","last_peered":"2024-03-12T03:38:55.680499+0000","last_clean":"2024-03-12T03:38:55.680499+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:38:55.680499+0000","last_undegraded":"2024-03-12T03:38:55.680499+0000","last_fullsized":"2024-03-12T03:38:55.680499+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:28.377664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:28.377664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:28.649951+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'17","reported_seq":283,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.655811+0000","last_change":"2024-03-12T03:36:12.216521+0000","last_active":"2024-03-12T03:38:55.655811+0000","last_peered":"2024-03-12T03:38:55.655811+0000","last_clean":"2024-03-12T03:38:55.655811+0000","last_became_active":"2024-03-12T03:33:23.552812+0000","last_became_peered":"2024-03-12T03:33:23.552812+0000","last_unstale":"2024-03-12T03:38:55.655811+0000","last_undegraded":"2024-03-12T03:38:55.655811+0000","last_fullsized":"2024-03-12T03:38:55.655811+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:36:12.216269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:12.216269+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796783+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":268,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578788+0000","last_change":"2024-03-12T03:38:35.589089+0000","last_active":"2024-03-12T03:38:55.578788+0000","last_peered":"2024-03-12T03:38:55.578788+0000","last_clean":"2024-03-12T03:38:55.578788+0000","last_became_active":"2024-03-12T03:33:21.097218+0000","last_became_peered":"2024-03-12T03:33:21.097218+0000","last_unstale":"2024-03-12T03:38:55.578788+0000","last_undegraded":"2024-03-12T03:38:55.578788+0000","last_fullsized":"2024-03-12T03:38:55.578788+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:38:35.588969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:35.588969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:45.081888+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":328,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578525+0000","last_change":"2024-03-12T03:36:00.063359+0000","last_active":"2024-03-12T03:38:55.578525+0000","last_peered":"2024-03-12T03:38:55.578525+0000","last_clean":"2024-03-12T03:38:55.578525+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:38:55.578525+0000","last_undegraded":"2024-03-12T03:38:55.578525+0000","last_fullsized":"2024-03-12T03:38:55.578525+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:36:00.063228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:00.063228+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":258,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578370+0000","last_change":"2024-03-12T03:38:18.237278+0000","last_active":"2024-03-12T03:38:55.578370+0000","last_peered":"2024-03-12T03:38:55.578370+0000","last_clean":"2024-03-12T03:38:55.578370+0000","last_became_active":"2024-03-12T03:38:18.222322+0000","last_became_peered":"2024-03-12T03:38:18.222322+0000","last_unstale":"2024-03-12T03:38:55.578370+0000","last_undegraded":"2024-03-12T03:38:55.578370+0000","last_fullsized":"2024-03-12T03:38:55.578370+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:35:57.050605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:57.050605+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":282,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574157+0000","last_change":"2024-03-12T03:36:28.133412+0000","last_active":"2024-03-12T03:38:55.574157+0000","last_peered":"2024-03-12T03:38:55.574157+0000","last_clean":"2024-03-12T03:38:55.574157+0000","last_became_active":"2024-03-12T03:33:23.162427+0000","last_became_peered":"2024-03-12T03:33:23.162427+0000","last_unstale":"2024-03-12T03:38:55.574157+0000","last_undegraded":"2024-03-12T03:38:55.574157+0000","last_fullsized":"2024-03-12T03:38:55.574157+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:36:28.133261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.133261+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":273,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575490+0000","last_change":"2024-03-12T03:38:29.234776+0000","last_active":"2024-03-12T03:38:55.575490+0000","last_peered":"2024-03-12T03:38:55.575490+0000","last_clean":"2024-03-12T03:38:55.575490+0000","last_became_active":"2024-03-12T03:38:18.219713+0000","last_became_peered":"2024-03-12T03:38:18.219713+0000","last_unstale":"2024-03-12T03:38:55.575490+0000","last_undegraded":"2024-03-12T03:38:55.575490+0000","last_fullsized":"2024-03-12T03:38:55.575490+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:29.234641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:29.234641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:29.607880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":265,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.576010+0000","last_change":"2024-03-12T03:38:30.271818+0000","last_active":"2024-03-12T03:38:55.576010+0000","last_peered":"2024-03-12T03:38:55.576010+0000","last_clean":"2024-03-12T03:38:55.576010+0000","last_became_active":"2024-03-12T03:38:18.221780+0000","last_became_peered":"2024-03-12T03:38:18.221780+0000","last_unstale":"2024-03-12T03:38:55.576010+0000","last_undegraded":"2024-03-12T03:38:55.576010+0000","last_fullsized":"2024-03-12T03:38:55.576010+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:30.271687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:38:30.271687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:49.336364+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575878+0000","last_change":"2024-03-12T03:38:16.588878+0000","last_active":"2024-03-12T03:38:55.575878+0000","last_peered":"2024-03-12T03:38:55.575878+0000","last_clean":"2024-03-12T03:38:55.575878+0000","last_became_active":"2024-03-12T03:33:27.380880+0000","last_became_peered":"2024-03-12T03:33:27.380880+0000","last_unstale":"2024-03-12T03:38:55.575878+0000","last_undegraded":"2024-03-12T03:38:55.575878+0000","last_fullsized":"2024-03-12T03:38:55.575878+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:16.588551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:16.588551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:23.425488+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":118,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.678430+0000","last_change":"2024-03-12T03:38:21.342385+0000","last_active":"2024-03-12T03:38:55.678430+0000","last_peered":"2024-03-12T03:38:55.678430+0000","last_clean":"2024-03-12T03:38:55.678430+0000","last_became_active":"2024-03-12T03:38:18.237347+0000","last_became_peered":"2024-03-12T03:38:18.237347+0000","last_unstale":"2024-03-12T03:38:55.678430+0000","last_undegraded":"2024-03-12T03:38:55.678430+0000","last_fullsized":"2024-03-12T03:38:55.678430+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:38:21.342254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:21.342254+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:22.100123+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578002+0000","last_change":"2024-03-12T03:36:08.094593+0000","last_active":"2024-03-12T03:38:55.578002+0000","last_peered":"2024-03-12T03:38:55.578002+0000","last_clean":"2024-03-12T03:38:55.578002+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:38:55.578002+0000","last_undegraded":"2024-03-12T03:38:55.578002+0000","last_fullsized":"2024-03-12T03:38:55.578002+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:08.094475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:08.094475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796725+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":258,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.680403+0000","last_change":"2024-03-12T03:38:15.383205+0000","last_active":"2024-03-12T03:38:55.680403+0000","last_peered":"2024-03-12T03:38:55.680403+0000","last_clean":"2024-03-12T03:38:55.680403+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:38:55.680403+0000","last_undegraded":"2024-03-12T03:38:55.680403+0000","last_fullsized":"2024-03-12T03:38:55.680403+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:15.382972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:38:15.382972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:19.518813+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.576275+0000","last_change":"2024-03-12T03:38:21.337081+0000","last_active":"2024-03-12T03:38:55.576275+0000","last_peered":"2024-03-12T03:38:55.576275+0000","last_clean":"2024-03-12T03:38:55.576275+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:38:55.576275+0000","last_undegraded":"2024-03-12T03:38:55.576275+0000","last_fullsized":"2024-03-12T03:38:55.576275+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:21.336952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:21.336952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:28.924342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"86'14","reported_seq":316,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575718+0000","last_change":"2024-03-12T03:38:28.333862+0000","last_active":"2024-03-12T03:38:55.575718+0000","last_peered":"2024-03-12T03:38:55.575718+0000","last_clean":"2024-03-12T03:38:55.575718+0000","last_became_active":"2024-03-12T03:38:18.212636+0000","last_became_peered":"2024-03-12T03:38:18.212636+0000","last_unstale":"2024-03-12T03:38:55.575718+0000","last_undegraded":"2024-03-12T03:38:55.575718+0000","last_fullsized":"2024-03-12T03:38:55.575718+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:38:28.333713+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:38:28.333713+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:52.754970+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574823+0000","last_change":"2024-03-12T03:36:18.202204+0000","last_active":"2024-03-12T03:38:55.574823+0000","last_peered":"2024-03-12T03:38:55.574823+0000","last_clean":"2024-03-12T03:38:55.574823+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:38:55.574823+0000","last_undegraded":"2024-03-12T03:38:55.574823+0000","last_fullsized":"2024-03-12T03:38:55.574823+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:18.202082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:18.202082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":243,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578184+0000","last_change":"2024-03-12T03:35:11.909632+0000","last_active":"2024-03-12T03:38:55.578184+0000","last_peered":"2024-03-12T03:38:55.578184+0000","last_clean":"2024-03-12T03:38:55.578184+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:38:55.578184+0000","last_undegraded":"2024-03-12T03:38:55.578184+0000","last_fullsized":"2024-03-12T03:38:55.578184+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:11.909537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:11.909537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.573908+0000","last_change":"2024-03-12T03:38:41.613951+0000","last_active":"2024-03-12T03:38:55.573908+0000","last_peered":"2024-03-12T03:38:55.573908+0000","last_clean":"2024-03-12T03:38:55.573908+0000","last_became_active":"2024-03-12T03:33:27.187943+0000","last_became_peered":"2024-03-12T03:33:27.187943+0000","last_unstale":"2024-03-12T03:38:55.573908+0000","last_undegraded":"2024-03-12T03:38:55.573908+0000","last_fullsized":"2024-03-12T03:38:55.573908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:41.613829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:41.613829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:40:10.967563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"182'10067","reported_seq":20592,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:56.126289+0000","last_change":"2024-03-12T03:38:36.970648+0000","last_active":"2024-03-12T03:38:56.126289+0000","last_peered":"2024-03-12T03:38:56.126289+0000","last_clean":"2024-03-12T03:38:56.126289+0000","last_became_active":"2024-03-12T03:38:36.396720+0000","last_became_peered":"2024-03-12T03:38:36.396720+0000","last_unstale":"2024-03-12T03:38:56.126289+0000","last_undegraded":"2024-03-12T03:38:56.126289+0000","last_fullsized":"2024-03-12T03:38:56.126289+0000","mapping_epoch":171,"log_start":"110'5205","ondisk_log_start":"110'5205","created":119,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'9361","last_scrub_stamp":"2024-03-12T03:38:36.970477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:36.970477+0000","objects_scrubbed":1307,"log_size":4862,"log_dups_size":0,"ondisk_log_size":4862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:48.381065+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39387136,"num_objects":601,"num_object_clones":0,"num_object_copies":1803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":706,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"74'21","reported_seq":295,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575517+0000","last_change":"2024-03-12T03:37:13.158914+0000","last_active":"2024-03-12T03:38:55.575517+0000","last_peered":"2024-03-12T03:38:55.575517+0000","last_clean":"2024-03-12T03:38:55.575517+0000","last_became_active":"2024-03-12T03:33:23.162749+0000","last_became_peered":"2024-03-12T03:33:23.162749+0000","last_unstale":"2024-03-12T03:38:55.575517+0000","last_undegraded":"2024-03-12T03:38:55.575517+0000","last_fullsized":"2024-03-12T03:38:55.575517+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:37:13.158814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:37:13.158814+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.084496+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574536+0000","last_change":"2024-03-12T03:36:21.101794+0000","last_active":"2024-03-12T03:38:55.574536+0000","last_peered":"2024-03-12T03:38:55.574536+0000","last_clean":"2024-03-12T03:38:55.574536+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:38:55.574536+0000","last_undegraded":"2024-03-12T03:38:55.574536+0000","last_fullsized":"2024-03-12T03:38:55.574536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:21.101693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:36:21.101693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574420+0000","last_change":"2024-03-12T03:36:16.187277+0000","last_active":"2024-03-12T03:38:55.574420+0000","last_peered":"2024-03-12T03:38:55.574420+0000","last_clean":"2024-03-12T03:38:55.574420+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:38:55.574420+0000","last_undegraded":"2024-03-12T03:38:55.574420+0000","last_fullsized":"2024-03-12T03:38:55.574420+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:16.187145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:36:16.187145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.577007+0000","last_change":"2024-03-12T03:36:26.107553+0000","last_active":"2024-03-12T03:38:55.577007+0000","last_peered":"2024-03-12T03:38:55.577007+0000","last_clean":"2024-03-12T03:38:55.577007+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:38:55.577007+0000","last_undegraded":"2024-03-12T03:38:55.577007+0000","last_fullsized":"2024-03-12T03:38:55.577007+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:26.107439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:26.107439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.797125+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"182'10119","reported_seq":26985,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.916249+0000","last_change":"2024-03-12T03:38:39.630872+0000","last_active":"2024-03-12T03:38:55.916249+0000","last_peered":"2024-03-12T03:38:55.916249+0000","last_clean":"2024-03-12T03:38:55.916249+0000","last_became_active":"2024-03-12T03:38:26.270640+0000","last_became_peered":"2024-03-12T03:38:26.270640+0000","last_unstale":"2024-03-12T03:38:55.916249+0000","last_undegraded":"2024-03-12T03:38:55.916249+0000","last_fullsized":"2024-03-12T03:38:55.916249+0000","mapping_epoch":167,"log_start":"106'4680","ondisk_log_start":"106'4680","created":119,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":5,"last_scrub":"175'9538","last_scrub_stamp":"2024-03-12T03:38:39.630681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:22.761495+0000","last_clean_scrub_stamp":"2024-03-12T03:38:39.630681+0000","objects_scrubbed":1199,"log_size":5439,"log_dups_size":0,"ondisk_log_size":5439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:59.810065+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":618,"num_object_clones":0,"num_object_copies":1854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":581,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"94'50","reported_seq":361,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575388+0000","last_change":"2024-03-12T03:38:18.274198+0000","last_active":"2024-03-12T03:38:55.575388+0000","last_peered":"2024-03-12T03:38:55.575388+0000","last_clean":"2024-03-12T03:38:55.575388+0000","last_became_active":"2024-03-12T03:38:18.239506+0000","last_became_peered":"2024-03-12T03:38:18.239506+0000","last_unstale":"2024-03-12T03:38:55.575388+0000","last_undegraded":"2024-03-12T03:38:55.575388+0000","last_fullsized":"2024-03-12T03:38:55.575388+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.042030+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":257,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574082+0000","last_change":"2024-03-12T03:38:18.223415+0000","last_active":"2024-03-12T03:38:55.574082+0000","last_peered":"2024-03-12T03:38:55.574082+0000","last_clean":"2024-03-12T03:38:55.574082+0000","last_became_active":"2024-03-12T03:38:18.223071+0000","last_became_peered":"2024-03-12T03:38:18.223071+0000","last_unstale":"2024-03-12T03:38:55.574082+0000","last_undegraded":"2024-03-12T03:38:55.574082+0000","last_fullsized":"2024-03-12T03:38:55.574082+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:34:41.142994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":251,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574225+0000","last_change":"2024-03-12T03:37:18.123502+0000","last_active":"2024-03-12T03:38:55.574225+0000","last_peered":"2024-03-12T03:38:55.574225+0000","last_clean":"2024-03-12T03:38:55.574225+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:38:55.574225+0000","last_undegraded":"2024-03-12T03:38:55.574225+0000","last_fullsized":"2024-03-12T03:38:55.574225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:37:18.123378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.084478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":239,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574822+0000","last_change":"2024-03-12T03:36:19.237362+0000","last_active":"2024-03-12T03:38:55.574822+0000","last_peered":"2024-03-12T03:38:55.574822+0000","last_clean":"2024-03-12T03:38:55.574822+0000","last_became_active":"2024-03-12T03:33:27.382050+0000","last_became_peered":"2024-03-12T03:33:27.382050+0000","last_unstale":"2024-03-12T03:38:55.574822+0000","last_undegraded":"2024-03-12T03:38:55.574822+0000","last_fullsized":"2024-03-12T03:38:55.574822+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.236997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.236997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":283,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.574118+0000","last_change":"2024-03-12T03:38:27.323873+0000","last_active":"2024-03-12T03:38:55.574118+0000","last_peered":"2024-03-12T03:38:55.574118+0000","last_clean":"2024-03-12T03:38:55.574118+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:38:55.574118+0000","last_undegraded":"2024-03-12T03:38:55.574118+0000","last_fullsized":"2024-03-12T03:38:55.574118+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:38:27.323765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:27.323765+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:40.277807+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":265,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.656017+0000","last_change":"2024-03-12T03:38:18.241896+0000","last_active":"2024-03-12T03:38:55.656017+0000","last_peered":"2024-03-12T03:38:55.656017+0000","last_clean":"2024-03-12T03:38:55.656017+0000","last_became_active":"2024-03-12T03:38:18.241308+0000","last_became_peered":"2024-03-12T03:38:18.241308+0000","last_unstale":"2024-03-12T03:38:55.656017+0000","last_undegraded":"2024-03-12T03:38:55.656017+0000","last_fullsized":"2024-03-12T03:38:55.656017+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:28.208044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:28.208044+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.208044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:38:59.796881+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":267,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.578951+0000","last_change":"2024-03-12T03:38:41.747376+0000","last_active":"2024-03-12T03:38:55.578951+0000","last_peered":"2024-03-12T03:38:55.578951+0000","last_clean":"2024-03-12T03:38:55.578951+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:38:55.578951+0000","last_undegraded":"2024-03-12T03:38:55.578951+0000","last_fullsized":"2024-03-12T03:38:55.578951+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:41.747258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:26.625326+0000","last_clean_scrub_stamp":"2024-03-12T03:38:41.747258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:45.155585+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":247,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-12T03:38:55.575332+0000","last_change":"2024-03-12T03:37:12.106676+0000","last_active":"2024-03-12T03:38:55.575332+0000","last_peered":"2024-03-12T03:38:55.575332+0000","last_clean":"2024-03-12T03:38:55.575332+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:38:55.575332+0000","last_undegraded":"2024-03-12T03:38:55.575332+0000","last_fullsized":"2024-03-12T03:38:55.575332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:37:12.106550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:37:12.106550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:39:00.084433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":1321238552,"num_objects":20162,"num_object_clones":0,"num_object_copies":60486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2562,"num_objects_unfound":0,"num_objects_dirty":20162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135230,"num_write_kb":4249249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23230,"num_bytes_recovered":1522139136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5526073344,"data_stored":5526061128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120492,"ondisk_log_size":120492,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":329,"num_read_kb":324,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":985,"num_read_kb":710,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":1596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":631,"ondisk_log_size":631,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070536,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":853396,"kb_used_data":645892,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92891756,"statfs":{"total":95995035648,"available":95121158144,"internally_reserved":0,"allocated":661393408,"data_stored":660848678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":212466122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332173,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":837080,"kb_used_data":647628,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92908072,"statfs":{"total":95995035648,"available":95137865728,"internally_reserved":0,"allocated":663171072,"data_stored":662618134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":193984966},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696500,"kb_used_data":502880,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93048652,"statfs":{"total":95995035648,"available":95281819648,"internally_reserved":0,"allocated":514949120,"data_stored":514375629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899104,"kb_used_data":661072,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92846048,"statfs":{"total":95995035648,"available":95074353152,"internally_reserved":0,"allocated":676937728,"data_stored":676380208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411424,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117083,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1299372,"kb_used_data":1097436,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92445780,"statfs":{"total":95995035648,"available":94664478720,"internally_reserved":0,"allocated":1123774464,"data_stored":1123205436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":206764487},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953560,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":802764,"kb_used_data":685500,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92942388,"statfs":{"total":95995035648,"available":95173005312,"internally_reserved":0,"allocated":701952000,"data_stored":701408026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659218,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":667988,"kb_used_data":514628,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93077164,"statfs":{"total":95995035648,"available":95311015936,"internally_reserved":0,"allocated":526979072,"data_stored":526433704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":661327872,"data_stored":661323800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":518852608,"data_stored":518848536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":719126528,"data_stored":719126528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1106247680,"data_stored":1106247680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":691732480,"data_stored":691732480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":520552448,"data_stored":520552448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":653987840,"data_stored":653983768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":654245888,"data_stored":654245888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:39:03.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-03-12T03:39:03.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:03.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:03.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:03.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:03.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:03.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:03 smithi169 ceph-mon[39949]: pgmap v257: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 945 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 968 B/s rd, 0 B/s wr, 956 op/s; 1595/46023 objects misplaced (3.466%); 4.5 MiB/s, 72 objects/s recovering 2024-03-12T03:39:03.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:03 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:03.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:03 smithi169 ceph-mon[29344]: pgmap v257: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 945 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 968 B/s rd, 0 B/s wr, 956 op/s; 1595/46023 objects misplaced (3.466%); 4.5 MiB/s, 72 objects/s recovering 2024-03-12T03:39:03.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:03 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:03.729 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:03.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:03.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:03.882 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:03.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:03.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:03 smithi173 ceph-mon[33544]: pgmap v257: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 945 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 968 B/s rd, 0 B/s wr, 956 op/s; 1595/46023 objects misplaced (3.466%); 4.5 MiB/s, 72 objects/s recovering 2024-03-12T03:39:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:03 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:04.021 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:04.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:04.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:04.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:04.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:04.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:04.305 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:04.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:04.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:04.434 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:39:04.434 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:39:04.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.434 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:224352 7.d e186/97 7:b887235a:::benchmark_data_smithi169_79822_object106935:head v 186'14073, mlcod=186'14071)", 2024-03-12T03:39:04.434 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:39:04.418530+0000", 2024-03-12T03:39:04.434 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0097354099999999999, 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0097687269999999996, 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.418530+0000", 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.418530+0000", 2024-03-12T03:39:04.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.418531+0000", 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4700000000000001e-07 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.418540+0000", 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9039999999999998e-06 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.418561+0000", 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1118999999999999e-05 2024-03-12T03:39:04.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.418568+0000", 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9129999999999997e-06 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.418734+0000", 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016588400000000001 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.418865+0000", 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013152899999999999 2024-03-12T03:39:04.437 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:224346 7.15 e186/163 7:a9c3d55f:::benchmark_data_smithi169_79822_object106929:head v 186'10417, mlcod=186'10415)", 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:39:04.420273+0000", 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0079925670000000008, 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080963660000000007, 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420273+0000", 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420273+0000", 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420274+0000", 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7400000000000002e-07 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420283+0000", 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0359999999999995e-06 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420293+0000", 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0202e-05 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420300+0000", 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0140000000000001e-06 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420483+0000", 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000182523 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.423130+0000", 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026476189999999999 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:224349 7.5 e186/182 7:a27defe6:::benchmark_data_smithi169_79822_object106932:head v 186'10527, mlcod=186'10525)", 2024-03-12T03:39:04.441 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:39:04.420318+0000", 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0079474960000000001, 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0081131640000000008, 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420318+0000", 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420318+0000", 2024-03-12T03:39:04.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420319+0000", 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3999999999999997e-07 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420323+0000", 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9110000000000003e-06 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420328+0000", 2024-03-12T03:39:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9429999999999999e-06 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.420332+0000", 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3989999999999997e-06 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.424406+0000", 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040736660000000001 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.424566+0000", 2024-03-12T03:39:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000160526 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:224351 7.5 e186/182 7:a1492c88:::benchmark_data_smithi169_79822_object106934:head v 186'10528, mlcod=186'10525)", 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:39:04.423015+0000", 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0052503979999999999, 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054758419999999999, 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.423015+0000", 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.423015+0000", 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.423016+0000", 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9999999999999997e-07 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.423026+0000", 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6520000000000001e-06 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.423035+0000", 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1859999999999998e-06 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.423041+0000", 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1489999999999999e-06 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.427224+0000", 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041829290000000002 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.427244+0000", 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0321000000000001e-05 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:224356 7.d e186/97 7:bcc0f820:::benchmark_data_smithi169_79822_object106939:head v 186'14074, mlcod=186'14071)", 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:39:04.424449+0000", 2024-03-12T03:39:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0038167840000000001, 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041030069999999997, 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.424449+0000", 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.424449+0000", 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.424450+0000", 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4899999999999995e-07 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.424459+0000", 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8580000000000006e-06 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.424467+0000", 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2530000000000004e-06 2024-03-12T03:39:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.424474+0000", 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2269999999999996e-06 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.427247+0000", 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027728259999999999 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.427268+0000", 2024-03-12T03:39:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1588999999999999e-05 2024-03-12T03:39:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:224357 7.d e186/97)", 2024-03-12T03:39:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:39:04.426394+0000", 2024-03-12T03:39:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.001871976, 2024-03-12T03:39:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002214485, 2024-03-12T03:39:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "queued for pg", 2024-03-12T03:39:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426394+0000", 2024-03-12T03:39:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426394+0000", 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426395+0000", 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0999999999999997e-07 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426404+0000", 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1179999999999998e-06 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426417+0000", 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3499999999999999e-05 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426425+0000", 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5830000000000001e-06 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:224358 7.a e186/185 7:5a87a2f0:::benchmark_data_smithi169_79822_object106941:head v 186'14131, mlcod=186'14129)", 2024-03-12T03:39:04.456 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:39:04.426579+0000", 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0016872899999999999, 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002078529, 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426579+0000", 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426579+0000", 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426579+0000", 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5899999999999996e-07 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426585+0000", 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5589999999999996e-06 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426597+0000", 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2452999999999999e-05 2024-03-12T03:39:04.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426604+0000", 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8639999999999998e-06 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426880+0000", 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00027630800000000001 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:39:04.426927+0000", 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6726e-05 2024-03-12T03:39:04.459 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.460 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:04.460 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.460 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:04.460 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:39:04.460 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 7 2024-03-12T03:39:04.460 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:39:04.461 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:04.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:04.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:04.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:04.609 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:04.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:04.695 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:39:04.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:04.733 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:39:04.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:04.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:04.778 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:39:04.823 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:39:04.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:04.865 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:39:04.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:04.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:04.896 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:04 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:39:04.897 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:04 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:39:04.897 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:39:04.933 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:39:04.978 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:39:04.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:04 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:39:04.989 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:39:04.990 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:39:04.990 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:39:04.990 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:39:04.990 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:39:04.990 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:39:04.990 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:39:04.990 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:39:05.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:05.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:05.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:05.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:05.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:05.163 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:05.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:05.264 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:05.303 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:05.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:05.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:05.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:05.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-12T03:39:05.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:05.583 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:05.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:05.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:05 smithi169 ceph-mon[39949]: pgmap v258: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 945 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 817 B/s rd, 0 B/s wr, 807 op/s; 1595/46023 objects misplaced (3.466%); 3.8 MiB/s, 61 objects/s recovering 2024-03-12T03:39:05.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:05 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3497923717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:05 smithi169 ceph-mon[29344]: pgmap v258: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 945 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 817 B/s rd, 0 B/s wr, 807 op/s; 1595/46023 objects misplaced (3.466%); 3.8 MiB/s, 61 objects/s recovering 2024-03-12T03:39:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:05 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3497923717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:05.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:05.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:05.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:05.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:05.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:05.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:05.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:05 smithi173 ceph-mon[33544]: pgmap v258: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 945 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 817 B/s rd, 0 B/s wr, 807 op/s; 1595/46023 objects misplaced (3.466%); 3.8 MiB/s, 61 objects/s recovering 2024-03-12T03:39:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:05 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3497923717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:05.989 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:05.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:06.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:06.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:06.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:06.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:06.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:06.276 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:06.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:06.335 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:06.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:06.397 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:06.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:06.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:06.428 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:06.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:06.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:06.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:06.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:06.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:06.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:06.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:06.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:06.814 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:06.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:06.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:06.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:06.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:06.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:07.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:07.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:07.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:07.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:07.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:07 smithi169 ceph-mon[29344]: pgmap v259: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 810 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 902 op/s; 1144/39549 objects misplaced (2.893%); 3.0 MiB/s, 47 objects/s recovering 2024-03-12T03:39:07.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:07 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:07.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:07 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:07.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:07 smithi169 ceph-mon[39949]: pgmap v259: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 810 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 902 op/s; 1144/39549 objects misplaced (2.893%); 3.0 MiB/s, 47 objects/s recovering 2024-03-12T03:39:07.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:07 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:07.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:07 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:07.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:07.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:07.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:07.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:07.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:07.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:07.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:07.482 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:07.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:07.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:07 smithi173 ceph-mon[33544]: pgmap v259: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 810 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 902 op/s; 1144/39549 objects misplaced (2.893%); 3.0 MiB/s, 47 objects/s recovering 2024-03-12T03:39:07.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:07 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:07.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:07 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:07.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:07.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:07.627 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:07.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:07.728 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:07.767 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:07.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:07.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:07.905 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:07.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:08.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:08.048 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:08.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:08.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:08.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:08 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:08.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:08 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:39:08.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:08 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:08.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:08 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:08.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:08 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:39:08.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:08 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:08.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:08.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:08.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:08.285 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:08.325 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:08.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:08.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:39:08.427 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:08.470 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:08.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:08.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:08.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:08 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:08.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:08 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:39:08.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:08 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:08.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:08.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:08.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:08.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:08.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:08.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:08.755 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:08.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:08.856 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:08.885 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:08.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:08.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:08.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:09.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:09.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:09.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:09.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:09.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:09.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:09.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:09.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:09.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:09.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:09 smithi169 ceph-mon[29344]: pgmap v260: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 516 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 836/25422 objects misplaced (3.288%); 2.9 MiB/s, 46 objects/s recovering 2024-03-12T03:39:09.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:09 smithi169 ceph-mon[39949]: pgmap v260: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 516 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 836/25422 objects misplaced (3.288%); 2.9 MiB/s, 46 objects/s recovering 2024-03-12T03:39:09.434 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:09.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:09 smithi173 ceph-mon[33544]: pgmap v260: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 516 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 836/25422 objects misplaced (3.288%); 2.9 MiB/s, 46 objects/s recovering 2024-03-12T03:39:09.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:09.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:09.578 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:09.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:09.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:09.713 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:09.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:09.815 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:09.855 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:09.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:09.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:09.982 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:09.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:10.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:10.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:10.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:10.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:10.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:10.268 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:10.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:10.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:10.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:10.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:10.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:10.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:10.546 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:10.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:10.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:10.685 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:10.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:10.787 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:10.836 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:10.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:10.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:10.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:10.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:10.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:11.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:11.108 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:11.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:11.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:11.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:11.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:11.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:11.383 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:11.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:11.485 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:11.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:11 smithi173 ceph-mon[33544]: pgmap v261: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 516 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 836/25422 objects misplaced (3.288%); 2.4 MiB/s, 38 objects/s recovering 2024-03-12T03:39:11.492 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:11 smithi169 ceph-mon[29344]: pgmap v261: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 516 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 836/25422 objects misplaced (3.288%); 2.4 MiB/s, 38 objects/s recovering 2024-03-12T03:39:11.492 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:11 smithi169 ceph-mon[39949]: pgmap v261: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 516 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 836/25422 objects misplaced (3.288%); 2.4 MiB/s, 38 objects/s recovering 2024-03-12T03:39:11.574 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:11.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:11.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:11.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:11.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:11.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:11.833 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:11.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:11.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:11.975 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:11.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:12.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:12.092 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:39:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:39:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:39:12.118 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:12.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:12.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:12.219 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:12.260 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:12.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:12.303 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Removed 117414 objects 2024-03-12T03:39:12.304 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Clean up completed and total clean up time :130.7 2024-03-12T03:39:12.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:12.385 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:12.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:12.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:12 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:39:12.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:12 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:39:12.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:12 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/417264914' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:39:12.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:12 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:39:12.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:12 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:39:12.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:12 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:39:12.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:12 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/417264914' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:39:12.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:12 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:39:12.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:12 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:39:12.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:12 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:39:12.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:12 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/417264914' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:39:12.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:12 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:39:12.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:12.511 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:12.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:12.538 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:39:12.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:12.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:12.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:12.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:12.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:12.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:12.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:12.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:12.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:12.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:12.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:12.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:39:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:39:12.992 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:13.015 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:13.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:13.117 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:13.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:13.142 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:13.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:13.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:13.266 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:13.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:13.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:13.368 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:13.391 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:13.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:13.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:13.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:13.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:13.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:13.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:13.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:13.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[29344]: pgmap v262: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 88 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.67k op/s; 119/4887 objects misplaced (2.435%); 2.1 MiB/s, 34 objects/s recovering 2024-03-12T03:39:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:39:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:39:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[29344]: osdmap e187: 8 total, 8 up, 7 in 2024-03-12T03:39:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[29344]: 3.1 scrub starts 2024-03-12T03:39:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[29344]: 3.1 scrub ok 2024-03-12T03:39:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[29344]: 7.14 scrub starts 2024-03-12T03:39:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[29344]: 7.14 scrub ok 2024-03-12T03:39:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[29344]: osdmap e188: 8 total, 8 up, 7 in 2024-03-12T03:39:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[39949]: pgmap v262: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 88 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.67k op/s; 119/4887 objects misplaced (2.435%); 2.1 MiB/s, 34 objects/s recovering 2024-03-12T03:39:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:39:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:39:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[39949]: osdmap e187: 8 total, 8 up, 7 in 2024-03-12T03:39:13.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:13.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[39949]: 3.1 scrub starts 2024-03-12T03:39:13.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[39949]: 3.1 scrub ok 2024-03-12T03:39:13.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[39949]: 7.14 scrub starts 2024-03-12T03:39:13.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[39949]: 7.14 scrub ok 2024-03-12T03:39:13.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:13 smithi169 ceph-mon[39949]: osdmap e188: 8 total, 8 up, 7 in 2024-03-12T03:39:13.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:13 smithi173 ceph-mon[33544]: pgmap v262: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 88 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.67k op/s; 119/4887 objects misplaced (2.435%); 2.1 MiB/s, 34 objects/s recovering 2024-03-12T03:39:13.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:13 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:39:13.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:13 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:39:13.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:13 smithi173 ceph-mon[33544]: osdmap e187: 8 total, 8 up, 7 in 2024-03-12T03:39:13.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:13 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:13.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:13 smithi173 ceph-mon[33544]: 3.1 scrub starts 2024-03-12T03:39:13.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:13 smithi173 ceph-mon[33544]: 3.1 scrub ok 2024-03-12T03:39:13.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:13 smithi173 ceph-mon[33544]: 7.14 scrub starts 2024-03-12T03:39:13.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:13 smithi173 ceph-mon[33544]: 7.14 scrub ok 2024-03-12T03:39:13.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:13 smithi173 ceph-mon[33544]: osdmap e188: 8 total, 8 up, 7 in 2024-03-12T03:39:13.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:13.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:13.773 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:13.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:13.874 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:13.898 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:13.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:14.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:14.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:14.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:14.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:14.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:14.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:14.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:14.215 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:14.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:14.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:14.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:14.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:14.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:14.406 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:14.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:14.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:14 smithi169 ceph-mon[39949]: 7.c scrub starts 2024-03-12T03:39:14.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:14 smithi169 ceph-mon[39949]: 7.c scrub ok 2024-03-12T03:39:14.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:14 smithi169 ceph-mon[39949]: 7.5 scrub starts 2024-03-12T03:39:14.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:14 smithi169 ceph-mon[39949]: 7.5 scrub ok 2024-03-12T03:39:14.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:14 smithi169 ceph-mon[39949]: 7.13 scrub starts 2024-03-12T03:39:14.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:14 smithi169 ceph-mon[39949]: 7.13 scrub ok 2024-03-12T03:39:14.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:14 smithi169 ceph-mon[29344]: 7.c scrub starts 2024-03-12T03:39:14.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:14 smithi169 ceph-mon[29344]: 7.c scrub ok 2024-03-12T03:39:14.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:14 smithi169 ceph-mon[29344]: 7.5 scrub starts 2024-03-12T03:39:14.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:14 smithi169 ceph-mon[29344]: 7.5 scrub ok 2024-03-12T03:39:14.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:14 smithi169 ceph-mon[29344]: 7.13 scrub starts 2024-03-12T03:39:14.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:14 smithi169 ceph-mon[29344]: 7.13 scrub ok 2024-03-12T03:39:14.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:14.531 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:14.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:14.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:14.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:14.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:14.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:14.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:14 smithi173 ceph-mon[33544]: 7.c scrub starts 2024-03-12T03:39:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:14 smithi173 ceph-mon[33544]: 7.c scrub ok 2024-03-12T03:39:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:14 smithi173 ceph-mon[33544]: 7.5 scrub starts 2024-03-12T03:39:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:14 smithi173 ceph-mon[33544]: 7.5 scrub ok 2024-03-12T03:39:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:14 smithi173 ceph-mon[33544]: 7.13 scrub starts 2024-03-12T03:39:14.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:14 smithi173 ceph-mon[33544]: 7.13 scrub ok 2024-03-12T03:39:14.760 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:14.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:14.784 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:14.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:14.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:14.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:14.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:15.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:15.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:15.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:15.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:15.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:15.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:15.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:15.264 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:15.289 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:15.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:15.312 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:15 smithi169 ceph-mon[29344]: pgmap v265: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 88 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.45k op/s; 119/4887 objects misplaced (2.435%) 2024-03-12T03:39:15.312 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:15 smithi169 ceph-mon[29344]: 4.16 scrub starts 2024-03-12T03:39:15.312 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:15 smithi169 ceph-mon[29344]: 4.16 scrub ok 2024-03-12T03:39:15.313 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:15 smithi169 ceph-mon[39949]: pgmap v265: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 88 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.45k op/s; 119/4887 objects misplaced (2.435%) 2024-03-12T03:39:15.313 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:15 smithi169 ceph-mon[39949]: 4.16 scrub starts 2024-03-12T03:39:15.313 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:15 smithi169 ceph-mon[39949]: 4.16 scrub ok 2024-03-12T03:39:15.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:15.416 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:15.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:15.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:15.542 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:15.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:15.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:15.670 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:15.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:15.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:15 smithi173 ceph-mon[33544]: pgmap v265: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 88 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.45k op/s; 119/4887 objects misplaced (2.435%) 2024-03-12T03:39:15.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:15 smithi173 ceph-mon[33544]: 4.16 scrub starts 2024-03-12T03:39:15.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:15 smithi173 ceph-mon[33544]: 4.16 scrub ok 2024-03-12T03:39:15.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:15.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:15.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:15.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:15.923 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:15.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:16.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:16.048 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:16.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:16.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:16.175 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:16.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:16.262 INFO:teuthology.orchestra.run.smithi169.stderr:set pool 1 pgp_num to 11 2024-03-12T03:39:16.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:16.301 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:16.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:16.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:16.326 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:39:16.326 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:39:16.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:16.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:16.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:16.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:16.465 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:16.465 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1464821171' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-12T03:39:16.465 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[29344]: 7.f scrub starts 2024-03-12T03:39:16.466 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[29344]: 7.f scrub ok 2024-03-12T03:39:16.466 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[29344]: 5.13 scrub starts 2024-03-12T03:39:16.466 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[29344]: 5.13 scrub ok 2024-03-12T03:39:16.466 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[29344]: 7.d scrub starts 2024-03-12T03:39:16.466 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[29344]: 7.d scrub ok 2024-03-12T03:39:16.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:16.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1464821171' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-12T03:39:16.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[39949]: 7.f scrub starts 2024-03-12T03:39:16.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[39949]: 7.f scrub ok 2024-03-12T03:39:16.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[39949]: 5.13 scrub starts 2024-03-12T03:39:16.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[39949]: 5.13 scrub ok 2024-03-12T03:39:16.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[39949]: 7.d scrub starts 2024-03-12T03:39:16.467 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:16 smithi169 ceph-mon[39949]: 7.d scrub ok 2024-03-12T03:39:16.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:16.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:16.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:16.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:16.687 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:16.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:16.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:16.734 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:39:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:16 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:16 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1464821171' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-12T03:39:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:16 smithi173 ceph-mon[33544]: 7.f scrub starts 2024-03-12T03:39:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:16 smithi173 ceph-mon[33544]: 7.f scrub ok 2024-03-12T03:39:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:16 smithi173 ceph-mon[33544]: 5.13 scrub starts 2024-03-12T03:39:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:16 smithi173 ceph-mon[33544]: 5.13 scrub ok 2024-03-12T03:39:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:16 smithi173 ceph-mon[33544]: 7.d scrub starts 2024-03-12T03:39:16.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:16 smithi173 ceph-mon[33544]: 7.d scrub ok 2024-03-12T03:39:16.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:16.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:16.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:16.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:16.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:16.944 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:16.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:17.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:17.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:17.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:17.176 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:17.201 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:17.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:17.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:17.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-12T03:39:17.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:17.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:17.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:17.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:17.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:17.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:17.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:17.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:17.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:17.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:17.597 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: pgmap v266: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 57 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 54/3381 objects misplaced (1.597%) 2024-03-12T03:39:17.597 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: 5.4 scrub starts 2024-03-12T03:39:17.597 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: 5.4 scrub ok 2024-03-12T03:39:17.597 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: 7.1 scrub starts 2024-03-12T03:39:17.597 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: 7.1 scrub ok 2024-03-12T03:39:17.597 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1464821171' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-12T03:39:17.597 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: osdmap e189: 8 total, 8 up, 7 in 2024-03-12T03:39:17.597 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: 5.5 scrub starts 2024-03-12T03:39:17.597 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: 5.5 scrub ok 2024-03-12T03:39:17.598 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: 7.0 scrub starts 2024-03-12T03:39:17.598 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: 7.0 scrub ok 2024-03-12T03:39:17.598 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: 3.10 scrub starts 2024-03-12T03:39:17.598 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[29344]: 3.10 scrub ok 2024-03-12T03:39:17.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: pgmap v266: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 57 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 54/3381 objects misplaced (1.597%) 2024-03-12T03:39:17.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: 5.4 scrub starts 2024-03-12T03:39:17.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: 5.4 scrub ok 2024-03-12T03:39:17.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: 7.1 scrub starts 2024-03-12T03:39:17.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: 7.1 scrub ok 2024-03-12T03:39:17.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1464821171' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-12T03:39:17.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: osdmap e189: 8 total, 8 up, 7 in 2024-03-12T03:39:17.599 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: 5.5 scrub starts 2024-03-12T03:39:17.599 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: 5.5 scrub ok 2024-03-12T03:39:17.599 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: 7.0 scrub starts 2024-03-12T03:39:17.599 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: 7.0 scrub ok 2024-03-12T03:39:17.599 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: 3.10 scrub starts 2024-03-12T03:39:17.599 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:17 smithi169 ceph-mon[39949]: 3.10 scrub ok 2024-03-12T03:39:17.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:17.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:17.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: pgmap v266: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 57 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 54/3381 objects misplaced (1.597%) 2024-03-12T03:39:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: 5.4 scrub starts 2024-03-12T03:39:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: 5.4 scrub ok 2024-03-12T03:39:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: 7.1 scrub starts 2024-03-12T03:39:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: 7.1 scrub ok 2024-03-12T03:39:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1464821171' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-12T03:39:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: osdmap e189: 8 total, 8 up, 7 in 2024-03-12T03:39:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: 5.5 scrub starts 2024-03-12T03:39:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: 5.5 scrub ok 2024-03-12T03:39:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: 7.0 scrub starts 2024-03-12T03:39:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: 7.0 scrub ok 2024-03-12T03:39:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: 3.10 scrub starts 2024-03-12T03:39:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:17 smithi173 ceph-mon[33544]: 3.10 scrub ok 2024-03-12T03:39:17.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:17.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:17.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:17.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:17.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:17.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:18.028 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-12T03:39:18.028 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-12T03:39:18.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:18.097 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:18.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:18.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:18.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:18.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:18.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:18.268 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:18 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:39:18.268 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:18 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:39:18.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:18.355 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:18.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:18.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:18.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:18.482 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:18.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:18.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:18.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:18 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:39:18.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:18 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:39:18.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:18 smithi169 ceph-mon[39949]: 4.13 scrub starts 2024-03-12T03:39:18.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:18 smithi169 ceph-mon[39949]: 4.13 scrub ok 2024-03-12T03:39:18.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:18 smithi169 ceph-mon[39949]: osdmap e190: 8 total, 8 up, 7 in 2024-03-12T03:39:18.521 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:18 smithi169 ceph-mon[29344]: 4.13 scrub starts 2024-03-12T03:39:18.521 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:18 smithi169 ceph-mon[29344]: 4.13 scrub ok 2024-03-12T03:39:18.521 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:18 smithi169 ceph-mon[29344]: osdmap e190: 8 total, 8 up, 7 in 2024-03-12T03:39:18.538 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:39:18.560 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:39:18.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:18.585 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:39:18.606 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:39:18.608 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:18.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:18.632 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:39:18.658 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:39:18.668 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:18.670 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:18.673 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:18.683 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:39:18.710 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:18.710 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:39:18.712 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:39:18.713 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:39:18.713 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:39:18.713 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:39:18.713 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:39:18.713 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:39:18.713 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:39:18.713 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:39:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:18 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:39:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:18 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:39:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:18 smithi173 ceph-mon[33544]: 4.13 scrub starts 2024-03-12T03:39:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:18 smithi173 ceph-mon[33544]: 4.13 scrub ok 2024-03-12T03:39:18.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:18 smithi173 ceph-mon[33544]: osdmap e190: 8 total, 8 up, 7 in 2024-03-12T03:39:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:18.738 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:18.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:18.838 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:18.867 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:18.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:18.968 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:18.992 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:18.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:19.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:19.117 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:19.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:19.219 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:19.242 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:19.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:19.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:19.368 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:19.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:19.414 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:19 smithi169 ceph-mon[29344]: 5.11 scrub starts 2024-03-12T03:39:19.414 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:19 smithi169 ceph-mon[29344]: 5.11 scrub ok 2024-03-12T03:39:19.414 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:19 smithi169 ceph-mon[29344]: pgmap v269: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 909 B/s rd, 0 op/s; 35/795 objects misplaced (4.403%) 2024-03-12T03:39:19.414 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:19 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2892330408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:19.414 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:19 smithi169 ceph-mon[29344]: 6.d scrub starts 2024-03-12T03:39:19.415 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:19 smithi169 ceph-mon[29344]: 6.d scrub ok 2024-03-12T03:39:19.415 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:19 smithi169 ceph-mon[39949]: 5.11 scrub starts 2024-03-12T03:39:19.415 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:19 smithi169 ceph-mon[39949]: 5.11 scrub ok 2024-03-12T03:39:19.415 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:19 smithi169 ceph-mon[39949]: pgmap v269: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 909 B/s rd, 0 op/s; 35/795 objects misplaced (4.403%) 2024-03-12T03:39:19.415 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:19 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2892330408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:19.416 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:19 smithi169 ceph-mon[39949]: 6.d scrub starts 2024-03-12T03:39:19.416 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:19 smithi169 ceph-mon[39949]: 6.d scrub ok 2024-03-12T03:39:19.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:19.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:19.495 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:19.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:19.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:19.620 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:19.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:19.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:19 smithi173 ceph-mon[33544]: 5.11 scrub starts 2024-03-12T03:39:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:19 smithi173 ceph-mon[33544]: 5.11 scrub ok 2024-03-12T03:39:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:19 smithi173 ceph-mon[33544]: pgmap v269: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 909 B/s rd, 0 op/s; 35/795 objects misplaced (4.403%) 2024-03-12T03:39:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:19 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2892330408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:19 smithi173 ceph-mon[33544]: 6.d scrub starts 2024-03-12T03:39:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:19 smithi173 ceph-mon[33544]: 6.d scrub ok 2024-03-12T03:39:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:19.745 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:19.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:19.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:19.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:19.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:19.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:20.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:20.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:20.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:20.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:20.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:20.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:20.231 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:20.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:20.256 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:20.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:20.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:20.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:20.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:20.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:20.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:20 smithi169 ceph-mon[29344]: 4.e scrub starts 2024-03-12T03:39:20.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:20 smithi169 ceph-mon[29344]: 4.e scrub ok 2024-03-12T03:39:20.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:20 smithi169 ceph-mon[29344]: 3.17 scrub starts 2024-03-12T03:39:20.482 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:20 smithi169 ceph-mon[29344]: 3.17 scrub ok 2024-03-12T03:39:20.482 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:20 smithi169 ceph-mon[29344]: 7.12 scrub starts 2024-03-12T03:39:20.482 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:20 smithi169 ceph-mon[29344]: 7.12 scrub ok 2024-03-12T03:39:20.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:20 smithi169 ceph-mon[39949]: 4.e scrub starts 2024-03-12T03:39:20.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:20 smithi169 ceph-mon[39949]: 4.e scrub ok 2024-03-12T03:39:20.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:20 smithi169 ceph-mon[39949]: 3.17 scrub starts 2024-03-12T03:39:20.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:20 smithi169 ceph-mon[39949]: 3.17 scrub ok 2024-03-12T03:39:20.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:20 smithi169 ceph-mon[39949]: 7.12 scrub starts 2024-03-12T03:39:20.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:20 smithi169 ceph-mon[39949]: 7.12 scrub ok 2024-03-12T03:39:20.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:20.506 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:20.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:20.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:20.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:20.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:20.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:20.734 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:20 smithi173 ceph-mon[33544]: 4.e scrub starts 2024-03-12T03:39:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:20 smithi173 ceph-mon[33544]: 4.e scrub ok 2024-03-12T03:39:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:20 smithi173 ceph-mon[33544]: 3.17 scrub starts 2024-03-12T03:39:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:20 smithi173 ceph-mon[33544]: 3.17 scrub ok 2024-03-12T03:39:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:20 smithi173 ceph-mon[33544]: 7.12 scrub starts 2024-03-12T03:39:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:20 smithi173 ceph-mon[33544]: 7.12 scrub ok 2024-03-12T03:39:20.758 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:20.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:20.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:20.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:20.888 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:20.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:20.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:21.014 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:21.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:21.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:21.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:21.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:21.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:21.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:21.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:21.272 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:21.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:21.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:21.397 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:21.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:21.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:21.523 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:21.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:21.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:21.625 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:21.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:21.650 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:21.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:21.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-mon[29344]: pgmap v270: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 770 B/s rd, 0 op/s; 35/795 objects misplaced (4.403%) 2024-03-12T03:39:21.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-mon[29344]: 7.15 scrub starts 2024-03-12T03:39:21.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-mon[29344]: 7.15 scrub ok 2024-03-12T03:39:21.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-mon[29344]: osdmap e191: 8 total, 8 up, 7 in 2024-03-12T03:39:21.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-mon[29344]: 5.1d scrub starts 2024-03-12T03:39:21.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-mon[29344]: 5.1d scrub ok 2024-03-12T03:39:21.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-mon[39949]: pgmap v270: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 770 B/s rd, 0 op/s; 35/795 objects misplaced (4.403%) 2024-03-12T03:39:21.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-mon[39949]: 7.15 scrub starts 2024-03-12T03:39:21.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-mon[39949]: 7.15 scrub ok 2024-03-12T03:39:21.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-mon[39949]: osdmap e191: 8 total, 8 up, 7 in 2024-03-12T03:39:21.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-mon[39949]: 5.1d scrub starts 2024-03-12T03:39:21.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-mon[39949]: 5.1d scrub ok 2024-03-12T03:39:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:21 smithi173 ceph-mon[33544]: pgmap v270: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.8 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 770 B/s rd, 0 op/s; 35/795 objects misplaced (4.403%) 2024-03-12T03:39:21.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:21 smithi173 ceph-mon[33544]: 7.15 scrub starts 2024-03-12T03:39:21.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:21 smithi173 ceph-mon[33544]: 7.15 scrub ok 2024-03-12T03:39:21.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:21 smithi173 ceph-mon[33544]: osdmap e191: 8 total, 8 up, 7 in 2024-03-12T03:39:21.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:21 smithi173 ceph-mon[33544]: 5.1d scrub starts 2024-03-12T03:39:21.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:21 smithi173 ceph-mon[33544]: 5.1d scrub ok 2024-03-12T03:39:21.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:21.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:21.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:21.880 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:21.904 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:21.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:22.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:22.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:22.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:22.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:22.135 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:22.159 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:22.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:22.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:22.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:39:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:39:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:39:22.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:22.284 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:22.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:22.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:22.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:22.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:22.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:22.502 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:22 smithi169 ceph-mon[39949]: osdmap e192: 8 total, 8 up, 7 in 2024-03-12T03:39:22.502 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:22 smithi169 ceph-mon[39949]: 7.b scrub starts 2024-03-12T03:39:22.502 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:22 smithi169 ceph-mon[39949]: 7.b scrub ok 2024-03-12T03:39:22.502 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:22 smithi169 ceph-mon[29344]: osdmap e192: 8 total, 8 up, 7 in 2024-03-12T03:39:22.502 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:22 smithi169 ceph-mon[29344]: 7.b scrub starts 2024-03-12T03:39:22.502 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:22 smithi169 ceph-mon[29344]: 7.b scrub ok 2024-03-12T03:39:22.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:22.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:22.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:22.634 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:39:22.634 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:39:22.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:39:22.634 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14156.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:31:06.365908+0000", 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 496.26375192500001, 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.31863010800000002, 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14156", 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/197210445", 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 1 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.365908+0000", 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.365908+0000", 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.365909+0000", 2024-03-12T03:39:22.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.339e-06 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.365916+0000", 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7710000000000001e-06 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.365925+0000", 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.533e-06 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.365936+0000", 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0984e-05 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.366008+0000", 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1540999999999998e-05 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.366018+0000", 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0288999999999999e-05 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.683593+0000", 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.31757436999999999 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.683711+0000", 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000118478 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,2", 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.683845+0000", 2024-03-12T03:39:22.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013388500000000001 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.684219+0000", 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037372300000000001 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.684423+0000", 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020421799999999999 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.684472+0000", 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8965999999999998e-05 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.684507+0000", 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4904999999999999e-05 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:31:06.684538+0000", 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1106000000000003e-05 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.641 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14445.0:2 3.0 3:00000000::::head [pgnls start_epoch 67] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e67)", 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:33:21.103943+0000", 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 361.52571655899999, 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46950572800000001, 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14445", 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/3748937004", 2024-03-12T03:39:22.642 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 2 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103943+0000", 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103943+0000", 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103945+0000", 2024-03-12T03:39:22.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5710000000000001e-06 2024-03-12T03:39:22.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103951+0000", 2024-03-12T03:39:22.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8420000000000001e-06 2024-03-12T03:39:22.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103961+0000", 2024-03-12T03:39:22.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0139000000000001e-05 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103972+0000", 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1579e-05 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.104099+0000", 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000126329 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.104106+0000", 2024-03-12T03:39:22.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7100000000000007e-06 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.573181+0000", 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46907466599999997 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.573220+0000", 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9076999999999999e-05 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.573449+0000", 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000228815 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14436.0:4 3.0 3:00000000::::head [pgnls start_epoch 67] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e67)", 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:33:21.109874+0000", 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 361.51978532200002, 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46447891499999999, 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14436", 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1557333783", 2024-03-12T03:39:22.647 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109874+0000", 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109874+0000", 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109876+0000", 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.694e-06 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109882+0000", 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.716e-06 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109894+0000", 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1758e-05 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109908+0000", 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4017e-05 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109959+0000", 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1094000000000002e-05 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109966+0000", 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5780000000000001e-06 2024-03-12T03:39:22.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.574147+0000", 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46418114100000002 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.574184+0000", 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6148e-05 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.574353+0000", 2024-03-12T03:39:22.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016976900000000001 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:124830 7.12 7:4ef4e82f:::benchmark_data_smithi169_79822_object7413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:10.294730+0000", 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 132.33492961799999, 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15294748399999999, 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.652 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 124830 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.294730+0000", 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.294730+0000", 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.294732+0000", 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.398e-06 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.294736+0000", 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2350000000000001e-06 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.294746+0000", 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8479999999999996e-06 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.294753+0000", 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5510000000000002e-06 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.295064+0000", 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000310802 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.295189+0000", 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000124726 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,2,3,6", 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.295450+0000", 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000261609 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.298665+0000", 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032148260000000001 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.298689+0000", 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3717999999999999e-05 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.299116+0000", 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000427367 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.423150+0000", 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12403383599999999 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.447498+0000", 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024348300999999999 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.447558+0000", 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9973999999999999e-05 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.447678+0000", 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000119293 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.658 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:131596 7.b 7:d6222529:::benchmark_data_smithi169_79822_object14179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:18.417672+0000", 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 124.211987361, 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.164573473, 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 131596 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.659 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.417672+0000", 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.417672+0000", 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.417674+0000", 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5430000000000001e-06 2024-03-12T03:39:22.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.417679+0000", 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.429e-06 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.417696+0000", 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6387000000000001e-05 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.417706+0000", 2024-03-12T03:39:22.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:22.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0129000000000001e-05 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.548605+0000", 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.130899234 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.548715+0000", 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010977899999999999 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.548827+0000", 2024-03-12T03:39:22.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011176 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.555915+0000", 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.007088184 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.555956+0000", 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0850999999999999e-05 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.582079+0000", 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02612364 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.582168+0000", 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8654999999999997e-05 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:18.582246+0000", 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7881999999999997e-05 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.666 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:132618 7.b 7:db626f7b:::benchmark_data_smithi169_79822_object15201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:19.825119+0000", 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 122.804540768, 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15801670100000001, 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.667 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 132618 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.825119+0000", 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.825119+0000", 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.825120+0000", 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.305e-06 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.825126+0000", 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7529999999999998e-06 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.825136+0000", 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9450000000000005e-06 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.825144+0000", 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9410000000000007e-06 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.843043+0000", 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.017898954000000002 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.843171+0000", 2024-03-12T03:39:22.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012786399999999999 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.843295+0000", 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012401900000000001 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.851598+0000", 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083028259999999993 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.851637+0000", 2024-03-12T03:39:22.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9932000000000003e-05 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.983026+0000", 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13138839099999999 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.983076+0000", 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0556000000000002e-05 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:19.983136+0000", 2024-03-12T03:39:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9215000000000001e-05 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:134456 7.b 7:d99b6212:::benchmark_data_smithi169_79822_object17039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:22.033741+0000", 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 120.595918873, 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15225271400000001, 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.673 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 134456 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033741+0000", 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033741+0000", 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033743+0000", 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7799999999999999e-06 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033749+0000", 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1090000000000003e-06 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033766+0000", 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6852999999999999e-05 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033774+0000", 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6999999999999997e-06 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.079783+0000", 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.046008618000000001 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.079900+0000", 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116976 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.080012+0000", 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000111994 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.087870+0000", 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00785836 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.087911+0000", 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0843000000000002e-05 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.185707+0000", 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.097795982000000004 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.185924+0000", 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000217037 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.185994+0000", 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9461999999999993e-05 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.678 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:134457 7.b 7:dacf57cb:::benchmark_data_smithi169_79822_object17040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:22.033790+0000", 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 120.59586995399999, 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15247941300000001, 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 134457 2024-03-12T03:39:22.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033790+0000", 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033790+0000", 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033790+0000", 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2899999999999999e-07 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033793+0000", 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7829999999999999e-06 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033798+0000", 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4700000000000001e-06 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033804+0000", 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2499999999999997e-06 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.082582+0000", 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.048778469999999997 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.082700+0000", 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011765999999999999 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.082899+0000", 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019871799999999999 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.087954+0000", 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050558169999999998 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.087989+0000", 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5126000000000002e-05 2024-03-12T03:39:22.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.186071+0000", 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.098081724999999995 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.186201+0000", 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013020999999999999 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.186269+0000", 2024-03-12T03:39:22.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7755000000000004e-05 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:134459 7.b 7:d204cf26:::benchmark_data_smithi169_79822_object17042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:22.033810+0000", 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 120.59584998699999, 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15260665400000001, 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.685 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 134459 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033810+0000", 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033810+0000", 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033810+0000", 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4999999999999998e-07 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033812+0000", 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0499999999999999e-06 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033817+0000", 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5619999999999997e-06 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033821+0000", 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5419999999999999e-06 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.085199+0000", 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.051377447999999999 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.085300+0000", 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101461 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.085406+0000", 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000106038 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.088026+0000", 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026194970000000001 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.088062+0000", 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6556e-05 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.186279+0000", 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.098216997 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.186364+0000", 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4376000000000007e-05 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.186416+0000", 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2777e-05 2024-03-12T03:39:22.690 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:134460 7.b 7:d6144070:::benchmark_data_smithi169_79822_object17043:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:22.033826+0000", 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 120.595833263, 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15272477200000001, 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 134460 2024-03-12T03:39:22.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033826+0000", 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033826+0000", 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033827+0000", 2024-03-12T03:39:22.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2800000000000003e-07 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033829+0000", 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0229999999999999e-06 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033844+0000", 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5379e-05 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.033849+0000", 2024-03-12T03:39:22.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1059999999999999e-06 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.088094+0000", 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.054244835999999998 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.088215+0000", 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121266 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.088329+0000", 2024-03-12T03:39:22.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011346 2024-03-12T03:39:22.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.695 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.090824+0000", 2024-03-12T03:39:22.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024951280000000001 2024-03-12T03:39:22.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.695 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.090866+0000", 2024-03-12T03:39:22.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2364000000000003e-05 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.186423+0000", 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.095556729000000007 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.186500+0000", 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7353000000000003e-05 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.186551+0000", 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0800000000000002e-05 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:134461 7.b 7:d0c8436b:::benchmark_data_smithi169_79822_object17044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:22.034580+0000", 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 120.595079893, 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15212452000000001, 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.697 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 134461 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.034580+0000", 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.034580+0000", 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.034581+0000", 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2600000000000001e-07 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.034587+0000", 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2040000000000002e-06 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.034600+0000", 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3371e-05 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.034608+0000", 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9340000000000006e-06 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.090895+0000", 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.05628673 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.091002+0000", 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000107308 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.091119+0000", 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116688 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.095005+0000", 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038861619999999999 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.095040+0000", 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4748e-05 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.186558+0000", 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.091518506999999999 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.186652+0000", 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3962000000000006e-05 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:22.186704+0000", 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2080000000000003e-05 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.702 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155212 7.b 7:daf2e0a1:::benchmark_data_smithi169_79822_object37795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.126624+0000", 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 90.503035530999995, 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17515635500000001, 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155212 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.126624+0000", 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.126624+0000", 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.126626+0000", 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9800000000000001e-06 2024-03-12T03:39:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.126632+0000", 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0979999999999999e-06 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.126646+0000", 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3241000000000001e-05 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.126654+0000", 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2460000000000003e-06 2024-03-12T03:39:22.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.150688+0000", 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024034356999999999 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.150850+0000", 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000161872 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.150992+0000", 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014215699999999999 2024-03-12T03:39:22.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.159091+0000", 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080992319999999996 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.159145+0000", 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3517999999999998e-05 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.301618+0000", 2024-03-12T03:39:22.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.142473396 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.301710+0000", 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2077000000000006e-05 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.301781+0000", 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0180999999999998e-05 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155216 7.b 7:d9daa9fb:::benchmark_data_smithi169_79822_object37799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.130863+0000", 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 90.498796720000001, 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17106501099999999, 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155216 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.130863+0000", 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.130863+0000", 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.130864+0000", 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4139999999999999e-06 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.130871+0000", 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.579e-06 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.130888+0000", 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6852999999999999e-05 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.130897+0000", 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.789e-06 2024-03-12T03:39:22.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.155023+0000", 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024126765000000001 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.155138+0000", 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011487799999999999 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.155262+0000", 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012404399999999999 2024-03-12T03:39:22.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.159184+0000", 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0039218320000000001 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.159221+0000", 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6797e-05 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.301791+0000", 2024-03-12T03:39:22.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14257020100000001 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.301875+0000", 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4220000000000003e-05 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.301928+0000", 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2639000000000002e-05 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155217 7.b 7:d76bb7ca:::benchmark_data_smithi169_79822_object37800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.132019+0000", 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 90.497640579000006, 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.170045373, 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155217 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.132019+0000", 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.132019+0000", 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.132020+0000", 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6000000000000002e-07 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.132024+0000", 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8519999999999997e-06 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.132032+0000", 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3259999999999996e-06 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.132039+0000", 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3509999999999998e-06 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.717 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.159258+0000", 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027219779 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.159403+0000", 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014457700000000001 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.159561+0000", 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015768100000000001 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.170580+0000", 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011019849999999999 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.170623+0000", 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3018e-05 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.301937+0000", 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.131313187 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.302014+0000", 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7127999999999997e-05 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.302065+0000", 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0763999999999997e-05 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155309 7.b 7:d08aa399:::benchmark_data_smithi169_79822_object37892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.293060+0000", 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 90.336600204000007, 2024-03-12T03:39:22.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15881757699999999, 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155309 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.293060+0000", 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.721 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.293060+0000", 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.293061+0000", 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.669e-06 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.293069+0000", 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4200000000000001e-06 2024-03-12T03:39:22.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.293082+0000", 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3699999999999999e-05 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.293091+0000", 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9069999999999996e-06 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.297792+0000", 2024-03-12T03:39:22.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0047005880000000003 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.298143+0000", 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000350813 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.298324+0000", 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018139799999999999 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.301597+0000", 2024-03-12T03:39:22.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032729730000000002 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.332828+0000", 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031230843000000001 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.451190+0000", 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.118362516 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.451801+0000", 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00061027200000000003 2024-03-12T03:39:22.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.451877+0000", 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6477999999999995e-05 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155418 7.b 7:d1e27e05:::benchmark_data_smithi169_79822_object38001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.439143+0000", 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 90.190517172, 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16604084799999999, 2024-03-12T03:39:22.726 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155418 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.439143+0000", 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.439143+0000", 2024-03-12T03:39:22.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.439144+0000", 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6479999999999999e-06 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.439151+0000", 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7739999999999999e-06 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.439163+0000", 2024-03-12T03:39:22.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2078e-05 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.439171+0000", 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8800000000000008e-06 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.456558+0000", 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.017386697999999999 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.456672+0000", 2024-03-12T03:39:22.729 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000114751 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.456789+0000", 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116354 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.462092+0000", 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053035319999999997 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.462127+0000", 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5188000000000003e-05 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.605062+0000", 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14293483500000001 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.605118+0000", 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5448000000000002e-05 2024-03-12T03:39:22.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.605183+0000", 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5661999999999996e-05 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155689 7.b 7:d77d3091:::benchmark_data_smithi169_79822_object38272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.729513+0000", 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.900146789999994, 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.199297788, 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.732 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155689 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729513+0000", 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729513+0000", 2024-03-12T03:39:22.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729514+0000", 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4300000000000001e-06 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729519+0000", 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2549999999999999e-06 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729527+0000", 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1410000000000005e-06 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729533+0000", 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0410000000000003e-06 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.790125+0000", 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.060591712999999998 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.790253+0000", 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000128052 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.790394+0000", 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000141812 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.798494+0000", 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080992140000000004 2024-03-12T03:39:22.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.798540+0000", 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6567999999999999e-05 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928691+0000", 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.130150604 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928746+0000", 2024-03-12T03:39:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5340000000000002e-05 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928811+0000", 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4617999999999997e-05 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155695 7.b 7:d612ca62:::benchmark_data_smithi169_79822_object38278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.732453+0000", 2024-03-12T03:39:22.738 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.897206948999994, 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19647717000000001, 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155695 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732453+0000", 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732453+0000", 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732454+0000", 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.671e-06 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732461+0000", 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8179999999999998e-06 2024-03-12T03:39:22.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732477+0000", 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6169999999999999e-05 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732486+0000", 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3790000000000006e-06 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.795096+0000", 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062610618000000007 2024-03-12T03:39:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.795213+0000", 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011686199999999999 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.795338+0000", 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012426200000000001 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.798582+0000", 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032448910000000002 2024-03-12T03:39:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.744 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.744 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.805288+0000", 2024-03-12T03:39:22.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067054109999999997 2024-03-12T03:39:22.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.744 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.744 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928826+0000", 2024-03-12T03:39:22.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12353776499999999 2024-03-12T03:39:22.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928877+0000", 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.147e-05 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928930+0000", 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2852999999999997e-05 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155703 7.b 7:db75e65d:::benchmark_data_smithi169_79822_object38286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:39:22.745 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.751100+0000", 2024-03-12T03:39:22.746 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.878560019000005, 2024-03-12T03:39:22.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17840450999999999, 2024-03-12T03:39:22.746 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.746 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.746 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.746 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.746 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.746 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155703 2024-03-12T03:39:22.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.746 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.751100+0000", 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.751100+0000", 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.751101+0000", 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.637e-06 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.751106+0000", 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1540000000000001e-06 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.751134+0000", 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7906999999999999e-05 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.751145+0000", 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0611999999999999e-05 2024-03-12T03:39:22.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.798617+0000", 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.047471507000000003 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.798746+0000", 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012913399999999999 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.798896+0000", 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015039200000000001 2024-03-12T03:39:22.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.805330+0000", 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0064336200000000001 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.805365+0000", 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5002999999999997e-05 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.929404+0000", 2024-03-12T03:39:22.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.124039444 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.929450+0000", 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5512000000000001e-05 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.929504+0000", 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4588e-05 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155711 7.b 7:dd1554a5:::benchmark_data_smithi169_79822_object38294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.762311+0000", 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.867348700999997, 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.167563198, 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155711 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.762311+0000", 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.762311+0000", 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.762313+0000", 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5829999999999999e-06 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.762318+0000", 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1769999999999997e-06 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.762330+0000", 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2707999999999999e-05 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.762338+0000", 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3200000000000002e-06 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.802202+0000", 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.039863876999999999 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.802337+0000", 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013518800000000001 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.802472+0000", 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013463 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.805400+0000", 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029282050000000001 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.809051+0000", 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003651357 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.929512+0000", 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12046064099999999 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.929553+0000", 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1087000000000002e-05 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.929874+0000", 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00032142500000000002 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:39:22.757 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:39:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:22.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:22.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:22.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:39:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:39:22.759 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:22 smithi173 ceph-mon[33544]: osdmap e192: 8 total, 8 up, 7 in 2024-03-12T03:39:22.759 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:22 smithi173 ceph-mon[33544]: 7.b scrub starts 2024-03-12T03:39:22.759 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:22 smithi173 ceph-mon[33544]: 7.b scrub ok 2024-03-12T03:39:22.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:22.885 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:22.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:22.987 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:23.012 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:23.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:23.029 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:39:23.029 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:39:23.029 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-03-12T03:39:23.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd in 2 2024-03-12T03:39:23.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:23.136 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:23.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:23.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:23.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:23.266 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:23.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:23.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:23.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:23.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:23.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:23.536 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:23.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:23.586 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[29344]: pgmap v273: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 912 B/s rd, 0 op/s 2024-03-12T03:39:23.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[29344]: 5.1f scrub starts 2024-03-12T03:39:23.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[29344]: 5.1f scrub ok 2024-03-12T03:39:23.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[29344]: 3.15 scrub starts 2024-03-12T03:39:23.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[29344]: 3.15 scrub ok 2024-03-12T03:39:23.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[29344]: 3.3 scrub starts 2024-03-12T03:39:23.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[29344]: 3.3 scrub ok 2024-03-12T03:39:23.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[29344]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:39:23.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:23.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[39949]: pgmap v273: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 912 B/s rd, 0 op/s 2024-03-12T03:39:23.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[39949]: 5.1f scrub starts 2024-03-12T03:39:23.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[39949]: 5.1f scrub ok 2024-03-12T03:39:23.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[39949]: 3.15 scrub starts 2024-03-12T03:39:23.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[39949]: 3.15 scrub ok 2024-03-12T03:39:23.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[39949]: 3.3 scrub starts 2024-03-12T03:39:23.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[39949]: 3.3 scrub ok 2024-03-12T03:39:23.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[39949]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:39:23.588 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:23 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:23.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:23.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:23.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:23.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:39:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:23 smithi173 ceph-mon[33544]: pgmap v273: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 912 B/s rd, 0 op/s 2024-03-12T03:39:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:23 smithi173 ceph-mon[33544]: 5.1f scrub starts 2024-03-12T03:39:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:23 smithi173 ceph-mon[33544]: 5.1f scrub ok 2024-03-12T03:39:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:23 smithi173 ceph-mon[33544]: 3.15 scrub starts 2024-03-12T03:39:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:23 smithi173 ceph-mon[33544]: 3.15 scrub ok 2024-03-12T03:39:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:23 smithi173 ceph-mon[33544]: 3.3 scrub starts 2024-03-12T03:39:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:23 smithi173 ceph-mon[33544]: 3.3 scrub ok 2024-03-12T03:39:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:23 smithi173 ceph-mon[33544]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:39:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:23 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:23.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:23.794 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:23.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:23.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:23.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:23.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:24.024 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:24.047 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:24.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:24.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:24.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:24.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:24.174 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:24.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:24.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:24.299 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:24.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:24.302 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:39:24.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:24.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:24.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:24.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:24.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:24.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:24.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:24.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:24.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:24 smithi169 ceph-mon[29344]: 6.8 scrub starts 2024-03-12T03:39:24.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:24 smithi169 ceph-mon[29344]: 6.8 scrub ok 2024-03-12T03:39:24.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:24 smithi169 ceph-mon[29344]: 5.3 scrub starts 2024-03-12T03:39:24.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:24 smithi169 ceph-mon[29344]: 5.3 scrub ok 2024-03-12T03:39:24.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:24 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/385668439' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:39:24.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:24 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:39:24.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:24 smithi169 ceph-mon[39949]: 6.8 scrub starts 2024-03-12T03:39:24.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:24 smithi169 ceph-mon[39949]: 6.8 scrub ok 2024-03-12T03:39:24.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:24 smithi169 ceph-mon[39949]: 5.3 scrub starts 2024-03-12T03:39:24.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:24 smithi169 ceph-mon[39949]: 5.3 scrub ok 2024-03-12T03:39:24.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:24 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/385668439' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:39:24.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:24 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:39:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:24.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:24.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-12T03:39:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:24 smithi173 ceph-mon[33544]: 6.8 scrub starts 2024-03-12T03:39:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:24 smithi173 ceph-mon[33544]: 6.8 scrub ok 2024-03-12T03:39:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:24 smithi173 ceph-mon[33544]: 5.3 scrub starts 2024-03-12T03:39:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:24 smithi173 ceph-mon[33544]: 5.3 scrub ok 2024-03-12T03:39:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:24 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/385668439' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:39:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:24 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:39:24.765 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:24.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:24.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:24.813 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:24.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:24.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:24.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:24.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:25.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:25.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:25.086 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:25.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:25.187 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:25.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:25.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:25.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:25.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[29344]: pgmap v274: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 775 B/s rd, 0 op/s 2024-03-12T03:39:25.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[29344]: 5.2 scrub starts 2024-03-12T03:39:25.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[29344]: 5.2 scrub ok 2024-03-12T03:39:25.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:39:25.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:25.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:25.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:25.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:25.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:25.465 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:25.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:25.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:39:25.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[29344]: osdmap e193: 8 total, 8 up, 7 in 2024-03-12T03:39:25.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[29344]: 5.b scrub starts 2024-03-12T03:39:25.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[29344]: 5.b scrub ok 2024-03-12T03:39:25.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[29344]: 1.3 scrub starts 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[29344]: 1.3 scrub ok 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[29344]: 7.e scrub starts 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[29344]: 7.e scrub ok 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[39949]: pgmap v274: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 775 B/s rd, 0 op/s 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[39949]: 5.2 scrub starts 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[39949]: 5.2 scrub ok 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[39949]: osdmap e193: 8 total, 8 up, 7 in 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[39949]: 5.b scrub starts 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[39949]: 5.b scrub ok 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[39949]: 1.3 scrub starts 2024-03-12T03:39:25.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[39949]: 1.3 scrub ok 2024-03-12T03:39:25.553 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[39949]: 7.e scrub starts 2024-03-12T03:39:25.553 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:25 smithi169 ceph-mon[39949]: 7.e scrub ok 2024-03-12T03:39:25.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:25.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:25.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:25.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:25.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:25.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:25 smithi173 ceph-mon[33544]: pgmap v274: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 775 B/s rd, 0 op/s 2024-03-12T03:39:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:25 smithi173 ceph-mon[33544]: 5.2 scrub starts 2024-03-12T03:39:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:25 smithi173 ceph-mon[33544]: 5.2 scrub ok 2024-03-12T03:39:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:25 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:39:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:25 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:39:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:25 smithi173 ceph-mon[33544]: osdmap e193: 8 total, 8 up, 7 in 2024-03-12T03:39:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:25 smithi173 ceph-mon[33544]: 5.b scrub starts 2024-03-12T03:39:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:25 smithi173 ceph-mon[33544]: 5.b scrub ok 2024-03-12T03:39:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:25 smithi173 ceph-mon[33544]: 1.3 scrub starts 2024-03-12T03:39:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:25 smithi173 ceph-mon[33544]: 1.3 scrub ok 2024-03-12T03:39:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:25 smithi173 ceph-mon[33544]: 7.e scrub starts 2024-03-12T03:39:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:25 smithi173 ceph-mon[33544]: 7.e scrub ok 2024-03-12T03:39:25.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:25.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:25.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:25.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:25.968 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:25.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:25.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:26.070 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:26.096 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:26.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:26.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:26.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:26.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:26.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:26.326 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'unique_pool_0' removed 2024-03-12T03:39:26.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:26.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:26.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:26.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:26.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:26.569 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[29344]: 3.a scrub starts 2024-03-12T03:39:26.569 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[29344]: 3.a scrub ok 2024-03-12T03:39:26.569 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[29344]: osdmap e194: 8 total, 8 up, 7 in 2024-03-12T03:39:26.569 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1150101884' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:39:26.569 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:39:26.570 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[29344]: 4.1c scrub starts 2024-03-12T03:39:26.570 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[29344]: 4.1c scrub ok 2024-03-12T03:39:26.570 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[39949]: 3.a scrub starts 2024-03-12T03:39:26.570 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[39949]: 3.a scrub ok 2024-03-12T03:39:26.570 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[39949]: osdmap e194: 8 total, 8 up, 7 in 2024-03-12T03:39:26.570 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1150101884' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:39:26.570 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:39:26.570 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[39949]: 4.1c scrub starts 2024-03-12T03:39:26.570 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:26 smithi169 ceph-mon[39949]: 4.1c scrub ok 2024-03-12T03:39:26.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:26.599 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:26.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:26.701 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:26.724 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:26.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:26.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:26 smithi173 ceph-mon[33544]: 3.a scrub starts 2024-03-12T03:39:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:26 smithi173 ceph-mon[33544]: 3.a scrub ok 2024-03-12T03:39:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:26 smithi173 ceph-mon[33544]: osdmap e194: 8 total, 8 up, 7 in 2024-03-12T03:39:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:26 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1150101884' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:39:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:26 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:39:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:26 smithi173 ceph-mon[33544]: 4.1c scrub starts 2024-03-12T03:39:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:26 smithi173 ceph-mon[33544]: 4.1c scrub ok 2024-03-12T03:39:26.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:26.850 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:26.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:26.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:26.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:26.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:27.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:27.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:27.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:27.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:27.229 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:27.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:27.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:27.240 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:27.331 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:27.355 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:27.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:27.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:27.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:27.480 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:27.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:27.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:27.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:27.601 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:27 smithi169 ceph-mon[29344]: pgmap v277: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-12T03:39:27.601 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:27 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:39:27.601 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:27 smithi169 ceph-mon[29344]: osdmap e195: 8 total, 8 up, 7 in 2024-03-12T03:39:27.601 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:27 smithi169 ceph-mon[29344]: 3.1e scrub starts 2024-03-12T03:39:27.601 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:27 smithi169 ceph-mon[29344]: 3.1e scrub ok 2024-03-12T03:39:27.602 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:27 smithi169 ceph-mon[39949]: pgmap v277: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-12T03:39:27.602 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:27 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:39:27.602 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:27 smithi169 ceph-mon[39949]: osdmap e195: 8 total, 8 up, 7 in 2024-03-12T03:39:27.602 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:27 smithi169 ceph-mon[39949]: 3.1e scrub starts 2024-03-12T03:39:27.602 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:27 smithi169 ceph-mon[39949]: 3.1e scrub ok 2024-03-12T03:39:27.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:27.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:27.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:27.624 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-12T03:39:27.624 INFO:tasks.radosbench:Beginning radosbench... 2024-03-12T03:39:27.625 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-12T03:39:27.625 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-12T03:39:27.707 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:27.730 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:27.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:27 smithi173 ceph-mon[33544]: pgmap v277: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-12T03:39:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:27 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:39:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:27 smithi173 ceph-mon[33544]: osdmap e195: 8 total, 8 up, 7 in 2024-03-12T03:39:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:27 smithi173 ceph-mon[33544]: 3.1e scrub starts 2024-03-12T03:39:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:27 smithi173 ceph-mon[33544]: 3.1e scrub ok 2024-03-12T03:39:27.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:27.855 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:27.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:27.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:27.940 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:27.957 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:27.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:27.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:27.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:28.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:28.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:28.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:28.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:28.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:28.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:28.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:28.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:28.358 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:28.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:28.436 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:28 smithi169 ceph-mon[29344]: 4.a deep-scrub starts 2024-03-12T03:39:28.437 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:28 smithi169 ceph-mon[29344]: 4.a deep-scrub ok 2024-03-12T03:39:28.437 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:28 smithi169 ceph-mon[29344]: 6.17 scrub starts 2024-03-12T03:39:28.437 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:28 smithi169 ceph-mon[29344]: 6.17 scrub ok 2024-03-12T03:39:28.437 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:28 smithi169 ceph-mon[29344]: osdmap e196: 8 total, 8 up, 7 in 2024-03-12T03:39:28.437 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:28 smithi169 ceph-mon[39949]: 4.a deep-scrub starts 2024-03-12T03:39:28.437 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:28 smithi169 ceph-mon[39949]: 4.a deep-scrub ok 2024-03-12T03:39:28.437 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:28 smithi169 ceph-mon[39949]: 6.17 scrub starts 2024-03-12T03:39:28.437 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:28 smithi169 ceph-mon[39949]: 6.17 scrub ok 2024-03-12T03:39:28.437 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:28 smithi169 ceph-mon[39949]: osdmap e196: 8 total, 8 up, 7 in 2024-03-12T03:39:28.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:28.485 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:28.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:28.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:28.597 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:39:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:28.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:28.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:28.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:28 smithi173 ceph-mon[33544]: 4.a deep-scrub starts 2024-03-12T03:39:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:28 smithi173 ceph-mon[33544]: 4.a deep-scrub ok 2024-03-12T03:39:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:28 smithi173 ceph-mon[33544]: 6.17 scrub starts 2024-03-12T03:39:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:28 smithi173 ceph-mon[33544]: 6.17 scrub ok 2024-03-12T03:39:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:28 smithi173 ceph-mon[33544]: osdmap e196: 8 total, 8 up, 7 in 2024-03-12T03:39:28.738 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:28.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:28.834 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:39:28.834 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:39:28.834 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:39:28.834 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:39:28.834 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:39:28.834 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:39:28.834 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:39:28.834 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:39:28.835 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:39:28.835 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:39:28.835 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:39:28.835 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:39:28.835 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:39:28.835 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:39:28.835 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:39:28.835 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:39:28.840 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:28.864 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:28.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:28.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:28.988 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:28.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:29.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:29.114 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:29.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:29.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:29.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:29.239 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:29.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:29.341 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:29.364 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:29.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:29.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:29.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:29.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:29.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:29.523 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[39949]: pgmap v279: 142 pgs: 142 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:39:29.523 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[39949]: 5.9 scrub starts 2024-03-12T03:39:29.523 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[39949]: 5.9 scrub ok 2024-03-12T03:39:29.523 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[39949]: 6.2 scrub starts 2024-03-12T03:39:29.523 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[39949]: 6.2 scrub ok 2024-03-12T03:39:29.523 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[39949]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-12T03:39:29.523 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[39949]: Cluster is now healthy 2024-03-12T03:39:29.523 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1961034355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:29.523 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[39949]: 3.1b scrub starts 2024-03-12T03:39:29.523 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[39949]: 3.1b scrub ok 2024-03-12T03:39:29.524 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[29344]: pgmap v279: 142 pgs: 142 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:39:29.524 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[29344]: 5.9 scrub starts 2024-03-12T03:39:29.524 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[29344]: 5.9 scrub ok 2024-03-12T03:39:29.524 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[29344]: 6.2 scrub starts 2024-03-12T03:39:29.524 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[29344]: 6.2 scrub ok 2024-03-12T03:39:29.524 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[29344]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-12T03:39:29.524 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[29344]: Cluster is now healthy 2024-03-12T03:39:29.524 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1961034355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:29.524 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[29344]: 3.1b scrub starts 2024-03-12T03:39:29.524 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:29 smithi169 ceph-mon[29344]: 3.1b scrub ok 2024-03-12T03:39:29.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:29.616 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:29.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:29.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:29.637 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:29 smithi173 ceph-mon[33544]: pgmap v279: 142 pgs: 142 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:39:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:29 smithi173 ceph-mon[33544]: 5.9 scrub starts 2024-03-12T03:39:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:29 smithi173 ceph-mon[33544]: 5.9 scrub ok 2024-03-12T03:39:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:29 smithi173 ceph-mon[33544]: 6.2 scrub starts 2024-03-12T03:39:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:29 smithi173 ceph-mon[33544]: 6.2 scrub ok 2024-03-12T03:39:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:29 smithi173 ceph-mon[33544]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-12T03:39:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:29 smithi173 ceph-mon[33544]: Cluster is now healthy 2024-03-12T03:39:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:29 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1961034355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:29 smithi173 ceph-mon[33544]: 3.1b scrub starts 2024-03-12T03:39:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:29 smithi173 ceph-mon[33544]: 3.1b scrub ok 2024-03-12T03:39:29.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:29.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:29.742 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:29.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:29.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:29.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:29.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:29.947 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:29.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:29.993 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:29.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:30.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:30.119 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:30.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:30.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:30.244 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:30.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:30.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:30.370 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:30.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:30.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:30.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:30.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:30.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:30.622 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:30.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:30.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:30 smithi169 ceph-mon[29344]: 5.10 scrub starts 2024-03-12T03:39:30.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:30 smithi169 ceph-mon[29344]: 5.10 scrub ok 2024-03-12T03:39:30.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:30 smithi169 ceph-mon[29344]: 4.5 scrub starts 2024-03-12T03:39:30.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:30 smithi169 ceph-mon[29344]: 4.5 scrub ok 2024-03-12T03:39:30.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:30 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:30.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:30 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:30.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:30 smithi169 ceph-mon[39949]: 5.10 scrub starts 2024-03-12T03:39:30.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:30 smithi169 ceph-mon[39949]: 5.10 scrub ok 2024-03-12T03:39:30.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:30 smithi169 ceph-mon[39949]: 4.5 scrub starts 2024-03-12T03:39:30.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:30 smithi169 ceph-mon[39949]: 4.5 scrub ok 2024-03-12T03:39:30.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:30 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:30.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:30 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:30.724 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:30.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:30 smithi173 ceph-mon[33544]: 5.10 scrub starts 2024-03-12T03:39:30.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:30 smithi173 ceph-mon[33544]: 5.10 scrub ok 2024-03-12T03:39:30.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:30 smithi173 ceph-mon[33544]: 4.5 scrub starts 2024-03-12T03:39:30.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:30 smithi173 ceph-mon[33544]: 4.5 scrub ok 2024-03-12T03:39:30.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:30 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:30.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:30 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:30.747 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:30.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:30.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:30.873 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:30.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:30.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:30.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:31.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:31.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:31.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:31.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:31.132 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:31.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:31.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:31.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:31.259 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:31.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:31.360 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:31.384 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:31.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:31.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:31.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:31.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:31.549 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:39:31.574 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:31 smithi169 ceph-mon[39949]: pgmap v281: 142 pgs: 142 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 734 B/s rd, 0 op/s 2024-03-12T03:39:31.574 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:31 smithi169 ceph-mon[29344]: pgmap v281: 142 pgs: 142 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 734 B/s rd, 0 op/s 2024-03-12T03:39:31.616 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:31.640 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:31.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:31.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:31 smithi173 ceph-mon[33544]: pgmap v281: 142 pgs: 142 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 734 B/s rd, 0 op/s 2024-03-12T03:39:31.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:31.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:31.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:31.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:31.836 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a[29340]: 2024-03-12T03:39:31.762+0000 7f7050ca9700 -1 mon.a@0(leader).osd e196 definitely_dead 0 2024-03-12T03:39:31.837 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:39:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:39:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:39:31.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:31.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:31.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:31.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:32.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:32.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:32.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:32.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:32.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:32.154 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:32.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:32.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:32.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:32.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:32.334 INFO:teuthology.orchestra.run.smithi169.stderr:marked in osd.2. 2024-03-12T03:39:32.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:32.408 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:32.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:32.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:32.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:32.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:32.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:32.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:32.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:32 smithi169 ceph-mon[29344]: 4.2 scrub starts 2024-03-12T03:39:32.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:32 smithi169 ceph-mon[29344]: 4.2 scrub ok 2024-03-12T03:39:32.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:32 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4116469773' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-12T03:39:32.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:32 smithi169 ceph-mon[39949]: 4.2 scrub starts 2024-03-12T03:39:32.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:32 smithi169 ceph-mon[39949]: 4.2 scrub ok 2024-03-12T03:39:32.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:32 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4116469773' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-12T03:39:32.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:32.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:32.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:32 smithi173 ceph-mon[33544]: 4.2 scrub starts 2024-03-12T03:39:32.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:32 smithi173 ceph-mon[33544]: 4.2 scrub ok 2024-03-12T03:39:32.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:32 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4116469773' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-12T03:39:32.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:39:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:39:32.766 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:32.789 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:32.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:32.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:32.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:32.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:32.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:32.967 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-03-12T03:39:33.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:33.043 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:33.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:33.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:33.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:33.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:33.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:33.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:33.242 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'unique_pool_1' created 2024-03-12T03:39:33.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:33.310 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:33.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:33.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:33.412 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:33.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:33.437 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:33.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:33.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:33.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:33.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:33.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[29344]: 6.4 scrub starts 2024-03-12T03:39:33.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[29344]: 6.4 scrub ok 2024-03-12T03:39:33.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[29344]: pgmap v282: 142 pgs: 142 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-12T03:39:33.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4116469773' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-12T03:39:33.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[29344]: osdmap e197: 8 total, 8 up, 8 in 2024-03-12T03:39:33.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:33.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/611820857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-12T03:39:33.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/611820857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-12T03:39:33.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[29344]: osdmap e198: 8 total, 8 up, 8 in 2024-03-12T03:39:33.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[39949]: 6.4 scrub starts 2024-03-12T03:39:33.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[39949]: 6.4 scrub ok 2024-03-12T03:39:33.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[39949]: pgmap v282: 142 pgs: 142 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-12T03:39:33.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4116469773' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-12T03:39:33.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[39949]: osdmap e197: 8 total, 8 up, 8 in 2024-03-12T03:39:33.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:33.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/611820857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-12T03:39:33.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/611820857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-12T03:39:33.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:33 smithi169 ceph-mon[39949]: osdmap e198: 8 total, 8 up, 8 in 2024-03-12T03:39:33.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:33.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:33.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:33.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:33 smithi173 ceph-mon[33544]: 6.4 scrub starts 2024-03-12T03:39:33.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:33 smithi173 ceph-mon[33544]: 6.4 scrub ok 2024-03-12T03:39:33.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:33 smithi173 ceph-mon[33544]: pgmap v282: 142 pgs: 142 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-12T03:39:33.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:33 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4116469773' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-12T03:39:33.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:33 smithi173 ceph-mon[33544]: osdmap e197: 8 total, 8 up, 8 in 2024-03-12T03:39:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:33 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:33 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/611820857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-12T03:39:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:33 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/611820857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-12T03:39:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:33 smithi173 ceph-mon[33544]: osdmap e198: 8 total, 8 up, 8 in 2024-03-12T03:39:33.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:33.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:33.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:33.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:33.940 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:33.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:34.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:34.046 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-12T03:39:34.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:34.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:34.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:34.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:34.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:34.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:34.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:34.318 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:34.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:34.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:34.445 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:34.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:34.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:34.572 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:34.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:34.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[29344]: 4.1f scrub starts 2024-03-12T03:39:34.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[29344]: 4.1f scrub ok 2024-03-12T03:39:34.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[29344]: 3.11 scrub starts 2024-03-12T03:39:34.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[29344]: 3.11 scrub ok 2024-03-12T03:39:34.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[29344]: 3.16 deep-scrub starts 2024-03-12T03:39:34.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[29344]: 3.16 deep-scrub ok 2024-03-12T03:39:34.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[29344]: 6.f scrub starts 2024-03-12T03:39:34.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[29344]: 6.f scrub ok 2024-03-12T03:39:34.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[29344]: osdmap e199: 8 total, 8 up, 8 in 2024-03-12T03:39:34.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[39949]: 4.1f scrub starts 2024-03-12T03:39:34.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[39949]: 4.1f scrub ok 2024-03-12T03:39:34.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[39949]: 3.11 scrub starts 2024-03-12T03:39:34.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[39949]: 3.11 scrub ok 2024-03-12T03:39:34.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[39949]: 3.16 deep-scrub starts 2024-03-12T03:39:34.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[39949]: 3.16 deep-scrub ok 2024-03-12T03:39:34.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[39949]: 6.f scrub starts 2024-03-12T03:39:34.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[39949]: 6.f scrub ok 2024-03-12T03:39:34.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:34 smithi169 ceph-mon[39949]: osdmap e199: 8 total, 8 up, 8 in 2024-03-12T03:39:34.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:34.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:34.699 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:34.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:34 smithi173 ceph-mon[33544]: 4.1f scrub starts 2024-03-12T03:39:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:34 smithi173 ceph-mon[33544]: 4.1f scrub ok 2024-03-12T03:39:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:34 smithi173 ceph-mon[33544]: 3.11 scrub starts 2024-03-12T03:39:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:34 smithi173 ceph-mon[33544]: 3.11 scrub ok 2024-03-12T03:39:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:34 smithi173 ceph-mon[33544]: 3.16 deep-scrub starts 2024-03-12T03:39:34.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:34 smithi173 ceph-mon[33544]: 3.16 deep-scrub ok 2024-03-12T03:39:34.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:34 smithi173 ceph-mon[33544]: 6.f scrub starts 2024-03-12T03:39:34.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:34 smithi173 ceph-mon[33544]: 6.f scrub ok 2024-03-12T03:39:34.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:34 smithi173 ceph-mon[33544]: osdmap e199: 8 total, 8 up, 8 in 2024-03-12T03:39:34.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:34.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:34.827 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:34.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:34.928 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:34.952 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:34.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:35.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:35.058 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:35.059 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:39:35.060 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:39:35.079 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:35.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:35.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:35.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:35.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:35.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:35.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:35.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:35.323 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:35.335 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:35.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:35.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:35.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:35.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:35.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:35.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:35.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:35.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:35.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:35.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:35.626 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[29344]: 6.1d scrub starts 2024-03-12T03:39:35.626 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[29344]: 6.1d scrub ok 2024-03-12T03:39:35.626 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[29344]: pgmap v285: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:35.626 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[29344]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:39:35.626 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[29344]: 3.5 scrub starts 2024-03-12T03:39:35.626 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[29344]: 3.5 scrub ok 2024-03-12T03:39:35.626 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[29344]: 5.6 scrub starts 2024-03-12T03:39:35.626 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[29344]: 5.6 scrub ok 2024-03-12T03:39:35.627 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[29344]: osdmap e200: 8 total, 8 up, 8 in 2024-03-12T03:39:35.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[39949]: 6.1d scrub starts 2024-03-12T03:39:35.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[39949]: 6.1d scrub ok 2024-03-12T03:39:35.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[39949]: pgmap v285: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:35.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[39949]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:39:35.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[39949]: 3.5 scrub starts 2024-03-12T03:39:35.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[39949]: 3.5 scrub ok 2024-03-12T03:39:35.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[39949]: 5.6 scrub starts 2024-03-12T03:39:35.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[39949]: 5.6 scrub ok 2024-03-12T03:39:35.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:35 smithi169 ceph-mon[39949]: osdmap e200: 8 total, 8 up, 8 in 2024-03-12T03:39:35.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:35.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:35.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:35 smithi173 ceph-mon[33544]: 6.1d scrub starts 2024-03-12T03:39:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:35 smithi173 ceph-mon[33544]: 6.1d scrub ok 2024-03-12T03:39:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:35 smithi173 ceph-mon[33544]: pgmap v285: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:35 smithi173 ceph-mon[33544]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:39:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:35 smithi173 ceph-mon[33544]: 3.5 scrub starts 2024-03-12T03:39:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:35 smithi173 ceph-mon[33544]: 3.5 scrub ok 2024-03-12T03:39:35.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:35 smithi173 ceph-mon[33544]: 5.6 scrub starts 2024-03-12T03:39:35.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:35 smithi173 ceph-mon[33544]: 5.6 scrub ok 2024-03-12T03:39:35.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:35 smithi173 ceph-mon[33544]: osdmap e200: 8 total, 8 up, 8 in 2024-03-12T03:39:35.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:35.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:35.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:35.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:35.970 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:35.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:36.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:36.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:36.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:36.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:36.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:36.226 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:36.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:36.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:36.352 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:36.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:36.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:36.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:36.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:36.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:36.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:36.513 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[29344]: 5.1a deep-scrub starts 2024-03-12T03:39:36.513 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[29344]: 5.1a deep-scrub ok 2024-03-12T03:39:36.514 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[29344]: 5.c deep-scrub starts 2024-03-12T03:39:36.514 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[29344]: 5.c deep-scrub ok 2024-03-12T03:39:36.514 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[29344]: 4.6 scrub starts 2024-03-12T03:39:36.514 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[29344]: 4.6 scrub ok 2024-03-12T03:39:36.514 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[29344]: osdmap e201: 8 total, 8 up, 8 in 2024-03-12T03:39:36.514 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[39949]: 5.1a deep-scrub starts 2024-03-12T03:39:36.514 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[39949]: 5.1a deep-scrub ok 2024-03-12T03:39:36.514 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[39949]: 5.c deep-scrub starts 2024-03-12T03:39:36.514 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[39949]: 5.c deep-scrub ok 2024-03-12T03:39:36.514 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[39949]: 4.6 scrub starts 2024-03-12T03:39:36.514 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[39949]: 4.6 scrub ok 2024-03-12T03:39:36.515 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:36 smithi169 ceph-mon[39949]: osdmap e201: 8 total, 8 up, 8 in 2024-03-12T03:39:36.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:39:36.605 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:36.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:36.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:39:36.707 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:36.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:36.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:36 smithi173 ceph-mon[33544]: 5.1a deep-scrub starts 2024-03-12T03:39:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:36 smithi173 ceph-mon[33544]: 5.1a deep-scrub ok 2024-03-12T03:39:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:36 smithi173 ceph-mon[33544]: 5.c deep-scrub starts 2024-03-12T03:39:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:36 smithi173 ceph-mon[33544]: 5.c deep-scrub ok 2024-03-12T03:39:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:36 smithi173 ceph-mon[33544]: 4.6 scrub starts 2024-03-12T03:39:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:36 smithi173 ceph-mon[33544]: 4.6 scrub ok 2024-03-12T03:39:36.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:36 smithi173 ceph-mon[33544]: osdmap e201: 8 total, 8 up, 8 in 2024-03-12T03:39:36.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:36.861 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:36.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:36.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:36.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:36.988 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:36.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:37.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:37.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:37.115 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:37.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:37.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:37.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:37.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:37.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:37.369 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:37.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:37.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:37.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:37.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:37.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:37.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:37.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:37.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:37.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: 3.f scrub starts 2024-03-12T03:39:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: 3.f scrub ok 2024-03-12T03:39:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: 5.12 scrub starts 2024-03-12T03:39:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: 5.12 scrub ok 2024-03-12T03:39:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: pgmap v288: 158 pgs: 5 peering, 12 unknown, 141 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 65 KiB/s, 5 objects/s recovering 2024-03-12T03:39:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: 4.1 deep-scrub starts 2024-03-12T03:39:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: 4.1 deep-scrub ok 2024-03-12T03:39:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: 4.3 scrub starts 2024-03-12T03:39:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: 4.3 scrub ok 2024-03-12T03:39:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:39:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:39:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: 5.0 scrub starts 2024-03-12T03:39:37.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[29344]: 5.0 scrub ok 2024-03-12T03:39:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: 3.f scrub starts 2024-03-12T03:39:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: 3.f scrub ok 2024-03-12T03:39:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: 5.12 scrub starts 2024-03-12T03:39:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: 5.12 scrub ok 2024-03-12T03:39:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: pgmap v288: 158 pgs: 5 peering, 12 unknown, 141 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 65 KiB/s, 5 objects/s recovering 2024-03-12T03:39:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: 4.1 deep-scrub starts 2024-03-12T03:39:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: 4.1 deep-scrub ok 2024-03-12T03:39:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: 4.3 scrub starts 2024-03-12T03:39:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: 4.3 scrub ok 2024-03-12T03:39:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:39:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:39:37.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: 5.0 scrub starts 2024-03-12T03:39:37.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:37 smithi169 ceph-mon[39949]: 5.0 scrub ok 2024-03-12T03:39:37.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: 3.f scrub starts 2024-03-12T03:39:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: 3.f scrub ok 2024-03-12T03:39:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: 5.12 scrub starts 2024-03-12T03:39:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: 5.12 scrub ok 2024-03-12T03:39:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: pgmap v288: 158 pgs: 5 peering, 12 unknown, 141 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 65 KiB/s, 5 objects/s recovering 2024-03-12T03:39:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: 4.1 deep-scrub starts 2024-03-12T03:39:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: 4.1 deep-scrub ok 2024-03-12T03:39:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: 4.3 scrub starts 2024-03-12T03:39:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: 4.3 scrub ok 2024-03-12T03:39:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:39:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:39:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: 5.0 scrub starts 2024-03-12T03:39:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:37 smithi173 ceph-mon[33544]: 5.0 scrub ok 2024-03-12T03:39:37.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:37.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:37.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:37.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-12T03:39:37.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:37.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:37.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:37.968 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:39:37.968 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:39:37.968 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-12T03:39:37.968 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-12T03:39:37.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3 2024-03-12T03:39:37.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:38.038 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:38.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:38.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:38.087 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:38 smithi169 systemd[1]: Stopping Ceph osd.3 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:39:38.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:38.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:38.166 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:38.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:38.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:38.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:38.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:38.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:38.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:38.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:38.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:38.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:38.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:38.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:38.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:38.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:38.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:38.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:38 smithi169 ceph-mon[39949]: osdmap e202: 8 total, 8 up, 8 in 2024-03-12T03:39:38.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:38 smithi169 ceph-mon[39949]: 4.1e scrub starts 2024-03-12T03:39:38.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:38 smithi169 ceph-mon[39949]: 4.1e scrub ok 2024-03-12T03:39:38.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:38 smithi169 ceph-mon[39949]: 6.13 scrub starts 2024-03-12T03:39:38.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:38 smithi169 ceph-mon[39949]: 6.13 scrub ok 2024-03-12T03:39:38.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:38 smithi169 ceph-mon[29344]: osdmap e202: 8 total, 8 up, 8 in 2024-03-12T03:39:38.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:38 smithi169 ceph-mon[29344]: 4.1e scrub starts 2024-03-12T03:39:38.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:38 smithi169 ceph-mon[29344]: 4.1e scrub ok 2024-03-12T03:39:38.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:38 smithi169 ceph-mon[29344]: 6.13 scrub starts 2024-03-12T03:39:38.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:38 smithi169 ceph-mon[29344]: 6.13 scrub ok 2024-03-12T03:39:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:38 smithi173 ceph-mon[33544]: osdmap e202: 8 total, 8 up, 8 in 2024-03-12T03:39:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:38 smithi173 ceph-mon[33544]: 4.1e scrub starts 2024-03-12T03:39:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:38 smithi173 ceph-mon[33544]: 4.1e scrub ok 2024-03-12T03:39:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:38 smithi173 ceph-mon[33544]: 6.13 scrub starts 2024-03-12T03:39:38.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:38 smithi173 ceph-mon[33544]: 6.13 scrub ok 2024-03-12T03:39:38.773 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:38.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:38.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:38.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:38.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:38.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:38.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:39.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:39.049 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:39.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:39.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:39.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:39.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:39.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:39.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:39.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:39.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:39.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:39.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:39.435 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:39.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:39.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:39.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:39.560 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:39.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:39.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:39.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:39.663 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[29344]: pgmap v291: 158 pgs: 5 peering, 153 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 65 KiB/s, 10 objects/s recovering 2024-03-12T03:39:39.664 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[29344]: 6.1a deep-scrub starts 2024-03-12T03:39:39.664 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[29344]: 6.1a deep-scrub ok 2024-03-12T03:39:39.664 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[29344]: 6.10 deep-scrub starts 2024-03-12T03:39:39.664 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[29344]: 6.10 deep-scrub ok 2024-03-12T03:39:39.664 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[29344]: 5.f scrub starts 2024-03-12T03:39:39.664 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[29344]: 5.f scrub ok 2024-03-12T03:39:39.664 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[29344]: 6.6 scrub starts 2024-03-12T03:39:39.664 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[29344]: 6.6 scrub ok 2024-03-12T03:39:39.664 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[39949]: pgmap v291: 158 pgs: 5 peering, 153 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 65 KiB/s, 10 objects/s recovering 2024-03-12T03:39:39.664 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[39949]: 6.1a deep-scrub starts 2024-03-12T03:39:39.665 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[39949]: 6.1a deep-scrub ok 2024-03-12T03:39:39.665 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[39949]: 6.10 deep-scrub starts 2024-03-12T03:39:39.665 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[39949]: 6.10 deep-scrub ok 2024-03-12T03:39:39.665 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[39949]: 5.f scrub starts 2024-03-12T03:39:39.665 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[39949]: 5.f scrub ok 2024-03-12T03:39:39.665 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[39949]: 6.6 scrub starts 2024-03-12T03:39:39.665 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:39 smithi169 ceph-mon[39949]: 6.6 scrub ok 2024-03-12T03:39:39.685 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:39.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:39 smithi173 ceph-mon[33544]: pgmap v291: 158 pgs: 5 peering, 153 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 65 KiB/s, 10 objects/s recovering 2024-03-12T03:39:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:39 smithi173 ceph-mon[33544]: 6.1a deep-scrub starts 2024-03-12T03:39:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:39 smithi173 ceph-mon[33544]: 6.1a deep-scrub ok 2024-03-12T03:39:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:39 smithi173 ceph-mon[33544]: 6.10 deep-scrub starts 2024-03-12T03:39:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:39 smithi173 ceph-mon[33544]: 6.10 deep-scrub ok 2024-03-12T03:39:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:39 smithi173 ceph-mon[33544]: 5.f scrub starts 2024-03-12T03:39:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:39 smithi173 ceph-mon[33544]: 5.f scrub ok 2024-03-12T03:39:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:39 smithi173 ceph-mon[33544]: 6.6 scrub starts 2024-03-12T03:39:39.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:39 smithi173 ceph-mon[33544]: 6.6 scrub ok 2024-03-12T03:39:39.787 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:39.812 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:39.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:39.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:39.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:39.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:40.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:40.072 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:40.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:40.173 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:40.199 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:40.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:40.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:40.304 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:40.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:40.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:40.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:40.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:40.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:40.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:40.462 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:40 smithi169 ceph-mon[29344]: 4.b scrub starts 2024-03-12T03:39:40.462 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:40 smithi169 ceph-mon[29344]: 4.b scrub ok 2024-03-12T03:39:40.462 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:40 smithi169 ceph-mon[39949]: 4.b scrub starts 2024-03-12T03:39:40.462 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:40 smithi169 ceph-mon[39949]: 4.b scrub ok 2024-03-12T03:39:40.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:40.586 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:40.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:40.687 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:40.714 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:40.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:40.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:40 smithi173 ceph-mon[33544]: 4.b scrub starts 2024-03-12T03:39:40.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:40 smithi173 ceph-mon[33544]: 4.b scrub ok 2024-03-12T03:39:40.815 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:40.838 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:40.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:40.940 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:40.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:40.964 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:40.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:41.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:41.093 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:41.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:41.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:41.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:41.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:41.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:41.348 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:41.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:41.370 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:39:41.384 INFO:teuthology.orchestra.run.smithi169.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-12T03:39:41.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:41.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:41.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:41.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:41.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:41.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:41.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:41.674 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[61852]: 2024-03-12T03:39:41.312+0000 7f07f402c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:39:41.674 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[61852]: 2024-03-12T03:39:41.312+0000 7f07f402c700 -1 osd.3 202 *** Got signal Terminated *** 2024-03-12T03:39:41.674 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[61852]: 2024-03-12T03:39:41.312+0000 7f07f402c700 -1 osd.3 202 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:39:41.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[29344]: 6.1 scrub starts 2024-03-12T03:39:41.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[29344]: 6.1 scrub ok 2024-03-12T03:39:41.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[29344]: pgmap v292: 158 pgs: 5 peering, 153 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 909 B/s rd, 0 op/s; 46 KiB/s, 7 objects/s recovering 2024-03-12T03:39:41.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4116947091' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[29344]: 6.0 scrub starts 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1701383786' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[29344]: 6.0 scrub ok 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[29344]: 4.15 scrub starts 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[29344]: 4.15 scrub ok 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[29344]: osd.3 marked itself down and dead 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[39949]: 6.1 scrub starts 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[39949]: 6.1 scrub ok 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[39949]: pgmap v292: 158 pgs: 5 peering, 153 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 909 B/s rd, 0 op/s; 46 KiB/s, 7 objects/s recovering 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4116947091' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[39949]: 6.0 scrub starts 2024-03-12T03:39:41.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1701383786' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:39:41.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:39:41.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[39949]: 6.0 scrub ok 2024-03-12T03:39:41.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[39949]: 4.15 scrub starts 2024-03-12T03:39:41.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[39949]: 4.15 scrub ok 2024-03-12T03:39:41.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-mon[39949]: osd.3 marked itself down and dead 2024-03-12T03:39:41.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:41.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:41.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:41.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:41.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:41 smithi173 ceph-mon[33544]: 6.1 scrub starts 2024-03-12T03:39:41.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:41 smithi173 ceph-mon[33544]: 6.1 scrub ok 2024-03-12T03:39:41.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:41 smithi173 ceph-mon[33544]: pgmap v292: 158 pgs: 5 peering, 153 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 909 B/s rd, 0 op/s; 46 KiB/s, 7 objects/s recovering 2024-03-12T03:39:41.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:41 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4116947091' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:39:41.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:41 smithi173 ceph-mon[33544]: 6.0 scrub starts 2024-03-12T03:39:41.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:41 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1701383786' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:39:41.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:41 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:39:41.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:41 smithi173 ceph-mon[33544]: 6.0 scrub ok 2024-03-12T03:39:41.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:41 smithi173 ceph-mon[33544]: 4.15 scrub starts 2024-03-12T03:39:41.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:41 smithi173 ceph-mon[33544]: 4.15 scrub ok 2024-03-12T03:39:41.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:41 smithi173 ceph-mon[33544]: osd.3 marked itself down and dead 2024-03-12T03:39:41.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:41.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:41.860 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:39:41.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:41.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:41.880 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:39:41.899 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:39:41.919 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:39:41.939 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:39:41.958 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:39:41.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:41.979 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:39:41.979 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:39:41.979 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:39:41.979 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:39:41.979 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:39:41.979 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:39:41.980 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:39:41.980 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:39:41.980 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:39:41.981 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:39:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:39:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:39:41.986 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:41.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:41.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:42.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:42.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:42.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:42.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:42.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:42.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:42.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:42.448 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:42.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:42.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:42.577 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:42.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:42.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:42.663 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:42 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:39:42.663 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:42 smithi169 ceph-mon[29344]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T03:39:42.663 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:42 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4116947091' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:39:42.663 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:42 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:39:42.663 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:42 smithi169 ceph-mon[29344]: osdmap e203: 8 total, 7 up, 8 in 2024-03-12T03:39:42.663 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:42 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1701700935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:42.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:42 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:39:42.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:42 smithi169 ceph-mon[39949]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T03:39:42.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:42 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4116947091' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:39:42.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:42 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:39:42.664 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:42 smithi169 ceph-mon[39949]: osdmap e203: 8 total, 7 up, 8 in 2024-03-12T03:39:42.664 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:42 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1701700935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:42.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:39:42.691 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:39:42.707 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:42.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:42.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:39:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:39:42.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:42 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:39:42.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:42 smithi173 ceph-mon[33544]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T03:39:42.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:42 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4116947091' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:39:42.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:42 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:39:42.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:42 smithi173 ceph-mon[33544]: osdmap e203: 8 total, 7 up, 8 in 2024-03-12T03:39:42.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:42 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1701700935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:42.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:42.833 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:42.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:42.923 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:42 smithi169 podman[106052]: 2024-03-12 03:39:42.673964938 +0000 UTC m=+2.099707170 container died 56494484c3df9c6908d4f8e1ae2f00a3dcb90803493de5d7ecaf7bdb27542759 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, RELEASE=main-2b0101e, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-12T03:39:42.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:42.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:42.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:42.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:43.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:43.091 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:43.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:43.192 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:43.216 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:43.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:43.300 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:43 smithi169 podman[106052]: 2024-03-12 03:39:43.04678372 +0000 UTC m=+2.472525958 container cleanup 56494484c3df9c6908d4f8e1ae2f00a3dcb90803493de5d7ecaf7bdb27542759 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-12T03:39:43.300 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:43 smithi169 bash[106052]: ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3 2024-03-12T03:39:43.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:43.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:43.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:43.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:43.476 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:43.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:43.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:43.578 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:43.605 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:43 smithi169 ceph-mon[29344]: pgmap v294: 158 pgs: 11 peering, 22 stale+active+clean, 125 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 36 B/s, 3 objects/s recovering 2024-03-12T03:39:43.605 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:43 smithi169 ceph-mon[29344]: Health check failed: Reduced data availability: 2 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-12T03:39:43.605 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:43 smithi169 ceph-mon[29344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:39:43.605 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:43 smithi169 ceph-mon[29344]: osdmap e204: 8 total, 7 up, 8 in 2024-03-12T03:39:43.605 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:43 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:43.605 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:43 smithi169 ceph-mon[39949]: pgmap v294: 158 pgs: 11 peering, 22 stale+active+clean, 125 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 36 B/s, 3 objects/s recovering 2024-03-12T03:39:43.605 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:43 smithi169 ceph-mon[39949]: Health check failed: Reduced data availability: 2 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-12T03:39:43.605 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:43 smithi169 ceph-mon[39949]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:39:43.605 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:43 smithi169 ceph-mon[39949]: osdmap e204: 8 total, 7 up, 8 in 2024-03-12T03:39:43.605 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:43 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:43.606 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:43 smithi169 podman[106209]: 2024-03-12 03:39:43.298501283 +0000 UTC m=+1.452222106 container remove 56494484c3df9c6908d4f8e1ae2f00a3dcb90803493de5d7ecaf7bdb27542759 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-12T03:39:43.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:43.607 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:43.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:43.707 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:43.732 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:43.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:43.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:43 smithi173 ceph-mon[33544]: pgmap v294: 158 pgs: 11 peering, 22 stale+active+clean, 125 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 36 B/s, 3 objects/s recovering 2024-03-12T03:39:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:43 smithi173 ceph-mon[33544]: Health check failed: Reduced data availability: 2 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-12T03:39:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:43 smithi173 ceph-mon[33544]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:39:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:43 smithi173 ceph-mon[33544]: osdmap e204: 8 total, 7 up, 8 in 2024-03-12T03:39:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:43 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:43.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:43.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:43.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:43.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:43.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:43.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:43.991 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:43.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:44.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:44.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:44.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:44.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:44.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:44.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:44.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:44.352 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:44.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:44.378 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:44.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:44.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:44.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:44.508 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:44.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:44.609 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:44.635 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:44.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:44.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:39:44.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:44.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:44.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:44.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:44.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:44.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:44.941 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-12T03:39:44.944 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-12T03:39:44.944 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-12T03:39:44.991 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:hints = 1 2024-03-12T03:39:44.992 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-12T03:39:44.992 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Object prefix: benchmark_data_smithi169_106492 2024-03-12T03:39:44.992 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:39:44.992 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 0 0 0 0 0 0 - 0 2024-03-12T03:39:44.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:45.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:45.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:45.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:45.196 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:45.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:45.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:45.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:45.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:45.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:45.436 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:45.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:45.481 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:45.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:45.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:45.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:45.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:45.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:45.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:45 smithi169 ceph-mon[29344]: pgmap v296: 158 pgs: 11 peering, 22 stale+active+clean, 125 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 470 B/s rd, 0 op/s 2024-03-12T03:39:45.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:45 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:45.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:45 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:45.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:45 smithi169 ceph-mon[39949]: pgmap v296: 158 pgs: 11 peering, 22 stale+active+clean, 125 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 470 B/s rd, 0 op/s 2024-03-12T03:39:45.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:45 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:45.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:45 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:45.674 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:45 smithi169 podman[106617]: 2024-03-12 03:39:45.404873998 +0000 UTC m=+0.214429390 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:39:45.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:45 smithi173 ceph-mon[33544]: pgmap v296: 158 pgs: 11 peering, 22 stale+active+clean, 125 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 470 B/s rd, 0 op/s 2024-03-12T03:39:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:45 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:39:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:45 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:39:45.756 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:45.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:45.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:45.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:45.897 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:45.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:45.971 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:45 smithi169 podman[106617]: 2024-03-12 03:39:45.715097405 +0000 UTC m=+0.524652780 container create 0347095c4ef253140a6246689f5b532a75eafce3754711757634993d71a7f795 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=) 2024-03-12T03:39:45.992 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 1 16 3770 3754 234.612 234.625 0.00518385 0.00425177 2024-03-12T03:39:45.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:46.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:46.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:46.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:46.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:46.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-03-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-03-12T03:39:46.190 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:39:46.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:46.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:46.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:46.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:46.451 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:46.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:46.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:39:46.535 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:46 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:39:46.535 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:46 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:39:46.535 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:46 smithi169 ceph-mon[29344]: osdmap e205: 8 total, 7 up, 8 in 2024-03-12T03:39:46.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:46 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:39:46.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:46 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:39:46.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:46 smithi169 ceph-mon[39949]: osdmap e205: 8 total, 7 up, 8 in 2024-03-12T03:39:46.536 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:46 smithi169 podman[106617]: 2024-03-12 03:39:46.244922541 +0000 UTC m=+1.054477933 container init 0347095c4ef253140a6246689f5b532a75eafce3754711757634993d71a7f795 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:39:46.536 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:46 smithi169 podman[106617]: 2024-03-12 03:39:46.252917412 +0000 UTC m=+1.062472791 container start 0347095c4ef253140a6246689f5b532a75eafce3754711757634993d71a7f795 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-12T03:39:46.536 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:46 smithi169 podman[106617]: 2024-03-12 03:39:46.353765542 +0000 UTC m=+1.163320924 container attach 0347095c4ef253140a6246689f5b532a75eafce3754711757634993d71a7f795 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:39:46.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:46.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:46.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:46.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:46.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:46.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:46 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:39:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:46 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:39:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:46 smithi173 ceph-mon[33544]: osdmap e205: 8 total, 7 up, 8 in 2024-03-12T03:39:46.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:46.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:46.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:46.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:46.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:46.992 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 2 16 7282 7266 227.048 219.5 0.00584724 0.00439538 2024-03-12T03:39:47.003 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:47.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:47.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:47.144 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:47.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:47.173 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:39:47 smithi169 podman[106617]: 2024-03-12 03:39:47.003823988 +0000 UTC m=+1.813379359 container died 0347095c4ef253140a6246689f5b532a75eafce3754711757634993d71a7f795 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, RELEASE=main-2b0101e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-12T03:39:47.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:47.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:47.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:47.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:47.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-12T03:39:47.426 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:47.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:47.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:47.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:47.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:47.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:47.664 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:47.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:47 smithi169 ceph-mon[29344]: pgmap v298: 158 pgs: 15 active+undersized+degraded, 11 peering, 10 stale+active+clean, 17 active+undersized, 105 active+clean; 77 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 13 MiB/s wr, 209 op/s; 651/4350 objects degraded (14.966%); 26 KiB/s, 0 objects/s recovering 2024-03-12T03:39:47.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:47 smithi169 ceph-mon[29344]: Health check failed: Degraded data redundancy: 651/4350 objects degraded (14.966%), 15 pgs degraded (PG_DEGRADED) 2024-03-12T03:39:47.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:47 smithi169 ceph-mon[39949]: pgmap v298: 158 pgs: 15 active+undersized+degraded, 11 peering, 10 stale+active+clean, 17 active+undersized, 105 active+clean; 77 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 13 MiB/s wr, 209 op/s; 651/4350 objects degraded (14.966%); 26 KiB/s, 0 objects/s recovering 2024-03-12T03:39:47.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:47 smithi169 ceph-mon[39949]: Health check failed: Degraded data redundancy: 651/4350 objects degraded (14.966%), 15 pgs degraded (PG_DEGRADED) 2024-03-12T03:39:47.708 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:47.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:47.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:47 smithi173 ceph-mon[33544]: pgmap v298: 158 pgs: 15 active+undersized+degraded, 11 peering, 10 stale+active+clean, 17 active+undersized, 105 active+clean; 77 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 13 MiB/s wr, 209 op/s; 651/4350 objects degraded (14.966%); 26 KiB/s, 0 objects/s recovering 2024-03-12T03:39:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:47 smithi173 ceph-mon[33544]: Health check failed: Degraded data redundancy: 651/4350 objects degraded (14.966%), 15 pgs degraded (PG_DEGRADED) 2024-03-12T03:39:47.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:47.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:47.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:47.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:47.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:47.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:47.992 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 3 15 10776 10761 224.172 218.438 0.0034474 0.00445423 2024-03-12T03:39:48.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:48.127 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:48.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:48.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:48.265 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:48.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:48.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:48.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:48.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:48.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:48.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:48.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:48.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:48.686 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:48.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:48.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3.service' 2024-03-12T03:39:48.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:48.828 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:39:48.828 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-12T03:39:48.828 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-03-12T03:39:48.828 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi169.front.sepia.ceph.com 2024-03-12T03:39:48.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.3 --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-03-12T03:39:48.839 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:48.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:48.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:48.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:48.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:48.992 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 4 16 14141 14125 220.687 210.25 0.00468398 0.00452506 2024-03-12T03:39:49.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:49.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:49.124 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:49.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:49.223 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/osd.3/config 2024-03-12T03:39:49.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:49.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:49.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:49.275 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:49.360 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:49.404 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:49.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:49.501 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:49 smithi169 ceph-mon[29344]: pgmap v299: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 262 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 40 MiB/s wr, 646 op/s; 1656/13227 objects degraded (12.520%); 24 KiB/s, 0 objects/s recovering 2024-03-12T03:39:49.501 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:49 smithi169 ceph-mon[29344]: osdmap e206: 8 total, 7 up, 8 in 2024-03-12T03:39:49.501 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:49 smithi169 ceph-mon[29344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 6 pgs peering) 2024-03-12T03:39:49.501 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:49 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:39:49.501 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:49 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:39:49.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:49 smithi169 ceph-mon[39949]: pgmap v299: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 262 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 40 MiB/s wr, 646 op/s; 1656/13227 objects degraded (12.520%); 24 KiB/s, 0 objects/s recovering 2024-03-12T03:39:49.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:49 smithi169 ceph-mon[39949]: osdmap e206: 8 total, 7 up, 8 in 2024-03-12T03:39:49.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:49 smithi169 ceph-mon[39949]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 6 pgs peering) 2024-03-12T03:39:49.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:49 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:39:49.502 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:49 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:39:49.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:49.546 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:49.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:49.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:49.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:49.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:49.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:49 smithi173 ceph-mon[33544]: pgmap v299: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 262 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 40 MiB/s wr, 646 op/s; 1656/13227 objects degraded (12.520%); 24 KiB/s, 0 objects/s recovering 2024-03-12T03:39:49.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:49 smithi173 ceph-mon[33544]: osdmap e206: 8 total, 7 up, 8 in 2024-03-12T03:39:49.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:49 smithi173 ceph-mon[33544]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 6 pgs peering) 2024-03-12T03:39:49.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:49 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:39:49.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:49 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:39:49.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:49.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:49.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:49.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:49.957 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:49.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:49.992 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 5 15 17276 17261 215.746 196 0.00498115 0.00462826 2024-03-12T03:39:50.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:50.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:50.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:50.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:50.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:50.228 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:50.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:50.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:50.367 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:50.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:50.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:50.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:50.508 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:50.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:50.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:50.644 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:50.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:50.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:50 smithi169 ceph-mon[29344]: osdmap e207: 8 total, 7 up, 8 in 2024-03-12T03:39:50.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:50 smithi169 ceph-mon[39949]: osdmap e207: 8 total, 7 up, 8 in 2024-03-12T03:39:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:50 smithi173 ceph-mon[33544]: osdmap e207: 8 total, 7 up, 8 in 2024-03-12T03:39:50.739 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:39:50.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:50.780 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:39:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:50.785 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:50.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:50.820 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:39:50.860 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:39:50.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:50.899 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:39:50.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:50.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:50.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:50.938 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:39:50.976 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:39:50.976 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:39:50.977 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:39:50.977 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:39:50.977 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:39:50.977 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:39:50.977 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:39:50.977 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:39:50.977 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:39:50.992 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 6 16 20144 20128 209.651 179.188 0.00452434 0.00476401 2024-03-12T03:39:51.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:51.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:51.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:51.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:51.159 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:51.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:51.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:51.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:51.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:51.333 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:51.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:51.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:51.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:51.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:51.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:51.572 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:51.609 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:51.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:51.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:51 smithi169 ceph-mon[29344]: pgmap v302: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 262 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 44 MiB/s wr, 701 op/s; 1656/13227 objects degraded (12.520%); 26 KiB/s, 0 objects/s recovering 2024-03-12T03:39:51.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:51 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2515014365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:51.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:51 smithi169 ceph-mon[39949]: pgmap v302: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 262 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 44 MiB/s wr, 701 op/s; 1656/13227 objects degraded (12.520%); 26 KiB/s, 0 objects/s recovering 2024-03-12T03:39:51.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:51 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2515014365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:51.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:51.711 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:39:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:51 smithi173 ceph-mon[33544]: pgmap v302: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 262 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 44 MiB/s wr, 701 op/s; 1656/13227 objects degraded (12.520%); 26 KiB/s, 0 objects/s recovering 2024-03-12T03:39:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:51 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2515014365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:39:51.757 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:51.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:51.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:51.896 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:51.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:51.992 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 7 16 23096 23080 206.056 184.5 0.00377288 0.00484758 2024-03-12T03:39:51.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:52.040 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:52.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:52.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:52.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:39:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:39:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:39:52.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:52.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:52.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:52.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:52.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:39:52.372 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:39:52.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:52.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:52.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:52.551 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:52.592 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:52.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:52.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:52 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:39:52.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:52 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:39:52.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:52 smithi169 ceph-mon[29344]: osdmap e208: 8 total, 7 up, 8 in 2024-03-12T03:39:52.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:52 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:39:52.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:52 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:39:52.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:52 smithi169 ceph-mon[39949]: osdmap e208: 8 total, 7 up, 8 in 2024-03-12T03:39:52.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:52.733 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:52.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:52.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:39:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:39:52.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:52 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:39:52.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:52 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:39:52.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:52 smithi173 ceph-mon[33544]: osdmap e208: 8 total, 7 up, 8 in 2024-03-12T03:39:52.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:52.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:52.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:52.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:52.993 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 8 15 25939 25924 202.515 177.75 0.0068228 0.00493226 2024-03-12T03:39:53.010 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:53.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:53.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:53.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:53.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:53.151 INFO:teuthology.orchestra.run.smithi169.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-12T03:39:53.251 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:53.288 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:53.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:53.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:53.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:53.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:53.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:53.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:53.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:53.643 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:53 smithi169 ceph-mon[29344]: pgmap v304: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 201 MiB/s wr, 3.21k op/s; 7779/62205 objects degraded (12.505%) 2024-03-12T03:39:53.643 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:53 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 7779/62205 objects degraded (12.505%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:39:53.643 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:53 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:53.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:53 smithi169 ceph-mon[39949]: pgmap v304: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 201 MiB/s wr, 3.21k op/s; 7779/62205 objects degraded (12.505%) 2024-03-12T03:39:53.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:53 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 7779/62205 objects degraded (12.505%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:39:53.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:53 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:53.667 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:53.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:53.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:53.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:53 smithi173 ceph-mon[33544]: pgmap v304: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 201 MiB/s wr, 3.21k op/s; 7779/62205 objects degraded (12.505%) 2024-03-12T03:39:53.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:53 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 7779/62205 objects degraded (12.505%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:39:53.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:53 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:53.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:53.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:53.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:53.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:53.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:53.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:53.992 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 9 16 28577 28561 198.325 164.812 0.00481411 0.00503648 2024-03-12T03:39:54.079 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:54.121 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:54.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:54.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:54.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:54.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:54.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:54.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:54.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:39:54.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:54.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:54.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:54.537 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:54.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:54.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:54.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:54.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:54.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:54.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:54.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:54.914 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:54.957 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:54.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:54.993 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 10 16 30863 30847 192.778 142.875 0.00600026 0.00518247 2024-03-12T03:39:55.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:55.091 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:55.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:55.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:55.236 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:55.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:55.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:55.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:55.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:55.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:55.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:55.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:55.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:55.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:55.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:55.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:55.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:55 smithi173 ceph-mon[33544]: pgmap v305: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 170 MiB/s wr, 2.72k op/s; 7779/62205 objects degraded (12.505%) 2024-03-12T03:39:55.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:55.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:55.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:55.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:55.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:55.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:55.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:55.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:55 smithi169 ceph-mon[29344]: pgmap v305: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 170 MiB/s wr, 2.72k op/s; 7779/62205 objects degraded (12.505%) 2024-03-12T03:39:55.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:55 smithi169 ceph-mon[39949]: pgmap v305: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 170 MiB/s wr, 2.72k op/s; 7779/62205 objects degraded (12.505%) 2024-03-12T03:39:55.993 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 11 15 32855 32840 186.573 124.562 0.00513522 0.00535449 2024-03-12T03:39:56.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:56.059 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:56.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:56.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:56.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:56.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:56.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:56.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:56.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:56.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:56.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:56.451 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:39:56.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:56.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:56.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:56.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:56.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:56.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:56.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:56.735 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:56.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:56.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:56.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:56 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1142673635' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:39:56.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:56.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:56.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:56.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:56 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1142673635' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:39:56.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:56 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1142673635' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:39:56.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:56.993 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 12 15 34652 34637 180.376 112.312 0.0118436 0.00553922 2024-03-12T03:39:57.012 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:57.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:57.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:57.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:57.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:57.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:57.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:57.297 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:57.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:57.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:57.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:57.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:57.524 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:57.569 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:57.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:57.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:57.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:57 smithi169 ceph-mon[29344]: pgmap v306: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 1.5 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 173 MiB/s wr, 2.77k op/s; 10196/76611 objects degraded (13.309%) 2024-03-12T03:39:57.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:57 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:39:57.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:57 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1142673635' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:39:57.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:57 smithi169 ceph-mon[29344]: osdmap e209: 8 total, 7 up, 8 in 2024-03-12T03:39:57.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:57 smithi169 ceph-mon[39949]: pgmap v306: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 1.5 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 173 MiB/s wr, 2.77k op/s; 10196/76611 objects degraded (13.309%) 2024-03-12T03:39:57.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:57 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:39:57.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:57 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1142673635' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:39:57.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:57 smithi169 ceph-mon[39949]: osdmap e209: 8 total, 7 up, 8 in 2024-03-12T03:39:57.694 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:57.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:57.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:57 smithi173 ceph-mon[33544]: pgmap v306: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 1.5 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 173 MiB/s wr, 2.77k op/s; 10196/76611 objects degraded (13.309%) 2024-03-12T03:39:57.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:57 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:39:57.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:57 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1142673635' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:39:57.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:57 smithi173 ceph-mon[33544]: osdmap e209: 8 total, 7 up, 8 in 2024-03-12T03:39:57.757 INFO:teuthology.orchestra.run.smithi169.stdout:8.f 2024-03-12T03:39:57.757 INFO:teuthology.orchestra.run.smithi169.stdout:8.c 2024-03-12T03:39:57.757 INFO:teuthology.orchestra.run.smithi169.stdout:8.b 2024-03-12T03:39:57.757 INFO:teuthology.orchestra.run.smithi169.stdout:8.8 2024-03-12T03:39:57.757 INFO:teuthology.orchestra.run.smithi169.stdout:8.7 2024-03-12T03:39:57.757 INFO:teuthology.orchestra.run.smithi169.stdout:8.0 2024-03-12T03:39:57.757 INFO:teuthology.orchestra.run.smithi169.stdout:7.17 2024-03-12T03:39:57.757 INFO:teuthology.orchestra.run.smithi169.stdout:6.6 2024-03-12T03:39:57.757 INFO:teuthology.orchestra.run.smithi169.stdout:6.c 2024-03-12T03:39:57.757 INFO:teuthology.orchestra.run.smithi169.stdout:6.4 2024-03-12T03:39:57.757 INFO:teuthology.orchestra.run.smithi169.stdout:6.f 2024-03-12T03:39:57.757 INFO:teuthology.orchestra.run.smithi169.stdout:6.2 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:6.1f 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:6.16 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:6.7 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:6.11 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:5.a 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:7.13 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:5.1c 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:5.14 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:6.0 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:5.16 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:5.13 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:5.12 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:5.0 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:3.1e 2024-03-12T03:39:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:6.5 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:4.16 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:1.7 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:4.1e 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:6.b 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:3.2 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:3.8 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:4.4 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:4.c 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:3.17 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:6.13 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:3.5 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:6.19 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:4.0 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:3.15 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:1.6 2024-03-12T03:39:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:2.1 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:5.1 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:3.18 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:6.8 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:1.9 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:3.b 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:4.10 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:4.11 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:3.12 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:4.9 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:4.17 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:4.14 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:3.1 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:4.19 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:6.1b 2024-03-12T03:39:57.760 INFO:teuthology.orchestra.run.smithi169.stdout:3.7 2024-03-12T03:39:57.761 INFO:teuthology.orchestra.run.smithi169.stdout:3.19 2024-03-12T03:39:57.761 INFO:teuthology.orchestra.run.smithi169.stdout:1.8 2024-03-12T03:39:57.761 INFO:teuthology.orchestra.run.smithi169.stdout:3.c 2024-03-12T03:39:57.761 INFO:teuthology.orchestra.run.smithi169.stdout:4.1a 2024-03-12T03:39:57.761 INFO:teuthology.orchestra.run.smithi169.stdout:5.17 2024-03-12T03:39:57.761 INFO:teuthology.orchestra.run.smithi169.stdout:4.1b 2024-03-12T03:39:57.761 INFO:teuthology.orchestra.run.smithi169.stdout:7.12 2024-03-12T03:39:57.761 INFO:teuthology.orchestra.run.smithi169.stdout:4.15 2024-03-12T03:39:57.761 INFO:teuthology.orchestra.run.smithi169.stdout:4.1c 2024-03-12T03:39:57.761 INFO:teuthology.orchestra.run.smithi169.stdout:1.4 2024-03-12T03:39:57.761 INFO:teuthology.orchestra.run.smithi169.stdout:4.1d 2024-03-12T03:39:57.761 INFO:teuthology.orchestra.run.smithi169.stdout:4.f 2024-03-12T03:39:57.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:57.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:57.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:57.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:57.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:57.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:39:57.993 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 13 16 36315 36299 174.491 103.875 0.00383899 0.00572594 2024-03-12T03:39:58.068 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:58.091 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:58.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:58.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:58.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:58.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:58.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:58.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:58.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:58.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:58.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:58.494 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:58.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:58.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:39:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:58.629 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:58.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:39:58.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:58 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 10196/76611 objects degraded (13.309%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:39:58.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:58 smithi169 ceph-mon[29344]: osdmap e210: 8 total, 7 up, 8 in 2024-03-12T03:39:58.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:58 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 10196/76611 objects degraded (13.309%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:39:58.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:58 smithi169 ceph-mon[39949]: osdmap e210: 8 total, 7 up, 8 in 2024-03-12T03:39:58.731 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:58.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:58 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 10196/76611 objects degraded (13.309%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:39:58.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:58 smithi173 ceph-mon[33544]: osdmap e210: 8 total, 7 up, 8 in 2024-03-12T03:39:58.765 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:58.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:58.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:39:58.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:58.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:39:58.994 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 14 16 38105 38089 170.018 111.875 0.00369491 0.00587639 2024-03-12T03:39:59.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:59.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:59.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:59.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:59.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:59.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:59.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:59.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:59.280 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:39:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:59.319 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:59.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:39:59.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:39:59.457 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:39:59.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:59.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:39:59.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:59.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:59.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:59.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:59.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:39:59 smithi169 ceph-mon[29344]: pgmap v308: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 225 MiB/s wr, 3.60k op/s; 12491/99681 objects degraded (12.531%) 2024-03-12T03:39:59.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:39:59 smithi169 ceph-mon[39949]: pgmap v308: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 225 MiB/s wr, 3.60k op/s; 12491/99681 objects degraded (12.531%) 2024-03-12T03:39:59.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:39:59.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:59.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:39:59.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:39:59 smithi173 ceph-mon[33544]: pgmap v308: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 225 MiB/s wr, 3.60k op/s; 12491/99681 objects degraded (12.531%) 2024-03-12T03:39:59.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:39:59.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:59.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:39:59.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:39:59.994 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 15 16 40258 40242 167.653 134.562 0.00744934 0.00595995 2024-03-12T03:39:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:39:59.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:39:59.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:00.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:00.132 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:00.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:00.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:00.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:00.273 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:00.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:00.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:00.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:00.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:00.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:00.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:00.548 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:00.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:00.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:00.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:00.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:00.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:00.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 12491/99681 objects degraded (12.531%), 29 pgs degraded 2024-03-12T03:40:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-12T03:40:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: [WRN] OSD_DOWN: 1 osds down 2024-03-12T03:40:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: osd.3 (root=default,host=smithi169) is down 2024-03-12T03:40:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: [WRN] PG_DEGRADED: Degraded data redundancy: 12491/99681 objects degraded (12.531%), 29 pgs degraded 2024-03-12T03:40:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 1.8 is active+undersized+degraded, acting [5,2] 2024-03-12T03:40:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.4 is active+undersized+degraded, acting [5,2] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.9 is active+undersized+degraded, acting [4,1] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.c is active+undersized+degraded, acting [4,6] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.f is active+undersized+degraded, acting [1,4] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.10 is active+undersized+degraded, acting [1,6] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.11 is active+undersized+degraded, acting [7,6] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.15 is active+undersized+degraded, acting [5,7] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.16 is active+undersized+degraded, acting [0,7] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.17 is active+undersized+degraded, acting [7,1] 2024-03-12T03:40:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.19 is active+undersized+degraded, acting [2,0] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.1a is active+undersized+degraded, acting [4,0] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.1b is active+undersized+degraded, acting [4,1] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.1c is active+undersized+degraded, acting [2,1] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.1d is active+undersized+degraded, acting [1,4] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 4.1e is active+undersized+degraded, acting [0,7] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 5.0 is active+undersized+degraded, acting [1,4] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 5.14 is active+undersized+degraded, acting [7,2] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 6.5 is active+undersized+degraded, acting [7,6] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 8.0 is active+undersized+degraded, acting [5,1] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 8.7 is active+undersized+degraded, acting [1,6] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 8.8 is active+undersized+degraded, acting [5,7] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 8.b is active+undersized+degraded, acting [4,1] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 8.c is active+undersized+degraded, acting [0,7] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: pg 8.f is active+undersized+degraded, acting [2,1] 2024-03-12T03:40:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:00.740 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:00.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:00.824 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:00.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:00.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 12491/99681 objects degraded (12.531%), 29 pgs degraded 2024-03-12T03:40:00.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-12T03:40:00.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: [WRN] OSD_DOWN: 1 osds down 2024-03-12T03:40:00.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: osd.3 (root=default,host=smithi169) is down 2024-03-12T03:40:00.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: [WRN] PG_DEGRADED: Degraded data redundancy: 12491/99681 objects degraded (12.531%), 29 pgs degraded 2024-03-12T03:40:00.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 1.8 is active+undersized+degraded, acting [5,2] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.4 is active+undersized+degraded, acting [5,2] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.9 is active+undersized+degraded, acting [4,1] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.c is active+undersized+degraded, acting [4,6] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.f is active+undersized+degraded, acting [1,4] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.10 is active+undersized+degraded, acting [1,6] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.11 is active+undersized+degraded, acting [7,6] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.15 is active+undersized+degraded, acting [5,7] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.16 is active+undersized+degraded, acting [0,7] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.17 is active+undersized+degraded, acting [7,1] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.19 is active+undersized+degraded, acting [2,0] 2024-03-12T03:40:00.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.1a is active+undersized+degraded, acting [4,0] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.1b is active+undersized+degraded, acting [4,1] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.1c is active+undersized+degraded, acting [2,1] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.1d is active+undersized+degraded, acting [1,4] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 4.1e is active+undersized+degraded, acting [0,7] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 5.0 is active+undersized+degraded, acting [1,4] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 5.14 is active+undersized+degraded, acting [7,2] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 6.5 is active+undersized+degraded, acting [7,6] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 8.0 is active+undersized+degraded, acting [5,1] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 8.7 is active+undersized+degraded, acting [1,6] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 8.8 is active+undersized+degraded, acting [5,7] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 8.b is active+undersized+degraded, acting [4,1] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 8.c is active+undersized+degraded, acting [0,7] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: pg 8.f is active+undersized+degraded, acting [2,1] 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:00.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:00.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 12491/99681 objects degraded (12.531%), 29 pgs degraded 2024-03-12T03:40:00.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-12T03:40:00.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: [WRN] OSD_DOWN: 1 osds down 2024-03-12T03:40:00.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: osd.3 (root=default,host=smithi169) is down 2024-03-12T03:40:00.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: [WRN] PG_DEGRADED: Degraded data redundancy: 12491/99681 objects degraded (12.531%), 29 pgs degraded 2024-03-12T03:40:00.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 1.8 is active+undersized+degraded, acting [5,2] 2024-03-12T03:40:00.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-03-12T03:40:00.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-03-12T03:40:00.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-03-12T03:40:00.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.4 is active+undersized+degraded, acting [5,2] 2024-03-12T03:40:00.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.9 is active+undersized+degraded, acting [4,1] 2024-03-12T03:40:00.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.c is active+undersized+degraded, acting [4,6] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.f is active+undersized+degraded, acting [1,4] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.10 is active+undersized+degraded, acting [1,6] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.11 is active+undersized+degraded, acting [7,6] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.15 is active+undersized+degraded, acting [5,7] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.16 is active+undersized+degraded, acting [0,7] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.17 is active+undersized+degraded, acting [7,1] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.19 is active+undersized+degraded, acting [2,0] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.1a is active+undersized+degraded, acting [4,0] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.1b is active+undersized+degraded, acting [4,1] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.1c is active+undersized+degraded, acting [2,1] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.1d is active+undersized+degraded, acting [1,4] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 4.1e is active+undersized+degraded, acting [0,7] 2024-03-12T03:40:00.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 5.0 is active+undersized+degraded, acting [1,4] 2024-03-12T03:40:00.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 5.14 is active+undersized+degraded, acting [7,2] 2024-03-12T03:40:00.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 6.5 is active+undersized+degraded, acting [7,6] 2024-03-12T03:40:00.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 8.0 is active+undersized+degraded, acting [5,1] 2024-03-12T03:40:00.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 8.7 is active+undersized+degraded, acting [1,6] 2024-03-12T03:40:00.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 8.8 is active+undersized+degraded, acting [5,7] 2024-03-12T03:40:00.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 8.b is active+undersized+degraded, acting [4,1] 2024-03-12T03:40:00.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 8.c is active+undersized+degraded, acting [0,7] 2024-03-12T03:40:00.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: pg 8.f is active+undersized+degraded, acting [2,1] 2024-03-12T03:40:00.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:00.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:00.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:00.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:00.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:00.994 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 16 16 42374 42358 165.44 132.25 0.0105844 0.00603939 2024-03-12T03:40:01.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:01.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:01.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:01.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:01.245 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:01.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:01.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:01.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:01.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:01.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:01.519 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:01.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:01.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:01.620 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:01.660 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:01.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:01 smithi173 ceph-mon[33544]: pgmap v310: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 98 MiB/s wr, 1.56k op/s; 12491/99681 objects degraded (12.531%) 2024-03-12T03:40:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:01 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:40:01.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:01 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:40:01.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:01 smithi173 ceph-mon[33544]: osdmap e211: 8 total, 7 up, 8 in 2024-03-12T03:40:01.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:01.798 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:01.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:01.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:01.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:01 smithi169 ceph-mon[29344]: pgmap v310: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 98 MiB/s wr, 1.56k op/s; 12491/99681 objects degraded (12.531%) 2024-03-12T03:40:01.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:01 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:40:01.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:01 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:40:01.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:01 smithi169 ceph-mon[29344]: osdmap e211: 8 total, 7 up, 8 in 2024-03-12T03:40:01.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:01 smithi169 ceph-mon[39949]: pgmap v310: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.0 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 98 MiB/s wr, 1.56k op/s; 12491/99681 objects degraded (12.531%) 2024-03-12T03:40:01.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:01 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:40:01.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:01 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:40:01.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:01 smithi169 ceph-mon[39949]: osdmap e211: 8 total, 7 up, 8 in 2024-03-12T03:40:01.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:01.942 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:01.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:01.994 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 17 16 44399 44383 163.152 126.562 0.00238581 0.00611351 2024-03-12T03:40:02.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:02.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:02.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:02.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:40:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:40:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:40:02.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:02.223 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:02.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:02.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:02.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:02.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:02.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:02.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:02.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:02.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:02.603 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:02.641 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:02.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:02.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:02.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:02.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:02.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:02.881 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:02.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:02.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:02.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:40:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:40:02.994 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 18 15 46330 46315 160.795 120.75 0.0080128 0.00621474 2024-03-12T03:40:03.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:03.067 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:03.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:03.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:03.206 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:03.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:03.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:03.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:03.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:03.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:03.406 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:03.443 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.3 --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 4.4 --file /var/log/ceph/exp.4.4.3 2024-03-12T03:40:03.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:03.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:03.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:03.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:03.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:03.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:03.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:03 smithi169 ceph-mon[29344]: pgmap v312: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 180 MiB/s wr, 2.88k op/s; 16073/128526 objects degraded (12.506%) 2024-03-12T03:40:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:03 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:03.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:03 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 16073/128526 objects degraded (12.506%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:03.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:03 smithi169 ceph-mon[29344]: osdmap e212: 8 total, 7 up, 8 in 2024-03-12T03:40:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:03 smithi169 ceph-mon[39949]: pgmap v312: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 180 MiB/s wr, 2.88k op/s; 16073/128526 objects degraded (12.506%) 2024-03-12T03:40:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:03 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:03 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 16073/128526 objects degraded (12.506%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:03 smithi169 ceph-mon[39949]: osdmap e212: 8 total, 7 up, 8 in 2024-03-12T03:40:03.725 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:03 smithi173 ceph-mon[33544]: pgmap v312: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 180 MiB/s wr, 2.88k op/s; 16073/128526 objects degraded (12.506%) 2024-03-12T03:40:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:03 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:03 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 16073/128526 objects degraded (12.506%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:03 smithi173 ceph-mon[33544]: osdmap e212: 8 total, 7 up, 8 in 2024-03-12T03:40:03.762 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:03.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:03.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:03.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:03.899 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:03.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:03.994 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 19 16 48115 48099 158.2 111.5 0.00336703 0.00631571 2024-03-12T03:40:04.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:04.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:04.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:04.088 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/osd.3/config 2024-03-12T03:40:04.110 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:40:04.111 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:40:04.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:04.179 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:04.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:04.280 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:04.328 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:04.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:04.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:04.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:04.464 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:04.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:04.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:04.566 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:04.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:04.607 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:04.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:04.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:04.738 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:04.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:04.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:04.880 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:04.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:04.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:04.994 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:40:04.993647+0000 min lat: 0.00107765 max lat: 0.15041 avg lat: 0.00639087 2024-03-12T03:40:04.995 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:40:04.995 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 20 16 50056 50040 156.352 121.312 0.00455294 0.00639087 2024-03-12T03:40:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:05.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:05.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:05.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:40:05.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:05.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:05.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:05.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:05.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:05.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:05.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:05.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:05.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:05.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:05.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:05.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:05.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:05.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:05.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:05.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:05 smithi173 ceph-mon[33544]: pgmap v314: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s; 16073/128526 objects degraded (12.506%) 2024-03-12T03:40:05.739 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:05 smithi169 ceph-mon[29344]: pgmap v314: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s; 16073/128526 objects degraded (12.506%) 2024-03-12T03:40:05.740 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:05 smithi169 ceph-mon[39949]: pgmap v314: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s; 16073/128526 objects degraded (12.506%) 2024-03-12T03:40:05.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:05.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:05.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:05.947 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:05.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:05.986 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:05.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:05.995 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 21 16 52010 51994 154.722 122.125 0.011184 0.0064587 2024-03-12T03:40:06.087 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:06.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:06.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:06.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:06.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-12T03:40:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:06.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:06.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:06.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:06.401 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:06.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:06.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:06.540 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:06.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:06.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:06.694 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:06.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:06.697 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:40:06.738 INFO:teuthology.orchestra.run.smithi169.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-12T03:40:06.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:06.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:06.841 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:06.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:06.943 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:06.983 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:06.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:06.995 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 22 16 53808 53792 152.797 112.375 0.0116201 0.0065399 2024-03-12T03:40:07.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:07.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:07.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:07.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:07.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:07.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:07.271 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:07.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:07.301 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:07.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:07.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:07.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:07.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:07 smithi169 ceph-mon[29344]: pgmap v315: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 16631/131994 objects degraded (12.600%) 2024-03-12T03:40:07.484 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:07 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:40:07.516 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:07.564 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:07.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:07.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:07.699 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:07.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:07 smithi173 ceph-mon[33544]: pgmap v315: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 16631/131994 objects degraded (12.600%) 2024-03-12T03:40:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:07 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:40:07.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:07.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:07.875 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:07 smithi169 ceph-mon[39949]: pgmap v315: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 16631/131994 objects degraded (12.600%) 2024-03-12T03:40:07.875 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:07 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:40:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:07.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:07.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:07.995 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 23 16 55361 55345 150.373 97.0625 0.00814251 0.00664471 2024-03-12T03:40:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:08.021 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:08.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:08.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:08.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:08.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:08.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:08.321 INFO:teuthology.orchestra.run.smithi169.stderr:Exporting 4.4 info 4.4( v 201'37 (0'0,201'37] local-lis/les=197/198 n=11 ec=68/68 lis/c=197/197 les/c/f=198/198/0 sis=197) 2024-03-12T03:40:08.323 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:08.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:08.330 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:2166d8e1:gc::gc.3:head# 2024-03-12T03:40:08.334 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:227f63b5:gc::gc.14:head# 2024-03-12T03:40:08.340 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:22d26bf9:::data_loggenerations_metadata:head# 2024-03-12T03:40:08.340 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:24c8e1ae:gc::gc.19:head# 2024-03-12T03:40:08.340 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:2533b45a:::obj_delete_at_hint.0000000076:head# 2024-03-12T03:40:08.340 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:258fd845:::obj_delete_at_hint.0000000113:head# 2024-03-12T03:40:08.341 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:25ca39bb:::obj_delete_at_hint.0000000005:head# 2024-03-12T03:40:08.341 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:264a8785:::obj_delete_at_hint.0000000011:head# 2024-03-12T03:40:08.341 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:26625e58:lc::lc.26:head# 2024-03-12T03:40:08.341 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:26903331:::obj_delete_at_hint.0000000058:head# 2024-03-12T03:40:08.341 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:26e74a3e:::obj_delete_at_hint.0000000052:head# 2024-03-12T03:40:08.341 INFO:teuthology.orchestra.run.smithi169.stderr:Export successful 2024-03-12T03:40:08.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:08.457 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:08.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:08.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:08.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:08.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:08.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:08.704 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:08.733 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:08.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:08.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:08.803 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:40:08.803 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:40:08.834 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:08.878 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:08.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:08.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:08.995 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 24 16 56837 56821 147.951 92.25 0.00540602 0.00675425 2024-03-12T03:40:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:09.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:09.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:09.131 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:09.162 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:09.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:09.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:09.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:09.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:09.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:09.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:09.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:09 smithi173 ceph-mon[33544]: pgmap v316: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 150 MiB/s wr, 2.41k op/s; 19594/157440 objects degraded (12.445%) 2024-03-12T03:40:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:09.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:09 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 16631/131994 objects degraded (12.600%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:09.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:09.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:09.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:09.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:09.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:09.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:09 smithi169 ceph-mon[29344]: pgmap v316: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 150 MiB/s wr, 2.41k op/s; 19594/157440 objects degraded (12.445%) 2024-03-12T03:40:09.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:09.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:09.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:09 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 16631/131994 objects degraded (12.600%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:09.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:09 smithi169 ceph-mon[39949]: pgmap v316: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 150 MiB/s wr, 2.41k op/s; 19594/157440 objects degraded (12.445%) 2024-03-12T03:40:09.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:09.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:09.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:09 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 16631/131994 objects degraded (12.600%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:09.699 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:09.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:09.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:09.830 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:09.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:09.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:09.932 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:09.980 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:09.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:09.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:09.995 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 25 16 58107 58091 145.208 79.375 0.00712344 0.00688225 2024-03-12T03:40:10.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:10.109 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:10.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:10.119 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:40:10.150 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:40:10.193 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:40:10.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:10.237 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:40:10.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:10.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:10.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:10.278 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:40:10.322 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:10 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:40:10.322 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:10 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:40:10.322 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:10 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:10.322 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:10 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1044609821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:10.322 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:10 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:40:10.322 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:10 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:40:10.322 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:10 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:10.323 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:10 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1044609821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:10.323 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:40:10.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:10.362 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:40:10.362 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:40:10.362 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:40:10.362 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:40:10.362 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:40:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:40:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:40:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:40:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:40:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:10.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:10.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:10.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:10 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:40:10.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:10 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:40:10.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:10 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:10.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:10 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1044609821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:10.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:10.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:10.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:10.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:10.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:10.656 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:10.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:10.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-12T03:40:10.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:10.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:10.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:10.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.3 --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 4.4 2024-03-12T03:40:10.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:10.957 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:10.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:10.995 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 26 15 59552 59537 143.099 90.375 0.00869183 0.00698345 2024-03-12T03:40:11.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:11.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:11.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:11.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:11.131 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:40:11.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:11.241 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:11.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:11.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:11.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:11.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:11.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:11.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:11 smithi169 ceph-mon[29344]: pgmap v317: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 128 MiB/s wr, 2.04k op/s; 19594/157440 objects degraded (12.445%) 2024-03-12T03:40:11.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:11 smithi169 ceph-mon[39949]: pgmap v317: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 128 MiB/s wr, 2.04k op/s; 19594/157440 objects degraded (12.445%) 2024-03-12T03:40:11.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:40:11.497 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:40:11.497 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/osd.3/config 2024-03-12T03:40:11.532 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:11.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:11.556 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:11 smithi173 ceph-mon[33544]: pgmap v317: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 128 MiB/s wr, 2.04k op/s; 19594/157440 objects degraded (12.445%) 2024-03-12T03:40:11.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:11.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:11.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:11.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:11.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:11.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:11.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:11.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:11.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:11.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:11.978 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:40:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:40:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:40:11.995 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 27 16 61183 61167 141.572 101.875 0.0220494 0.00705884 2024-03-12T03:40:12.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:12.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:12.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:12.172 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:12.217 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:12.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:12.307 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:40:12.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:12.346 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:12.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:12.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:12.448 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:12.482 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:12.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:12.533 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:12 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2228074900' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:40:12.534 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:12 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2228074900' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:40:12.567 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:12 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2228074900' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:40:12.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:12.619 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:12.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:12.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:12.756 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:12.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:12.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:12.899 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:12.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:12.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:12.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:40:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:40:12.995 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 28 16 62538 62522 139.54 84.6875 0.00843305 0.00716222 2024-03-12T03:40:13.001 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:13.039 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:13.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:13.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:13.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:13.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:13.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:13.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:13.323 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:13.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:13.326 INFO:teuthology.orchestra.run.smithi169.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-12T03:40:13.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:13.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:13.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:13.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:40:13.536 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[29344]: pgmap v318: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1022 B/s rd, 107 MiB/s wr, 1.71k op/s; 22398/179982 objects degraded (12.445%) 2024-03-12T03:40:13.536 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:40:13.536 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2228074900' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:40:13.536 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[29344]: osdmap e213: 8 total, 7 up, 8 in 2024-03-12T03:40:13.536 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:13.536 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 22398/179982 objects degraded (12.445%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:13.536 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[29344]: osdmap e214: 8 total, 7 up, 8 in 2024-03-12T03:40:13.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[39949]: pgmap v318: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1022 B/s rd, 107 MiB/s wr, 1.71k op/s; 22398/179982 objects degraded (12.445%) 2024-03-12T03:40:13.537 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:40:13.537 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2228074900' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:40:13.537 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[39949]: osdmap e213: 8 total, 7 up, 8 in 2024-03-12T03:40:13.537 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:13.537 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 22398/179982 objects degraded (12.445%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:13.537 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:13 smithi169 ceph-mon[39949]: osdmap e214: 8 total, 7 up, 8 in 2024-03-12T03:40:13.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:13.597 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:13 smithi173 ceph-mon[33544]: pgmap v318: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1022 B/s rd, 107 MiB/s wr, 1.71k op/s; 22398/179982 objects degraded (12.445%) 2024-03-12T03:40:13.597 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:13 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:40:13.597 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:13 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2228074900' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:40:13.597 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:13 smithi173 ceph-mon[33544]: osdmap e213: 8 total, 7 up, 8 in 2024-03-12T03:40:13.597 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:13 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:13.597 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:13 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 22398/179982 objects degraded (12.445%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:13.597 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:13 smithi173 ceph-mon[33544]: osdmap e214: 8 total, 7 up, 8 in 2024-03-12T03:40:13.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:13.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:13.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:13.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:13.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:13.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:13.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:13.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:13.898 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:13.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:13.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:13.995 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 29 16 63950 63934 137.771 88.25 0.0143782 0.00725367 2024-03-12T03:40:14.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:14.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:14.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:14.090 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:40:14.090 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:40:14.090 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:40:14.090 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:40:14.090 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:40:14.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:14.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:14.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:14.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:14.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:14.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:14.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:14.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:14.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:14.473 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:14 smithi169 ceph-mon[29344]: 5.e scrub starts 2024-03-12T03:40:14.473 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:14 smithi169 ceph-mon[29344]: 5.e scrub ok 2024-03-12T03:40:14.474 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:14 smithi169 ceph-mon[29344]: 5.8 scrub starts 2024-03-12T03:40:14.474 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:14 smithi169 ceph-mon[29344]: 5.8 scrub ok 2024-03-12T03:40:14.474 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:14 smithi169 ceph-mon[39949]: 5.e scrub starts 2024-03-12T03:40:14.474 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:14 smithi169 ceph-mon[39949]: 5.e scrub ok 2024-03-12T03:40:14.474 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:14 smithi169 ceph-mon[39949]: 5.8 scrub starts 2024-03-12T03:40:14.474 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:14 smithi169 ceph-mon[39949]: 5.8 scrub ok 2024-03-12T03:40:14.489 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:14.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:14.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:14.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:14.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:14.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:14.727 INFO:teuthology.orchestra.run.smithi169.stdout: marking collection for removal 2024-03-12T03:40:14.727 INFO:teuthology.orchestra.run.smithi169.stdout:setting '_remove' omap key 2024-03-12T03:40:14.727 INFO:teuthology.orchestra.run.smithi169.stdout:finish_remove_pgs 4.4_head removing 4.4 2024-03-12T03:40:14.732 INFO:teuthology.orchestra.run.smithi169.stdout:Remove successful 2024-03-12T03:40:14.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:14.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:14 smithi173 ceph-mon[33544]: 5.e scrub starts 2024-03-12T03:40:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:14 smithi173 ceph-mon[33544]: 5.e scrub ok 2024-03-12T03:40:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:14 smithi173 ceph-mon[33544]: 5.8 scrub starts 2024-03-12T03:40:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:14 smithi173 ceph-mon[33544]: 5.8 scrub ok 2024-03-12T03:40:14.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:14.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-12T03:40:14.870 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:14.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:14.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:14.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:14.996 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 30 15 65497 65482 136.403 96.75 0.00965535 0.00732484 2024-03-12T03:40:15.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:15.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:15.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:15.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:15.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:15.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:15.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:15.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:15.310 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:40:15.310 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:15.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:15.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:15.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[29344]: pgmap v321: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 125 MiB/s wr, 2.00k op/s; 22398/179982 objects degraded (12.445%) 2024-03-12T03:40:15.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[29344]: 6.12 scrub starts 2024-03-12T03:40:15.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[29344]: 6.12 scrub ok 2024-03-12T03:40:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[29344]: 5.18 scrub starts 2024-03-12T03:40:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[29344]: 5.18 scrub ok 2024-03-12T03:40:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[29344]: 4.8 scrub starts 2024-03-12T03:40:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[29344]: 4.8 scrub ok 2024-03-12T03:40:15.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[39949]: pgmap v321: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 125 MiB/s wr, 2.00k op/s; 22398/179982 objects degraded (12.445%) 2024-03-12T03:40:15.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[39949]: 6.12 scrub starts 2024-03-12T03:40:15.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[39949]: 6.12 scrub ok 2024-03-12T03:40:15.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[39949]: 5.18 scrub starts 2024-03-12T03:40:15.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[39949]: 5.18 scrub ok 2024-03-12T03:40:15.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[39949]: 4.8 scrub starts 2024-03-12T03:40:15.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:15 smithi169 ceph-mon[39949]: 4.8 scrub ok 2024-03-12T03:40:15.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:15.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:15.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:15.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:15.580 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:15.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:15.634 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:15 smithi173 ceph-mon[33544]: pgmap v321: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 125 MiB/s wr, 2.00k op/s; 22398/179982 objects degraded (12.445%) 2024-03-12T03:40:15.634 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:15 smithi173 ceph-mon[33544]: 6.12 scrub starts 2024-03-12T03:40:15.634 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:15 smithi173 ceph-mon[33544]: 6.12 scrub ok 2024-03-12T03:40:15.634 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:15 smithi173 ceph-mon[33544]: 5.18 scrub starts 2024-03-12T03:40:15.634 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:15 smithi173 ceph-mon[33544]: 5.18 scrub ok 2024-03-12T03:40:15.634 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:15 smithi173 ceph-mon[33544]: 4.8 scrub starts 2024-03-12T03:40:15.634 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:15 smithi173 ceph-mon[33544]: 4.8 scrub ok 2024-03-12T03:40:15.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:15.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:15.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:15.824 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:15.861 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:15.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:15.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:40:15.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:15.996 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 31 16 66994 66978 135.019 93.5 0.0104043 0.0074014 2024-03-12T03:40:16.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:16.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:16.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:16.141 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:16.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:16.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:16.270 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:16.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:16.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:16.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:16.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:16.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:16.519 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:16.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:16.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:16.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:16.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:16 smithi169 ceph-mon[29344]: 6.1c scrub starts 2024-03-12T03:40:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:16 smithi169 ceph-mon[29344]: 6.1c scrub ok 2024-03-12T03:40:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:16 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:16 smithi169 ceph-mon[29344]: 5.15 scrub starts 2024-03-12T03:40:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:16 smithi169 ceph-mon[29344]: 5.15 scrub ok 2024-03-12T03:40:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:16 smithi169 ceph-mon[39949]: 6.1c scrub starts 2024-03-12T03:40:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:16 smithi169 ceph-mon[39949]: 6.1c scrub ok 2024-03-12T03:40:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:16 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:16 smithi169 ceph-mon[39949]: 5.15 scrub starts 2024-03-12T03:40:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:16 smithi169 ceph-mon[39949]: 5.15 scrub ok 2024-03-12T03:40:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:16 smithi173 ceph-mon[33544]: 6.1c scrub starts 2024-03-12T03:40:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:16 smithi173 ceph-mon[33544]: 6.1c scrub ok 2024-03-12T03:40:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:16 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:16 smithi173 ceph-mon[33544]: 5.15 scrub starts 2024-03-12T03:40:16.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:16 smithi173 ceph-mon[33544]: 5.15 scrub ok 2024-03-12T03:40:16.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:16.802 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:16.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:16.903 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:16.937 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:16.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:16.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:16.996 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 32 16 68304 68288 133.358 81.875 0.00293602 0.00748665 2024-03-12T03:40:17.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:17.066 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:17.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:17.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:17.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:17.205 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:17.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:17.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:17.346 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:17.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:17.448 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:17.485 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:17.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:17.586 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:17.625 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:17.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:17.669 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:17 smithi169 ceph-mon[29344]: pgmap v322: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 22837/182769 objects degraded (12.495%) 2024-03-12T03:40:17.669 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:17 smithi169 ceph-mon[29344]: 4.d scrub starts 2024-03-12T03:40:17.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:17 smithi169 ceph-mon[29344]: 4.d scrub ok 2024-03-12T03:40:17.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:17 smithi169 ceph-mon[29344]: 2.2 scrub starts 2024-03-12T03:40:17.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:17 smithi169 ceph-mon[29344]: 2.2 scrub ok 2024-03-12T03:40:17.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:17 smithi169 ceph-mon[39949]: pgmap v322: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 22837/182769 objects degraded (12.495%) 2024-03-12T03:40:17.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:17 smithi169 ceph-mon[39949]: 4.d scrub starts 2024-03-12T03:40:17.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:17 smithi169 ceph-mon[39949]: 4.d scrub ok 2024-03-12T03:40:17.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:17 smithi169 ceph-mon[39949]: 2.2 scrub starts 2024-03-12T03:40:17.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:17 smithi169 ceph-mon[39949]: 2.2 scrub ok 2024-03-12T03:40:17.727 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.3 --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.4.3 2024-03-12T03:40:17.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:17 smithi173 ceph-mon[33544]: pgmap v322: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 22837/182769 objects degraded (12.495%) 2024-03-12T03:40:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:17 smithi173 ceph-mon[33544]: 4.d scrub starts 2024-03-12T03:40:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:17 smithi173 ceph-mon[33544]: 4.d scrub ok 2024-03-12T03:40:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:17 smithi173 ceph-mon[33544]: 2.2 scrub starts 2024-03-12T03:40:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:17 smithi173 ceph-mon[33544]: 2.2 scrub ok 2024-03-12T03:40:17.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:17.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:17.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:17.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:17.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:17.997 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 33 16 69743 69727 132.037 89.9375 0.00878368 0.00756917 2024-03-12T03:40:18.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:18.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:18.043 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:18.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:18.056 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/osd.3/config 2024-03-12T03:40:18.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:18.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:18.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:18.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:18.314 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:18.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:18.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:18.456 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:18.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:18.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:18.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:18.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:18.613 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:18 smithi173 ceph-mon[33544]: 1.5 scrub starts 2024-03-12T03:40:18.613 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:18 smithi173 ceph-mon[33544]: 1.5 scrub ok 2024-03-12T03:40:18.613 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:18 smithi173 ceph-mon[33544]: 3.1c scrub starts 2024-03-12T03:40:18.613 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:18 smithi173 ceph-mon[33544]: 3.1c scrub ok 2024-03-12T03:40:18.614 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:18 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 22837/182769 objects degraded (12.495%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:18 smithi169 ceph-mon[29344]: 1.5 scrub starts 2024-03-12T03:40:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:18 smithi169 ceph-mon[29344]: 1.5 scrub ok 2024-03-12T03:40:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:18 smithi169 ceph-mon[29344]: 3.1c scrub starts 2024-03-12T03:40:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:18 smithi169 ceph-mon[29344]: 3.1c scrub ok 2024-03-12T03:40:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:18 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 22837/182769 objects degraded (12.495%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:18 smithi169 ceph-mon[39949]: 1.5 scrub starts 2024-03-12T03:40:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:18 smithi169 ceph-mon[39949]: 1.5 scrub ok 2024-03-12T03:40:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:18 smithi169 ceph-mon[39949]: 3.1c scrub starts 2024-03-12T03:40:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:18 smithi169 ceph-mon[39949]: 3.1c scrub ok 2024-03-12T03:40:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:18 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 22837/182769 objects degraded (12.495%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:18.705 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:18.740 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:18.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:18.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:18.883 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:18.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:18.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:18.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:18.997 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 34 16 71150 71134 130.739 87.9375 0.00829526 0.00764354 2024-03-12T03:40:19.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:19.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:19.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:19.110 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:40:19.110 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:40:19.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:29234 7.1 7:81685624:::benchmark_data_smithi169_79822_object29233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:43.518062+0000", 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.58555527499999, 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13080905500000001, 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29234 2024-03-12T03:40:19.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518062+0000", 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518062+0000", 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518064+0000", 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5510000000000001e-06 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518107+0000", 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3503e-05 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518119+0000", 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1826e-05 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518125+0000", 2024-03-12T03:40:19.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7880000000000003e-06 2024-03-12T03:40:19.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518177+0000", 2024-03-12T03:40:19.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1598000000000003e-05 2024-03-12T03:40:19.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518279+0000", 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102298 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518366+0000", 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6800999999999998e-05 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.519415+0000", 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010492139999999999 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.519580+0000", 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016457 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.648770+0000", 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12919019900000001 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.648836+0000", 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6403999999999996e-05 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.648871+0000", 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5302999999999997e-05 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.14799.0:36191 7.4 e101/97 7:280025ea:::benchmark_data_smithi169_79822_object36190:head v 101'2294, mlcod=101'2291)", 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:47.695220+0000", 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 271.40839749100002, 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15715553299999999, 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.117 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695220+0000", 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695220+0000", 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695221+0000", 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9699999999999996e-07 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695280+0000", 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8925000000000002e-05 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695288+0000", 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9580000000000007e-06 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695294+0000", 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4570000000000002e-06 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695341+0000", 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6600000000000001e-05 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695361+0000", 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0122000000000001e-05 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852339+0000", 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15697844699999999 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852376+0000", 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6427000000000002e-05 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:36201 7.1 7:8b7d9225:::benchmark_data_smithi169_79822_object36200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:40:19.121 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:47.695582+0000", 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 271.40803572800002, 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15702228200000001, 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36201 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695582+0000", 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695582+0000", 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695583+0000", 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6700000000000003e-07 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695655+0000", 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2690999999999999e-05 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695662+0000", 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0090000000000001e-06 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695667+0000", 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9169999999999997e-06 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695712+0000", 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4870999999999998e-05 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695830+0000", 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011733199999999999 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.695957+0000", 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012791300000000001 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852365+0000", 2024-03-12T03:40:19.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15640720699999999 2024-03-12T03:40:19.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852459+0000", 2024-03-12T03:40:19.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4413999999999994e-05 2024-03-12T03:40:19.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852509+0000", 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9487000000000002e-05 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852556+0000", 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7018000000000003e-05 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852604+0000", 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8656000000000003e-05 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.14799.0:36201 7.1 e101/97 ondisk, result = 0)", 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:47.697416+0000", 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 271.40620163800003, 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.155048456, 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.697416+0000", 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.697416+0000", 2024-03-12T03:40:19.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.697417+0000", 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1800000000000005e-07 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.697421+0000", 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4340000000000002e-06 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852318+0000", 2024-03-12T03:40:19.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15489645599999999 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852332+0000", 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4231e-05 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852450+0000", 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011798699999999999 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852455+0000", 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5559999999999998e-06 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852465+0000", 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0739999999999999e-06 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.14799.0:36205 7.9 e101/97 7:92884a1f:::benchmark_data_smithi169_79822_object36204:head v 101'2227, mlcod=101'2225)", 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:47.698428+0000", 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 271.40518922799998, 2024-03-12T03:40:19.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15713771500000001, 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.698428+0000", 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.698428+0000", 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.698429+0000", 2024-03-12T03:40:19.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5300000000000004e-07 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.698477+0000", 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7682000000000002e-05 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852317+0000", 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15384040900000001 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852331+0000", 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4256e-05 2024-03-12T03:40:19.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852494+0000", 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016242700000000001 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.852514+0000", 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9772e-05 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.855521+0000", 2024-03-12T03:40:19.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003007352 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:47.855566+0000", 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5263999999999999e-05 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107719 7.1 7:806d079c:::benchmark_data_smithi169_79822_object107718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.171905+0000", 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 210.93171290500001, 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.095925204, 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.135 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107719 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.171905+0000", 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.171905+0000", 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.171906+0000", 2024-03-12T03:40:19.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5969999999999999e-06 2024-03-12T03:40:19.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:19.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.171952+0000", 2024-03-12T03:40:19.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5763999999999998e-05 2024-03-12T03:40:19.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.171973+0000", 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0755999999999999e-05 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.171981+0000", 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4700000000000002e-06 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.172031+0000", 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9626000000000002e-05 2024-03-12T03:40:19.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.172130+0000", 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8590000000000003e-05 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.172226+0000", 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6107999999999995e-05 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.174669+0000", 2024-03-12T03:40:19.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024434600000000002 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.175103+0000", 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00043432900000000001 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.267813+0000", 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.092709535999999995 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.267830+0000", 2024-03-12T03:40:19.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6968000000000001e-05 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107722 7.1 7:8728bd8e:::benchmark_data_smithi169_79822_object107721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.175115+0000", 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 210.92850227, 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.092770225999999997, 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.142 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107722 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.175115+0000", 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.175115+0000", 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.175117+0000", 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2899999999999999e-06 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.175164+0000", 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7116000000000001e-05 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.175173+0000", 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1409999999999994e-06 2024-03-12T03:40:19.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.175180+0000", 2024-03-12T03:40:19.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1489999999999997e-06 2024-03-12T03:40:19.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.175237+0000", 2024-03-12T03:40:19.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7379000000000001e-05 2024-03-12T03:40:19.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.175378+0000", 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000140054 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.175492+0000", 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114836 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.267874+0000", 2024-03-12T03:40:19.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.092381931 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.267886+0000", 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.133e-05 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107719 7.1 7:806d079c:::benchmark_data_smithi169_79822_object107718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.268589+0000", 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 210.835028714, 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.037164685, 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.147 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107719 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268589+0000", 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268589+0000", 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268596+0000", 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1659999999999997e-06 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268665+0000", 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8821999999999999e-05 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268695+0000", 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0086e-05 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268707+0000", 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.183e-05 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268930+0000", 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022354699999999999 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268945+0000", 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5031999999999999e-05 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.305654+0000", 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0367086379999999 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.305754+0000", 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9563999999999997e-05 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107722 7.1 7:8728bd8e:::benchmark_data_smithi169_79822_object107721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:40:19.151 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.268729+0000", 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 210.83488822300001, 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.037482821, 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107722 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268729+0000", 2024-03-12T03:40:19.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268729+0000", 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268730+0000", 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9400000000000005e-07 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268762+0000", 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2036999999999998e-05 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268773+0000", 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0767999999999999e-05 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268779+0000", 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3960000000000003e-06 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268981+0000", 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000201311 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268987+0000", 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8789999999999997e-06 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.306145+0000", 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.037157731 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.306212+0000", 2024-03-12T03:40:19.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7105000000000002e-05 2024-03-12T03:40:19.156 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.156 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.156 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.156 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107731 7.1 7:846066e2:::benchmark_data_smithi169_79822_object107730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:40:19.156 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.268789+0000", 2024-03-12T03:40:19.156 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 210.83482867000001, 2024-03-12T03:40:19.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0546209849999999, 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107731 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268789+0000", 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268789+0000", 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268806+0000", 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7019000000000001e-05 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268859+0000", 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2778000000000002e-05 2024-03-12T03:40:19.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268869+0000", 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0071000000000001e-05 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.268875+0000", 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9000000000000003e-06 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269009+0000", 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000134578 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269017+0000", 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4959999999999999e-06 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.306244+0000", 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.037226765 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.306353+0000", 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000109724 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.306606+0000", 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025288600000000002 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.309952+0000", 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033455400000000001 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.310001+0000", 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9063000000000001e-05 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.323294+0000", 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013293208000000001 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.323356+0000", 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2261000000000006e-05 2024-03-12T03:40:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.323410+0000", 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3696000000000002e-05 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107738 7.1 7:86a5cf4a:::benchmark_data_smithi169_79822_object107737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.452704+0000", 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 210.650914083, 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87188821599999999, 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107738 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.452704+0000", 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.452704+0000", 2024-03-12T03:40:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.452705+0000", 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7039999999999999e-06 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.452736+0000", 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0994999999999997e-05 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.452753+0000", 2024-03-12T03:40:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6578999999999998e-05 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.452762+0000", 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7590000000000003e-06 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.452829+0000", 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7021999999999996e-05 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.452843+0000", 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3910999999999999e-05 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.307660+0000", 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85481703099999995 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.307777+0000", 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117794 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.308042+0000", 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026420600000000001 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.310004+0000", 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001962493 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.311441+0000", 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001436959 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.324447+0000", 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013005937 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.324503+0000", 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5727999999999999e-05 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.324592+0000", 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9098000000000007e-05 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107740 7.1 7:81a02028:::benchmark_data_smithi169_79822_object107739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.489507+0000", 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 210.614110608, 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83526112900000005, 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107740 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.489507+0000", 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.489507+0000", 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.489510+0000", 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1269999999999999e-06 2024-03-12T03:40:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.489583+0000", 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3077999999999994e-05 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.489623+0000", 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9313000000000001e-05 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.489648+0000", 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5488999999999999e-05 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.489824+0000", 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017577200000000001 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.489838+0000", 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4313e-05 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.308969+0000", 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.81913061399999998 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.309067+0000", 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8488999999999998e-05 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.309234+0000", 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016625400000000001 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.310099+0000", 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086521199999999997 2024-03-12T03:40:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.311729+0000", 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001630259 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.324640+0000", 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012911188000000001 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.324723+0000", 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3170999999999994e-05 2024-03-12T03:40:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.324768+0000", 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4849999999999999e-05 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:158457 7.16 7:6d3c0171:::benchmark_data_smithi169_79822_object41040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:56.312979+0000", 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 142.79063897699999, 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.113480214, 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 158457 2024-03-12T03:40:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.312979+0000", 2024-03-12T03:40:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.312979+0000", 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.312980+0000", 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1260000000000001e-06 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.312985+0000", 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3340000000000001e-06 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.312998+0000", 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2541000000000001e-05 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.313005+0000", 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8800000000000002e-06 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.313058+0000", 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.295e-05 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.313198+0000", 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014002700000000001 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.313548+0000", 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035049099999999998 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.315315+0000", 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017665739999999999 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.315871+0000", 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00055623300000000005 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.426361+0000", 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.110489799 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.426412+0000", 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1227999999999998e-05 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:56.426459+0000", 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7030999999999997e-05 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:187348 7.16 7:6a9d974d:::benchmark_data_smithi169_79822_object69931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:28.446669+0000", 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 110.656948547, 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.094973828999999996, 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 187348 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.446669+0000", 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.446669+0000", 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.446670+0000", 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0410000000000001e-06 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.446674+0000", 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3109999999999999e-06 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.446685+0000", 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0256e-05 2024-03-12T03:40:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.446693+0000", 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6889999999999993e-06 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.446753+0000", 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0035999999999999e-05 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.446903+0000", 2024-03-12T03:40:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149073 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.447237+0000", 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000334066 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.449158+0000", 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019214239999999999 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.449805+0000", 2024-03-12T03:40:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00064683799999999995 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.541449+0000", 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.091644259000000006 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.541576+0000", 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012637400000000001 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:28.541643+0000", 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7461999999999999e-05 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:188246 7.1 7:81b8f398:::benchmark_data_smithi169_79822_object70829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:29.348838+0000", 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 109.754780076, 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12411238199999999, 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 188246 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.348838+0000", 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.348838+0000", 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.348838+0000", 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5499999999999997e-07 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.348846+0000", 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6040000000000004e-06 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.348855+0000", 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6349999999999995e-06 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.348874+0000", 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9891e-05 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.348950+0000", 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5273999999999998e-05 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.349085+0000", 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013476499999999999 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.349452+0000", 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000367866 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.352055+0000", 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026027229999999999 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.353280+0000", 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012246850000000001 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.472827+0000", 2024-03-12T03:40:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.119547153 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.472889+0000", 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1722000000000003e-05 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.472950+0000", 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1309000000000005e-05 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:188252 7.1 7:85c4ac98:::benchmark_data_smithi169_79822_object70835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:29.355700+0000", 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 109.74791763, 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11748104500000001, 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 188252 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.355700+0000", 2024-03-12T03:40:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.355700+0000", 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.355702+0000", 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.466e-06 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.355709+0000", 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8590000000000005e-06 2024-03-12T03:40:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.355725+0000", 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5599000000000001e-05 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.355734+0000", 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7499999999999992e-06 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.355808+0000", 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4306000000000003e-05 2024-03-12T03:40:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.355936+0000", 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012830099999999999 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.356334+0000", 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000397799 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.358348+0000", 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020137890000000002 2024-03-12T03:40:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.358772+0000", 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00042443800000000002 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.473085+0000", 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11431250799999999 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.473139+0000", 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3696000000000002e-05 2024-03-12T03:40:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.473181+0000", 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2534000000000003e-05 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:191083 7.16 7:6cc47aa5:::benchmark_data_smithi169_79822_object73666:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:32.766378+0000", 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 106.337239626, 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.107351285, 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 191083 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.766378+0000", 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.766378+0000", 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.766380+0000", 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5960000000000001e-06 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.766385+0000", 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4909999999999996e-06 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.766400+0000", 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.456e-05 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.766408+0000", 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9270000000000005e-06 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.766536+0000", 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012807899999999999 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.766659+0000", 2024-03-12T03:40:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123779 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.767022+0000", 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000362123 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.768897+0000", 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018758469999999999 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.769461+0000", 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056329000000000004 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.873665+0000", 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104203885 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.873698+0000", 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3181e-05 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.873729+0000", 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1526999999999998e-05 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:192322 7.1 7:87574a8c:::benchmark_data_smithi169_79822_object74905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:34.224605+0000", 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 104.879012769, 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.147904281, 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 192322 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.224605+0000", 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.224605+0000", 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.224606+0000", 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.248e-06 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.224610+0000", 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8870000000000002e-06 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.224618+0000", 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3650000000000004e-06 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.224626+0000", 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7659999999999998e-06 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.224678+0000", 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1425999999999999e-05 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.224804+0000", 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000126453 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.225175+0000", 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00037115 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.227256+0000", 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020803290000000001 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.227663+0000", 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040696599999999998 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.372371+0000", 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.144708213 2024-03-12T03:40:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.372452+0000", 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1304999999999995e-05 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.372509+0000", 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7173000000000003e-05 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:192326 7.1 7:85f0fb8a:::benchmark_data_smithi169_79822_object74909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:34.230906+0000", 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 104.87271166399999, 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14634765799999999, 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 192326 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.230906+0000", 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.230906+0000", 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.230907+0000", 2024-03-12T03:40:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.449e-06 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.230914+0000", 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2480000000000001e-06 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.230929+0000", 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5302e-05 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.230939+0000", 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8970000000000003e-06 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.231008+0000", 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9512999999999997e-05 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.231150+0000", 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014142899999999999 2024-03-12T03:40:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.231553+0000", 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040346799999999998 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.234436+0000", 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028829670000000002 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.234768+0000", 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00033195399999999997 2024-03-12T03:40:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.377168+0000", 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14239943999999999 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.377208+0000", 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9981999999999998e-05 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:34.377254+0000", 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6008999999999999e-05 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:213895 7.16 7:6d6b2ff5:::benchmark_data_smithi169_79822_object96478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:54.426417+0000", 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 84.677201061000005, 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.096323958000000001, 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 213895 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.426417+0000", 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.426417+0000", 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.426418+0000", 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.452e-06 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.426424+0000", 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8069999999999996e-06 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.426440+0000", 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5709000000000002e-05 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.426447+0000", 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8830000000000006e-06 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.426520+0000", 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2433000000000003e-05 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.426647+0000", 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000126766 2024-03-12T03:40:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.427016+0000", 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036975400000000002 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.429104+0000", 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020875770000000002 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.431840+0000", 2024-03-12T03:40:19.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002736282 2024-03-12T03:40:19.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:19.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.522653+0000", 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.090812778999999996 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.522703+0000", 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0136000000000002e-05 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.522741+0000", 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7379999999999998e-05 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:19.227 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:19.227 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:40:19.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:19.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:19.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:19.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:19.364 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:19.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:19.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:19.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:19 smithi173 ceph-mon[33544]: 4.12 scrub starts 2024-03-12T03:40:19.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:19 smithi173 ceph-mon[33544]: 4.12 scrub ok 2024-03-12T03:40:19.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:19 smithi173 ceph-mon[33544]: pgmap v323: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s; 25138/202194 objects degraded (12.433%) 2024-03-12T03:40:19.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:19 smithi173 ceph-mon[33544]: 2.0 scrub starts 2024-03-12T03:40:19.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:19 smithi173 ceph-mon[33544]: 2.0 scrub ok 2024-03-12T03:40:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:19 smithi173 ceph-mon[33544]: 6.1e scrub starts 2024-03-12T03:40:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:19 smithi173 ceph-mon[33544]: 6.1e scrub ok 2024-03-12T03:40:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:19 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:40:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:19 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:40:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:19 smithi173 ceph-mon[33544]: 5.d scrub starts 2024-03-12T03:40:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:19 smithi173 ceph-mon[33544]: 5.d scrub ok 2024-03-12T03:40:19.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:19.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:19.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:19.641 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:19.646 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:19.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:19.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[39949]: 4.12 scrub starts 2024-03-12T03:40:19.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[39949]: 4.12 scrub ok 2024-03-12T03:40:19.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[39949]: pgmap v323: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s; 25138/202194 objects degraded (12.433%) 2024-03-12T03:40:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[39949]: 2.0 scrub starts 2024-03-12T03:40:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[39949]: 2.0 scrub ok 2024-03-12T03:40:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[39949]: 6.1e scrub starts 2024-03-12T03:40:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[39949]: 6.1e scrub ok 2024-03-12T03:40:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:40:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:40:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[39949]: 5.d scrub starts 2024-03-12T03:40:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[39949]: 5.d scrub ok 2024-03-12T03:40:19.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[29344]: 4.12 scrub starts 2024-03-12T03:40:19.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[29344]: 4.12 scrub ok 2024-03-12T03:40:19.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[29344]: pgmap v323: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s; 25138/202194 objects degraded (12.433%) 2024-03-12T03:40:19.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[29344]: 2.0 scrub starts 2024-03-12T03:40:19.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[29344]: 2.0 scrub ok 2024-03-12T03:40:19.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[29344]: 6.1e scrub starts 2024-03-12T03:40:19.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[29344]: 6.1e scrub ok 2024-03-12T03:40:19.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:40:19.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:40:19.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[29344]: 5.d scrub starts 2024-03-12T03:40:19.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:19 smithi169 ceph-mon[29344]: 5.d scrub ok 2024-03-12T03:40:19.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:19.788 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:19.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:19.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:19.930 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:19.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:19.997 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 35 16 72624 72608 129.636 92.125 0.00529748 0.00770912 2024-03-12T03:40:20.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:20.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:20.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:20.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:20.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:20.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:20.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:20.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:20.252 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:40:20.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-12T03:40:20.286 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:40:20.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:20.329 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:40:20.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:20.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:20.367 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:40:20.418 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:40:20.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:20.456 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:40:20.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:20 smithi173 ceph-mon[33544]: 4.7 scrub starts 2024-03-12T03:40:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:20 smithi173 ceph-mon[33544]: 4.7 scrub ok 2024-03-12T03:40:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:20 smithi173 ceph-mon[33544]: 3.d scrub starts 2024-03-12T03:40:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:20 smithi173 ceph-mon[33544]: 3.d scrub ok 2024-03-12T03:40:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:20 smithi173 ceph-mon[33544]: osdmap e215: 8 total, 7 up, 8 in 2024-03-12T03:40:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:20 smithi173 ceph-mon[33544]: 6.9 scrub starts 2024-03-12T03:40:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:20 smithi173 ceph-mon[33544]: 6.9 scrub ok 2024-03-12T03:40:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:20 smithi173 ceph-mon[33544]: 6.14 scrub starts 2024-03-12T03:40:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:20 smithi173 ceph-mon[33544]: 6.14 scrub ok 2024-03-12T03:40:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:20 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/640728112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:20.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:20.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:20.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:20.490 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:40:20.490 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:40:20.490 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:40:20.490 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:40:20.490 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:40:20.490 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:40:20.491 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:40:20.491 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:40:20.491 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:40:20.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:20.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:20.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:20.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:20.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[29344]: 4.7 scrub starts 2024-03-12T03:40:20.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[29344]: 4.7 scrub ok 2024-03-12T03:40:20.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[29344]: 3.d scrub starts 2024-03-12T03:40:20.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[29344]: 3.d scrub ok 2024-03-12T03:40:20.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[29344]: osdmap e215: 8 total, 7 up, 8 in 2024-03-12T03:40:20.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[29344]: 6.9 scrub starts 2024-03-12T03:40:20.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[29344]: 6.9 scrub ok 2024-03-12T03:40:20.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[29344]: 6.14 scrub starts 2024-03-12T03:40:20.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[29344]: 6.14 scrub ok 2024-03-12T03:40:20.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/640728112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:20.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[39949]: 4.7 scrub starts 2024-03-12T03:40:20.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[39949]: 4.7 scrub ok 2024-03-12T03:40:20.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[39949]: 3.d scrub starts 2024-03-12T03:40:20.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[39949]: 3.d scrub ok 2024-03-12T03:40:20.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[39949]: osdmap e215: 8 total, 7 up, 8 in 2024-03-12T03:40:20.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[39949]: 6.9 scrub starts 2024-03-12T03:40:20.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[39949]: 6.9 scrub ok 2024-03-12T03:40:20.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[39949]: 6.14 scrub starts 2024-03-12T03:40:20.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[39949]: 6.14 scrub ok 2024-03-12T03:40:20.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:20 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/640728112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:20.693 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:40:20.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:20.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:20.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:20.906 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:20.940 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:20.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:20.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:20.998 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 36 16 74046 74030 128.503 88.875 0.00913678 0.0077772 2024-03-12T03:40:21.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:21.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:21.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:21.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:21.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:21.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:21.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:21.244 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:40:21.244 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:40:21.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:21.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:21.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:21.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:21.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:21.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:21.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:21.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:21.627 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:21.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:21.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:21 smithi169 ceph-mon[29344]: pgmap v325: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 674 B/s rd, 61 MiB/s wr, 975 op/s; 25138/202194 objects degraded (12.433%) 2024-03-12T03:40:21.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:21 smithi169 ceph-mon[39949]: pgmap v325: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 674 B/s rd, 61 MiB/s wr, 975 op/s; 25138/202194 objects degraded (12.433%) 2024-03-12T03:40:21.728 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:21 smithi173 ceph-mon[33544]: pgmap v325: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 674 B/s rd, 61 MiB/s wr, 975 op/s; 25138/202194 objects degraded (12.433%) 2024-03-12T03:40:21.768 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:21.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:21.803 INFO:teuthology.orchestra.run.smithi169.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-12T03:40:21.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:21.918 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:21.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:21.997 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 37 16 75540 75524 127.554 93.375 0.00946944 0.00783519 2024-03-12T03:40:22.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:22.054 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:22.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:22.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:22.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:22.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:40:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:40:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:40:22.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:22.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:22.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:22.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:22.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:22.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:22.429 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:22 smithi173 ceph-mon[33544]: 5.1e scrub starts 2024-03-12T03:40:22.429 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:22 smithi173 ceph-mon[33544]: 5.1e scrub ok 2024-03-12T03:40:22.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:22.481 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:22.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:22.568 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:22 smithi169 ceph-mon[29344]: 5.1e scrub starts 2024-03-12T03:40:22.568 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:22 smithi169 ceph-mon[29344]: 5.1e scrub ok 2024-03-12T03:40:22.568 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:22 smithi169 ceph-mon[39949]: 5.1e scrub starts 2024-03-12T03:40:22.568 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:22 smithi169 ceph-mon[39949]: 5.1e scrub ok 2024-03-12T03:40:22.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:22.609 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:22.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:22.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:40:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:40:22.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:22.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:22.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:22.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:22.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:22.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:22.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:22.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:22.910 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:40:22.911 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:40:22.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:22.998 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 38 16 76931 76915 126.485 86.9375 0.00969278 0.00790136 2024-03-12T03:40:23.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:23.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:23.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:23.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:23.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:23.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:23.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:40:23.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:23.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:23.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:23.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:23.408 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:23.451 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:23.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:23.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:23.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[29344]: pgmap v326: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 27819/223479 objects degraded (12.448%) 2024-03-12T03:40:23.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[29344]: 3.e scrub starts 2024-03-12T03:40:23.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[29344]: 3.e scrub ok 2024-03-12T03:40:23.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:23.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[29344]: 5.7 scrub starts 2024-03-12T03:40:23.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[29344]: 5.7 scrub ok 2024-03-12T03:40:23.482 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 27819/223479 objects degraded (12.448%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:23.482 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[29344]: osdmap e216: 8 total, 7 up, 8 in 2024-03-12T03:40:23.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[39949]: pgmap v326: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 27819/223479 objects degraded (12.448%) 2024-03-12T03:40:23.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[39949]: 3.e scrub starts 2024-03-12T03:40:23.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[39949]: 3.e scrub ok 2024-03-12T03:40:23.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:23.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[39949]: 5.7 scrub starts 2024-03-12T03:40:23.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[39949]: 5.7 scrub ok 2024-03-12T03:40:23.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 27819/223479 objects degraded (12.448%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:23.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:23 smithi169 ceph-mon[39949]: osdmap e216: 8 total, 7 up, 8 in 2024-03-12T03:40:23.508 INFO:teuthology.orchestra.run.smithi169.stdout:get_pg_num_history pg_num_history pg_num_history(e202 pg_nums {1={20=1,149=11},2={65=3},3={66=32},4={68=32},5={70=32},6={72=32},7={97=16,119=26},8={198=16}} deleted_pools 195,7) 2024-03-12T03:40:23.508 INFO:teuthology.orchestra.run.smithi169.stdout:Importing pgid 4.4 2024-03-12T03:40:23.508 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:2166d8e1:gc::gc.3:head# 2024-03-12T03:40:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:40:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:227f63b5:gc::gc.14:head# 2024-03-12T03:40:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:40:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:22d26bf9:::data_loggenerations_metadata:head# 2024-03-12T03:40:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:40:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:24c8e1ae:gc::gc.19:head# 2024-03-12T03:40:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:40:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:2533b45a:::obj_delete_at_hint.0000000076:head# 2024-03-12T03:40:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:40:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:258fd845:::obj_delete_at_hint.0000000113:head# 2024-03-12T03:40:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:40:23.510 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:25ca39bb:::obj_delete_at_hint.0000000005:head# 2024-03-12T03:40:23.510 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:40:23.510 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:264a8785:::obj_delete_at_hint.0000000011:head# 2024-03-12T03:40:23.510 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:40:23.510 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:26625e58:lc::lc.26:head# 2024-03-12T03:40:23.511 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:40:23.511 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:26903331:::obj_delete_at_hint.0000000058:head# 2024-03-12T03:40:23.511 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:40:23.511 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:26e74a3e:::obj_delete_at_hint.0000000052:head# 2024-03-12T03:40:23.511 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:40:23.511 INFO:teuthology.orchestra.run.smithi169.stdout:write_pg epoch 201 info 4.4( v 201'37 (0'0,201'37] local-lis/les=197/198 n=11 ec=68/68 lis/c=197/197 les/c/f=198/198/0 sis=197) 2024-03-12T03:40:23.512 INFO:teuthology.orchestra.run.smithi169.stdout:Import successful 2024-03-12T03:40:23.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:23.599 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:23.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:23.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:23.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:23.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:23.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:23 smithi173 ceph-mon[33544]: pgmap v326: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 27819/223479 objects degraded (12.448%) 2024-03-12T03:40:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:23 smithi173 ceph-mon[33544]: 3.e scrub starts 2024-03-12T03:40:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:23 smithi173 ceph-mon[33544]: 3.e scrub ok 2024-03-12T03:40:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:23 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:23 smithi173 ceph-mon[33544]: 5.7 scrub starts 2024-03-12T03:40:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:23 smithi173 ceph-mon[33544]: 5.7 scrub ok 2024-03-12T03:40:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:23 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 27819/223479 objects degraded (12.448%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:23 smithi173 ceph-mon[33544]: osdmap e216: 8 total, 7 up, 8 in 2024-03-12T03:40:23.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-12T03:40:23.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:23.891 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:23.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:23.928 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:23.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:23.998 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 39 16 78260 78244 125.37 83.0625 0.00558306 0.00797124 2024-03-12T03:40:24.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:24.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:24.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:24.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:24.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:24.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:24.248 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:40:24.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:24.333 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:24.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:24.387 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:40:24.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:24.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:24.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:24.572 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:24.605 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:24.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:24.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:24.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:24 smithi169 ceph-mon[29344]: 3.0 scrub starts 2024-03-12T03:40:24.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:24 smithi169 ceph-mon[29344]: 3.0 scrub ok 2024-03-12T03:40:24.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:24 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3049116505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:40:24.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:24 smithi169 ceph-mon[39949]: 3.0 scrub starts 2024-03-12T03:40:24.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:24 smithi169 ceph-mon[39949]: 3.0 scrub ok 2024-03-12T03:40:24.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:24 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3049116505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:40:24.703 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:24 smithi173 ceph-mon[33544]: 3.0 scrub starts 2024-03-12T03:40:24.703 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:24 smithi173 ceph-mon[33544]: 3.0 scrub ok 2024-03-12T03:40:24.703 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:24 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3049116505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:40:24.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:24.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:24.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:24.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:24.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:24.889 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:24.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:24.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:24.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:24.998 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:40:24.997364+0000 min lat: 0.00107765 max lat: 0.15041 avg lat: 0.00804054 2024-03-12T03:40:24.998 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:40:24.998 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 40 16 79577 79561 124.293 82.3125 0.0118575 0.00804054 2024-03-12T03:40:25.025 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:25.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:25.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:25.162 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:25.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:25.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:25.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:25.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:25.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:25.421 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:25.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:25.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:25.554 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:25.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:25.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:25.665 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:25 smithi173 ceph-mon[33544]: pgmap v328: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.70k op/s; 27819/223479 objects degraded (12.448%) 2024-03-12T03:40:25.665 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:25 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:40:25.665 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:25 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:40:25.665 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:25 smithi173 ceph-mon[33544]: 6.e scrub starts 2024-03-12T03:40:25.665 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:25 smithi173 ceph-mon[33544]: 6.e scrub ok 2024-03-12T03:40:25.665 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:25 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:40:25.665 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:25 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3049116505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:40:25.666 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:25 smithi173 ceph-mon[33544]: osdmap e217: 8 total, 7 up, 8 in 2024-03-12T03:40:25.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[39949]: pgmap v328: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.70k op/s; 27819/223479 objects degraded (12.448%) 2024-03-12T03:40:25.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:40:25.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:40:25.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[39949]: 6.e scrub starts 2024-03-12T03:40:25.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[39949]: 6.e scrub ok 2024-03-12T03:40:25.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:40:25.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3049116505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:40:25.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[39949]: osdmap e217: 8 total, 7 up, 8 in 2024-03-12T03:40:25.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[29344]: pgmap v328: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.70k op/s; 27819/223479 objects degraded (12.448%) 2024-03-12T03:40:25.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:40:25.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:40:25.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[29344]: 6.e scrub starts 2024-03-12T03:40:25.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[29344]: 6.e scrub ok 2024-03-12T03:40:25.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:40:25.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3049116505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:40:25.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:25 smithi169 ceph-mon[29344]: osdmap e217: 8 total, 7 up, 8 in 2024-03-12T03:40:25.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:25.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:25.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:25.817 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:25.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:25.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:25.956 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:25.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:25.998 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 41 16 80720 80704 123.004 71.4375 0.00460264 0.00812448 2024-03-12T03:40:26.057 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:26.087 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:26.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:26.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:26.206 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:40:26.206 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:40:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:26.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:26.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:26.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:26.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:26.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:26.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:26.472 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/exp.4.4.3 2024-03-12T03:40:26.497 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:26.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:26.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:26.634 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:26.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:26.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:26.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:26 smithi169 ceph-mon[29344]: 3.13 scrub starts 2024-03-12T03:40:26.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:26 smithi169 ceph-mon[29344]: 3.13 scrub ok 2024-03-12T03:40:26.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:26 smithi169 ceph-mon[29344]: 3.6 scrub starts 2024-03-12T03:40:26.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:26 smithi169 ceph-mon[29344]: 3.6 scrub ok 2024-03-12T03:40:26.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:26 smithi169 ceph-mon[29344]: 3.1d scrub starts 2024-03-12T03:40:26.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:26 smithi169 ceph-mon[29344]: 3.1d scrub ok 2024-03-12T03:40:26.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:26 smithi169 ceph-mon[39949]: 3.13 scrub starts 2024-03-12T03:40:26.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:26 smithi169 ceph-mon[39949]: 3.13 scrub ok 2024-03-12T03:40:26.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:26 smithi169 ceph-mon[39949]: 3.6 scrub starts 2024-03-12T03:40:26.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:26 smithi169 ceph-mon[39949]: 3.6 scrub ok 2024-03-12T03:40:26.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:26 smithi169 ceph-mon[39949]: 3.1d scrub starts 2024-03-12T03:40:26.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:26 smithi169 ceph-mon[39949]: 3.1d scrub ok 2024-03-12T03:40:26.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:26.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:26 smithi173 ceph-mon[33544]: 3.13 scrub starts 2024-03-12T03:40:26.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:26 smithi173 ceph-mon[33544]: 3.13 scrub ok 2024-03-12T03:40:26.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:26 smithi173 ceph-mon[33544]: 3.6 scrub starts 2024-03-12T03:40:26.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:26 smithi173 ceph-mon[33544]: 3.6 scrub ok 2024-03-12T03:40:26.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:26 smithi173 ceph-mon[33544]: 3.1d scrub starts 2024-03-12T03:40:26.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:26 smithi173 ceph-mon[33544]: 3.1d scrub ok 2024-03-12T03:40:26.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:26.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:26.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:26.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:26.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:26.999 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 42 15 82060 82045 122.071 83.8125 0.00642528 0.00818568 2024-03-12T03:40:27.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:27.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:27.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:27.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:27.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:27.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:27.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:27.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:27.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:27.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:27.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:27.472 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:27.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:27.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:27.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:27.612 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:27.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:27.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:27 smithi169 ceph-mon[29344]: pgmap v330: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 76 MiB/s wr, 1.21k op/s; 28237/225963 objects degraded (12.496%) 2024-03-12T03:40:27.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:27 smithi169 ceph-mon[29344]: 6.3 scrub starts 2024-03-12T03:40:27.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:27 smithi169 ceph-mon[29344]: 6.3 scrub ok 2024-03-12T03:40:27.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:27 smithi169 ceph-mon[39949]: pgmap v330: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 76 MiB/s wr, 1.21k op/s; 28237/225963 objects degraded (12.496%) 2024-03-12T03:40:27.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:27 smithi169 ceph-mon[39949]: 6.3 scrub starts 2024-03-12T03:40:27.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:27 smithi169 ceph-mon[39949]: 6.3 scrub ok 2024-03-12T03:40:27.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:27 smithi173 ceph-mon[33544]: pgmap v330: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 76 MiB/s wr, 1.21k op/s; 28237/225963 objects degraded (12.496%) 2024-03-12T03:40:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:27 smithi173 ceph-mon[33544]: 6.3 scrub starts 2024-03-12T03:40:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:27 smithi173 ceph-mon[33544]: 6.3 scrub ok 2024-03-12T03:40:27.742 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:27.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:27.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:27.883 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:27.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:27.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:27.999 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 43 16 83409 83393 121.19 84.25 0.020199 0.00824668 2024-03-12T03:40:28.019 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:28.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:28.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:28.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:28.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:28.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:28.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:28.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:28.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:28.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-12T03:40:28.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:28.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:28.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:28.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:28.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:28.605 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:28.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:28.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:28.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:28 smithi169 ceph-mon[29344]: 3.1f scrub starts 2024-03-12T03:40:28.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:28 smithi169 ceph-mon[29344]: 3.1f scrub ok 2024-03-12T03:40:28.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:28 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 28237/225963 objects degraded (12.496%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:28.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:28 smithi169 ceph-mon[29344]: osdmap e218: 8 total, 7 up, 8 in 2024-03-12T03:40:28.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:28 smithi169 ceph-mon[39949]: 3.1f scrub starts 2024-03-12T03:40:28.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:28 smithi169 ceph-mon[39949]: 3.1f scrub ok 2024-03-12T03:40:28.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:28 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 28237/225963 objects degraded (12.496%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:28.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:28 smithi169 ceph-mon[39949]: osdmap e218: 8 total, 7 up, 8 in 2024-03-12T03:40:28.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:28.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:28 smithi173 ceph-mon[33544]: 3.1f scrub starts 2024-03-12T03:40:28.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:28 smithi173 ceph-mon[33544]: 3.1f scrub ok 2024-03-12T03:40:28.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:28 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 28237/225963 objects degraded (12.496%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:28.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:28 smithi173 ceph-mon[33544]: osdmap e218: 8 total, 7 up, 8 in 2024-03-12T03:40:28.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:28.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:28.798 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:40:28.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:28.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:28.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:28.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:29.000 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 44 16 84749 84733 120.339 83.75 0.00454344 0.00830547 2024-03-12T03:40:29.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:29.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:29.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:29.068 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:40:29.102 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:40:29.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:29.136 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:40:29.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:29.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:29.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:29.174 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:40:29.214 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:40:29.250 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:40:29.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:29.285 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:40:29.286 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:40:29.286 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:40:29.286 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:40:29.286 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:40:29.286 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:40:29.286 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:40:29.286 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:40:29.286 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:40:29.312 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:29.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:29.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:29.452 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:29.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:29.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:40:29.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:29 smithi173 ceph-mon[33544]: pgmap v331: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s; 30308/243462 objects degraded (12.449%) 2024-03-12T03:40:29.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:29 smithi173 ceph-mon[33544]: 6.a scrub starts 2024-03-12T03:40:29.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:29 smithi173 ceph-mon[33544]: 6.a scrub ok 2024-03-12T03:40:29.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:29 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1024747861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:29.519 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:29 smithi169 ceph-mon[29344]: pgmap v331: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s; 30308/243462 objects degraded (12.449%) 2024-03-12T03:40:29.519 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:29 smithi169 ceph-mon[29344]: 6.a scrub starts 2024-03-12T03:40:29.519 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:29 smithi169 ceph-mon[29344]: 6.a scrub ok 2024-03-12T03:40:29.519 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:29 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1024747861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:29.520 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:29 smithi169 ceph-mon[39949]: pgmap v331: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s; 30308/243462 objects degraded (12.449%) 2024-03-12T03:40:29.520 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:29 smithi169 ceph-mon[39949]: 6.a scrub starts 2024-03-12T03:40:29.520 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:29 smithi169 ceph-mon[39949]: 6.a scrub ok 2024-03-12T03:40:29.520 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:29 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1024747861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:29.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:29.592 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:29.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:29.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:29.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:29.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:29.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:29.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:29.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:29.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:29.999 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 45 16 86036 86020 119.452 80.4375 0.0196825 0.00836607 2024-03-12T03:40:30.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:30.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:40:30.039 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:40:30.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:30.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:30.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:30.197 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:30.250 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:30.282 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:30.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:30.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:30.424 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:30.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:30.443 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:30 smithi173 ceph-mon[33544]: 4.18 deep-scrub starts 2024-03-12T03:40:30.443 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:30 smithi173 ceph-mon[33544]: 4.18 deep-scrub ok 2024-03-12T03:40:30.443 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:30 smithi173 ceph-mon[33544]: 5.1b scrub starts 2024-03-12T03:40:30.443 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:30 smithi173 ceph-mon[33544]: 5.1b scrub ok 2024-03-12T03:40:30.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:30.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:30.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:30.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:30.622 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:30 smithi169 ceph-mon[29344]: 4.18 deep-scrub starts 2024-03-12T03:40:30.622 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:30 smithi169 ceph-mon[29344]: 4.18 deep-scrub ok 2024-03-12T03:40:30.623 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:30 smithi169 ceph-mon[29344]: 5.1b scrub starts 2024-03-12T03:40:30.623 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:30 smithi169 ceph-mon[29344]: 5.1b scrub ok 2024-03-12T03:40:30.623 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:30 smithi169 ceph-mon[39949]: 4.18 deep-scrub starts 2024-03-12T03:40:30.623 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:30 smithi169 ceph-mon[39949]: 4.18 deep-scrub ok 2024-03-12T03:40:30.623 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:30 smithi169 ceph-mon[39949]: 5.1b scrub starts 2024-03-12T03:40:30.623 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:30 smithi169 ceph-mon[39949]: 5.1b scrub ok 2024-03-12T03:40:30.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:30.707 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:30.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:30.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:30.835 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:30.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:30.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:30.951 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:40:30.951 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:40:30.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:40:30.951 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:40:30.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24359.0:1322 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94067641432704 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:33:49.138906+0000", 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.80568796199998, 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0179765549999999, 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24359", 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 1322 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.138906+0000", 2024-03-12T03:40:30.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.138906+0000", 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.138908+0000", 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6869999999999999e-06 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.138916+0000", 2024-03-12T03:40:30.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8509999999999999e-06 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.138933+0000", 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7326999999999999e-05 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.138943+0000", 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0159e-05 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:30.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.143672+0000", 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047292829999999999 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:49.143684+0000", 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1768e-05 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.151966+0000", 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.008281574 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for missing object", 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.152130+0000", 2024-03-12T03:40:30.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000164096 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.153140+0000", 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001010431 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.153217+0000", 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6908000000000003e-05 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.153476+0000", 2024-03-12T03:40:30.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025934299999999999 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.153537+0000", 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0319000000000003e-05 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:33:50.156883+0000", 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033458089999999999 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.957 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:29236 7.0 7:065c5898:::benchmark_data_smithi169_79822_object29235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:43.518470+0000", 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 287.42612355099999, 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13041738999999999, 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29236 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518470+0000", 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518470+0000", 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518480+0000", 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4930000000000004e-06 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518535+0000", 2024-03-12T03:40:30.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4787999999999998e-05 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518551+0000", 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5985e-05 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518558+0000", 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7389999999999999e-06 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518614+0000", 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5525000000000001e-05 2024-03-12T03:40:30.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518718+0000", 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103692 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518806+0000", 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7848000000000003e-05 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.520467+0000", 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016614310000000001 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.523282+0000", 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028148890000000001 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.648772+0000", 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12549065300000001 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.648840+0000", 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7738999999999996e-05 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.648888+0000", 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7608000000000002e-05 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:29239 7.0 7:0bde76fb:::benchmark_data_smithi169_79822_object29238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:35:43.518868+0000", 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 287.42572574000002, 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.130465672, 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:30.963 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29239 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518868+0000", 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:30.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518868+0000", 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518869+0000", 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3900000000000004e-07 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518953+0000", 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3907999999999997e-05 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518971+0000", 2024-03-12T03:40:30.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8162999999999999e-05 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.518979+0000", 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8050000000000007e-06 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.519707+0000", 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00072848100000000001 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:30.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.519786+0000", 2024-03-12T03:40:30.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8532e-05 2024-03-12T03:40:30.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:40:30.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.519881+0000", 2024-03-12T03:40:30.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4783000000000004e-05 2024-03-12T03:40:30.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:30.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.520596+0000", 2024-03-12T03:40:30.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000715432 2024-03-12T03:40:30.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.523409+0000", 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028132949999999999 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649240+0000", 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12583079 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649295+0000", 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4503999999999999e-05 2024-03-12T03:40:30.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:35:43.649334+0000", 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9339999999999999e-05 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107734 7.10 7:09752c19:::benchmark_data_smithi169_79822_object107733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.406825+0000", 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 222.53776857099999, 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90505219699999995, 2024-03-12T03:40:30.969 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107734 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406825+0000", 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406825+0000", 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406827+0000", 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6139999999999999e-06 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406856+0000", 2024-03-12T03:40:30.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9417999999999999e-05 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406870+0000", 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3414999999999999e-05 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406881+0000", 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1232e-05 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406940+0000", 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8978999999999999e-05 2024-03-12T03:40:30.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406951+0000", 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0831e-05 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.301975+0000", 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89502379700000001 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.302104+0000", 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012923399999999999 2024-03-12T03:40:30.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.302398+0000", 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029388400000000001 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.303969+0000", 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001571476 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.309180+0000", 2024-03-12T03:40:30.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0052102030000000001 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.311767+0000", 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025875160000000002 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.311830+0000", 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2638000000000006e-05 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.311878+0000", 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7960000000000002e-05 2024-03-12T03:40:30.975 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:123455 7.0 7:04f681fe:::benchmark_data_smithi169_79822_object6038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:08.405591+0000", 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 202.539003347, 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15875210000000001, 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 123455 2024-03-12T03:40:30.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.977 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:30.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:30.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405591+0000", 2024-03-12T03:40:30.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405591+0000", 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405592+0000", 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.44e-06 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405599+0000", 2024-03-12T03:40:30.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6950000000000003e-06 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405610+0000", 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1488000000000001e-05 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405619+0000", 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2619999999999998e-06 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405664+0000", 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5083999999999997e-05 2024-03-12T03:40:30.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.405765+0000", 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010178099999999999 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.406019+0000", 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025336500000000002 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.407608+0000", 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015897179999999999 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.408103+0000", 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00049444199999999999 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.564247+0000", 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.156143808 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.564293+0000", 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6502999999999998e-05 2024-03-12T03:40:30.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:08.564343+0000", 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9514000000000001e-05 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:182823 7.e 7:70e611a8:::benchmark_data_smithi169_79822_object65406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:23.250919+0000", 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 127.69367538, 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.095654971000000005, 2024-03-12T03:40:30.982 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 182823 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.250919+0000", 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.250919+0000", 2024-03-12T03:40:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.250919+0000", 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7999999999999995e-07 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.250927+0000", 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9589999999999995e-06 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.250935+0000", 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1710000000000002e-06 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.250942+0000", 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8140000000000003e-06 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.253535+0000", 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025926899999999999 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:30.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.253665+0000", 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000129596 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2,3", 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.253816+0000", 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000151163 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.256187+0000", 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023712749999999999 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.259117+0000", 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002929564 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.259166+0000", 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9113000000000002e-05 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.346366+0000", 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.087200387000000004 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.346435+0000", 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9321999999999998e-05 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:23.346574+0000", 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013823700000000001 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:188240 7.0 7:0288c969:::benchmark_data_smithi169_79822_object70823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:29.346892+0000", 2024-03-12T03:40:30.988 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.59770208899999, 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12420619200000001, 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 188240 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.346892+0000", 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.346892+0000", 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.346893+0000", 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.437e-06 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.346898+0000", 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0440000000000003e-06 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.346913+0000", 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4329e-05 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.346921+0000", 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9980000000000003e-06 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.346976+0000", 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5433000000000003e-05 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.347087+0000", 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110776 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.347437+0000", 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034968299999999999 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.352694+0000", 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0052572419999999996 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.354559+0000", 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018654570000000001 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.470989+0000", 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.116429356 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.471048+0000", 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9676000000000003e-05 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:30.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.471098+0000", 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9761e-05 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:188248 7.e 7:7946d1e4:::benchmark_data_smithi169_79822_object70831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:29.349277+0000", 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.595317364, 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12366983400000001, 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 188248 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.349277+0000", 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.349277+0000", 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:30.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.349278+0000", 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.344e-06 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.349282+0000", 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4920000000000004e-06 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.349297+0000", 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.454e-05 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.349305+0000", 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7519999999999996e-06 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.351475+0000", 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021704490000000001 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.351601+0000", 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012587199999999999 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2,3", 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.351754+0000", 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000152523 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.353993+0000", 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022393140000000001 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.354732+0000", 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073932299999999996 2024-03-12T03:40:30.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.355307+0000", 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057454200000000004 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.472721+0000", 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.117414308 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.472825+0000", 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010402 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:30.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:29.472946+0000", 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012135500000000001 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:189349 7.e 7:78c2a40a:::benchmark_data_smithi169_79822_object71932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:30.631242+0000", 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 120.313351903, 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19704829900000001, 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:31.000 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 189349 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631242+0000", 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631242+0000", 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631243+0000", 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.15e-06 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631249+0000", 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5809999999999996e-06 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631260+0000", 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0712e-05 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:31.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631267+0000", 2024-03-12T03:40:31.003 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 46 16 87194 87178 118.429 72.375 0.0129988 0.00843905 2024-03-12T03:40:31.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.277e-06 2024-03-12T03:40:31.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631324+0000", 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7567e-05 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631472+0000", 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014747800000000001 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2,3", 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631643+0000", 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000171435 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.635257+0000", 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036141390000000002 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.636921+0000", 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001663072 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.637788+0000", 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086715000000000002 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.828001+0000", 2024-03-12T03:40:31.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19021288 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.828134+0000", 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000133619 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.828290+0000", 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015623899999999999 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:189350 7.e 7:7b91d00a:::benchmark_data_smithi169_79822_object71933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:30.633610+0000", 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 120.310983995, 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19495178699999999, 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 189350 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633610+0000", 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633610+0000", 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633611+0000", 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6400000000000001e-07 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633617+0000", 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2450000000000003e-06 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633628+0000", 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0404999999999999e-05 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633636+0000", 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3969999999999994e-06 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633748+0000", 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112451 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633883+0000", 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000134535 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2,3", 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.634034+0000", 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000150956 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.635379+0000", 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013455209999999999 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.636983+0000", 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016039610000000001 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.637883+0000", 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00089951699999999996 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.828366+0000", 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19048279300000001 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.828439+0000", 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3374000000000006e-05 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.828562+0000", 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012276799999999999 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.012 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:189788 7.0 7:03f1b61e:::benchmark_data_smithi169_79822_object72371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:31.267688+0000", 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 119.67690614599999, 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15011001099999999, 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 189788 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:31.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.267688+0000", 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.267688+0000", 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.267689+0000", 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.598e-06 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.267695+0000", 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1909999999999999e-06 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.267708+0000", 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2853999999999999e-05 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.267716+0000", 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1179999999999992e-06 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.268920+0000", 2024-03-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012045960000000001 2024-03-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:31.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:31.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.269044+0000", 2024-03-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123924 2024-03-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.269406+0000", 2024-03-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036179700000000002 2024-03-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.271879+0000", 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024729880000000002 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.274787+0000", 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029085230000000001 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.417690+0000", 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14290281299999999 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:31.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.417751+0000", 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0807999999999998e-05 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.417798+0000", 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6801000000000001e-05 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:190106 7.e 7:7676f7d0:::benchmark_data_smithi169_79822_object72689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:31.650786+0000", 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 119.293808109, 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12671031099999999, 2024-03-12T03:40:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 190106 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.650786+0000", 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.650786+0000", 2024-03-12T03:40:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.650787+0000", 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.451e-06 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.650793+0000", 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5430000000000001e-06 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:31.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.650819+0000", 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6157e-05 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.650835+0000", 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5892999999999998e-05 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.651048+0000", 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000212776 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.651155+0000", 2024-03-12T03:40:31.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010725 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2,3", 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.651289+0000", 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013358900000000001 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.654295+0000", 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0030061990000000002 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.654890+0000", 2024-03-12T03:40:31.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000595328 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.661135+0000", 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0062453769999999999 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.777278+0000", 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.116142287 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:31.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.777351+0000", 2024-03-12T03:40:31.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3079999999999998e-05 2024-03-12T03:40:31.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.777496+0000", 2024-03-12T03:40:31.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014538100000000001 2024-03-12T03:40:31.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.026 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:190119 7.e 7:7d9077f1:::benchmark_data_smithi169_79822_object72702:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:31.658941+0000", 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 119.28565339399999, 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.118723865, 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 190119 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.027 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.658941+0000", 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.658941+0000", 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.658942+0000", 2024-03-12T03:40:31.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3850000000000001e-06 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.658948+0000", 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5969999999999999e-06 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.658965+0000", 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7300999999999999e-05 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:31.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.658980+0000", 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5041999999999999e-05 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.659057+0000", 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7513000000000001e-05 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.659158+0000", 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010026000000000001 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2,3", 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.659255+0000", 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7241e-05 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.662412+0000", 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0031575050000000001 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.663715+0000", 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013030380000000001 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.690545+0000", 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026829201 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.777547+0000", 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.087002654999999998 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.777586+0000", 2024-03-12T03:40:31.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8235999999999998e-05 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.777664+0000", 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8891000000000001e-05 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:190120 7.e 7:763297bf:::benchmark_data_smithi169_79822_object72703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:31.660561+0000", 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 119.284032854, 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.117326233, 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:31.033 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 190120 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.660561+0000", 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.660561+0000", 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.660562+0000", 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.82e-07 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.660566+0000", 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7740000000000002e-06 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.660574+0000", 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2549999999999997e-06 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.660579+0000", 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0560000000000004e-06 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.661170+0000", 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059158099999999999 2024-03-12T03:40:31.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.661264+0000", 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3646000000000004e-05 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2,3", 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.661349+0000", 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4770000000000003e-05 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.662492+0000", 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001142948 2024-03-12T03:40:31.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.663745+0000", 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001253311 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.693001+0000", 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.029256250000000001 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.777709+0000", 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.084707831999999997 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.777770+0000", 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0850999999999997e-05 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:31.777887+0000", 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117377 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:190613 7.e 7:745b3594:::benchmark_data_smithi169_79822_object73196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:32.197405+0000", 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 118.74718905, 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13463789300000001, 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 190613 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:31.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.197405+0000", 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.197405+0000", 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.197406+0000", 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.265e-06 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:31.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.197411+0000", 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6240000000000001e-06 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.197421+0000", 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.064e-05 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.197436+0000", 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4943000000000001e-05 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.197495+0000", 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8900000000000002e-05 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.197648+0000", 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000152647 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2,3", 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.197812+0000", 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016416400000000001 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:31.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.200600+0000", 2024-03-12T03:40:31.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027874810000000001 2024-03-12T03:40:31.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.200988+0000", 2024-03-12T03:40:31.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038877200000000001 2024-03-12T03:40:31.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.268243+0000", 2024-03-12T03:40:31.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.067254653999999997 2024-03-12T03:40:31.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.331915+0000", 2024-03-12T03:40:31.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.063672212000000006 2024-03-12T03:40:31.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.331958+0000", 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2259999999999999e-05 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.332043+0000", 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5331e-05 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.046 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:190618 7.e 7:7de79505:::benchmark_data_smithi169_79822_object73201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:32.201980+0000", 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 118.742614284, 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13020414899999999, 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 190618 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.201980+0000", 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.201980+0000", 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.201981+0000", 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5510000000000001e-06 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.201987+0000", 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9000000000000003e-06 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.202001+0000", 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3579e-05 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.202009+0000", 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9719999999999992e-06 2024-03-12T03:40:31.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:31.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.202062+0000", 2024-03-12T03:40:31.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2779999999999999e-05 2024-03-12T03:40:31.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:31.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.202212+0000", 2024-03-12T03:40:31.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015040600000000001 2024-03-12T03:40:31.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2,3", 2024-03-12T03:40:31.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.202349+0000", 2024-03-12T03:40:31.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000136833 2024-03-12T03:40:31.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:31.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.204783+0000", 2024-03-12T03:40:31.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002434009 2024-03-12T03:40:31.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.205029+0000", 2024-03-12T03:40:31.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024637900000000002 2024-03-12T03:40:31.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.268307+0000", 2024-03-12T03:40:31.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.063278198999999993 2024-03-12T03:40:31.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.332078+0000", 2024-03-12T03:40:31.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.063770460000000001 2024-03-12T03:40:31.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:31.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.332113+0000", 2024-03-12T03:40:31.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4748e-05 2024-03-12T03:40:31.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:32.332184+0000", 2024-03-12T03:40:31.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1333000000000003e-05 2024-03-12T03:40:31.055 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.055 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.055 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:211752 7.e 7:7301e914:::benchmark_data_smithi169_79822_object94335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:52.541164+0000", 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 98.403430181999994, 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.097028918000000006, 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 211752 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541164+0000", 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541164+0000", 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:31.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541165+0000", 2024-03-12T03:40:31.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.46e-06 2024-03-12T03:40:31.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:31.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541172+0000", 2024-03-12T03:40:31.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0190000000000001e-06 2024-03-12T03:40:31.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:31.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541191+0000", 2024-03-12T03:40:31.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8884000000000001e-05 2024-03-12T03:40:31.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541201+0000", 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8840000000000006e-06 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541374+0000", 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000172912 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:31.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541747+0000", 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00037311199999999999 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541863+0000", 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000116366 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.546773+0000", 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049092060000000002 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.597452+0000", 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.050679305000000001 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.638074+0000", 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.040622221 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.638140+0000", 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5607999999999999e-05 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.638193+0000", 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2941e-05 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:211753 7.e 7:7f632651:::benchmark_data_smithi169_79822_object94336:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:52.541224+0000", 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 98.40337049, 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.097059287999999994, 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:31.062 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 211753 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541224+0000", 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541224+0000", 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541224+0000", 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7899999999999999e-07 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541227+0000", 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7480000000000002e-06 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541236+0000", 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9539999999999993e-06 2024-03-12T03:40:31.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.541242+0000", 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0349999999999998e-06 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.544231+0000", 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029897499999999998 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.544352+0000", 2024-03-12T03:40:31.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012096599999999999 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.544488+0000", 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000135198 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.547063+0000", 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002574977 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.597502+0000", 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.050439802999999998 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.638226+0000", 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.040723720999999997 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.638255+0000", 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9434e-05 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:52.638283+0000", 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7322999999999999e-05 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:213887 7.e 7:7531dbe0:::benchmark_data_smithi169_79822_object96470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:54.420493+0000", 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 96.524101005999995, 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102613071, 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:31.068 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 213887 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.420493+0000", 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.420493+0000", 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.420494+0000", 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4929999999999999e-06 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.420502+0000", 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8660000000000006e-06 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.420517+0000", 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4479e-05 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.420525+0000", 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6839999999999996e-06 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.420800+0000", 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00027514599999999998 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.420922+0000", 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012197300000000001 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.421031+0000", 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000109059 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.426904+0000", 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058731149999999999 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.516688+0000", 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.089784535999999998 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.523003+0000", 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006314231 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.523060+0000", 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7052999999999997e-05 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.523106+0000", 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6436000000000001e-05 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:213891 7.e 7:7a552860:::benchmark_data_smithi169_79822_object96474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:54.423886+0000", 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 96.520708142999993, 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.099346023000000006, 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 213891 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.423886+0000", 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.423886+0000", 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.423887+0000", 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.359e-06 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.423894+0000", 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.302e-06 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.423909+0000", 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.562e-05 2024-03-12T03:40:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.423919+0000", 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5200000000000003e-06 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.423982+0000", 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3452000000000002e-05 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.424107+0000", 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000124933 2024-03-12T03:40:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.424215+0000", 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010820299999999999 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.426911+0000", 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026955350000000002 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.516726+0000", 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.089815584000000004 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.523157+0000", 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00643086 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.523193+0000", 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6256e-05 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:54.523232+0000", 2024-03-12T03:40:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8399000000000003e-05 2024-03-12T03:40:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:31.080 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:40:31.119 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:31.154 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:31.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:31.256 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:31.298 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:31.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:31.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:31.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:31.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:31.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:31.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:31.540 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-12T03:40:31.540 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-03-12T03:40:31.540 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-03-12T03:40:31.541 DEBUG:teuthology.orchestra.run.smithi169:osd.3> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3.service 2024-03-12T03:40:31.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl start ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3 2024-03-12T03:40:31.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:31.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:31.629 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:31.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[29344]: 5.11 scrub starts 2024-03-12T03:40:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[29344]: 5.11 scrub ok 2024-03-12T03:40:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[29344]: pgmap v333: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 771 B/s rd, 63 MiB/s wr, 1.00k op/s; 30308/243462 objects degraded (12.449%) 2024-03-12T03:40:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[29344]: 3.1a scrub starts 2024-03-12T03:40:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[29344]: 3.1a scrub ok 2024-03-12T03:40:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[29344]: 5.19 scrub starts 2024-03-12T03:40:31.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[29344]: 5.19 scrub ok 2024-03-12T03:40:31.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[39949]: 5.11 scrub starts 2024-03-12T03:40:31.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[39949]: 5.11 scrub ok 2024-03-12T03:40:31.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[39949]: pgmap v333: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 771 B/s rd, 63 MiB/s wr, 1.00k op/s; 30308/243462 objects degraded (12.449%) 2024-03-12T03:40:31.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[39949]: 3.1a scrub starts 2024-03-12T03:40:31.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[39949]: 3.1a scrub ok 2024-03-12T03:40:31.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:31.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:31.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[39949]: 5.19 scrub starts 2024-03-12T03:40:31.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-mon[39949]: 5.19 scrub ok 2024-03-12T03:40:31.676 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:31 smithi169 systemd[1]: Starting Ceph osd.3 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:40:31.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:31.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:31 smithi173 ceph-mon[33544]: 5.11 scrub starts 2024-03-12T03:40:31.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:31 smithi173 ceph-mon[33544]: 5.11 scrub ok 2024-03-12T03:40:31.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:31 smithi173 ceph-mon[33544]: pgmap v333: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 771 B/s rd, 63 MiB/s wr, 1.00k op/s; 30308/243462 objects degraded (12.449%) 2024-03-12T03:40:31.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:31 smithi173 ceph-mon[33544]: 3.1a scrub starts 2024-03-12T03:40:31.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:31 smithi173 ceph-mon[33544]: 3.1a scrub ok 2024-03-12T03:40:31.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:31 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:31.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:31.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:31 smithi173 ceph-mon[33544]: 5.19 scrub starts 2024-03-12T03:40:31.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:31 smithi173 ceph-mon[33544]: 5.19 scrub ok 2024-03-12T03:40:31.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:31.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:31.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:31.916 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:31.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:32.000 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 47 15 88478 88463 117.617 80.3125 0.010247 0.00849619 2024-03-12T03:40:32.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:32.040 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:40:32.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-12T03:40:32.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:32.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:32.082 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:40:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:40:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:40:32.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:32.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:32.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:32.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:32.372 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:32.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:32.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:32.458 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:40:32.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:32.527 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:32.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:32.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:32.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:32.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:32.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:32 smithi169 ceph-mon[29344]: 6.15 scrub starts 2024-03-12T03:40:32.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:32 smithi169 ceph-mon[29344]: 6.15 scrub ok 2024-03-12T03:40:32.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:32 smithi169 ceph-mon[29344]: 6.18 scrub starts 2024-03-12T03:40:32.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:32 smithi169 ceph-mon[29344]: 6.18 scrub ok 2024-03-12T03:40:32.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:32 smithi169 ceph-mon[39949]: 6.15 scrub starts 2024-03-12T03:40:32.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:32 smithi169 ceph-mon[39949]: 6.15 scrub ok 2024-03-12T03:40:32.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:32 smithi169 ceph-mon[39949]: 6.18 scrub starts 2024-03-12T03:40:32.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:32 smithi169 ceph-mon[39949]: 6.18 scrub ok 2024-03-12T03:40:32.674 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:32 smithi169 podman[112423]: 2024-03-12 03:40:32.290895285 +0000 UTC m=+0.118500547 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:40:32.674 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:32 smithi169 podman[112423]: 2024-03-12 03:40:32.442356949 +0000 UTC m=+0.269962204 container create f03cab09a2b41e2a42b212ccd2b9864fde7323d455f3461dd5da771af1be1240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5) 2024-03-12T03:40:32.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:32 smithi173 ceph-mon[33544]: 6.15 scrub starts 2024-03-12T03:40:32.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:32 smithi173 ceph-mon[33544]: 6.15 scrub ok 2024-03-12T03:40:32.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:32 smithi173 ceph-mon[33544]: 6.18 scrub starts 2024-03-12T03:40:32.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:32 smithi173 ceph-mon[33544]: 6.18 scrub ok 2024-03-12T03:40:32.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:40:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:40:32.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:32.813 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:32.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:32.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:32.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:32.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:33.000 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 48 16 89744 89728 116.813 79.0625 0.0104298 0.00855635 2024-03-12T03:40:33.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:33.091 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:33.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:33.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:33.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:33.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:33.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:33.326 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 podman[112423]: 2024-03-12 03:40:33.213265621 +0000 UTC m=+1.040870888 container init f03cab09a2b41e2a42b212ccd2b9864fde7323d455f3461dd5da771af1be1240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-12T03:40:33.326 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 podman[112423]: 2024-03-12 03:40:33.220969189 +0000 UTC m=+1.048574449 container start f03cab09a2b41e2a42b212ccd2b9864fde7323d455f3461dd5da771af1be1240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-12T03:40:33.326 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 podman[112423]: 2024-03-12 03:40:33.324341115 +0000 UTC m=+1.151946373 container attach f03cab09a2b41e2a42b212ccd2b9864fde7323d455f3461dd5da771af1be1240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-03-12T03:40:33.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:33.355 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:33.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:33.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:33.423 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:33.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:33.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:33.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:33.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:33.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:33.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:33.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[29344]: pgmap v334: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 32726/262851 objects degraded (12.450%) 2024-03-12T03:40:33.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[29344]: 1.2 scrub starts 2024-03-12T03:40:33.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[29344]: 1.2 scrub ok 2024-03-12T03:40:33.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[29344]: 3.9 scrub starts 2024-03-12T03:40:33.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[29344]: 3.9 scrub ok 2024-03-12T03:40:33.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:33.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[29344]: 3.4 deep-scrub starts 2024-03-12T03:40:33.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[29344]: 3.4 deep-scrub ok 2024-03-12T03:40:33.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 32726/262851 objects degraded (12.450%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:33.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[39949]: pgmap v334: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 32726/262851 objects degraded (12.450%) 2024-03-12T03:40:33.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[39949]: 1.2 scrub starts 2024-03-12T03:40:33.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[39949]: 1.2 scrub ok 2024-03-12T03:40:33.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[39949]: 3.9 scrub starts 2024-03-12T03:40:33.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[39949]: 3.9 scrub ok 2024-03-12T03:40:33.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:33.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[39949]: 3.4 deep-scrub starts 2024-03-12T03:40:33.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[39949]: 3.4 deep-scrub ok 2024-03-12T03:40:33.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 32726/262851 objects degraded (12.450%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:33.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:33 smithi173 ceph-mon[33544]: pgmap v334: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 32726/262851 objects degraded (12.450%) 2024-03-12T03:40:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:33 smithi173 ceph-mon[33544]: 1.2 scrub starts 2024-03-12T03:40:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:33 smithi173 ceph-mon[33544]: 1.2 scrub ok 2024-03-12T03:40:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:33 smithi173 ceph-mon[33544]: 3.9 scrub starts 2024-03-12T03:40:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:33 smithi173 ceph-mon[33544]: 3.9 scrub ok 2024-03-12T03:40:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:33 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:33 smithi173 ceph-mon[33544]: 3.4 deep-scrub starts 2024-03-12T03:40:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:33 smithi173 ceph-mon[33544]: 3.4 deep-scrub ok 2024-03-12T03:40:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:33 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 32726/262851 objects degraded (12.450%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:33.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:33.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:33.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:33.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:33.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:34.000 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 49 16 90957 90941 115.977 75.8125 0.00955554 0.00861755 2024-03-12T03:40:34.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:34.051 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:34.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:34.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:34.174 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[112468]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-12T03:40:34.174 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[112468]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-12T03:40:34.174 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 bash[112423]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-12T03:40:34.174 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 bash[112423]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-12T03:40:34.174 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 bash[112423]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-12T03:40:34.174 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 bash[112423]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-12T03:40:34.174 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 bash[112423]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-12T03:40:34.174 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 bash[112423]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-12T03:40:34.174 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 bash[112423]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-12T03:40:34.174 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[112468]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-12T03:40:34.174 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[112468]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-12T03:40:34.174 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[112468]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-12T03:40:34.175 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[112468]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-12T03:40:34.175 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:33 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[112468]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-12T03:40:34.191 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:34.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:34.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:34.323 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:34.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:34.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:34.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:34.463 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:34.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:34.467 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:40:34.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:34.571 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:34 smithi173 ceph-mon[33544]: 5.2 deep-scrub starts 2024-03-12T03:40:34.571 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:34 smithi173 ceph-mon[33544]: 5.2 deep-scrub ok 2024-03-12T03:40:34.571 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:34 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4264022124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:40:34.571 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:34 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:40:34.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:34.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:34.709 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:34.717 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:34 smithi169 ceph-mon[29344]: 5.2 deep-scrub starts 2024-03-12T03:40:34.717 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:34 smithi169 ceph-mon[29344]: 5.2 deep-scrub ok 2024-03-12T03:40:34.717 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:34 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4264022124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:40:34.717 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:34 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:40:34.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:34 smithi169 ceph-mon[39949]: 5.2 deep-scrub starts 2024-03-12T03:40:34.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:34 smithi169 ceph-mon[39949]: 5.2 deep-scrub ok 2024-03-12T03:40:34.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:34 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4264022124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:40:34.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:34 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:40:34.744 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:34.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:34.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:34.884 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:34.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:34.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:35.000 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 50 16 92314 92298 115.353 84.8125 0.0137638 0.00866422 2024-03-12T03:40:35.014 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:34 smithi169 podman[112423]: 2024-03-12 03:40:34.799110402 +0000 UTC m=+2.626715666 container died f03cab09a2b41e2a42b212ccd2b9864fde7323d455f3461dd5da771af1be1240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:40:35.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:35.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:35.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:35.038 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:40:35.038 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:40:35.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:35.038 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24914.0:92360 8.7 e219/203)", 2024-03-12T03:40:35.038 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:40:35.035737+0000", 2024-03-12T03:40:35.038 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.6408000000000003e-05, 2024-03-12T03:40:35.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012971400000000001, 2024-03-12T03:40:35.038 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:35.038 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:40:35.035737+0000", 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:40:35.035737+0000", 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:40:35.035738+0000", 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0699999999999996e-07 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:40:35.035797+0000", 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9067999999999998e-05 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:40:35.035812+0000", 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4959e-05 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:35.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:40:35.035821+0000", 2024-03-12T03:40:35.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5739999999999996e-06 2024-03-12T03:40:35.041 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:35.041 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:35.041 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:35.041 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:35.041 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-03-12T03:40:35.041 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-03-12T03:40:35.041 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:40:35.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:35.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:35.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:35.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:35.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:35.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:35.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:35.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:35.452 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:35.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:35.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:35.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:35.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:35.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:35.729 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:35.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:35.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:35.734 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:35 smithi173 ceph-mon[33544]: pgmap v335: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 32726/262851 objects degraded (12.450%) 2024-03-12T03:40:35.735 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:35 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:40:35.735 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:35 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:40:35.735 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:35 smithi173 ceph-mon[33544]: osdmap e219: 8 total, 7 up, 8 in 2024-03-12T03:40:35.754 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:35 smithi169 ceph-mon[29344]: pgmap v335: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 32726/262851 objects degraded (12.450%) 2024-03-12T03:40:35.754 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:35 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:40:35.754 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:35 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:40:35.755 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:35 smithi169 ceph-mon[29344]: osdmap e219: 8 total, 7 up, 8 in 2024-03-12T03:40:35.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:35 smithi169 ceph-mon[39949]: pgmap v335: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 32726/262851 objects degraded (12.450%) 2024-03-12T03:40:35.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:35 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:40:35.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:35 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:40:35.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:35 smithi169 ceph-mon[39949]: osdmap e219: 8 total, 7 up, 8 in 2024-03-12T03:40:35.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-12T03:40:35.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:35.892 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:35.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:35.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:36.000 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 51 16 93693 93677 114.781 86.1875 0.00617943 0.00870785 2024-03-12T03:40:36.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:36.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:36.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:36.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:36.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:36.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:36.197 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:40:36.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:36.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:36.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:36.424 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:36 smithi169 podman[112681]: 2024-03-12 03:40:36.084586073 +0000 UTC m=+2.108505215 container remove f03cab09a2b41e2a42b212ccd2b9864fde7323d455f3461dd5da771af1be1240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-12T03:40:36.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:36.471 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:36.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:36.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:36.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:36.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:36.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:36.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:36.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:36.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:36.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:36.877 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:36.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:36.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:37.000 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 52 15 94962 94947 114.1 79.375 0.0207789 0.00875971 2024-03-12T03:40:37.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:37.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:37.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:37.152 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:37.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:37.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:37.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:37.290 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:37.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:37.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:37.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:37.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:37.528 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:40:37.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:37.552 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:40:37.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:37.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:37.583 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:40:37.583 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:37 smithi169 ceph-mon[39949]: pgmap v337: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 912 op/s; 33132/265347 objects degraded (12.486%) 2024-03-12T03:40:37.583 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:37 smithi169 ceph-mon[29344]: pgmap v337: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 912 op/s; 33132/265347 objects degraded (12.486%) 2024-03-12T03:40:37.630 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:40:37.639 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:40:37.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:37 smithi173 ceph-mon[33544]: pgmap v337: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 912 op/s; 33132/265347 objects degraded (12.486%) 2024-03-12T03:40:37.677 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:40:37.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:37.718 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:40:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:37.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:37.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:37.760 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:40:37.760 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:40:37.760 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:40:37.760 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:40:37.760 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:40:37.760 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:40:37.760 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:40:37.760 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:40:37.761 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:40:37.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:37.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:37.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:37.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:37.923 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:37 smithi169 podman[112941]: 2024-03-12 03:40:37.499303148 +0000 UTC m=+0.038795369 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:40:37.924 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:37 smithi169 podman[112941]: 2024-03-12 03:40:37.659386414 +0000 UTC m=+0.198878673 container create d630955caaecd5099827c47c69eba501c9a19f37d2a64b47df5df454feb96214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:40:37.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:38.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:38.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:38.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:38.001 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 53 16 96235 96219 113.447 79.5 0.0147849 0.00880958 2024-03-12T03:40:38.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:38.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:38.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:38.218 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:40:38.218 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:40:38.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:38.276 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:38.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:38.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:38.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:38.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:38.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:38.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:38.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:38.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:38.658 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:38.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:38.697 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:38.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:38 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1078893357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:38 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 33132/265347 objects degraded (12.486%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:38.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:38 smithi173 ceph-mon[33544]: osdmap e220: 8 total, 7 up, 8 in 2024-03-12T03:40:38.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:38.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:38 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1078893357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:38.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:38 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 33132/265347 objects degraded (12.486%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:38.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:38 smithi169 ceph-mon[29344]: osdmap e220: 8 total, 7 up, 8 in 2024-03-12T03:40:38.802 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:38 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1078893357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:38.802 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:38 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 33132/265347 objects degraded (12.486%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:38.802 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:38 smithi169 ceph-mon[39949]: osdmap e220: 8 total, 7 up, 8 in 2024-03-12T03:40:38.829 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:38.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:38.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:38.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:38.967 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:38.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:39.001 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 54 16 97587 97571 112.911 84.5 0.0169527 0.00885164 2024-03-12T03:40:39.067 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:39.104 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:39.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:39.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:39.206 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:39.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:39.240 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:39.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:39.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:39.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:39.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:39.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:39.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:39.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:39.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:39.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:39.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:39.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:39 smithi173 ceph-mon[33544]: pgmap v338: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 35181/282444 objects degraded (12.456%) 2024-03-12T03:40:39.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:39.777 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:39.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:39.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:39.879 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:39.904 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:39.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:39.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:39.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:39 smithi169 ceph-mon[29344]: pgmap v338: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 35181/282444 objects degraded (12.456%) 2024-03-12T03:40:39.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:39 smithi169 ceph-mon[39949]: pgmap v338: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 35181/282444 objects degraded (12.456%) 2024-03-12T03:40:40.001 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 55 16 98684 98668 112.105 68.5625 0.00627078 0.00891545 2024-03-12T03:40:40.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:40.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:40.043 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:40.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:40.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:40.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:40.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:40.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:40.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-12T03:40:40.283 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:40.313 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:40.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:40.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:40.452 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:40.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:40:40.510 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:40:40.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:40.588 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:40.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:40.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:40.668 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:40:40.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:40.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:40.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:40.748 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:40 smithi169 podman[112941]: 2024-03-12 03:40:40.645238338 +0000 UTC m=+3.184730557 container init d630955caaecd5099827c47c69eba501c9a19f37d2a64b47df5df454feb96214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-12T03:40:40.748 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:40 smithi169 podman[112941]: 2024-03-12 03:40:40.650233901 +0000 UTC m=+3.189726119 container start d630955caaecd5099827c47c69eba501c9a19f37d2a64b47df5df454feb96214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:40:40.748 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:40 smithi169 ceph-osd[113216]: -- 172.21.15.169:0/937727714 <== mon.2 v2:172.21.15.169:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55cc238dbba0 con 0x55cc238acc00 2024-03-12T03:40:40.748 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:40 smithi169 ceph-osd[113216]: -- 172.21.15.169:0/937727714 <== mon.2 v2:172.21.15.169:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cc238e3680 con 0x55cc238acc00 2024-03-12T03:40:40.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-12T03:40:40.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:40.848 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:40.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:40.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:40.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:40.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:41.001 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 56 15 99919 99904 111.482 77.25 0.0117724 0.00896515 2024-03-12T03:40:41.015 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:40 smithi169 bash[112941]: d630955caaecd5099827c47c69eba501c9a19f37d2a64b47df5df454feb96214 2024-03-12T03:40:41.015 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:40 smithi169 systemd[1]: Started Ceph osd.3 for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:40:41.026 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:41.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:41.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:41.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:41.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:41.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:41.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:41.293 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[113028]: 2024-03-12T03:40:41.209+0000 7fa23da837c0 -1 Falling back to public interface 2024-03-12T03:40:41.311 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:41.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:41.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:41.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:41.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:41.552 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:41.589 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:41.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:41.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:41 smithi169 ceph-mon[29344]: pgmap v340: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 816 op/s; 35181/282444 objects degraded (12.456%) 2024-03-12T03:40:41.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:41 smithi169 ceph-mon[39949]: pgmap v340: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 816 op/s; 35181/282444 objects degraded (12.456%) 2024-03-12T03:40:41.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:41.729 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:41.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:41.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:41 smithi173 ceph-mon[33544]: pgmap v340: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 816 op/s; 35181/282444 objects degraded (12.456%) 2024-03-12T03:40:41.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:41.861 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:41.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:41.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:42.001 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 57 16 101283 101267 111.021 85.1875 0.0225808 0.00900284 2024-03-12T03:40:42.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:42.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:42.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:42.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:42.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:42.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:40:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:40:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:40:42.243 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:42.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:42.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:42.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:42.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:42.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:42.440 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:42.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:42.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:40:42.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:42.559 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:40:42.559 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:40:42.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:42.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:42.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:42.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:42.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:42.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:42.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:40:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:40:42.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:42.861 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:42.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:42.962 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:43.002 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 58 16 102532 102516 110.452 78.0625 0.00743857 0.00904921 2024-03-12T03:40:43.035 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:43.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:43.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:43.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:43.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:43.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:43.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:43.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:43.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-12T03:40:43.307 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:43.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:43.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:43.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:43.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:43.464 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:43.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:43.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:43.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:43.624 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:43.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:43.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:43 smithi169 ceph-mon[29344]: pgmap v341: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 37500/301185 objects degraded (12.451%) 2024-03-12T03:40:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:43 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:43 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 37500/301185 objects degraded (12.451%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:43 smithi169 ceph-mon[39949]: pgmap v341: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 37500/301185 objects degraded (12.451%) 2024-03-12T03:40:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:43 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:43 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 37500/301185 objects degraded (12.451%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:43.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:43.726 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:40:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:43 smithi173 ceph-mon[33544]: pgmap v341: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 37500/301185 objects degraded (12.451%) 2024-03-12T03:40:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:43 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:43 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 37500/301185 objects degraded (12.451%), 29 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:43.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:43.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:43.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:43.931 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:43.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:43.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:44.001 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 59 16 103571 103555 109.681 64.9375 0.0202101 0.00911284 2024-03-12T03:40:44.033 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:44.069 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:44.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:44.171 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:44.217 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:44.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:44.282 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:44 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[113028]: 2024-03-12T03:40:44.008+0000 7fa23da837c0 -1 osd.3 202 log_to_monitors true 2024-03-12T03:40:44.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:44.354 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:44.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:44.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:44.481 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:40:44.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:44.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:44.540 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:44 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3855590815' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:40:44.540 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:44 smithi169 ceph-mon[29344]: from='osd.3 v2:172.21.15.169:6813/287982149' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:40:44.540 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:40:44 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[113028]: 2024-03-12T03:40:44.535+0000 7fa22eedb700 -1 osd.3 202 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:40:44.541 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:44 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3855590815' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:40:44.541 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:44 smithi169 ceph-mon[39949]: from='osd.3 v2:172.21.15.169:6813/287982149' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:40:44.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:44.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:44.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:44.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:44.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:44 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3855590815' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:40:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:44 smithi173 ceph-mon[33544]: from='osd.3 v2:172.21.15.169:6813/287982149' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:40:44.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:44.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:44.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:44.908 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:44.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:45.001 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:40:45.000124+0000 min lat: 0.00107765 max lat: 0.15041 avg lat: 0.0091598 2024-03-12T03:40:45.001 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:40:45.001 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 60 16 104776 104760 109.108 75.3125 0.00888461 0.0091598 2024-03-12T03:40:45.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:45.042 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:45.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:45.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:45.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:45.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:45.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:45.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:45.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:45.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:45.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:45.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:45.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:45.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:45.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:45.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:45.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:45.723 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:45.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:45 smithi173 ceph-mon[33544]: pgmap v342: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.35k op/s; 37500/301185 objects degraded (12.451%) 2024-03-12T03:40:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:45 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:40:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:45 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3855590815' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:40:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:45 smithi173 ceph-mon[33544]: from='osd.3 v2:172.21.15.169:6813/287982149' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:40:45.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:45 smithi173 ceph-mon[33544]: osdmap e221: 8 total, 7 up, 8 in 2024-03-12T03:40:45.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:45 smithi173 ceph-mon[33544]: from='osd.3 v2:172.21.15.169:6813/287982149' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:40:45.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:45 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:45.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:45 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:45.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[39949]: pgmap v342: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.35k op/s; 37500/301185 objects degraded (12.451%) 2024-03-12T03:40:45.806 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:40:45.806 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3855590815' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:40:45.806 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[39949]: from='osd.3 v2:172.21.15.169:6813/287982149' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:40:45.806 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[39949]: osdmap e221: 8 total, 7 up, 8 in 2024-03-12T03:40:45.806 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[39949]: from='osd.3 v2:172.21.15.169:6813/287982149' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:40:45.806 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:45.806 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:45.806 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[29344]: pgmap v342: 158 pgs: 29 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.35k op/s; 37500/301185 objects degraded (12.451%) 2024-03-12T03:40:45.806 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:40:45.806 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3855590815' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:40:45.807 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[29344]: from='osd.3 v2:172.21.15.169:6813/287982149' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:40:45.807 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[29344]: osdmap e221: 8 total, 7 up, 8 in 2024-03-12T03:40:45.807 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[29344]: from='osd.3 v2:172.21.15.169:6813/287982149' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:40:45.807 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:40:45.807 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:45 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:40:45.824 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:45.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:45.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:45.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:45.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:45.975 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:45.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:46.001 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 61 16 105451 105435 108.011 42.1875 0.00318737 0.00917939 2024-03-12T03:40:46.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:46.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:46.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:46.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:46.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:46.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:46.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:46.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:46.360 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:46.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:46.455 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:40:46.455 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:40:46.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:46.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:46.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:46.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:46.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:46.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:46.624 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:46 smithi169 ceph-mon[29344]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T03:40:46.624 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:46 smithi169 ceph-mon[29344]: osd.3 v2:172.21.15.169:6813/287982149 boot 2024-03-12T03:40:46.624 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:46 smithi169 ceph-mon[29344]: osdmap e222: 8 total, 8 up, 8 in 2024-03-12T03:40:46.624 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:46 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:40:46.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:46 smithi169 ceph-mon[39949]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T03:40:46.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:46 smithi169 ceph-mon[39949]: osd.3 v2:172.21.15.169:6813/287982149 boot 2024-03-12T03:40:46.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:46 smithi169 ceph-mon[39949]: osdmap e222: 8 total, 8 up, 8 in 2024-03-12T03:40:46.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:46 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:40:46.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:46 smithi173 ceph-mon[33544]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T03:40:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:46 smithi173 ceph-mon[33544]: osd.3 v2:172.21.15.169:6813/287982149 boot 2024-03-12T03:40:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:46 smithi173 ceph-mon[33544]: osdmap e222: 8 total, 8 up, 8 in 2024-03-12T03:40:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:46 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:40:46.737 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:46.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:46.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:46.843 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:40:46.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:46.862 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:40:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:46.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:46.881 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:40:46.901 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:40:46.921 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:40:46.940 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:40:46.960 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:40:46.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:46.979 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:40:46.981 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:40:46.981 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:40:46.981 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:40:46.981 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:40:46.981 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:40:46.981 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:40:46.981 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:40:46.981 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:40:46.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:46.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:46.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:47.001 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 62 16 105451 105435 106.269 0 - 0.00917939 2024-03-12T03:40:47.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-12T03:40:47.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:47.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:47.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:47.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:40:47.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:47.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:47.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:47.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:47.328 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:40:47.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:47.407 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:47.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:47.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:47.534 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:47.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:47.614 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:40:47.614 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:40:47.614 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-03-12T03:40:47.614 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:40:47.614 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:47 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:40:47.614 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:47 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:40:47.615 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:47 smithi169 ceph-mon[29344]: pgmap v345: 158 pgs: 11 peering, 24 active+undersized+degraded, 32 active+undersized, 91 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 54 MiB/s wr, 862 op/s; 37478/302151 objects degraded (12.404%) 2024-03-12T03:40:47.615 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:47 smithi169 ceph-mon[29344]: osdmap e223: 8 total, 8 up, 8 in 2024-03-12T03:40:47.615 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:47 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/697335873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:47.615 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:47 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:40:47.615 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:47 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:40:47.615 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:47 smithi169 ceph-mon[39949]: pgmap v345: 158 pgs: 11 peering, 24 active+undersized+degraded, 32 active+undersized, 91 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 54 MiB/s wr, 862 op/s; 37478/302151 objects degraded (12.404%) 2024-03-12T03:40:47.615 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:47 smithi169 ceph-mon[39949]: osdmap e223: 8 total, 8 up, 8 in 2024-03-12T03:40:47.615 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:47 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/697335873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:47.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:47.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:47.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:47.713 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:47 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:40:47.713 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:47 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:40:47.713 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:47 smithi173 ceph-mon[33544]: pgmap v345: 158 pgs: 11 peering, 24 active+undersized+degraded, 32 active+undersized, 91 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 54 MiB/s wr, 862 op/s; 37478/302151 objects degraded (12.404%) 2024-03-12T03:40:47.713 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:47 smithi173 ceph-mon[33544]: osdmap e223: 8 total, 8 up, 8 in 2024-03-12T03:40:47.713 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:47 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/697335873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:47.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:47.784 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:47.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:47.886 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:47.918 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:47.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:48.001 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 63 16 105512 105496 104.643 1.90625 0.00694168 0.00955157 2024-03-12T03:40:48.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:48.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:48.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:48.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:48.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:48.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:48.200 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:48.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:48.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:48.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-12T03:40:48.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:48.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:48.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:48.466 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:48.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:48.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:48.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:48.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:48.609 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:48.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:48.710 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:40:48.757 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:48.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:48.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:40:48.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:48.862 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:48 smithi173 ceph-mon[33544]: 8.b starting backfill to osd.3 from (0'0,0'0] MAX to 221'6450 2024-03-12T03:40:48.862 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:48 smithi173 ceph-mon[33544]: 8.0 starting backfill to osd.3 from (0'0,0'0] MAX to 221'6506 2024-03-12T03:40:48.862 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:48 smithi173 ceph-mon[33544]: osdmap e224: 8 total, 8 up, 8 in 2024-03-12T03:40:48.862 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:48 smithi173 ceph-mon[33544]: 8.8 starting backfill to osd.3 from (0'0,0'0] MAX to 221'6597 2024-03-12T03:40:48.862 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:48 smithi173 ceph-mon[33544]: 8.c starting backfill to osd.3 from (0'0,0'0] MAX to 221'6640 2024-03-12T03:40:48.862 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:48 smithi173 ceph-mon[33544]: 8.7 starting backfill to osd.3 from (0'0,0'0] MAX to 221'6592 2024-03-12T03:40:48.863 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:48 smithi173 ceph-mon[33544]: 8.f starting backfill to osd.3 from (0'0,0'0] MAX to 221'6545 2024-03-12T03:40:48.863 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:48 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 37478/302151 objects degraded (12.404%), 24 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-12T03:40:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:48.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:48.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[29344]: 8.b starting backfill to osd.3 from (0'0,0'0] MAX to 221'6450 2024-03-12T03:40:48.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[29344]: 8.0 starting backfill to osd.3 from (0'0,0'0] MAX to 221'6506 2024-03-12T03:40:48.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[29344]: osdmap e224: 8 total, 8 up, 8 in 2024-03-12T03:40:48.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[29344]: 8.8 starting backfill to osd.3 from (0'0,0'0] MAX to 221'6597 2024-03-12T03:40:48.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[29344]: 8.c starting backfill to osd.3 from (0'0,0'0] MAX to 221'6640 2024-03-12T03:40:48.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[29344]: 8.7 starting backfill to osd.3 from (0'0,0'0] MAX to 221'6592 2024-03-12T03:40:48.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[29344]: 8.f starting backfill to osd.3 from (0'0,0'0] MAX to 221'6545 2024-03-12T03:40:48.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 37478/302151 objects degraded (12.404%), 24 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-12T03:40:48.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[39949]: 8.b starting backfill to osd.3 from (0'0,0'0] MAX to 221'6450 2024-03-12T03:40:48.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[39949]: 8.0 starting backfill to osd.3 from (0'0,0'0] MAX to 221'6506 2024-03-12T03:40:48.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[39949]: osdmap e224: 8 total, 8 up, 8 in 2024-03-12T03:40:48.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[39949]: 8.8 starting backfill to osd.3 from (0'0,0'0] MAX to 221'6597 2024-03-12T03:40:48.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[39949]: 8.c starting backfill to osd.3 from (0'0,0'0] MAX to 221'6640 2024-03-12T03:40:48.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[39949]: 8.7 starting backfill to osd.3 from (0'0,0'0] MAX to 221'6592 2024-03-12T03:40:48.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[39949]: 8.f starting backfill to osd.3 from (0'0,0'0] MAX to 221'6545 2024-03-12T03:40:48.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:48 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 37478/302151 objects degraded (12.404%), 24 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-12T03:40:48.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:49.002 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 64 16 106599 106583 104.068 67.9375 0.0131752 0.00960434 2024-03-12T03:40:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:49.037 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:49.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:49.097 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:49.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:49.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:49.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:49.191 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:40:49.191 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:40:49.191 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:40:49.191 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:40:49.191 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:40:49.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:49.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:49.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:49.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:49.453 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:49.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:49.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:49.584 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:49.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:49.686 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:49.722 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:49.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:49.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:49.805 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:49.824 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:40:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:49.860 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:49.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:40:49.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:49 smithi169 ceph-mon[29344]: pgmap v348: 158 pgs: 39 peering, 13 active+undersized+degraded, 15 active+undersized, 91 active+clean; 6.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 18548/314016 objects degraded (5.907%) 2024-03-12T03:40:49.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:49 smithi169 ceph-mon[39949]: pgmap v348: 158 pgs: 39 peering, 13 active+undersized+degraded, 15 active+undersized, 91 active+clean; 6.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 18548/314016 objects degraded (5.907%) 2024-03-12T03:40:49.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:49.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:49 smithi173 ceph-mon[33544]: pgmap v348: 158 pgs: 39 peering, 13 active+undersized+degraded, 15 active+undersized, 91 active+clean; 6.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 18548/314016 objects degraded (5.907%) 2024-03-12T03:40:50.002 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 65 16 107561 107545 103.391 60.125 0.0133367 0.00966611 2024-03-12T03:40:50.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:50.008 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:50.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:50.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:50.143 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:50.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:50.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:50.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:50.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:50.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:50.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:50.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:50.557 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:50.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:50.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:50.659 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:50.698 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:50.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:50.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:50.836 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:50.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:50.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:50.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:50.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:50.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:50.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:51.003 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 66 16 108453 108437 102.669 55.75 0.062289 0.00972976 2024-03-12T03:40:51.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:40:51.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:51.115 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:51.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:51.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:51.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:51.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:51.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:51.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:51.396 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:51.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:51.496 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:51.528 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:51.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:51.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:51.630 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:40:51.665 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:51.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:40:51.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:51.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:51.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:51.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:51 smithi169 ceph-mon[29344]: pgmap v349: 158 pgs: 39 peering, 3 active+undersized+degraded, 116 active+clean; 6.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 944 B/s rd, 49 MiB/s wr, 789 op/s; 18499/314016 objects degraded (5.891%) 2024-03-12T03:40:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:51 smithi169 ceph-mon[39949]: pgmap v349: 158 pgs: 39 peering, 3 active+undersized+degraded, 116 active+clean; 6.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 944 B/s rd, 49 MiB/s wr, 789 op/s; 18499/314016 objects degraded (5.891%) 2024-03-12T03:40:51.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:51.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:51.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:51 smithi173 ceph-mon[33544]: pgmap v349: 158 pgs: 39 peering, 3 active+undersized+degraded, 116 active+clean; 6.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 944 B/s rd, 49 MiB/s wr, 789 op/s; 18499/314016 objects degraded (5.891%) 2024-03-12T03:40:52.003 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 67 16 109549 109533 102.159 68.5 0.0157534 0.00978445 2024-03-12T03:40:52.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:52.085 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:40:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:40:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:40:52.086 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:52.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:52.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:52.189 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:52.230 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:52.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:52.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-12T03:40:52.331 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:52.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:52.476 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:52.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:52.521 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:52.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:52.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:52.624 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:40:52.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:52.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:52.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:52.760 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:52 smithi169 ceph-mon[29344]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:40:52.760 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:52 smithi169 ceph-mon[39949]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:40:52.774 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:52.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:52.825 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:52.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:52.926 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:40:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:40:52.926 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:52 smithi173 ceph-mon[33544]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:40:52.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:52.974 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:52.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:52.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:53.003 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 68 16 110614 110598 101.636 66.5625 0.00919371 0.00983426 2024-03-12T03:40:53.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:53.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:53.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:53.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:53.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:53.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:53.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:53.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:53.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:53.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:53.517 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:53.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:53.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:53.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:53.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:53.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:53.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:53.751 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:53.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:53.806 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:53.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:53.875 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:53 smithi169 ceph-mon[29344]: pgmap v350: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.6 KiB/s rd, 83 MiB/s wr, 1.34k op/s; 39264/326871 objects degraded (12.012%); 13 MiB/s, 201 objects/s recovering 2024-03-12T03:40:53.876 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:53 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:53.876 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:53 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 39264/326871 objects degraded (12.012%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:53.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:53 smithi169 ceph-mon[39949]: pgmap v350: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.6 KiB/s rd, 83 MiB/s wr, 1.34k op/s; 39264/326871 objects degraded (12.012%); 13 MiB/s, 201 objects/s recovering 2024-03-12T03:40:53.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:53 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:53.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:53 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 39264/326871 objects degraded (12.012%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:53.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:53.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:53.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:53.937 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:53 smithi173 ceph-mon[33544]: pgmap v350: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.6 KiB/s rd, 83 MiB/s wr, 1.34k op/s; 39264/326871 objects degraded (12.012%); 13 MiB/s, 201 objects/s recovering 2024-03-12T03:40:53.937 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:53 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:53.937 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:53 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 39264/326871 objects degraded (12.012%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:54.003 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 69 16 111658 111642 101.108 65.25 0.0468165 0.00988369 2024-03-12T03:40:54.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:54.065 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:54.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:54.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:54.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:54.198 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:54.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:54.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:54.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:54.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:54.431 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:54.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:54.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:54.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:54.606 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:54.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:54.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:54.615 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:40:54.616 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.0 to 1 (10000) 2024-03-12T03:40:54.621 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:40:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:40:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:40:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:40:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:129154 7.15 7:abc81e19:::benchmark_data_smithi169_79822_object11737:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:40:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:15.204199+0000", 2024-03-12T03:40:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 219.41220813699999, 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0467048430000001, 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 129154 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204199+0000", 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204199+0000", 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204201+0000", 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.13e-06 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204205+0000", 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9250000000000003e-06 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204215+0000", 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9799999999999993e-06 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204221+0000", 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8950000000000003e-06 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204265+0000", 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3579999999999999e-05 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204404+0000", 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013930600000000001 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1,2", 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204657+0000", 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025272100000000002 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.205872+0000", 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012149540000000001 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.206748+0000", 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00087584100000000003 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.208928+0000", 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021800690000000002 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.231311+0000", 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.022383713999999999 2024-03-12T03:40:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.231324+0000", 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2982e-05 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.250844+0000", 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.019519925 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.250904+0000", 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9821000000000002e-05 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:129167 7.15 7:a82f6f0d:::benchmark_data_smithi169_79822_object11750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:15.216355+0000", 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 219.40005239000001, 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0349462650000001, 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 129167 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216355+0000", 2024-03-12T03:40:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216355+0000", 2024-03-12T03:40:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216357+0000", 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1009999999999999e-06 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216363+0000", 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4940000000000003e-06 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216375+0000", 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2055999999999999e-05 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216384+0000", 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1279999999999997e-06 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216440+0000", 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6094999999999997e-05 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216568+0000", 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012828199999999999 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1,2", 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216853+0000", 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028503600000000001 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.218510+0000", 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016566549999999999 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.220906+0000", 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023956540000000001 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.231461+0000", 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010555787000000001 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.231472+0000", 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1031e-05 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.251263+0000", 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0197909190000001 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.251301+0000", 2024-03-12T03:40:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8027e-05 2024-03-12T03:40:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:129180 7.15 7:a934eea3:::benchmark_data_smithi169_79822_object11763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:15.237990+0000", 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 219.37841732800001, 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0297124339999999, 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 129180 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.237990+0000", 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.237990+0000", 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.237992+0000", 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.497e-06 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.237997+0000", 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8030000000000002e-06 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.238013+0000", 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5271e-05 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.238021+0000", 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9889999999999992e-06 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.238076+0000", 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5757999999999997e-05 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.238084+0000", 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8599999999999993e-06 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.251326+0000", 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0132414750000001 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.251411+0000", 2024-03-12T03:40:54.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5657999999999999e-05 2024-03-12T03:40:54.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2", 2024-03-12T03:40:54.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.251593+0000", 2024-03-12T03:40:54.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018110700000000001 2024-03-12T03:40:54.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.255444+0000", 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038511219999999998 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.263712+0000", 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0082679350000000006 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267631+0000", 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003919769 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267663+0000", 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1665999999999998e-05 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267703+0000", 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9524000000000003e-05 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:129218 7.15 7:ac84363a:::benchmark_data_smithi169_79822_object11801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:15.278835+0000", 2024-03-12T03:40:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 219.337572029, 2024-03-12T03:40:54.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98898992100000005, 2024-03-12T03:40:54.642 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.642 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.642 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.642 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.642 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.642 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 129218 2024-03-12T03:40:54.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.642 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278835+0000", 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278835+0000", 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278837+0000", 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.443e-06 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278842+0000", 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1410000000000004e-06 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278856+0000", 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4423000000000001e-05 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278865+0000", 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4409999999999993e-06 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278919+0000", 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4038e-05 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278927+0000", 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2900000000000002e-06 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.252599+0000", 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97367206100000003 2024-03-12T03:40:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.252718+0000", 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118445 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2", 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.252881+0000", 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016331700000000001 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.257985+0000", 2024-03-12T03:40:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0051034950000000004 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.265507+0000", 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0075223210000000002 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267745+0000", 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022385999999999999 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267790+0000", 2024-03-12T03:40:54.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4743000000000002e-05 2024-03-12T03:40:54.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267825+0000", 2024-03-12T03:40:54.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5163000000000002e-05 2024-03-12T03:40:54.650 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.650 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.650 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.650 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:129221 7.15 7:abe42fdc:::benchmark_data_smithi169_79822_object11804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-12T03:40:54.650 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:15.285585+0000", 2024-03-12T03:40:54.650 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 219.33082300000001, 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98232308099999999, 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 129221 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285585+0000", 2024-03-12T03:40:54.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285585+0000", 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285586+0000", 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.44e-06 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285592+0000", 2024-03-12T03:40:54.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6300000000000003e-06 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285605+0000", 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2934e-05 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285620+0000", 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5019e-05 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285680+0000", 2024-03-12T03:40:54.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0352000000000001e-05 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285689+0000", 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.611e-06 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.253492+0000", 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96780261499999998 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.253563+0000", 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0690000000000003e-05 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2", 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.253679+0000", 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000116539 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.257991+0000", 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043116459999999997 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.265538+0000", 2024-03-12T03:40:54.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0075468699999999998 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267856+0000", 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023183660000000001 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267882+0000", 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6027e-05 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267908+0000", 2024-03-12T03:40:54.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5341999999999999e-05 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:129271 7.15 7:ab2867bc:::benchmark_data_smithi169_79822_object11854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:15.337635+0000", 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 219.278772242, 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93035491299999995, 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.657 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 129271 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337635+0000", 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337635+0000", 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337637+0000", 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8390000000000001e-06 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337642+0000", 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3839999999999996e-06 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337653+0000", 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0139999999999999e-05 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337659+0000", 2024-03-12T03:40:54.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8190000000000003e-06 2024-03-12T03:40:54.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337720+0000", 2024-03-12T03:40:54.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0788000000000001e-05 2024-03-12T03:40:54.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337729+0000", 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9500000000000007e-06 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.254290+0000", 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91656037300000004 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.254411+0000", 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012097200000000001 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2", 2024-03-12T03:40:54.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.254623+0000", 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021296100000000001 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.261126+0000", 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0065024999999999996 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.265577+0000", 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044507549999999998 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267935+0000", 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023586200000000001 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267968+0000", 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2815999999999999e-05 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267990+0000", 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1996000000000001e-05 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.663 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:157476 7.11 7:8b1b26a8:::benchmark_data_smithi169_79822_object40059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:54.829219+0000", 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 179.787188359, 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.939989089, 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 157476 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.664 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829219+0000", 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829219+0000", 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829220+0000", 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.004e-06 2024-03-12T03:40:54.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829224+0000", 2024-03-12T03:40:54.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8990000000000003e-06 2024-03-12T03:40:54.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829233+0000", 2024-03-12T03:40:54.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2720000000000003e-06 2024-03-12T03:40:54.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829239+0000", 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7180000000000001e-06 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829300+0000", 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0857999999999999e-05 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829306+0000", 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3289999999999999e-06 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.765307+0000", 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93600046199999998 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.765388+0000", 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1761999999999994e-05 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.765626+0000", 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000237128 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769059+0000", 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003432928 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769100+0000", 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1183000000000002e-05 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769145+0000", 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5318999999999998e-05 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769176+0000", 2024-03-12T03:40:54.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0868000000000003e-05 2024-03-12T03:40:54.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769208+0000", 2024-03-12T03:40:54.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2359e-05 2024-03-12T03:40:54.670 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:157480 7.11 7:8cdca059:::benchmark_data_smithi169_79822_object40063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:54.833037+0000", 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 179.78337060300001, 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94306616099999996, 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 157480 2024-03-12T03:40:54.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833037+0000", 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833037+0000", 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833039+0000", 2024-03-12T03:40:54.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7179999999999999e-06 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833044+0000", 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5779999999999998e-06 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833059+0000", 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4848000000000001e-05 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833068+0000", 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7220000000000005e-06 2024-03-12T03:40:54.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833132+0000", 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4257999999999994e-05 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833157+0000", 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4570999999999999e-05 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.767762+0000", 2024-03-12T03:40:54.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93460503100000003 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.767831+0000", 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9350999999999994e-05 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.768013+0000", 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018161499999999999 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.772936+0000", 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049235320000000004 2024-03-12T03:40:54.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.773001+0000", 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4875000000000005e-05 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776000+0000", 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002998775 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776056+0000", 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6474000000000002e-05 2024-03-12T03:40:54.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776103+0000", 2024-03-12T03:40:54.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6813000000000001e-05 2024-03-12T03:40:54.678 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.678 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.678 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.678 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:157528 7.11 7:8ad61dec:::benchmark_data_smithi169_79822_object40111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-12T03:40:54.678 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:54.893117+0000", 2024-03-12T03:40:54.678 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 179.72329053799999, 2024-03-12T03:40:54.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88316590900000003, 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 157528 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893117+0000", 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893117+0000", 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893118+0000", 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2279999999999999e-06 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893123+0000", 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9520000000000002e-06 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893133+0000", 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0115000000000001e-05 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893141+0000", 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3100000000000003e-06 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893181+0000", 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0275000000000003e-05 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893187+0000", 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8839999999999999e-06 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769313+0000", 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87612656099999997 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769427+0000", 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000113409 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769620+0000", 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000193556 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.773065+0000", 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034451809999999999 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.773080+0000", 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4368e-05 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776195+0000", 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0031153470000000001 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776241+0000", 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5568000000000001e-05 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776283+0000", 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2154999999999999e-05 2024-03-12T03:40:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:157552 7.11 7:893c4227:::benchmark_data_smithi169_79822_object40135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:54.924603+0000", 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 179.69180439799999, 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85183126099999995, 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 157552 2024-03-12T03:40:54.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924603+0000", 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924603+0000", 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924605+0000", 2024-03-12T03:40:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3499999999999999e-06 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924615+0000", 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2529999999999993e-06 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924643+0000", 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8082999999999999e-05 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924665+0000", 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2365999999999999e-05 2024-03-12T03:40:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924755+0000", 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9766999999999996e-05 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924762+0000", 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6270000000000002e-06 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.771023+0000", 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.846261025 2024-03-12T03:40:54.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.771137+0000", 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114169 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.771366+0000", 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022884700000000001 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.775926+0000", 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.004559942 2024-03-12T03:40:54.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.775939+0000", 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3706999999999999e-05 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776340+0000", 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040105499999999999 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776392+0000", 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1381999999999997e-05 2024-03-12T03:40:54.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776434+0000", 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2688000000000002e-05 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:157555 7.11 7:8d5a9d9d:::benchmark_data_smithi169_79822_object40138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:54.924744+0000", 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 179.691663804, 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.86213734799999997, 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.691 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 157555 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924744+0000", 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924744+0000", 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924744+0000", 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5400000000000001e-07 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924751+0000", 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6320000000000002e-06 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924758+0000", 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9229999999999996e-06 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924766+0000", 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.841e-06 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924826+0000", 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0337999999999997e-05 2024-03-12T03:40:54.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.924834+0000", 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4429999999999997e-06 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.773124+0000", 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.84829055900000006 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.773244+0000", 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119768 2024-03-12T03:40:54.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.773454+0000", 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000209676 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776142+0000", 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026887489999999998 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776150+0000", 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6410000000000002e-06 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.786799+0000", 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010648591000000001 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.786845+0000", 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6035000000000002e-05 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.786881+0000", 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6498000000000002e-05 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:193831 7.8 7:17000021:::benchmark_data_smithi169_79822_object76414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:36.449932+0000", 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 138.16647520000001, 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97274706300000002, 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.698 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193831 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.449932+0000", 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.449932+0000", 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.449934+0000", 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.074e-06 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.449944+0000", 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1200000000000008e-06 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.449956+0000", 2024-03-12T03:40:54.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.27e-05 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.449974+0000", 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8247999999999999e-05 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.450110+0000", 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013577700000000001 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.450119+0000", 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9120000000000004e-06 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.417091+0000", 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96697190899999996 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.417182+0000", 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0586000000000003e-05 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.417473+0000", 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029137100000000001 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422452+0000", 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049790820000000001 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422512+0000", 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9846000000000003e-05 2024-03-12T03:40:54.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422574+0000", 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2218999999999995e-05 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422636+0000", 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1651999999999999e-05 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422679+0000", 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3566999999999998e-05 2024-03-12T03:40:54.704 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:193840 7.8 7:12e91ce1:::benchmark_data_smithi169_79822_object76423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:36.459371+0000", 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 138.157036866, 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96367646900000004, 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193840 2024-03-12T03:40:54.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.459371+0000", 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.459371+0000", 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.459371+0000", 2024-03-12T03:40:54.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1900000000000001e-07 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.459375+0000", 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7620000000000001e-06 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.459383+0000", 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9400000000000002e-06 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.459389+0000", 2024-03-12T03:40:54.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2670000000000003e-06 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.460921+0000", 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015317810000000001 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.461000+0000", 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9171999999999993e-05 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.419254+0000", 2024-03-12T03:40:54.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95825351700000005 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.419390+0000", 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013585600000000001 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.419603+0000", 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000213017 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422764+0000", 2024-03-12T03:40:54.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003160726 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422819+0000", 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5849e-05 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422938+0000", 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119085 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.423002+0000", 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4008000000000001e-05 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.423047+0000", 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4669999999999998e-05 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:193879 7.8 7:16eafa45:::benchmark_data_smithi169_79822_object76462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:36.491804+0000", 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 138.12460357, 2024-03-12T03:40:54.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93352429599999998, 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193879 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491804+0000", 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491804+0000", 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491805+0000", 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1319999999999999e-06 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491810+0000", 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8760000000000004e-06 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491821+0000", 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1094000000000001e-05 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491830+0000", 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5299999999999996e-06 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491868+0000", 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8293000000000001e-05 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491878+0000", 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0455000000000001e-05 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.420888+0000", 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.92901003500000001 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.420990+0000", 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101359 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-03-12T03:40:54.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.421191+0000", 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020108600000000001 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422883+0000", 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016925880000000001 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422897+0000", 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3202e-05 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.425124+0000", 2024-03-12T03:40:54.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022270739999999999 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.425279+0000", 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015494599999999999 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.425328+0000", 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9626000000000002e-05 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:193909 7.8 7:113cb578:::benchmark_data_smithi169_79822_object76492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:36.511598+0000", 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 138.10480989199999, 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91856142100000004, 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193909 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511598+0000", 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511598+0000", 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511600+0000", 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.905e-06 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511605+0000", 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0189999999999997e-06 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511616+0000", 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1584e-05 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511624+0000", 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5410000000000003e-06 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511690+0000", 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6323999999999997e-05 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511699+0000", 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2820000000000002e-06 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.423085+0000", 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.911386166 2024-03-12T03:40:54.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.423157+0000", 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1669999999999997e-05 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.423363+0000", 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000205427 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.429984+0000", 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0066215149999999997 2024-03-12T03:40:54.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.430007+0000", 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3006999999999998e-05 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.430052+0000", 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5311e-05 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.430111+0000", 2024-03-12T03:40:54.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9051000000000001e-05 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.430159+0000", 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7618999999999999e-05 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:214218 7.5 7:a5b8afe0:::benchmark_data_smithi169_79822_object96801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:55.695612+0000", 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 118.920795424, 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90713400899999996, 2024-03-12T03:40:54.724 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 214218 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695612+0000", 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695612+0000", 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695613+0000", 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3239999999999999e-06 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695619+0000", 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7259999999999999e-06 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695631+0000", 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2241e-05 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695638+0000", 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7399999999999998e-06 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695697+0000", 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9312999999999999e-05 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695708+0000", 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0505e-05 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.597283+0000", 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90157510900000004 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.597378+0000", 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5352000000000005e-05 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.597649+0000", 2024-03-12T03:40:54.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000271026 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.601493+0000", 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038440229999999998 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602663+0000", 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00116942 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602688+0000", 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5034000000000001e-05 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602720+0000", 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1809e-05 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602746+0000", 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6387e-05 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:214243 7.5 7:a26f371a:::benchmark_data_smithi169_79822_object96826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:55.724411+0000", 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 118.89199656, 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87854072900000002, 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 214243 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.724411+0000", 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.724411+0000", 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.724412+0000", 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3600000000000003e-07 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.724423+0000", 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1112e-05 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.724694+0000", 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000270733 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.724702+0000", 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7190000000000007e-06 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.724770+0000", 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7471000000000005e-05 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.724778+0000", 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0590000000000003e-06 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.599259+0000", 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87448123899999997 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.599340+0000", 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0766999999999994e-05 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.599537+0000", 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019747699999999999 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602794+0000", 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032565010000000002 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602857+0000", 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2741000000000001e-05 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602898+0000", 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1635999999999999e-05 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602929+0000", 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0445000000000001e-05 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602952+0000", 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3093000000000001e-05 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:214250 7.5 7:a76b874e:::benchmark_data_smithi169_79822_object96833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:55.732552+0000", 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 118.88385503400001, 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87349016999999995, 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.737 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 214250 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.732552+0000", 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.732552+0000", 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.732554+0000", 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.049e-06 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.732557+0000", 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7529999999999999e-06 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.732565+0000", 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0709999999999995e-06 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.732572+0000", 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0319999999999998e-06 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.732643+0000", 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0487999999999993e-05 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.732652+0000", 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0059999999999998e-06 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.600453+0000", 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.86780115099999999 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.600518+0000", 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4517000000000006e-05 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:40:54.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.600661+0000", 2024-03-12T03:40:54.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000143793 2024-03-12T03:40:54.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602796+0000", 2024-03-12T03:40:54.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021351400000000002 2024-03-12T03:40:54.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602987+0000", 2024-03-12T03:40:54.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019028299999999999 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.605986+0000", 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029997270000000002 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.606018+0000", 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1788999999999997e-05 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.606043+0000", 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4371000000000001e-05 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:214269 7.5 7:a7f85f22:::benchmark_data_smithi169_79822_object96852:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:55.747977+0000", 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 118.86843053299999, 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85830349399999994, 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.745 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 214269 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.747977+0000", 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.747977+0000", 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.747979+0000", 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7460000000000001e-06 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.747985+0000", 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8209999999999998e-06 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.747999+0000", 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4569999999999999e-05 2024-03-12T03:40:54.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.748007+0000", 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1890000000000007e-06 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.748064+0000", 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6372999999999997e-05 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.748073+0000", 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9080000000000001e-06 2024-03-12T03:40:54.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.601533+0000", 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85346024399999998 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.601607+0000", 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3732000000000005e-05 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.601792+0000", 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018516999999999999 2024-03-12T03:40:54.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602958+0000", 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001165814 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.603021+0000", 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3639000000000005e-05 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.606161+0000", 2024-03-12T03:40:54.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0031395799999999999 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.606233+0000", 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2539999999999993e-05 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.606280+0000", 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7168e-05 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.751 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:214275 7.5 7:a7773ee6:::benchmark_data_smithi169_79822_object96858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:55.753197+0000", 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 118.863210831, 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85660615200000001, 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 214275 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:40:54.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.753197+0000", 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.753197+0000", 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.753198+0000", 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2640000000000001e-06 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.753205+0000", 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1350000000000003e-06 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.753217+0000", 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1439e-05 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.753225+0000", 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2269999999999993e-06 2024-03-12T03:40:54.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.753283+0000", 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7918999999999999e-05 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.753291+0000", 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7099999999999996e-06 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.603049+0000", 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.84975754599999997 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.603136+0000", 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7576000000000003e-05 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.603354+0000", 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000217617 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.606095+0000", 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027409169999999998 2024-03-12T03:40:54.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.606103+0000", 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2640000000000008e-06 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.609709+0000", 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036052940000000002 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.609752+0000", 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3319999999999999e-05 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:40:54.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:40:54.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:40:54.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.609803+0000", 2024-03-12T03:40:54.758 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0924000000000002e-05 2024-03-12T03:40:54.758 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.758 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.758 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.758 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:40:54.758 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:40:54.758 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:40:54.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:54.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:54.821 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:54 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1838574529' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:40:54.822 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:54 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3037282623' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:40:54.822 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:54 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:40:54.822 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:54 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1838574529' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:40:54.822 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:54 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3037282623' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:40:54.822 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:54 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:40:54.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:54.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:54.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:54 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1838574529' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:40:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:54 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3037282623' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:40:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:54 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:40:55.003 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 70 16 112656 112640 100.555 62.375 0.0128513 0.0099407 2024-03-12T03:40:55.027 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:40:55.068 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:55.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:55.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:40:55.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:55.198 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:55.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:55.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:55.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:55.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:55.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:55.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:55.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:55.481 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:55.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:55.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-12T03:40:55.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:55.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:55.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:55.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:55.715 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:55.772 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:55.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:55.792 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-12T03:40:55.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:55 smithi169 ceph-mon[29344]: pgmap v351: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s; 39264/326871 objects degraded (12.012%); 10 MiB/s, 161 objects/s recovering 2024-03-12T03:40:55.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:55 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:40:55.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:55 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1838574529' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-12T03:40:55.851 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:55 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:40:55.851 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:55 smithi169 ceph-mon[29344]: osdmap e225: 8 total, 8 up, 8 in 2024-03-12T03:40:55.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:55 smithi169 ceph-mon[39949]: pgmap v351: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s; 39264/326871 objects degraded (12.012%); 10 MiB/s, 161 objects/s recovering 2024-03-12T03:40:55.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:55 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:40:55.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:55 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1838574529' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-12T03:40:55.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:55 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:40:55.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:55 smithi169 ceph-mon[39949]: osdmap e225: 8 total, 8 up, 8 in 2024-03-12T03:40:55.874 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:40:55.954 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:55.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:55.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:40:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:55 smithi173 ceph-mon[33544]: pgmap v351: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s; 39264/326871 objects degraded (12.012%); 10 MiB/s, 161 objects/s recovering 2024-03-12T03:40:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:55 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:40:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:55 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1838574529' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-12T03:40:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:55 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:40:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:55 smithi173 ceph-mon[33544]: osdmap e225: 8 total, 8 up, 8 in 2024-03-12T03:40:56.004 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 71 16 113708 113692 100.064 65.75 0.00898101 0.00998914 2024-03-12T03:40:56.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:40:56.105 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:56.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:56.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:40:56.208 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:40:56.257 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:56.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:40:56.359 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:56.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:56.403 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:56.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:56.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:56.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:56.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:56.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:56.678 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:56.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:56.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:56.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:56.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:56.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:56.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:56.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:57.004 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 72 16 114826 114810 99.6446 69.875 0.00605861 0.0100312 2024-03-12T03:40:57.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:57.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:57.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:57.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:57.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:57.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:57.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:57.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:57.363 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:57.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:57.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:57.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:40:57.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:57.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:57.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:40:57.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:57.644 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:57.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:57.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:57.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:57.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:57.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:57.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:57.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:57.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:57 smithi169 ceph-mon[29344]: pgmap v353: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 32 MiB/s wr, 511 op/s; 39264/326871 objects degraded (12.012%); 9.7 MiB/s, 154 objects/s recovering 2024-03-12T03:40:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:57 smithi169 ceph-mon[29344]: 1.6 scrub starts 2024-03-12T03:40:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:57 smithi169 ceph-mon[29344]: 1.6 scrub ok 2024-03-12T03:40:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:57 smithi169 ceph-mon[29344]: 6.5 scrub starts 2024-03-12T03:40:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:57 smithi169 ceph-mon[29344]: 6.5 scrub ok 2024-03-12T03:40:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:57 smithi169 ceph-mon[29344]: 8.2 scrub starts 2024-03-12T03:40:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:57 smithi169 ceph-mon[39949]: pgmap v353: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 32 MiB/s wr, 511 op/s; 39264/326871 objects degraded (12.012%); 9.7 MiB/s, 154 objects/s recovering 2024-03-12T03:40:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:57 smithi169 ceph-mon[39949]: 1.6 scrub starts 2024-03-12T03:40:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:57 smithi169 ceph-mon[39949]: 1.6 scrub ok 2024-03-12T03:40:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:57 smithi169 ceph-mon[39949]: 6.5 scrub starts 2024-03-12T03:40:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:57 smithi169 ceph-mon[39949]: 6.5 scrub ok 2024-03-12T03:40:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:57 smithi169 ceph-mon[39949]: 8.2 scrub starts 2024-03-12T03:40:57.965 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:57 smithi173 ceph-mon[33544]: pgmap v353: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 32 MiB/s wr, 511 op/s; 39264/326871 objects degraded (12.012%); 9.7 MiB/s, 154 objects/s recovering 2024-03-12T03:40:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:57 smithi173 ceph-mon[33544]: 1.6 scrub starts 2024-03-12T03:40:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:57 smithi173 ceph-mon[33544]: 1.6 scrub ok 2024-03-12T03:40:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:57 smithi173 ceph-mon[33544]: 6.5 scrub starts 2024-03-12T03:40:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:57 smithi173 ceph-mon[33544]: 6.5 scrub ok 2024-03-12T03:40:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:57 smithi173 ceph-mon[33544]: 8.2 scrub starts 2024-03-12T03:40:58.005 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 73 16 115877 115861 99.1782 65.6875 0.0180014 0.0100784 2024-03-12T03:40:58.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:58.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:58.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:58.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:58.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:58.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:58.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:58.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:40:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:58.363 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:58.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:40:58.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:40:58.507 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:58.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:58.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:40:58.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:58.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:58.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:58.643 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:40:58.663 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:40:58.704 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:40:58.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:58.749 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:40:58.787 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:58.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:58.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:58.792 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:40:58.834 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:40:58.871 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:40:58.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:58.914 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:40:58.925 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:58.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:58.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:58.945 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:40:58.946 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:40:58.946 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:40:58.946 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:40:58.946 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:40:58.946 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:40:58.946 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:40:58.946 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:40:58.947 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:40:59.005 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 74 16 116865 116849 98.6724 61.75 0.00786486 0.0101302 2024-03-12T03:40:59.028 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:40:59.064 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:59.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:59.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:40:59.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:59.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:59.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:59.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:40:59.336 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:59.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:40:59.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:40:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:59.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:59.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:40:59.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:40:59.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:59.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:40:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[29344]: pgmap v354: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 955 op/s; 39612/342660 objects degraded (11.560%); 18 MiB/s, 279 objects/s recovering 2024-03-12T03:40:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[29344]: 1.7 scrub starts 2024-03-12T03:40:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[29344]: 1.7 scrub ok 2024-03-12T03:40:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[29344]: 8.2 scrub ok 2024-03-12T03:40:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[29344]: osdmap e226: 8 total, 8 up, 8 in 2024-03-12T03:40:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[29344]: 4.1a scrub starts 2024-03-12T03:40:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[29344]: 4.15 scrub starts 2024-03-12T03:40:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[29344]: 4.15 scrub ok 2024-03-12T03:40:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[29344]: 4.1a scrub ok 2024-03-12T03:40:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/329918356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 39612/342660 objects degraded (11.560%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[39949]: pgmap v354: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 955 op/s; 39612/342660 objects degraded (11.560%); 18 MiB/s, 279 objects/s recovering 2024-03-12T03:40:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[39949]: 1.7 scrub starts 2024-03-12T03:40:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[39949]: 1.7 scrub ok 2024-03-12T03:40:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[39949]: 8.2 scrub ok 2024-03-12T03:40:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[39949]: osdmap e226: 8 total, 8 up, 8 in 2024-03-12T03:40:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[39949]: 4.1a scrub starts 2024-03-12T03:40:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[39949]: 4.15 scrub starts 2024-03-12T03:40:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[39949]: 4.15 scrub ok 2024-03-12T03:40:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[39949]: 4.1a scrub ok 2024-03-12T03:40:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/329918356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:40:59 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 39612/342660 objects degraded (11.560%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:59.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:40:59.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:59 smithi173 ceph-mon[33544]: pgmap v354: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 955 op/s; 39612/342660 objects degraded (11.560%); 18 MiB/s, 279 objects/s recovering 2024-03-12T03:40:59.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:59 smithi173 ceph-mon[33544]: 1.7 scrub starts 2024-03-12T03:40:59.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:59 smithi173 ceph-mon[33544]: 1.7 scrub ok 2024-03-12T03:40:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:59 smithi173 ceph-mon[33544]: 8.2 scrub ok 2024-03-12T03:40:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:59 smithi173 ceph-mon[33544]: osdmap e226: 8 total, 8 up, 8 in 2024-03-12T03:40:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:59 smithi173 ceph-mon[33544]: 4.1a scrub starts 2024-03-12T03:40:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:59 smithi173 ceph-mon[33544]: 4.15 scrub starts 2024-03-12T03:40:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:59 smithi173 ceph-mon[33544]: 4.15 scrub ok 2024-03-12T03:40:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:59 smithi173 ceph-mon[33544]: 4.1a scrub ok 2024-03-12T03:40:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:59 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/329918356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:40:59.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:40:59 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 39612/342660 objects degraded (11.560%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:40:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:59.745 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:40:59.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:40:59.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:40:59.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:40:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:40:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:40:59.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:40:59.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:00.006 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 75 16 117881 117865 98.2034 63.5 0.0257001 0.0101781 2024-03-12T03:41:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:00.037 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:00.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:00.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:00.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:00.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:00.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:00.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:00.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:00.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:00.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:00.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:00.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:00.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:00.602 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:00.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:00.658 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:00 smithi169 ceph-mon[29344]: 8.9 scrub starts 2024-03-12T03:41:00.658 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:00 smithi169 ceph-mon[29344]: 1.9 scrub starts 2024-03-12T03:41:00.658 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:00 smithi169 ceph-mon[29344]: 1.9 scrub ok 2024-03-12T03:41:00.658 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:00 smithi169 ceph-mon[29344]: 3.c scrub starts 2024-03-12T03:41:00.658 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:00 smithi169 ceph-mon[29344]: 3.c scrub ok 2024-03-12T03:41:00.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:00 smithi169 ceph-mon[39949]: 8.9 scrub starts 2024-03-12T03:41:00.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:00 smithi169 ceph-mon[39949]: 1.9 scrub starts 2024-03-12T03:41:00.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:00 smithi169 ceph-mon[39949]: 1.9 scrub ok 2024-03-12T03:41:00.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:00 smithi169 ceph-mon[39949]: 3.c scrub starts 2024-03-12T03:41:00.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:00 smithi169 ceph-mon[39949]: 3.c scrub ok 2024-03-12T03:41:00.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:00.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:00 smithi173 ceph-mon[33544]: 8.9 scrub starts 2024-03-12T03:41:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:00 smithi173 ceph-mon[33544]: 1.9 scrub starts 2024-03-12T03:41:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:00 smithi173 ceph-mon[33544]: 1.9 scrub ok 2024-03-12T03:41:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:00 smithi173 ceph-mon[33544]: 3.c scrub starts 2024-03-12T03:41:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:00 smithi173 ceph-mon[33544]: 3.c scrub ok 2024-03-12T03:41:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:00.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:00.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:00.879 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:00.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:00.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:00.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:01.005 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 76 16 118923 118907 97.7681 65.125 0.00893227 0.0102235 2024-03-12T03:41:01.027 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:01.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:01.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:01.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:01.177 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:01.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:01.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:01.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:01.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:01.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:01.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:01.463 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:01.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:01.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:01.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:01.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:01.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:01.719 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[29344]: pgmap v356: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 657 op/s; 39612/342660 objects degraded (11.560%); 12 MiB/s, 187 objects/s recovering 2024-03-12T03:41:01.719 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[29344]: 4.5 scrub starts 2024-03-12T03:41:01.719 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[29344]: 4.5 scrub ok 2024-03-12T03:41:01.719 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[29344]: 1.0 scrub starts 2024-03-12T03:41:01.719 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[29344]: 1.0 scrub ok 2024-03-12T03:41:01.719 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[29344]: 6.7 scrub starts 2024-03-12T03:41:01.719 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[29344]: 6.7 scrub ok 2024-03-12T03:41:01.719 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:41:01.719 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:01.719 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:41:01.719 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:41:01.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[39949]: pgmap v356: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 657 op/s; 39612/342660 objects degraded (11.560%); 12 MiB/s, 187 objects/s recovering 2024-03-12T03:41:01.720 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[39949]: 4.5 scrub starts 2024-03-12T03:41:01.720 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[39949]: 4.5 scrub ok 2024-03-12T03:41:01.720 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[39949]: 1.0 scrub starts 2024-03-12T03:41:01.720 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[39949]: 1.0 scrub ok 2024-03-12T03:41:01.720 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[39949]: 6.7 scrub starts 2024-03-12T03:41:01.720 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[39949]: 6.7 scrub ok 2024-03-12T03:41:01.720 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:41:01.720 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:01.720 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:41:01.720 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:41:01.721 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:01 smithi173 ceph-mon[33544]: pgmap v356: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 657 op/s; 39612/342660 objects degraded (11.560%); 12 MiB/s, 187 objects/s recovering 2024-03-12T03:41:01.721 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:01 smithi173 ceph-mon[33544]: 4.5 scrub starts 2024-03-12T03:41:01.721 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:01 smithi173 ceph-mon[33544]: 4.5 scrub ok 2024-03-12T03:41:01.721 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:01 smithi173 ceph-mon[33544]: 1.0 scrub starts 2024-03-12T03:41:01.721 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:01 smithi173 ceph-mon[33544]: 1.0 scrub ok 2024-03-12T03:41:01.721 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:01 smithi173 ceph-mon[33544]: 6.7 scrub starts 2024-03-12T03:41:01.721 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:01 smithi173 ceph-mon[33544]: 6.7 scrub ok 2024-03-12T03:41:01.721 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:01 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:41:01.721 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:01 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:01.721 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:01 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:41:01.721 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:01 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:41:01.752 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:01.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:01.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:01.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:01.891 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:01.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:01.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:02.005 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 77 16 120030 120014 97.3969 69.1875 0.0113494 0.0102621 2024-03-12T03:41:02.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:02.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:02.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:02.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:02.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:02.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:02.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:41:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:41:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:41:02.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:02.307 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:02.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:02.409 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:02.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:02.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:02.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:02.563 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:02 smithi169 ceph-mon[29344]: 5.1 scrub starts 2024-03-12T03:41:02.563 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:02 smithi169 ceph-mon[29344]: 5.1 scrub ok 2024-03-12T03:41:02.563 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:02 smithi169 ceph-mon[29344]: 8.9 scrub ok 2024-03-12T03:41:02.563 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:02 smithi169 ceph-mon[29344]: 5.16 scrub starts 2024-03-12T03:41:02.563 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:02 smithi169 ceph-mon[29344]: 5.16 scrub ok 2024-03-12T03:41:02.563 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:02 smithi169 ceph-mon[29344]: osdmap e227: 8 total, 8 up, 8 in 2024-03-12T03:41:02.564 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:02 smithi169 ceph-mon[39949]: 5.1 scrub starts 2024-03-12T03:41:02.564 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:02 smithi169 ceph-mon[39949]: 5.1 scrub ok 2024-03-12T03:41:02.564 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:02 smithi169 ceph-mon[39949]: 8.9 scrub ok 2024-03-12T03:41:02.564 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:02 smithi169 ceph-mon[39949]: 5.16 scrub starts 2024-03-12T03:41:02.564 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:02 smithi169 ceph-mon[39949]: 5.16 scrub ok 2024-03-12T03:41:02.564 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:02 smithi169 ceph-mon[39949]: osdmap e227: 8 total, 8 up, 8 in 2024-03-12T03:41:02.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:02.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:02.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:02.705 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:41:02.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:02.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:02.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:02.863 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:02.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:02.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:02.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:02.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:41:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:41:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:02 smithi173 ceph-mon[33544]: 5.1 scrub starts 2024-03-12T03:41:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:02 smithi173 ceph-mon[33544]: 5.1 scrub ok 2024-03-12T03:41:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:02 smithi173 ceph-mon[33544]: 8.9 scrub ok 2024-03-12T03:41:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:02 smithi173 ceph-mon[33544]: 5.16 scrub starts 2024-03-12T03:41:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:02 smithi173 ceph-mon[33544]: 5.16 scrub ok 2024-03-12T03:41:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:02 smithi173 ceph-mon[33544]: osdmap e227: 8 total, 8 up, 8 in 2024-03-12T03:41:02.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:02.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:03.008 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 78 15 121143 121128 97.0379 69.625 0.0100458 0.0103005 2024-03-12T03:41:03.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:03.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:03.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:03.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:03.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:03.277 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:03.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:03.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:03.379 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:03.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:03.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:03.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:03.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:03.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:03.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[29344]: pgmap v358: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 7.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 90 MiB/s wr, 1.43k op/s; 39624/358314 objects degraded (11.058%); 28 MiB/s, 444 objects/s recovering 2024-03-12T03:41:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[29344]: 4.a scrub starts 2024-03-12T03:41:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[29344]: 4.a scrub ok 2024-03-12T03:41:03.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[29344]: 4.c scrub starts 2024-03-12T03:41:03.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[29344]: 4.c scrub ok 2024-03-12T03:41:03.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:03.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:41:03.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:41:03.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[29344]: osdmap e228: 8 total, 8 up, 8 in 2024-03-12T03:41:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[39949]: pgmap v358: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 7.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 90 MiB/s wr, 1.43k op/s; 39624/358314 objects degraded (11.058%); 28 MiB/s, 444 objects/s recovering 2024-03-12T03:41:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[39949]: 4.a scrub starts 2024-03-12T03:41:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[39949]: 4.a scrub ok 2024-03-12T03:41:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[39949]: 4.c scrub starts 2024-03-12T03:41:03.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[39949]: 4.c scrub ok 2024-03-12T03:41:03.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:03.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:41:03.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:41:03.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:03 smithi169 ceph-mon[39949]: osdmap e228: 8 total, 8 up, 8 in 2024-03-12T03:41:03.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:03.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:03.792 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:03.837 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:03.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:03.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:03.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:03.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:03 smithi173 ceph-mon[33544]: pgmap v358: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 7.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 90 MiB/s wr, 1.43k op/s; 39624/358314 objects degraded (11.058%); 28 MiB/s, 444 objects/s recovering 2024-03-12T03:41:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:03 smithi173 ceph-mon[33544]: 4.a scrub starts 2024-03-12T03:41:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:03 smithi173 ceph-mon[33544]: 4.a scrub ok 2024-03-12T03:41:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:03 smithi173 ceph-mon[33544]: 4.c scrub starts 2024-03-12T03:41:03.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:03 smithi173 ceph-mon[33544]: 4.c scrub ok 2024-03-12T03:41:03.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:03 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:03.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:03 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:41:03.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:03 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:41:03.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:03 smithi173 ceph-mon[33544]: osdmap e228: 8 total, 8 up, 8 in 2024-03-12T03:41:04.010 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 79 16 122261 122245 96.6901 69.8125 0.022579 0.0103377 2024-03-12T03:41:04.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:04.118 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:04.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:04.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:04.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:04.245 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:04.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:04.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:04.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:04.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:04.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:04.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:04.527 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:04.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:04.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:04.683 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:04.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:04.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:04.786 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:04.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:04 smithi169 ceph-mon[29344]: 3.11 scrub starts 2024-03-12T03:41:04.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:04 smithi169 ceph-mon[29344]: 3.11 scrub ok 2024-03-12T03:41:04.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:04 smithi169 ceph-mon[29344]: 1.8 scrub starts 2024-03-12T03:41:04.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:04 smithi169 ceph-mon[29344]: 1.8 scrub ok 2024-03-12T03:41:04.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:04 smithi169 ceph-mon[39949]: 3.11 scrub starts 2024-03-12T03:41:04.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:04 smithi169 ceph-mon[39949]: 3.11 scrub ok 2024-03-12T03:41:04.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:04 smithi169 ceph-mon[39949]: 1.8 scrub starts 2024-03-12T03:41:04.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:04 smithi169 ceph-mon[39949]: 1.8 scrub ok 2024-03-12T03:41:04.832 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:04.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:04.934 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:04.981 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:04.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:04 smithi173 ceph-mon[33544]: 3.11 scrub starts 2024-03-12T03:41:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:04 smithi173 ceph-mon[33544]: 3.11 scrub ok 2024-03-12T03:41:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:04 smithi173 ceph-mon[33544]: 1.8 scrub starts 2024-03-12T03:41:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:04 smithi173 ceph-mon[33544]: 1.8 scrub ok 2024-03-12T03:41:05.010 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:41:05.009379+0000 min lat: 0.00107765 max lat: 2.48103 avg lat: 0.0103842 2024-03-12T03:41:05.011 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:41:05.011 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 80 16 123257 123241 96.2596 62.25 0.00959388 0.0103842 2024-03-12T03:41:05.083 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:05.129 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:05.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:05.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:05.147 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:41:05.147 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:41:05.231 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:05.274 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:05.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:05.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:05.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:05.425 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:05.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:05.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:05.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:05.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:05.658 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:05.702 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:05.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:05.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:05.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:05.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[29344]: pgmap v360: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 7.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 869 op/s; 39624/358314 objects degraded (11.058%); 18 MiB/s, 291 objects/s recovering 2024-03-12T03:41:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[29344]: 3.a scrub starts 2024-03-12T03:41:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[29344]: 3.a scrub ok 2024-03-12T03:41:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[29344]: 4.1b scrub starts 2024-03-12T03:41:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[29344]: 4.1b scrub ok 2024-03-12T03:41:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[29344]: 1.a scrub starts 2024-03-12T03:41:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[29344]: 1.a scrub ok 2024-03-12T03:41:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:41:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:41:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[39949]: pgmap v360: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 7.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 869 op/s; 39624/358314 objects degraded (11.058%); 18 MiB/s, 291 objects/s recovering 2024-03-12T03:41:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[39949]: 3.a scrub starts 2024-03-12T03:41:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[39949]: 3.a scrub ok 2024-03-12T03:41:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[39949]: 4.1b scrub starts 2024-03-12T03:41:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[39949]: 4.1b scrub ok 2024-03-12T03:41:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[39949]: 1.a scrub starts 2024-03-12T03:41:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[39949]: 1.a scrub ok 2024-03-12T03:41:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:41:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:05 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:41:05.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:05.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:05.986 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:05.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:05 smithi173 ceph-mon[33544]: pgmap v360: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 7.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 869 op/s; 39624/358314 objects degraded (11.058%); 18 MiB/s, 291 objects/s recovering 2024-03-12T03:41:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:05 smithi173 ceph-mon[33544]: 3.a scrub starts 2024-03-12T03:41:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:05 smithi173 ceph-mon[33544]: 3.a scrub ok 2024-03-12T03:41:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:05 smithi173 ceph-mon[33544]: 4.1b scrub starts 2024-03-12T03:41:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:05 smithi173 ceph-mon[33544]: 4.1b scrub ok 2024-03-12T03:41:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:05 smithi173 ceph-mon[33544]: 1.a scrub starts 2024-03-12T03:41:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:05 smithi173 ceph-mon[33544]: 1.a scrub ok 2024-03-12T03:41:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:05 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:41:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:05 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:41:06.010 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 81 16 124313 124297 95.886 66 0.00434903 0.0104237 2024-03-12T03:41:06.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-12T03:41:06.029 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:06.087 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:06.091 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:06.150 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:06.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:06.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:06.297 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:06.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:06.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:06.454 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:06.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:06.545 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.1 to 1 (10000) 2024-03-12T03:41:06.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:06.586 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:06.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:06.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:06.688 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:06.712 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:06.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:06.814 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:06.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:06.862 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:06.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:06.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:06 smithi169 ceph-mon[29344]: osdmap e229: 8 total, 8 up, 8 in 2024-03-12T03:41:06.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:06 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/127587993' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:41:06.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:06 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:41:06.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:06 smithi169 ceph-mon[39949]: osdmap e229: 8 total, 8 up, 8 in 2024-03-12T03:41:06.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:06 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/127587993' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:41:06.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:06 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:41:06.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:06 smithi173 ceph-mon[33544]: osdmap e229: 8 total, 8 up, 8 in 2024-03-12T03:41:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:06 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/127587993' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:41:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:06 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:41:06.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:06.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:07.010 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 82 16 125265 125249 95.4422 59.5 0.0171784 0.0104729 2024-03-12T03:41:07.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:07.135 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:07.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:07.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:07.237 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:07.281 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:07.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:07.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:07.421 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:07.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:07.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:07.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:07.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:07.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:07.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:07.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:07.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:07.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:07.831 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:07.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:07 smithi169 ceph-mon[29344]: pgmap v362: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 925 op/s; 39624/359289 objects degraded (11.028%); 18 MiB/s, 291 objects/s recovering 2024-03-12T03:41:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:07 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 39624/359289 objects degraded (11.028%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:07 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-12T03:41:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:07 smithi169 ceph-mon[29344]: osdmap e230: 8 total, 8 up, 8 in 2024-03-12T03:41:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:07 smithi169 ceph-mon[39949]: pgmap v362: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 925 op/s; 39624/359289 objects degraded (11.028%); 18 MiB/s, 291 objects/s recovering 2024-03-12T03:41:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:07 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 39624/359289 objects degraded (11.028%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:07 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-12T03:41:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:07 smithi169 ceph-mon[39949]: osdmap e230: 8 total, 8 up, 8 in 2024-03-12T03:41:07.933 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:07.971 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:07.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:07 smithi173 ceph-mon[33544]: pgmap v362: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 925 op/s; 39624/359289 objects degraded (11.028%); 18 MiB/s, 291 objects/s recovering 2024-03-12T03:41:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:07 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 39624/359289 objects degraded (11.028%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:07 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-12T03:41:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:07 smithi173 ceph-mon[33544]: osdmap e230: 8 total, 8 up, 8 in 2024-03-12T03:41:08.011 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 83 16 126376 126360 95.1289 69.4375 0.0170823 0.0105076 2024-03-12T03:41:08.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:08.112 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:08.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:08.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:08.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:08.245 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:08.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:08.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:08.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:08.394 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:08.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:08.495 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:08.534 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:08.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:08.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:08.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:08.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:08.768 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:08.807 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:08.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:08.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:08.909 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:08.945 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:08.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:09.011 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 84 16 127476 127460 94.8148 68.75 0.0163965 0.0105423 2024-03-12T03:41:09.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:09.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:09.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:09.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:09.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:09.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:09.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:09.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:09.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:09.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:09.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:09.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:09.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:09.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:09.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:09.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:09.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:09.703 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-12T03:41:09.730 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:09.768 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[39949]: pgmap v364: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 895 op/s; 39993/374424 objects degraded (10.681%); 15 MiB/s, 233 objects/s recovering 2024-03-12T03:41:09.768 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[39949]: 3.10 scrub starts 2024-03-12T03:41:09.768 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[39949]: 3.10 scrub ok 2024-03-12T03:41:09.768 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[39949]: 5.4 scrub starts 2024-03-12T03:41:09.768 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[39949]: 5.4 scrub ok 2024-03-12T03:41:09.768 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[39949]: 5.1c scrub starts 2024-03-12T03:41:09.768 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[39949]: 5.1c scrub ok 2024-03-12T03:41:09.768 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:41:09.769 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[29344]: pgmap v364: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 895 op/s; 39993/374424 objects degraded (10.681%); 15 MiB/s, 233 objects/s recovering 2024-03-12T03:41:09.769 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[29344]: 3.10 scrub starts 2024-03-12T03:41:09.769 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[29344]: 3.10 scrub ok 2024-03-12T03:41:09.769 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[29344]: 5.4 scrub starts 2024-03-12T03:41:09.769 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[29344]: 5.4 scrub ok 2024-03-12T03:41:09.769 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[29344]: 5.1c scrub starts 2024-03-12T03:41:09.769 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[29344]: 5.1c scrub ok 2024-03-12T03:41:09.769 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:09 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:41:09.802 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:09.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:09.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:09.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:09.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:09 smithi173 ceph-mon[33544]: pgmap v364: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 895 op/s; 39993/374424 objects degraded (10.681%); 15 MiB/s, 233 objects/s recovering 2024-03-12T03:41:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:09 smithi173 ceph-mon[33544]: 3.10 scrub starts 2024-03-12T03:41:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:09 smithi173 ceph-mon[33544]: 3.10 scrub ok 2024-03-12T03:41:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:09 smithi173 ceph-mon[33544]: 5.4 scrub starts 2024-03-12T03:41:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:09 smithi173 ceph-mon[33544]: 5.4 scrub ok 2024-03-12T03:41:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:09 smithi173 ceph-mon[33544]: 5.1c scrub starts 2024-03-12T03:41:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:09 smithi173 ceph-mon[33544]: 5.1c scrub ok 2024-03-12T03:41:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:09 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:41:10.011 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 85 16 128469 128453 94.4295 62.0625 0.0052906 0.0105834 2024-03-12T03:41:10.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:10.101 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:10.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:10.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:10.255 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:10.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:10.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:10.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:10.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:10.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:10.492 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:10.543 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:10.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:10.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:10.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:10.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:10.775 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:10.836 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:10 smithi169 ceph-mon[29344]: 5.1f scrub starts 2024-03-12T03:41:10.836 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:10 smithi169 ceph-mon[29344]: 5.1f scrub ok 2024-03-12T03:41:10.836 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:10 smithi169 ceph-mon[39949]: 5.1f scrub starts 2024-03-12T03:41:10.837 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:10 smithi169 ceph-mon[39949]: 5.1f scrub ok 2024-03-12T03:41:10.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:10.837 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:10.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:10.931 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:10 smithi173 ceph-mon[33544]: 5.1f scrub starts 2024-03-12T03:41:10.931 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:10 smithi173 ceph-mon[33544]: 5.1f scrub ok 2024-03-12T03:41:10.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:10.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:10.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:10.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:11.011 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 86 16 129478 129462 94.0643 63.0625 0.0214217 0.0106263 2024-03-12T03:41:11.068 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:41:11.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:11.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:11.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:11.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:11.107 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:41:11.146 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:41:11.192 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:41:11.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:11.235 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:41:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:11.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:11.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:11.297 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:41:11.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:11.338 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:41:11.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:11.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:11.380 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:41:11.392 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:41:11.392 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:41:11.392 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:41:11.392 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:41:11.392 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:41:11.392 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:41:11.392 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:41:11.392 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:41:11.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:11.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:11.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:11.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:11.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:11.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:11.643 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:11.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:11.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[39949]: pgmap v365: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 772 B/s rd, 51 MiB/s wr, 810 op/s; 39993/374424 objects degraded (10.681%); 13 MiB/s, 211 objects/s recovering 2024-03-12T03:41:11.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:41:11.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:41:11.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:41:11.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:41:11.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:41:11.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3330105689' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:41:11.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[29344]: pgmap v365: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 772 B/s rd, 51 MiB/s wr, 810 op/s; 39993/374424 objects degraded (10.681%); 13 MiB/s, 211 objects/s recovering 2024-03-12T03:41:11.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:41:11.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:41:11.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:41:11.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:41:11.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:41:11.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3330105689' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:41:11.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:11.788 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:11.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:11.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:11.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:11.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:11.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:11 smithi173 ceph-mon[33544]: pgmap v365: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 772 B/s rd, 51 MiB/s wr, 810 op/s; 39993/374424 objects degraded (10.681%); 13 MiB/s, 211 objects/s recovering 2024-03-12T03:41:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:11 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:41:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:11 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:41:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:11 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:41:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:11 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:41:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:11 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:41:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:11 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3330105689' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:41:12.011 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 87 16 130472 130456 93.6971 62.125 0.0192356 0.0106685 2024-03-12T03:41:12.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:12.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:12.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:12.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:12.120 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:41:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:41:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:41:12.133 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:12.133 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:12.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:12.229 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:12.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:12.332 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:12.371 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:12.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:12.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:12.516 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:12.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:12.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:12.616 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:41:12.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:12.663 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:12.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:12 smithi169 ceph-mon[29344]: osdmap e231: 8 total, 8 up, 8 in 2024-03-12T03:41:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:12 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/906961479' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:41:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:12 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:41:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:12 smithi169 ceph-mon[39949]: osdmap e231: 8 total, 8 up, 8 in 2024-03-12T03:41:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:12 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/906961479' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:41:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:12 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:41:12.764 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:12.801 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:12.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:12.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:12.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:12.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:12.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:12.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:41:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:41:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:12 smithi173 ceph-mon[33544]: osdmap e231: 8 total, 8 up, 8 in 2024-03-12T03:41:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:12 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/906961479' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:41:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:12 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:41:13.011 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 88 16 131577 131561 93.4172 69.0625 0.0157045 0.0107002 2024-03-12T03:41:13.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:13.079 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:13.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:13.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:13.180 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:13.219 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:13.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:13.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:13.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:13.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:13.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:13.496 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:13.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:13.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:13.637 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:13.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:13.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:13.739 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:13.775 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:13.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:13.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:13.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:13.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:13 smithi169 ceph-mon[29344]: pgmap v367: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.6 KiB/s rd, 103 MiB/s wr, 1.65k op/s; 40090/389877 objects degraded (10.283%); 29 MiB/s, 456 objects/s recovering 2024-03-12T03:41:13.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:13 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 40090/389877 objects degraded (10.283%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:13.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:13 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:41:13.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:13 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:41:13.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:13 smithi169 ceph-mon[29344]: osdmap e232: 8 total, 8 up, 8 in 2024-03-12T03:41:13.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:13 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:13.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:13 smithi169 ceph-mon[39949]: pgmap v367: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.6 KiB/s rd, 103 MiB/s wr, 1.65k op/s; 40090/389877 objects degraded (10.283%); 29 MiB/s, 456 objects/s recovering 2024-03-12T03:41:13.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:13 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 40090/389877 objects degraded (10.283%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:13.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:13 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:41:13.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:13 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:41:13.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:13 smithi169 ceph-mon[39949]: osdmap e232: 8 total, 8 up, 8 in 2024-03-12T03:41:13.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:13 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:13.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:13.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:13 smithi173 ceph-mon[33544]: pgmap v367: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.6 KiB/s rd, 103 MiB/s wr, 1.65k op/s; 40090/389877 objects degraded (10.283%); 29 MiB/s, 456 objects/s recovering 2024-03-12T03:41:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:13 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 40090/389877 objects degraded (10.283%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:13 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:41:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:13 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:41:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:13 smithi173 ceph-mon[33544]: osdmap e232: 8 total, 8 up, 8 in 2024-03-12T03:41:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:13 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:14.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:14.012 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 89 16 132722 132706 93.1716 71.5625 0.0112422 0.0107285 2024-03-12T03:41:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:14.047 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:14.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:14.148 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:14.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:41:14.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:14.193 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:14.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:14.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:14.347 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:14.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:14.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:14.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:14.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:14.575 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:14.611 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:14.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:14.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:14.742 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:14.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:14.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:14.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:14.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:14 smithi169 ceph-mon[29344]: 4.1f deep-scrub starts 2024-03-12T03:41:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:14 smithi169 ceph-mon[29344]: 4.1f deep-scrub ok 2024-03-12T03:41:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:14 smithi169 ceph-mon[29344]: 3.14 scrub starts 2024-03-12T03:41:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:14 smithi169 ceph-mon[29344]: 3.14 scrub ok 2024-03-12T03:41:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:14 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:41:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:14 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:41:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:14 smithi169 ceph-mon[39949]: 4.1f deep-scrub starts 2024-03-12T03:41:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:14 smithi169 ceph-mon[39949]: 4.1f deep-scrub ok 2024-03-12T03:41:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:14 smithi169 ceph-mon[39949]: 3.14 scrub starts 2024-03-12T03:41:14.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:14 smithi169 ceph-mon[39949]: 3.14 scrub ok 2024-03-12T03:41:14.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:14 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:41:14.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:14 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:41:14.976 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:14 smithi173 ceph-mon[33544]: 4.1f deep-scrub starts 2024-03-12T03:41:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:14 smithi173 ceph-mon[33544]: 4.1f deep-scrub ok 2024-03-12T03:41:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:14 smithi173 ceph-mon[33544]: 3.14 scrub starts 2024-03-12T03:41:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:14 smithi173 ceph-mon[33544]: 3.14 scrub ok 2024-03-12T03:41:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:14 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:41:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:14 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:41:15.011 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:15.012 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Total time run: 90.0107 2024-03-12T03:41:15.012 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Total writes made: 133480 2024-03-12T03:41:15.012 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Write size: 65536 2024-03-12T03:41:15.012 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Object size: 65536 2024-03-12T03:41:15.012 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Bandwidth (MB/sec): 92.6835 2024-03-12T03:41:15.012 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev Bandwidth: 42.8278 2024-03-12T03:41:15.012 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max bandwidth (MB/sec): 234.625 2024-03-12T03:41:15.013 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min bandwidth (MB/sec): 0 2024-03-12T03:41:15.013 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Average IOPS: 1482 2024-03-12T03:41:15.013 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev IOPS: 685.257 2024-03-12T03:41:15.013 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max IOPS: 3754 2024-03-12T03:41:15.013 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min IOPS: 0 2024-03-12T03:41:15.013 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Average Latency(s): 0.0107852 2024-03-12T03:41:15.013 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev Latency(s): 0.0279869 2024-03-12T03:41:15.013 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max latency(s): 2.48103 2024-03-12T03:41:15.013 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min latency(s): 0.00107765 2024-03-12T03:41:15.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:15.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:15.043 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-03-12T03:41:15.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:15.153 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:15.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:15.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:15.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:15.281 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:15.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:15.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:15.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:15.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:15.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:15.513 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:15.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:15.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:15.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:15.678 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:15.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:15.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:15.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:15.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:15.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:15.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:15 smithi169 ceph-mon[29344]: pgmap v369: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 694 B/s rd, 61 MiB/s wr, 975 op/s; 40090/389877 objects degraded (10.283%); 16 MiB/s, 262 objects/s recovering 2024-03-12T03:41:15.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:15 smithi169 ceph-mon[29344]: osdmap e233: 8 total, 8 up, 8 in 2024-03-12T03:41:15.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:41:15.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:15 smithi169 ceph-mon[39949]: pgmap v369: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 694 B/s rd, 61 MiB/s wr, 975 op/s; 40090/389877 objects degraded (10.283%); 16 MiB/s, 262 objects/s recovering 2024-03-12T03:41:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:15 smithi169 ceph-mon[39949]: osdmap e233: 8 total, 8 up, 8 in 2024-03-12T03:41:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:41:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:15.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:41:15.954 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:15.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:15.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:15 smithi173 ceph-mon[33544]: pgmap v369: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 694 B/s rd, 61 MiB/s wr, 975 op/s; 40090/389877 objects degraded (10.283%); 16 MiB/s, 262 objects/s recovering 2024-03-12T03:41:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:15 smithi173 ceph-mon[33544]: osdmap e233: 8 total, 8 up, 8 in 2024-03-12T03:41:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:41:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:16.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:16.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:16.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:16.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:16.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:16.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:16.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:16.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:16.359 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:16.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:16.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:16.497 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:16.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:16.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:16.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:16.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:16.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:16.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:16.734 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:16.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:16.775 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:16.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:16.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:16.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:16.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:16.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:16 smithi169 ceph-mon[29344]: 8.5 scrub starts 2024-03-12T03:41:16.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:16 smithi169 ceph-mon[29344]: 4.9 scrub starts 2024-03-12T03:41:16.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:16 smithi169 ceph-mon[29344]: 4.9 scrub ok 2024-03-12T03:41:16.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:16 smithi169 ceph-mon[29344]: 6.19 scrub starts 2024-03-12T03:41:16.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:16 smithi169 ceph-mon[29344]: 6.19 scrub ok 2024-03-12T03:41:16.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:16 smithi169 ceph-mon[39949]: 8.5 scrub starts 2024-03-12T03:41:16.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:16 smithi169 ceph-mon[39949]: 4.9 scrub starts 2024-03-12T03:41:16.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:16 smithi169 ceph-mon[39949]: 4.9 scrub ok 2024-03-12T03:41:16.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:16 smithi169 ceph-mon[39949]: 6.19 scrub starts 2024-03-12T03:41:16.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:16 smithi169 ceph-mon[39949]: 6.19 scrub ok 2024-03-12T03:41:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:16 smithi173 ceph-mon[33544]: 8.5 scrub starts 2024-03-12T03:41:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:16 smithi173 ceph-mon[33544]: 4.9 scrub starts 2024-03-12T03:41:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:16 smithi173 ceph-mon[33544]: 4.9 scrub ok 2024-03-12T03:41:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:16 smithi173 ceph-mon[33544]: 6.19 scrub starts 2024-03-12T03:41:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:16 smithi173 ceph-mon[33544]: 6.19 scrub ok 2024-03-12T03:41:17.013 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:17.050 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:17.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:17.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:17.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:17.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:17.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:17.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:17.334 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:17.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:17.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:17.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:17.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:17.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:17.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:17.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:17.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:17.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:17.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:17.825 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:17.868 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:17.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:17.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:17 smithi169 ceph-mon[29344]: pgmap v371: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 54 MiB/s wr, 859 op/s; 40090/389877 objects degraded (10.283%); 16 MiB/s, 252 objects/s recovering 2024-03-12T03:41:17.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:17 smithi169 ceph-mon[29344]: 5.10 scrub starts 2024-03-12T03:41:17.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:17 smithi169 ceph-mon[29344]: 5.10 scrub ok 2024-03-12T03:41:17.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:17 smithi169 ceph-mon[29344]: 8.5 scrub ok 2024-03-12T03:41:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:17 smithi169 ceph-mon[39949]: pgmap v371: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 54 MiB/s wr, 859 op/s; 40090/389877 objects degraded (10.283%); 16 MiB/s, 252 objects/s recovering 2024-03-12T03:41:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:17 smithi169 ceph-mon[39949]: 5.10 scrub starts 2024-03-12T03:41:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:17 smithi169 ceph-mon[39949]: 5.10 scrub ok 2024-03-12T03:41:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:17 smithi169 ceph-mon[39949]: 8.5 scrub ok 2024-03-12T03:41:17.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:17 smithi173 ceph-mon[33544]: pgmap v371: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 54 MiB/s wr, 859 op/s; 40090/389877 objects degraded (10.283%); 16 MiB/s, 252 objects/s recovering 2024-03-12T03:41:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:17 smithi173 ceph-mon[33544]: 5.10 scrub starts 2024-03-12T03:41:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:17 smithi173 ceph-mon[33544]: 5.10 scrub ok 2024-03-12T03:41:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:17 smithi173 ceph-mon[33544]: 8.5 scrub ok 2024-03-12T03:41:17.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:17.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:17.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:18.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:18.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:18.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:18.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:18.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:18.280 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:18.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:18.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:18.410 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:18.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:18.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:18.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:18.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:18.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:18.676 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:18.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:18.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:18.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:18.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:18.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:18.912 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:18.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:18 smithi169 ceph-mon[29344]: 5.9 scrub starts 2024-03-12T03:41:18.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:18 smithi169 ceph-mon[29344]: 5.9 scrub ok 2024-03-12T03:41:18.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:18 smithi169 ceph-mon[29344]: osdmap e234: 8 total, 8 up, 8 in 2024-03-12T03:41:18.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:18 smithi169 ceph-mon[39949]: 5.9 scrub starts 2024-03-12T03:41:18.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:18 smithi169 ceph-mon[39949]: 5.9 scrub ok 2024-03-12T03:41:18.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:18 smithi169 ceph-mon[39949]: osdmap e234: 8 total, 8 up, 8 in 2024-03-12T03:41:18.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:18.951 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:18.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:18 smithi173 ceph-mon[33544]: 5.9 scrub starts 2024-03-12T03:41:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:18 smithi173 ceph-mon[33544]: 5.9 scrub ok 2024-03-12T03:41:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:18 smithi173 ceph-mon[33544]: osdmap e234: 8 total, 8 up, 8 in 2024-03-12T03:41:19.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:19.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:19.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:19.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:19.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:19.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:19.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:19.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:19.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:19.356 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:19.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:19.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:19.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:19.497 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:19.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:19.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:19.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:19.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:19.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:19.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:19.768 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:19.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:19.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:19.911 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:19.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:19.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:19 smithi169 ceph-mon[29344]: pgmap v372: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 970 B/s rd, 56 MiB/s wr, 1.10k op/s; 39182/397233 objects degraded (9.864%); 20 MiB/s, 322 objects/s recovering 2024-03-12T03:41:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:19 smithi169 ceph-mon[29344]: 3.15 deep-scrub starts 2024-03-12T03:41:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:19 smithi169 ceph-mon[29344]: 3.15 deep-scrub ok 2024-03-12T03:41:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:19 smithi169 ceph-mon[29344]: 4.e scrub starts 2024-03-12T03:41:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:19 smithi169 ceph-mon[29344]: 4.e scrub ok 2024-03-12T03:41:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:19 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 39182/397233 objects degraded (9.864%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:19 smithi169 ceph-mon[39949]: pgmap v372: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 970 B/s rd, 56 MiB/s wr, 1.10k op/s; 39182/397233 objects degraded (9.864%); 20 MiB/s, 322 objects/s recovering 2024-03-12T03:41:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:19 smithi169 ceph-mon[39949]: 3.15 deep-scrub starts 2024-03-12T03:41:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:19 smithi169 ceph-mon[39949]: 3.15 deep-scrub ok 2024-03-12T03:41:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:19 smithi169 ceph-mon[39949]: 4.e scrub starts 2024-03-12T03:41:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:19 smithi169 ceph-mon[39949]: 4.e scrub ok 2024-03-12T03:41:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:19 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 39182/397233 objects degraded (9.864%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:19 smithi173 ceph-mon[33544]: pgmap v372: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 970 B/s rd, 56 MiB/s wr, 1.10k op/s; 39182/397233 objects degraded (9.864%); 20 MiB/s, 322 objects/s recovering 2024-03-12T03:41:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:19 smithi173 ceph-mon[33544]: 3.15 deep-scrub starts 2024-03-12T03:41:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:19 smithi173 ceph-mon[33544]: 3.15 deep-scrub ok 2024-03-12T03:41:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:19 smithi173 ceph-mon[33544]: 4.e scrub starts 2024-03-12T03:41:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:19 smithi173 ceph-mon[33544]: 4.e scrub ok 2024-03-12T03:41:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:19 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 39182/397233 objects degraded (9.864%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:20.013 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:20.037 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:20.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:20.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:20.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:20.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:20.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:20.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:20.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:20.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:20.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:20.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:20.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:20.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:20.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:20.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:20.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:20.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:20.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:20.702 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:20.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:20.803 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:20.857 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:20.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:20 smithi169 ceph-mon[29344]: 6.17 scrub starts 2024-03-12T03:41:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:20 smithi169 ceph-mon[29344]: 6.17 scrub ok 2024-03-12T03:41:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:20 smithi169 ceph-mon[39949]: 6.17 scrub starts 2024-03-12T03:41:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:20 smithi169 ceph-mon[39949]: 6.17 scrub ok 2024-03-12T03:41:20.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:20 smithi173 ceph-mon[33544]: 6.17 scrub starts 2024-03-12T03:41:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:20 smithi173 ceph-mon[33544]: 6.17 scrub ok 2024-03-12T03:41:21.005 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:21.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:21.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:21.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:21.149 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:21.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:21.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:21.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:21.291 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:21.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:21.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:21.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:21.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:21.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:21.586 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:21.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:21.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:21.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:21.735 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:21.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:21.738 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:21 smithi169 ceph-mon[29344]: pgmap v374: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 839 B/s rd, 32 MiB/s wr, 690 op/s; 39182/397233 objects degraded (9.864%); 14 MiB/s, 217 objects/s recovering 2024-03-12T03:41:21.739 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:21 smithi169 ceph-mon[29344]: 3.f scrub starts 2024-03-12T03:41:21.739 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:21 smithi169 ceph-mon[29344]: 3.f scrub ok 2024-03-12T03:41:21.739 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:21 smithi169 ceph-mon[39949]: pgmap v374: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 839 B/s rd, 32 MiB/s wr, 690 op/s; 39182/397233 objects degraded (9.864%); 14 MiB/s, 217 objects/s recovering 2024-03-12T03:41:21.739 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:21 smithi169 ceph-mon[39949]: 3.f scrub starts 2024-03-12T03:41:21.739 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:21 smithi169 ceph-mon[39949]: 3.f scrub ok 2024-03-12T03:41:21.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:21.878 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:21.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:21.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:21 smithi173 ceph-mon[33544]: pgmap v374: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 839 B/s rd, 32 MiB/s wr, 690 op/s; 39182/397233 objects degraded (9.864%); 14 MiB/s, 217 objects/s recovering 2024-03-12T03:41:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:21 smithi173 ceph-mon[33544]: 3.f scrub starts 2024-03-12T03:41:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:21 smithi173 ceph-mon[33544]: 3.f scrub ok 2024-03-12T03:41:22.012 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:41:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:41:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:41:22.037 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:22.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:22.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:22.183 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:22.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:22.285 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:22.323 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:22.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:22.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:22.415 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:41:22.415 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:41:22.415 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-03-12T03:41:22.415 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-03-12T03:41:22.415 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:41:22.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:22.426 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:22.431 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:22.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:22.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:22.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:22.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:22.585 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:22.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:22.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:22.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:22.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:22.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:22.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:22.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:22.864 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:22.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:22.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:22 smithi169 ceph-mon[29344]: 4.13 scrub starts 2024-03-12T03:41:22.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:22 smithi169 ceph-mon[29344]: 4.13 scrub ok 2024-03-12T03:41:22.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:22 smithi169 ceph-mon[39949]: 4.13 scrub starts 2024-03-12T03:41:22.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:22 smithi169 ceph-mon[39949]: 4.13 scrub ok 2024-03-12T03:41:22.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:22 smithi173 ceph-mon[33544]: 4.13 scrub starts 2024-03-12T03:41:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:22 smithi173 ceph-mon[33544]: 4.13 scrub ok 2024-03-12T03:41:22.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:41:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:41:22.993 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:22.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:23.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:23.128 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:23.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:23.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:23.265 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:23.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:23.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:23.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:23.411 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:23.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:23.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:23.555 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:23.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:23.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:23.657 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:23.689 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.3 to 1 (10000) 2024-03-12T03:41:23.702 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:23.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:23.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:23.804 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:23.844 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:23.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:23.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:23 smithi169 ceph-mon[29344]: pgmap v375: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 29 MiB/s wr, 1.29k op/s; 36127/381330 objects degraded (9.474%); 24 MiB/s, 378 objects/s recovering 2024-03-12T03:41:23.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:23 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:23.871 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:23 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3292619607' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:41:23.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:23 smithi169 ceph-mon[39949]: pgmap v375: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 29 MiB/s wr, 1.29k op/s; 36127/381330 objects degraded (9.474%); 24 MiB/s, 378 objects/s recovering 2024-03-12T03:41:23.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:23 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:23.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:23 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3292619607' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:41:23.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:23 smithi173 ceph-mon[33544]: pgmap v375: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 29 MiB/s wr, 1.29k op/s; 36127/381330 objects degraded (9.474%); 24 MiB/s, 378 objects/s recovering 2024-03-12T03:41:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:23 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:23 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3292619607' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:41:23.997 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:23.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:24.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:24.140 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:24.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:24.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:24.276 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:24.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:24.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:24.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:24.408 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:24.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:24.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:24.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:24.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:24.627 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-12T03:41:24.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:24.755 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:24.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:24.856 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:24.860 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:24 smithi169 ceph-mon[29344]: 6.8 scrub starts 2024-03-12T03:41:24.860 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:24 smithi169 ceph-mon[29344]: 6.8 scrub ok 2024-03-12T03:41:24.860 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:24 smithi169 ceph-mon[29344]: 6.2 deep-scrub starts 2024-03-12T03:41:24.861 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:24 smithi169 ceph-mon[29344]: 6.2 deep-scrub ok 2024-03-12T03:41:24.861 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:24 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3292619607' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-12T03:41:24.861 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:24 smithi169 ceph-mon[29344]: osdmap e235: 8 total, 8 up, 8 in 2024-03-12T03:41:24.861 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:24 smithi169 ceph-mon[39949]: 6.8 scrub starts 2024-03-12T03:41:24.861 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:24 smithi169 ceph-mon[39949]: 6.8 scrub ok 2024-03-12T03:41:24.861 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:24 smithi169 ceph-mon[39949]: 6.2 deep-scrub starts 2024-03-12T03:41:24.861 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:24 smithi169 ceph-mon[39949]: 6.2 deep-scrub ok 2024-03-12T03:41:24.861 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:24 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3292619607' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-12T03:41:24.861 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:24 smithi169 ceph-mon[39949]: osdmap e235: 8 total, 8 up, 8 in 2024-03-12T03:41:24.901 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:24.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:24.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:24 smithi173 ceph-mon[33544]: 6.8 scrub starts 2024-03-12T03:41:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:24 smithi173 ceph-mon[33544]: 6.8 scrub ok 2024-03-12T03:41:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:24 smithi173 ceph-mon[33544]: 6.2 deep-scrub starts 2024-03-12T03:41:24.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:24 smithi173 ceph-mon[33544]: 6.2 deep-scrub ok 2024-03-12T03:41:24.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:24 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3292619607' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-12T03:41:24.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:24 smithi173 ceph-mon[33544]: osdmap e235: 8 total, 8 up, 8 in 2024-03-12T03:41:25.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:25.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:25.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:25.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:25.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:25.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:25.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:25.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:25.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:25.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:25.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:25.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:25.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:25.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:25.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:25.579 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:25.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:25.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:25.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:25.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:25.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:25.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:25.851 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:25.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:25.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:25.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:25.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:25.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:25.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:25 smithi173 ceph-mon[33544]: pgmap v377: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 767 B/s rd, 29 MiB/s wr, 1.29k op/s; 36127/381330 objects degraded (9.474%); 24 MiB/s, 378 objects/s recovering 2024-03-12T03:41:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:25 smithi173 ceph-mon[33544]: 5.1a scrub starts 2024-03-12T03:41:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:25 smithi173 ceph-mon[33544]: 5.1a scrub ok 2024-03-12T03:41:26.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:26.117 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:26.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:26.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:25 smithi169 ceph-mon[29344]: pgmap v377: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 767 B/s rd, 29 MiB/s wr, 1.29k op/s; 36127/381330 objects degraded (9.474%); 24 MiB/s, 378 objects/s recovering 2024-03-12T03:41:26.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:25 smithi169 ceph-mon[29344]: 5.1a scrub starts 2024-03-12T03:41:26.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:25 smithi169 ceph-mon[29344]: 5.1a scrub ok 2024-03-12T03:41:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:25 smithi169 ceph-mon[39949]: pgmap v377: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 767 B/s rd, 29 MiB/s wr, 1.29k op/s; 36127/381330 objects degraded (9.474%); 24 MiB/s, 378 objects/s recovering 2024-03-12T03:41:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:25 smithi169 ceph-mon[39949]: 5.1a scrub starts 2024-03-12T03:41:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:25 smithi169 ceph-mon[39949]: 5.1a scrub ok 2024-03-12T03:41:26.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:26.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:26.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:26.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:26.388 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:26.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:26.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:26.516 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:26.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:26.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:26.654 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:26.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:26.721 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:41:26.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:26.802 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:26.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:26.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:26.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:26.947 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:26.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:26 smithi173 ceph-mon[33544]: 6.12 scrub starts 2024-03-12T03:41:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:26 smithi173 ceph-mon[33544]: 6.12 scrub ok 2024-03-12T03:41:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:26 smithi173 ceph-mon[33544]: 6.1a scrub starts 2024-03-12T03:41:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:26 smithi173 ceph-mon[33544]: 6.1a scrub ok 2024-03-12T03:41:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:26 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3381525980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:41:27.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:27.056 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:26 smithi169 ceph-mon[29344]: 6.12 scrub starts 2024-03-12T03:41:27.056 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:26 smithi169 ceph-mon[29344]: 6.12 scrub ok 2024-03-12T03:41:27.056 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:26 smithi169 ceph-mon[29344]: 6.1a scrub starts 2024-03-12T03:41:27.057 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:26 smithi169 ceph-mon[29344]: 6.1a scrub ok 2024-03-12T03:41:27.057 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:26 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3381525980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:41:27.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:26 smithi169 ceph-mon[39949]: 6.12 scrub starts 2024-03-12T03:41:27.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:26 smithi169 ceph-mon[39949]: 6.12 scrub ok 2024-03-12T03:41:27.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:26 smithi169 ceph-mon[39949]: 6.1a scrub starts 2024-03-12T03:41:27.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:26 smithi169 ceph-mon[39949]: 6.1a scrub ok 2024-03-12T03:41:27.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:26 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3381525980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:41:27.083 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:27.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:27.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:27.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:27.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:27.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:27.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:27.340 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:27.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:27.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:27.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:27.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:27.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:27.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:27.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:27.618 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:27.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:27.719 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:27.759 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:27.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:27.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:27.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:27.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:27 smithi173 ceph-mon[33544]: pgmap v378: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 663 op/s; 36127/381330 objects degraded (9.474%); 11 MiB/s, 179 objects/s recovering 2024-03-12T03:41:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:27 smithi173 ceph-mon[33544]: 6.1c scrub starts 2024-03-12T03:41:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:27 smithi173 ceph-mon[33544]: 6.1c scrub ok 2024-03-12T03:41:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:27 smithi173 ceph-mon[33544]: 3.3 scrub starts 2024-03-12T03:41:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:27 smithi173 ceph-mon[33544]: 3.3 scrub ok 2024-03-12T03:41:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:27 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:41:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:27 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3381525980' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:41:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:27 smithi173 ceph-mon[33544]: osdmap e236: 8 total, 8 up, 8 in 2024-03-12T03:41:27.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:28.037 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:28.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:28.140 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:28.171 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:28.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:28.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:28.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[29344]: pgmap v378: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 663 op/s; 36127/381330 objects degraded (9.474%); 11 MiB/s, 179 objects/s recovering 2024-03-12T03:41:28.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[29344]: 6.1c scrub starts 2024-03-12T03:41:28.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[29344]: 6.1c scrub ok 2024-03-12T03:41:28.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[29344]: 3.3 scrub starts 2024-03-12T03:41:28.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[29344]: 3.3 scrub ok 2024-03-12T03:41:28.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:41:28.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3381525980' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:41:28.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[29344]: osdmap e236: 8 total, 8 up, 8 in 2024-03-12T03:41:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[39949]: pgmap v378: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 663 op/s; 36127/381330 objects degraded (9.474%); 11 MiB/s, 179 objects/s recovering 2024-03-12T03:41:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[39949]: 6.1c scrub starts 2024-03-12T03:41:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[39949]: 6.1c scrub ok 2024-03-12T03:41:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[39949]: 3.3 scrub starts 2024-03-12T03:41:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[39949]: 3.3 scrub ok 2024-03-12T03:41:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:41:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3381525980' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:41:28.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:27 smithi169 ceph-mon[39949]: osdmap e236: 8 total, 8 up, 8 in 2024-03-12T03:41:28.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:28.319 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:28.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:28.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:28.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:28.466 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:28.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:28.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:28.609 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:28.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:28.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:28.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:28.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:28.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:28.883 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:28.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:28.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:28.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:28 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 36127/381330 objects degraded (9.474%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:29.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:29.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:29.119 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:29.155 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:29.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:29.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:29.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:28 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 36127/381330 objects degraded (9.474%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:29.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:28 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 36127/381330 objects degraded (9.474%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:29.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:29.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:29.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:29.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:29.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:29.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:29.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:29.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:29.516 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:29.559 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:29.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:29.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:29.694 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:29.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:29.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:29.832 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:29.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:29.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:29.934 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:29.973 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:29.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:29.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:29 smithi173 ceph-mon[33544]: pgmap v380: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 33287/365127 objects degraded (9.117%); 22 MiB/s, 348 objects/s recovering 2024-03-12T03:41:30.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:30.122 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:30.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:30.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:29 smithi169 ceph-mon[29344]: pgmap v380: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 33287/365127 objects degraded (9.117%); 22 MiB/s, 348 objects/s recovering 2024-03-12T03:41:30.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:29 smithi169 ceph-mon[39949]: pgmap v380: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 33287/365127 objects degraded (9.117%); 22 MiB/s, 348 objects/s recovering 2024-03-12T03:41:30.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:30.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:30.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:30.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:30.357 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:30.400 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:30.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:30.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:30.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:30.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:30.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:30.670 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:30.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:30.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:30.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:30.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:30.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:30.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:30.943 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:30.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:31.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:31.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:31.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:31.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:31.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:31.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:31.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:31.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:31.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:31.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:31.411 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-12T03:41:31.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:31.515 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:31.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:31.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:31.646 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:31.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:31.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:31 smithi169 ceph-mon[29344]: pgmap v381: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 675 op/s; 33287/365127 objects degraded (9.117%); 11 MiB/s, 168 objects/s recovering 2024-03-12T03:41:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:31 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:41:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:31 smithi169 ceph-mon[39949]: pgmap v381: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 675 op/s; 33287/365127 objects degraded (9.117%); 11 MiB/s, 168 objects/s recovering 2024-03-12T03:41:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:31 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:41:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:31.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:31.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:31.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:31.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:31.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:31.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:31.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:31 smithi173 ceph-mon[33544]: pgmap v381: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 675 op/s; 33287/365127 objects degraded (9.117%); 11 MiB/s, 168 objects/s recovering 2024-03-12T03:41:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:31 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:41:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:32.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:32.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:32.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:32.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:32.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:32.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:32.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:32.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:41:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:41:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:41:32.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:32.316 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:32.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:32.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:32.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:32.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:32.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:32.479 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:32.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:32.602 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:32.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:32.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:32.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:32.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:32.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:32.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:32.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:32.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:32.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:32.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:32.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:41:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:41:33.008 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:33.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:33.110 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:33.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:33.151 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:33.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:33.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:33.289 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:33.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:33.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:33.431 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:33.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:33.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:33.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:33.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:33.573 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:33.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:33.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:33.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:33.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:33.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:33.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:33.815 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:33 smithi169 ceph-mon[29344]: pgmap v382: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 30249/348636 objects degraded (8.676%); 22 MiB/s, 349 objects/s recovering 2024-03-12T03:41:33.815 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:33 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:33.815 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:33 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 30249/348636 objects degraded (8.676%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:33.815 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:33 smithi169 ceph-mon[39949]: pgmap v382: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 30249/348636 objects degraded (8.676%); 22 MiB/s, 349 objects/s recovering 2024-03-12T03:41:33.815 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:33 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:33.815 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:33 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 30249/348636 objects degraded (8.676%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:33.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:33.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:33.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:33.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:33.985 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:33.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:33 smithi173 ceph-mon[33544]: pgmap v382: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 30249/348636 objects degraded (8.676%); 22 MiB/s, 349 objects/s recovering 2024-03-12T03:41:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:33 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:33 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 30249/348636 objects degraded (8.676%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:34.087 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:34.126 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:41:34.126 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:41:34.126 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:41:34.126 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:41:34.126 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:41:34.126 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:41:34.126 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:41:34.126 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:41:34.126 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:41:34.127 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:41:34.127 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:41:34.127 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:41:34.127 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:41:34.127 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:41:34.127 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:41:34.127 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:41:34.138 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:34.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:34.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:34.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:34.281 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:34.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:34.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:34.423 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:34.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:34.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:41:34.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:34.619 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:34.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:34.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:34 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2311234828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:41:34.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:34 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2311234828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:41:34.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:34.769 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:34.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:34.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:34.937 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:34.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:34.955 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:34.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:34.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:34 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2311234828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:41:35.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:35.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:35.114 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:35.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:35.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:35.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:35.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:35.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:35.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:35.389 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:35.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:35.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:35.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:35.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:35.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:35.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:35.670 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:35.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:35.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:35.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:35.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:35.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:35.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:35 smithi169 ceph-mon[29344]: pgmap v383: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 30249/348636 objects degraded (8.676%); 18 MiB/s, 287 objects/s recovering 2024-03-12T03:41:35.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:35 smithi169 ceph-mon[39949]: pgmap v383: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 30249/348636 objects degraded (8.676%); 18 MiB/s, 287 objects/s recovering 2024-03-12T03:41:35.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:35.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:35 smithi173 ceph-mon[33544]: pgmap v383: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 30249/348636 objects degraded (8.676%); 18 MiB/s, 287 objects/s recovering 2024-03-12T03:41:36.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:36.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:36.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:36.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:36.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:36.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:36.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:36.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:36.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:36.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:36.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:36.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:36.479 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:36.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.4 to 1 (10000) 2024-03-12T03:41:36.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:36.622 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:36.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:36.723 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:36.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:36.766 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:36.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:36.867 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:36.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:36.906 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:36.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:36.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:36 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/172292226' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:41:36.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:36 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/172292226' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:41:36.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:41:36.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:36 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/172292226' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:41:37.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:37.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:37.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:37.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:37.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:37.190 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:37.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:37.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:37.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:37.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:37.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:37.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:37.465 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:37.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:37.566 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:37.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:37.611 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:37.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:37.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:37.753 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:37.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:37.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:37.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:37.871 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:37 smithi169 ceph-mon[29344]: pgmap v384: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 30249/347682 objects degraded (8.700%); 18 MiB/s, 287 objects/s recovering 2024-03-12T03:41:37.871 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:37 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:41:37.871 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:37 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:41:37.871 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:37 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/172292226' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-12T03:41:37.871 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:37 smithi169 ceph-mon[29344]: osdmap e237: 8 total, 8 up, 8 in 2024-03-12T03:41:37.873 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:37 smithi169 ceph-mon[39949]: pgmap v384: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 30249/347682 objects degraded (8.700%); 18 MiB/s, 287 objects/s recovering 2024-03-12T03:41:37.873 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:37 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:41:37.873 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:37 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:41:37.873 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:37 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/172292226' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-12T03:41:37.873 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:37 smithi169 ceph-mon[39949]: osdmap e237: 8 total, 8 up, 8 in 2024-03-12T03:41:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:37.899 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:37.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:37.984 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:37.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:37 smithi173 ceph-mon[33544]: pgmap v384: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 30249/347682 objects degraded (8.700%); 18 MiB/s, 287 objects/s recovering 2024-03-12T03:41:37.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:37 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:41:37.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:37 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:41:37.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:37 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/172292226' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-12T03:41:37.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:37 smithi173 ceph-mon[33544]: osdmap e237: 8 total, 8 up, 8 in 2024-03-12T03:41:38.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:38.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:38.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:38.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:38.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:38.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:38.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:38.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:38.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:38.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:38.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:38.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:38.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:38.515 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-12T03:41:38.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:38.552 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:38.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:38.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:38.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:38.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:38.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:38.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:38.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:38.762 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:38 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 30249/347682 objects degraded (8.700%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:38.762 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:38 smithi169 ceph-mon[29344]: osdmap e238: 8 total, 8 up, 8 in 2024-03-12T03:41:38.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:38 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 30249/347682 objects degraded (8.700%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:38.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:38 smithi169 ceph-mon[39949]: osdmap e238: 8 total, 8 up, 8 in 2024-03-12T03:41:38.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:38.824 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:38.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:38.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:38.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:38.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:38.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:38.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:38 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 30249/347682 objects degraded (8.700%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:38.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:38 smithi173 ceph-mon[33544]: osdmap e238: 8 total, 8 up, 8 in 2024-03-12T03:41:39.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:39.091 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:39.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:39.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:39.237 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:39.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:39.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:39.380 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:39.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:39.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:39.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:39.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:39.528 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:39.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:39.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:39.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:39.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:39.764 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:39.803 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:39.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:39.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:39.917 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:39 smithi169 ceph-mon[29344]: pgmap v386: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 27476/332340 objects degraded (8.267%); 18 MiB/s, 286 objects/s recovering 2024-03-12T03:41:39.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:39 smithi169 ceph-mon[39949]: pgmap v386: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 27476/332340 objects degraded (8.267%); 18 MiB/s, 286 objects/s recovering 2024-03-12T03:41:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:39.947 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:39.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:39 smithi173 ceph-mon[33544]: pgmap v386: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 27476/332340 objects degraded (8.267%); 18 MiB/s, 286 objects/s recovering 2024-03-12T03:41:40.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:40.098 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:40.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:40.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:40.228 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:41:40.228 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:41:40.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:40.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:40.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:40.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:40.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:40.369 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:40.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:40.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:40.509 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:40.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:40.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:40.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:40.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:40.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:40.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:40.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:40.811 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:40.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:40.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:40.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:40.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:41.050 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:41.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:41.189 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:41.230 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:41.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:41.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:41.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:41.362 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:41.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:41.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:41.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:41.509 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:41.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:41.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:41.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:41.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:41.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:41 smithi169 ceph-mon[29344]: pgmap v388: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 679 op/s; 27476/332340 objects degraded (8.267%); 10 MiB/s, 167 objects/s recovering 2024-03-12T03:41:41.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:41 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:41:41.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:41 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:41:41.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:41 smithi169 ceph-mon[29344]: osdmap e239: 8 total, 8 up, 8 in 2024-03-12T03:41:41.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:41 smithi169 ceph-mon[39949]: pgmap v388: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 679 op/s; 27476/332340 objects degraded (8.267%); 10 MiB/s, 167 objects/s recovering 2024-03-12T03:41:41.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:41 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:41:41.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:41 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:41:41.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:41 smithi169 ceph-mon[39949]: osdmap e239: 8 total, 8 up, 8 in 2024-03-12T03:41:41.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:41.781 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:41.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:41.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:41.913 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:41.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:41.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:41 smithi173 ceph-mon[33544]: pgmap v388: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 679 op/s; 27476/332340 objects degraded (8.267%); 10 MiB/s, 167 objects/s recovering 2024-03-12T03:41:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:41 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:41:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:41 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:41:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:41 smithi173 ceph-mon[33544]: osdmap e239: 8 total, 8 up, 8 in 2024-03-12T03:41:42.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:42.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:42.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:42.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:42.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:42.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:41:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:41:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:41:42.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:42.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:42.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:42.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:42.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:42.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:42.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:42.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:42.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:42.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:42.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:42.610 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:41:42.683 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:42.730 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:42.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:42.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:42.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:42.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:42.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:42 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3680660099' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:41:42.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:42 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:41:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:42 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3680660099' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:41:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:42 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:41:42.957 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:42 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3680660099' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:41:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:42 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:41:42.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:41:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:41:42.989 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:42.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:43.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:43.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:43.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:43.227 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:43.255 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:43.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:43.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:43.356 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:43.392 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:43.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:43.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:43.516 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:43.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:43.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:43.660 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:43.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:43.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:43.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:43.793 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:43.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:43.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:43 smithi169 ceph-mon[29344]: pgmap v390: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.80k op/s; 24650/315303 objects degraded (7.818%); 29 MiB/s, 463 objects/s recovering 2024-03-12T03:41:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:43 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:41:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:43 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:41:43.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:43 smithi169 ceph-mon[29344]: osdmap e240: 8 total, 8 up, 8 in 2024-03-12T03:41:43.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:43 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:43.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:43 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 24650/315303 objects degraded (7.818%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:43.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:43 smithi169 ceph-mon[39949]: pgmap v390: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.80k op/s; 24650/315303 objects degraded (7.818%); 29 MiB/s, 463 objects/s recovering 2024-03-12T03:41:43.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:43 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:41:43.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:43 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:41:43.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:43 smithi169 ceph-mon[39949]: osdmap e240: 8 total, 8 up, 8 in 2024-03-12T03:41:43.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:43 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:43.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:43 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 24650/315303 objects degraded (7.818%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:43.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:43.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:43 smithi173 ceph-mon[33544]: pgmap v390: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.80k op/s; 24650/315303 objects degraded (7.818%); 29 MiB/s, 463 objects/s recovering 2024-03-12T03:41:43.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:43 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:41:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:43 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:41:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:43 smithi173 ceph-mon[33544]: osdmap e240: 8 total, 8 up, 8 in 2024-03-12T03:41:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:43 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:43 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 24650/315303 objects degraded (7.818%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:41:44.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:44.054 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:44.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:44.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:44.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:44.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:44.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:44.292 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:44.331 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:44.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:44.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:44.433 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:44.479 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:44.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:44.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:44.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:44.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:44.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:44.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:44.775 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:44.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:44.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:44.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:44.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:45.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:45.043 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:45.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:45.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:45.145 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:45.188 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:45.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:45.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:45.339 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:45.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:45.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-12T03:41:45.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:45.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:45.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:45.568 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:45.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:45.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:45.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:45.702 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:45.763 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:45.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:45.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:45.912 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:45.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:45.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:45.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:45 smithi169 ceph-mon[39949]: pgmap v392: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 946 op/s; 24650/315303 objects degraded (7.818%); 15 MiB/s, 240 objects/s recovering 2024-03-12T03:41:45.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:45 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:45.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:45 smithi169 ceph-mon[29344]: pgmap v392: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 946 op/s; 24650/315303 objects degraded (7.818%); 15 MiB/s, 240 objects/s recovering 2024-03-12T03:41:45.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:45 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:45.944 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:41:45.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:45 smithi173 ceph-mon[33544]: pgmap v392: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 946 op/s; 24650/315303 objects degraded (7.818%); 15 MiB/s, 240 objects/s recovering 2024-03-12T03:41:45.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:45 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:41:45.994 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:41:46.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:46.042 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:41:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:46.057 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:46.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:46.089 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:41:46.133 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:41:46.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:46.177 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:41:46.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:46.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:46.225 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:41:46.266 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:41:46.272 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:41:46.273 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:41:46.273 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:41:46.273 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:41:46.273 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:41:46.273 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:41:46.273 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:41:46.274 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:41:46.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:46.334 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:46.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:46.436 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:46.468 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:46.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:46.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:46.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:46.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:46.702 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:46.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:46.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:46.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:46.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:46.883 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:46.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:46.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3677258098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:41:46.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3677258098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:41:46.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:46.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3677258098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:41:47.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:47.028 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:47.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:47.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:47.173 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:47.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:47.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:47.318 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:47.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:47.419 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:47.462 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:47.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:47.564 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:47.606 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:47.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:47.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:47.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:47.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:47.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:47.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:47.814 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:47.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:47.899 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:47.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:47 smithi169 ceph-mon[29344]: pgmap v393: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 745 op/s; 24650/315303 objects degraded (7.818%); 12 MiB/s, 188 objects/s recovering 2024-03-12T03:41:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:47 smithi169 ceph-mon[39949]: pgmap v393: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 745 op/s; 24650/315303 objects degraded (7.818%); 12 MiB/s, 188 objects/s recovering 2024-03-12T03:41:47.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:47 smithi173 ceph-mon[33544]: pgmap v393: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 745 op/s; 24650/315303 objects degraded (7.818%); 12 MiB/s, 188 objects/s recovering 2024-03-12T03:41:48.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:48.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:48.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:48.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:48.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:48.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:48.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:48.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:48.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:48.310 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:48.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:48.411 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:48.458 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:48.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:48.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:48.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:48.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:48.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:48.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:41:48.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:48.746 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:48.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:48.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:48.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:48 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 21960/298950 objects degraded (7.346%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-12T03:41:48.893 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:48 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 21960/298950 objects degraded (7.346%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-12T03:41:48.895 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:48.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:48.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:48 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 21960/298950 objects degraded (7.346%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-12T03:41:48.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:49.041 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:49.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:49.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:49.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:49.180 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:49.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:49.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:49.327 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:49.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:49.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:49.463 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:49.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:49.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:49.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:49.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:49.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:49.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:49.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:49.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:49.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:41:49.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:49.898 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:49.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:49.957 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:49 smithi169 ceph-mon[29344]: pgmap v394: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 21960/298950 objects degraded (7.346%); 22 MiB/s, 355 objects/s recovering 2024-03-12T03:41:49.958 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:49 smithi169 ceph-mon[39949]: pgmap v394: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 21960/298950 objects degraded (7.346%); 22 MiB/s, 355 objects/s recovering 2024-03-12T03:41:49.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:49 smithi173 ceph-mon[33544]: pgmap v394: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 21960/298950 objects degraded (7.346%); 22 MiB/s, 355 objects/s recovering 2024-03-12T03:41:49.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:50.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:50.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:50.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:50.165 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:50.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:50.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:50.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:50.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:50.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:50.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:50.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:50.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:50.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:50.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:50.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:50.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:50.714 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:50.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:50.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:50.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:50.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:50.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:50.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:50.979 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:50.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:51.080 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:51.123 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:51.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:51.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:51.250 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:51.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:51.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:51.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:51.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:51.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:51.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:51.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:51.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:51.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:51.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:51.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:51.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:51.787 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:51.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:51.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:51.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:51.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:51.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:51 smithi169 ceph-mon[29344]: pgmap v395: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s; 21960/298950 objects degraded (7.346%); 19 MiB/s, 304 objects/s recovering 2024-03-12T03:41:51.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:51 smithi169 ceph-mon[39949]: pgmap v395: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s; 21960/298950 objects degraded (7.346%); 19 MiB/s, 304 objects/s recovering 2024-03-12T03:41:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:51.925 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:51.925 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:41:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:41:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:41:51.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:51 smithi173 ceph-mon[33544]: pgmap v395: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s; 21960/298950 objects degraded (7.346%); 19 MiB/s, 304 objects/s recovering 2024-03-12T03:41:52.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:52.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:52.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:52.159 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:52.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:52.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:52.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:52.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:52.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:52.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:52.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:52.498 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:52.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:52.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:52.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:52.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:52.735 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:52.779 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:52.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:52.881 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:52.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:52.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:52.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:52.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:41:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:41:53.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:53.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:53.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:53.151 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:53.189 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:53.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:53.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:53.340 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:53.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:53.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:53.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:53.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:53.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:53.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:53.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:53.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:53.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:53.726 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:53.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:53.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:53.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:53.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:53.860 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:53.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:53.942 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:53 smithi169 ceph-mon[29344]: pgmap v396: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s; 19389/282813 objects degraded (6.856%); 18 MiB/s, 281 objects/s recovering 2024-03-12T03:41:53.942 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:53 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:53.942 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:53 smithi169 ceph-mon[29344]: osdmap e241: 8 total, 8 up, 8 in 2024-03-12T03:41:53.942 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:53 smithi169 ceph-mon[39949]: pgmap v396: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s; 19389/282813 objects degraded (6.856%); 18 MiB/s, 281 objects/s recovering 2024-03-12T03:41:53.943 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:53 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:53.943 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:53 smithi169 ceph-mon[39949]: osdmap e241: 8 total, 8 up, 8 in 2024-03-12T03:41:53.962 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:53 smithi173 ceph-mon[33544]: pgmap v396: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s; 19389/282813 objects degraded (6.856%); 18 MiB/s, 281 objects/s recovering 2024-03-12T03:41:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:53 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:53 smithi173 ceph-mon[33544]: osdmap e241: 8 total, 8 up, 8 in 2024-03-12T03:41:53.990 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:53.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:54.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:54.121 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:54.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:54.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:54.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:54.250 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:54.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:54.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:54.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:54.377 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:41:54.377 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:41:54.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.377 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175342 8.c 8.47ae0eec (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.377 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.610499+0000", 2024-03-12T03:41:54.377 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.29797933300000001, 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.298004987, 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175342 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.610499+0000", 2024-03-12T03:41:54.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.610499+0000", 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.610500+0000", 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6840000000000001e-06 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.610507+0000", 2024-03-12T03:41:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8109999999999997e-06 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.610523+0000", 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6276000000000001e-05 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.610533+0000", 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9920000000000002e-06 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.610604+0000", 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0454e-05 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.610613+0000", 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8459999999999997e-06 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175245 8.c 8.2771aa5c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.449165+0000", 2024-03-12T03:41:54.381 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.45931253399999999, 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.45939968199999998, 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175245 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.449165+0000", 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.449165+0000", 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.449166+0000", 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.051e-06 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.449172+0000", 2024-03-12T03:41:54.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3739999999999997e-06 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.449212+0000", 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9824999999999999e-05 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.449219+0000", 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.985e-06 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.449276+0000", 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7182000000000002e-05 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.449286+0000", 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9899999999999992e-06 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175304 8.c 8.1e6c3b2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.510394+0000", 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.39808385200000002, 2024-03-12T03:41:54.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.39821012700000002, 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175304 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.510394+0000", 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.510394+0000", 2024-03-12T03:41:54.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.510395+0000", 2024-03-12T03:41:54.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1200000000000001e-07 2024-03-12T03:41:54.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.510400+0000", 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3399999999999997e-06 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.510415+0000", 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4985e-05 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.510425+0000", 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4949999999999997e-06 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.510484+0000", 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9633999999999999e-05 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.510503+0000", 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8751999999999999e-05 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175259 8.c 8.cd5e793c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.389 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.460443+0000", 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.44803462500000002, 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.44821033500000002, 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175259 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.460443+0000", 2024-03-12T03:41:54.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.460443+0000", 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.460445+0000", 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6169999999999999e-06 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.460451+0000", 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4629999999999998e-06 2024-03-12T03:41:54.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.460468+0000", 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7048999999999999e-05 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.460477+0000", 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7369999999999995e-06 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.460539+0000", 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1690000000000001e-05 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.460546+0000", 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1380000000000001e-06 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175223 8.c 8.22f4012c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.423213+0000", 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.48526511999999999, 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.48547987399999998, 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.393 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175223 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.423213+0000", 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.423213+0000", 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.423214+0000", 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7499999999999999e-07 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.423218+0000", 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3800000000000004e-06 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.423229+0000", 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0716999999999999e-05 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.423236+0000", 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2130000000000002e-06 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.423287+0000", 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1576999999999998e-05 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.423296+0000", 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2160000000000006e-06 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175305 8.c 8.5404b49c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.514018+0000", 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.39445954700000002, 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.39471119300000002, 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175305 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.514018+0000", 2024-03-12T03:41:54.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.514018+0000", 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.514020+0000", 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.553e-06 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.514027+0000", 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7320000000000001e-06 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.514494+0000", 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00046772200000000001 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.514507+0000", 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2257e-05 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.514717+0000", 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00021052199999999999 2024-03-12T03:41:54.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.514729+0000", 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1584e-05 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175180 8.c 8.d070b8ac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.386908+0000", 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.52156935599999998, 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.52185801099999996, 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.400 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175180 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.386908+0000", 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.386908+0000", 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.386909+0000", 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7899999999999999e-07 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.386912+0000", 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1889999999999999e-06 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.386920+0000", 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0819999999999999e-06 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.386926+0000", 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7509999999999997e-06 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.386999+0000", 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3102000000000006e-05 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.387010+0000", 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0934000000000001e-05 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.403 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175099 8.c 8.4ebb355c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.303953+0000", 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.60452507700000002, 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.60485251299999998, 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175099 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.303953+0000", 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.303953+0000", 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.303954+0000", 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4720000000000001e-06 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.303961+0000", 2024-03-12T03:41:54.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0829999999999998e-06 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.303972+0000", 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1054e-05 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.303982+0000", 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7010000000000008e-06 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.307066+0000", 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0030839510000000001 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.307079+0000", 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2697000000000001e-05 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175316 8.c 8.efbd650c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.528808+0000", 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.37967023300000002, 2024-03-12T03:41:54.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38004755699999998, 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175316 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.528808+0000", 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.528808+0000", 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.528810+0000", 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0999999999999998e-06 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.528818+0000", 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2689999999999999e-06 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.528848+0000", 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0355999999999999e-05 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.528860+0000", 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1304000000000001e-05 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.529016+0000", 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000156179 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.529030+0000", 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4365000000000001e-05 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175325 8.c 8.2e1735bc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.549134+0000", 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.35934397200000001, 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.35976056699999998, 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.411 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175325 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.549134+0000", 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.549134+0000", 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.549135+0000", 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5710000000000001e-06 2024-03-12T03:41:54.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.549143+0000", 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.277e-06 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.549157+0000", 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4375e-05 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.549164+0000", 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6699999999999997e-06 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.549226+0000", 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2027999999999996e-05 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.549236+0000", 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0586e-05 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175107 8.c 8.f25c021c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.314307+0000", 2024-03-12T03:41:54.414 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.59417058899999997, 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.59462419200000005, 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175107 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.314307+0000", 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.314307+0000", 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.314309+0000", 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.778e-06 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.314315+0000", 2024-03-12T03:41:54.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0939999999999996e-06 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.314326+0000", 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1307e-05 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.314334+0000", 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0299999999999994e-06 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.314445+0000", 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000110801 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.314455+0000", 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5640000000000002e-06 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175188 8.c 8.8539099c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.393388+0000", 2024-03-12T03:41:54.418 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.51508977199999995, 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.51558030700000002, 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175188 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.393388+0000", 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.393388+0000", 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.393390+0000", 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4670000000000001e-06 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.393395+0000", 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6010000000000002e-06 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.393410+0000", 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4419999999999999e-05 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.393418+0000", 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6419999999999996e-06 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.393939+0000", 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052109400000000005 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.393949+0000", 2024-03-12T03:41:54.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3030000000000005e-06 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175277 8.c 8.463daaec (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.482377+0000", 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.42610043800000003, 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.42662769699999997, 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.422 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175277 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.482377+0000", 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.482377+0000", 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.482379+0000", 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.313e-06 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.482384+0000", 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2689999999999999e-06 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.482396+0000", 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1603000000000001e-05 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.482404+0000", 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1790000000000008e-06 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.482461+0000", 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7626e-05 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.482470+0000", 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5099999999999998e-06 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175123 8.c 8.c14b6f9c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.338153+0000", 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.57032494600000005, 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.57088848299999995, 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175123 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.338153+0000", 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.338153+0000", 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.338155+0000", 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1600000000000001e-06 2024-03-12T03:41:54.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.338162+0000", 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3690000000000001e-06 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.338181+0000", 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8233999999999999e-05 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.338190+0000", 2024-03-12T03:41:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5089999999999999e-06 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.338314+0000", 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000123764 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.338322+0000", 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.571e-06 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175206 8.c 8.5c7ee4dc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.411081+0000", 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.49739723800000002, 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49804974899999999, 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175206 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.411081+0000", 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.411081+0000", 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.411082+0000", 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2999999999999999e-07 2024-03-12T03:41:54.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.411086+0000", 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4499999999999997e-06 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.411101+0000", 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4538e-05 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.411109+0000", 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7020000000000007e-06 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.411167+0000", 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7683999999999999e-05 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.411176+0000", 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6780000000000006e-06 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.434 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:175131 8.c 8.36764b1c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-03-12T03:41:54.434 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:41:53.345674+0000", 2024-03-12T03:41:54.434 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.56280364299999996, 2024-03-12T03:41:54.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.56352859799999999, 2024-03-12T03:41:54.434 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:41:54.434 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for peered", 2024-03-12T03:41:54.434 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:41:54.434 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:41:54.434 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:41:54.434 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 175131 2024-03-12T03:41:54.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.345674+0000", 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.345674+0000", 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.345676+0000", 2024-03-12T03:41:54.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5009999999999999e-06 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.345681+0000", 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3240000000000002e-06 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.345695+0000", 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4056999999999999e-05 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.345703+0000", 2024-03-12T03:41:54.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5900000000000002e-06 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.345785+0000", 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2319999999999998e-05 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:41:53.345793+0000", 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4370000000000007e-06 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:41:54.437 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:41:54.438 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 16 2024-03-12T03:41:54.438 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:41:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:54.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:54.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:54.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:54.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:54.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:54.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:54.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:54.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:54.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:54.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:54.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:54.924 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:54.952 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:54.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:55.054 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:55.082 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:55.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:55.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:55.215 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:55.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:55.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-12T03:41:55.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:55.357 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:55.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:55.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:55.480 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:55 smithi169 ceph-mon[39949]: pgmap v398: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s; 19389/282813 objects degraded (6.856%); 18 MiB/s, 281 objects/s recovering 2024-03-12T03:41:55.480 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:55 smithi169 ceph-mon[39949]: osdmap e242: 8 total, 8 up, 8 in 2024-03-12T03:41:55.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:55 smithi169 ceph-mon[29344]: pgmap v398: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s; 19389/282813 objects degraded (6.856%); 18 MiB/s, 281 objects/s recovering 2024-03-12T03:41:55.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:55 smithi169 ceph-mon[29344]: osdmap e242: 8 total, 8 up, 8 in 2024-03-12T03:41:55.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:55.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:55.591 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:55.636 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:55.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:55.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:55 smithi173 ceph-mon[33544]: pgmap v398: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s; 19389/282813 objects degraded (6.856%); 18 MiB/s, 281 objects/s recovering 2024-03-12T03:41:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:55 smithi173 ceph-mon[33544]: osdmap e242: 8 total, 8 up, 8 in 2024-03-12T03:41:55.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:55.784 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:55.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:55.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:55.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:55.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:55.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:56.009 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:56.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:56.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:56.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:56.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:56.069 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:41:56.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:56.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:56.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:56.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:56.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:56.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:56.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:56.335 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.5 to 1 (10000) 2024-03-12T03:41:56.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:56.463 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:56.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:56.465 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:56 smithi169 ceph-mon[29344]: osdmap e243: 8 total, 8 up, 8 in 2024-03-12T03:41:56.465 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:56 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/694164490' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:41:56.465 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:56 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:41:56.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:56 smithi169 ceph-mon[39949]: osdmap e243: 8 total, 8 up, 8 in 2024-03-12T03:41:56.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:56 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/694164490' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:41:56.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:56 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:41:56.566 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:56.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:56.608 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:56.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:56.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:56.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:56 smithi173 ceph-mon[33544]: osdmap e243: 8 total, 8 up, 8 in 2024-03-12T03:41:56.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:56 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/694164490' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:41:56.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:56 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:41:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:56.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:56.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:56.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:56.891 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:56.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:56.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:57.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:57.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:57.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:57.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:57.167 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:57.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:57.269 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:41:57.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:57.317 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:57.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:41:57.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:57.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:57.455 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:57.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:57.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:57.593 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:57 smithi169 ceph-mon[29344]: pgmap v401: 158 pgs: 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 961 op/s; 18824/281658 objects degraded (6.683%); 20 MiB/s, 327 objects/s recovering 2024-03-12T03:41:57.593 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:57 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 18824/281658 objects degraded (6.683%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-12T03:41:57.594 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:57 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-12T03:41:57.594 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:57 smithi169 ceph-mon[29344]: osdmap e244: 8 total, 8 up, 8 in 2024-03-12T03:41:57.594 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:57 smithi169 ceph-mon[39949]: pgmap v401: 158 pgs: 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 961 op/s; 18824/281658 objects degraded (6.683%); 20 MiB/s, 327 objects/s recovering 2024-03-12T03:41:57.594 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:57 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 18824/281658 objects degraded (6.683%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-12T03:41:57.595 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:57 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-12T03:41:57.595 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:57 smithi169 ceph-mon[39949]: osdmap e244: 8 total, 8 up, 8 in 2024-03-12T03:41:57.600 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:57.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:57.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:41:57.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:57.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:41:57.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:57 smithi173 ceph-mon[33544]: pgmap v401: 158 pgs: 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 961 op/s; 18824/281658 objects degraded (6.683%); 20 MiB/s, 327 objects/s recovering 2024-03-12T03:41:57.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:57 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 18824/281658 objects degraded (6.683%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-12T03:41:57.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:57 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-12T03:41:57.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:57 smithi173 ceph-mon[33544]: osdmap e244: 8 total, 8 up, 8 in 2024-03-12T03:41:57.834 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:57.874 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:57.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:57.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:57.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:58.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:58.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:58.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:58.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:58.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:58.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:58.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:58.284 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:58.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:58.300 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:41:58.386 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:58.421 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:58.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:58.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:41:58.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:58.559 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:58.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:41:58.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:41:58.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:58 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2653991655' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:41:58.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:58 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:41:58.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:58 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2653991655' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:41:58.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:58 smithi169 ceph-mon[39949]: osdmap e245: 8 total, 8 up, 8 in 2024-03-12T03:41:58.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:58 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2653991655' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:41:58.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:58 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:41:58.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:58 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2653991655' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:41:58.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:58 smithi169 ceph-mon[29344]: osdmap e245: 8 total, 8 up, 8 in 2024-03-12T03:41:58.689 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:58.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:58.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:41:58.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:58 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2653991655' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:41:58.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:58 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:41:58.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:58 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2653991655' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:41:58.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:58 smithi173 ceph-mon[33544]: osdmap e245: 8 total, 8 up, 8 in 2024-03-12T03:41:58.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:58.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:58.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:58.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:58.917 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:58.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:58.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:59.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:59.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:59.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:59.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:41:59.197 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:41:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:59.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:41:59.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:59.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:59.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:41:59.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:59.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:59.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:59.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:59.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:59.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:59.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:59.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:59 smithi173 ceph-mon[33544]: pgmap v403: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 peering, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 812 op/s; 15268/271527 objects degraded (5.623%); 46 MiB/s, 730 objects/s recovering 2024-03-12T03:41:59.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:59 smithi173 ceph-mon[33544]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:41:59.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:41:59 smithi173 ceph-mon[33544]: osdmap e246: 8 total, 8 up, 8 in 2024-03-12T03:41:59.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:59 smithi169 ceph-mon[29344]: pgmap v403: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 peering, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 812 op/s; 15268/271527 objects degraded (5.623%); 46 MiB/s, 730 objects/s recovering 2024-03-12T03:41:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:59 smithi169 ceph-mon[29344]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:41:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:41:59 smithi169 ceph-mon[29344]: osdmap e246: 8 total, 8 up, 8 in 2024-03-12T03:41:59.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:59 smithi169 ceph-mon[39949]: pgmap v403: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 peering, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 812 op/s; 15268/271527 objects degraded (5.623%); 46 MiB/s, 730 objects/s recovering 2024-03-12T03:41:59.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:59 smithi169 ceph-mon[39949]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:41:59.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:41:59 smithi169 ceph-mon[39949]: osdmap e246: 8 total, 8 up, 8 in 2024-03-12T03:41:59.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:41:59.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:59.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:59.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:41:59.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-12T03:41:59.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:41:59.860 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:59.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:41:59.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:41:59.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:41:59.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:41:59.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:00.095 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:00.124 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:00.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:00.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:00.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:00.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:00.281 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:00.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:00.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:00.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:00.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:00.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:00.550 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:00.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:00.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:00.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:00.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:00 smithi169 ceph-mon[29344]: 2.0 scrub starts 2024-03-12T03:42:00.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:00 smithi169 ceph-mon[29344]: 2.0 scrub ok 2024-03-12T03:42:00.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:00 smithi169 ceph-mon[39949]: 2.0 scrub starts 2024-03-12T03:42:00.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:00 smithi169 ceph-mon[39949]: 2.0 scrub ok 2024-03-12T03:42:00.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:00.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:00.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:00.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:00 smithi173 ceph-mon[33544]: 2.0 scrub starts 2024-03-12T03:42:00.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:00 smithi173 ceph-mon[33544]: 2.0 scrub ok 2024-03-12T03:42:00.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:00.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:00.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:00.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:00.951 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:00.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:00.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:01.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:01.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:01.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:01.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:01.224 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:01.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:01.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:01.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:01.358 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:01.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:01.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-mon[29344]: pgmap v406: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 155 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 862 op/s; 15183/270447 objects degraded (5.614%); 39 MiB/s, 631 objects/s recovering 2024-03-12T03:42:01.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-mon[29344]: 3.12 scrub starts 2024-03-12T03:42:01.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-mon[29344]: 3.12 scrub ok 2024-03-12T03:42:01.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:01.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-mon[29344]: 2.1 scrub starts 2024-03-12T03:42:01.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-mon[29344]: 2.1 scrub ok 2024-03-12T03:42:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-mon[39949]: pgmap v406: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 155 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 862 op/s; 15183/270447 objects degraded (5.614%); 39 MiB/s, 631 objects/s recovering 2024-03-12T03:42:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-mon[39949]: 3.12 scrub starts 2024-03-12T03:42:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-mon[39949]: 3.12 scrub ok 2024-03-12T03:42:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:01.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-mon[39949]: 2.1 scrub starts 2024-03-12T03:42:01.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-mon[39949]: 2.1 scrub ok 2024-03-12T03:42:01.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:01.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:01.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:01.594 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:01.631 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:01.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:01.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:01 smithi173 ceph-mon[33544]: pgmap v406: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 155 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 862 op/s; 15183/270447 objects degraded (5.614%); 39 MiB/s, 631 objects/s recovering 2024-03-12T03:42:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:01 smithi173 ceph-mon[33544]: 3.12 scrub starts 2024-03-12T03:42:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:01 smithi173 ceph-mon[33544]: 3.12 scrub ok 2024-03-12T03:42:01.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:01 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:01.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:01 smithi173 ceph-mon[33544]: 2.1 scrub starts 2024-03-12T03:42:01.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:01 smithi173 ceph-mon[33544]: 2.1 scrub ok 2024-03-12T03:42:01.767 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:01.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:01.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:01.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:01.896 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:01.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:01.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:02.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:02.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:02.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:02.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:02.158 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:02.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:02.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:42:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:42:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:42:02.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:02.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:02.295 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:02.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:02.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:02.438 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:02.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:02.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:02.569 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:02.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:02.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:02.688 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:02 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 12316/258594 objects degraded (4.763%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:02.688 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:02 smithi173 ceph-mon[33544]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-12T03:42:02.706 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:02.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:02.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:02.835 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:02.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:02.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:02 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 12316/258594 objects degraded (4.763%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:02.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:02 smithi169 ceph-mon[29344]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-12T03:42:02.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:02 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 12316/258594 objects degraded (4.763%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:02.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:02 smithi169 ceph-mon[39949]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-12T03:42:02.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:02.974 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:02.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:02.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:42:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:42:03.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:03.100 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:03.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:03.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:03.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:03.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:03.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:03.367 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:03.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:03.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:03.504 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:03.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:03.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:03.649 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:03.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:03 smithi173 ceph-mon[33544]: pgmap v407: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.28k op/s; 12316/258594 objects degraded (4.763%); 53 MiB/s, 842 objects/s recovering 2024-03-12T03:42:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:03 smithi173 ceph-mon[33544]: 6.15 scrub starts 2024-03-12T03:42:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:03 smithi173 ceph-mon[33544]: 6.15 scrub ok 2024-03-12T03:42:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:03 smithi173 ceph-mon[33544]: 5.2 scrub starts 2024-03-12T03:42:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:03 smithi173 ceph-mon[33544]: 5.2 scrub ok 2024-03-12T03:42:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:03 smithi173 ceph-mon[33544]: 4.1 scrub starts 2024-03-12T03:42:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:03 smithi173 ceph-mon[33544]: 4.1 scrub ok 2024-03-12T03:42:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:03 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:03 smithi173 ceph-mon[33544]: 5.a scrub starts 2024-03-12T03:42:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:03 smithi173 ceph-mon[33544]: 5.a scrub ok 2024-03-12T03:42:03.750 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:03.773 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:03.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:03.875 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:03.909 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:03.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:03.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[29344]: pgmap v407: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.28k op/s; 12316/258594 objects degraded (4.763%); 53 MiB/s, 842 objects/s recovering 2024-03-12T03:42:03.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[29344]: 6.15 scrub starts 2024-03-12T03:42:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[29344]: 6.15 scrub ok 2024-03-12T03:42:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[29344]: 5.2 scrub starts 2024-03-12T03:42:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[29344]: 5.2 scrub ok 2024-03-12T03:42:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[29344]: 4.1 scrub starts 2024-03-12T03:42:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[29344]: 4.1 scrub ok 2024-03-12T03:42:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[29344]: 5.a scrub starts 2024-03-12T03:42:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[29344]: 5.a scrub ok 2024-03-12T03:42:03.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[39949]: pgmap v407: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.28k op/s; 12316/258594 objects degraded (4.763%); 53 MiB/s, 842 objects/s recovering 2024-03-12T03:42:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[39949]: 6.15 scrub starts 2024-03-12T03:42:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[39949]: 6.15 scrub ok 2024-03-12T03:42:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[39949]: 5.2 scrub starts 2024-03-12T03:42:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[39949]: 5.2 scrub ok 2024-03-12T03:42:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[39949]: 4.1 scrub starts 2024-03-12T03:42:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[39949]: 4.1 scrub ok 2024-03-12T03:42:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[39949]: 5.a scrub starts 2024-03-12T03:42:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:03 smithi169 ceph-mon[39949]: 5.a scrub ok 2024-03-12T03:42:04.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:04.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:04.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:04.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:04.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:04.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:04.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:04.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:04.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:04.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:04.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:04.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:04.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:04.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:04.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:04.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:04.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:04.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:04.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:04.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:04 smithi173 ceph-mon[33544]: 4.1c scrub starts 2024-03-12T03:42:04.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:04 smithi173 ceph-mon[33544]: 4.1c scrub ok 2024-03-12T03:42:04.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:04.881 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:04.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:04.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:04.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:04 smithi169 ceph-mon[29344]: 4.1c scrub starts 2024-03-12T03:42:04.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:04 smithi169 ceph-mon[29344]: 4.1c scrub ok 2024-03-12T03:42:04.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:04 smithi169 ceph-mon[39949]: 4.1c scrub starts 2024-03-12T03:42:04.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:04 smithi169 ceph-mon[39949]: 4.1c scrub ok 2024-03-12T03:42:04.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:05.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:05.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:05.118 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:05.159 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:05.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:05.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:05.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:05.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:05.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:05.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:05.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:05.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:05.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:05.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:05.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:05.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:05.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:05 smithi169 ceph-mon[29344]: pgmap v408: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 672 B/s rd, 0 B/s wr, 853 op/s; 12316/258594 objects degraded (4.763%); 36 MiB/s, 579 objects/s recovering 2024-03-12T03:42:05.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:05 smithi169 ceph-mon[29344]: 6.3 scrub starts 2024-03-12T03:42:05.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:05 smithi169 ceph-mon[29344]: 6.3 scrub ok 2024-03-12T03:42:05.676 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:05 smithi169 ceph-mon[29344]: 5.e scrub starts 2024-03-12T03:42:05.677 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:05 smithi169 ceph-mon[29344]: 5.e scrub ok 2024-03-12T03:42:05.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:05 smithi169 ceph-mon[39949]: pgmap v408: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 672 B/s rd, 0 B/s wr, 853 op/s; 12316/258594 objects degraded (4.763%); 36 MiB/s, 579 objects/s recovering 2024-03-12T03:42:05.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:05 smithi169 ceph-mon[39949]: 6.3 scrub starts 2024-03-12T03:42:05.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:05 smithi169 ceph-mon[39949]: 6.3 scrub ok 2024-03-12T03:42:05.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:05 smithi169 ceph-mon[39949]: 5.e scrub starts 2024-03-12T03:42:05.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:05 smithi169 ceph-mon[39949]: 5.e scrub ok 2024-03-12T03:42:05.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:05.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:05.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:05.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:05 smithi173 ceph-mon[33544]: pgmap v408: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 672 B/s rd, 0 B/s wr, 853 op/s; 12316/258594 objects degraded (4.763%); 36 MiB/s, 579 objects/s recovering 2024-03-12T03:42:05.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:05 smithi173 ceph-mon[33544]: 6.3 scrub starts 2024-03-12T03:42:05.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:05 smithi173 ceph-mon[33544]: 6.3 scrub ok 2024-03-12T03:42:05.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:05 smithi173 ceph-mon[33544]: 5.e scrub starts 2024-03-12T03:42:05.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:05 smithi173 ceph-mon[33544]: 5.e scrub ok 2024-03-12T03:42:05.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:05.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:05.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:05.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:05.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:05.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:05.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:06.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:06.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:06.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:06.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:06.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:06.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:06.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:06.385 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:06.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:06.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:06.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:06.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:06.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:06.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:06.660 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:06.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:06.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:06.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:06.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:06.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:06.934 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:06.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:07.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:07.082 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:07.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:07.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:07.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:07.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:07.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:07.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:07.354 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:07.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:07.456 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:07.467 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:42:07.513 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:07.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:07.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:07.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:07.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:07.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:07.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:07.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:07 smithi169 ceph-mon[29344]: pgmap v409: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 611 op/s; 12316/256869 objects degraded (4.795%); 17 MiB/s, 277 objects/s recovering 2024-03-12T03:42:07.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:07 smithi169 ceph-mon[29344]: 5.11 scrub starts 2024-03-12T03:42:07.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:07 smithi169 ceph-mon[29344]: 5.11 scrub ok 2024-03-12T03:42:07.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:07 smithi169 ceph-mon[29344]: 6.f scrub starts 2024-03-12T03:42:07.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:07 smithi169 ceph-mon[39949]: pgmap v409: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 611 op/s; 12316/256869 objects degraded (4.795%); 17 MiB/s, 277 objects/s recovering 2024-03-12T03:42:07.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:07 smithi169 ceph-mon[39949]: 5.11 scrub starts 2024-03-12T03:42:07.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:07 smithi169 ceph-mon[39949]: 5.11 scrub ok 2024-03-12T03:42:07.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:07 smithi169 ceph-mon[39949]: 6.f scrub starts 2024-03-12T03:42:07.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:07 smithi173 ceph-mon[33544]: pgmap v409: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 611 op/s; 12316/256869 objects degraded (4.795%); 17 MiB/s, 277 objects/s recovering 2024-03-12T03:42:07.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:07 smithi173 ceph-mon[33544]: 5.11 scrub starts 2024-03-12T03:42:07.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:07 smithi173 ceph-mon[33544]: 5.11 scrub ok 2024-03-12T03:42:07.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:07 smithi173 ceph-mon[33544]: 6.f scrub starts 2024-03-12T03:42:07.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:07.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:07.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:07.855 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:07.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:07.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:07.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:07.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:07.946 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:42:07.987 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:42:08.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:08.025 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:42:08.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:08.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:08.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:08.065 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:42:08.105 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:42:08.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:08.158 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:42:08.188 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:08.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:08.203 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:42:08.242 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:42:08.248 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:42:08.248 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:42:08.248 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:42:08.248 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:42:08.249 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:42:08.249 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:42:08.249 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:42:08.249 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:42:08.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:08.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:08.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:08.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:08.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:08.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:08.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:08.515 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:08 smithi169 ceph-mon[29344]: 6.f scrub ok 2024-03-12T03:42:08.516 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:08 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4133383251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:42:08.516 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:08 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 12316/256869 objects degraded (4.795%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:08.516 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:08 smithi169 ceph-mon[39949]: 6.f scrub ok 2024-03-12T03:42:08.516 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:08 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4133383251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:42:08.516 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:08 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 12316/256869 objects degraded (4.795%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:08.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:08.600 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:08.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:08.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:08.729 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:42:08.729 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:42:08.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:08.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:08.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:08 smithi173 ceph-mon[33544]: 6.f scrub ok 2024-03-12T03:42:08.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:08 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4133383251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:42:08.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:08 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 12316/256869 objects degraded (4.795%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:08.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:08.868 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:08.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:08.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:09.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:09.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:09.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:09.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:09.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:09.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:09.264 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:09.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:09.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:09.407 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:09.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:09.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:09.546 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:09.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:09.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:09.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:09.691 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:09.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:09.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:09 smithi173 ceph-mon[33544]: pgmap v410: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 996 op/s; 10167/242757 objects degraded (4.188%); 24 MiB/s, 380 objects/s recovering 2024-03-12T03:42:09.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:09 smithi173 ceph-mon[33544]: 6.e scrub starts 2024-03-12T03:42:09.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:09 smithi173 ceph-mon[33544]: 6.e scrub ok 2024-03-12T03:42:09.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:09.813 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:09 smithi169 ceph-mon[29344]: pgmap v410: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 996 op/s; 10167/242757 objects degraded (4.188%); 24 MiB/s, 380 objects/s recovering 2024-03-12T03:42:09.814 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:09 smithi169 ceph-mon[29344]: 6.e scrub starts 2024-03-12T03:42:09.814 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:09 smithi169 ceph-mon[29344]: 6.e scrub ok 2024-03-12T03:42:09.814 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:09 smithi169 ceph-mon[39949]: pgmap v410: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 996 op/s; 10167/242757 objects degraded (4.188%); 24 MiB/s, 380 objects/s recovering 2024-03-12T03:42:09.814 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:09 smithi169 ceph-mon[39949]: 6.e scrub starts 2024-03-12T03:42:09.814 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:09 smithi169 ceph-mon[39949]: 6.e scrub ok 2024-03-12T03:42:09.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:09.839 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:09.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:09.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:09.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:09.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:09.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:10.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:10.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:10.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:10.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:10.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:10.248 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:10.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:10.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:10.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:10.380 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:10.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:10.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:10.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:10.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:10.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:10.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:10.666 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:10.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:10.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:10.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:10.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:10.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[29344]: 3.e scrub starts 2024-03-12T03:42:10.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[29344]: 3.e scrub ok 2024-03-12T03:42:10.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[29344]: 5.18 scrub starts 2024-03-12T03:42:10.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[29344]: 5.18 scrub ok 2024-03-12T03:42:10.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:42:10.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:42:10.871 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[29344]: 6.16 scrub starts 2024-03-12T03:42:10.871 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[29344]: 6.16 scrub ok 2024-03-12T03:42:10.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[39949]: 3.e scrub starts 2024-03-12T03:42:10.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[39949]: 3.e scrub ok 2024-03-12T03:42:10.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[39949]: 5.18 scrub starts 2024-03-12T03:42:10.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[39949]: 5.18 scrub ok 2024-03-12T03:42:10.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:42:10.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:42:10.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[39949]: 6.16 scrub starts 2024-03-12T03:42:10.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:10 smithi169 ceph-mon[39949]: 6.16 scrub ok 2024-03-12T03:42:10.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:10.936 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:10.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:10.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:10 smithi173 ceph-mon[33544]: 3.e scrub starts 2024-03-12T03:42:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:10 smithi173 ceph-mon[33544]: 3.e scrub ok 2024-03-12T03:42:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:10 smithi173 ceph-mon[33544]: 5.18 scrub starts 2024-03-12T03:42:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:10 smithi173 ceph-mon[33544]: 5.18 scrub ok 2024-03-12T03:42:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:10 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:42:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:10 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:42:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:10 smithi173 ceph-mon[33544]: 6.16 scrub starts 2024-03-12T03:42:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:10 smithi173 ceph-mon[33544]: 6.16 scrub ok 2024-03-12T03:42:11.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:11.078 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:11.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:11.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:11.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:11.238 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:11.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:11.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-12T03:42:11.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:11.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:11.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:11.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:11.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:11.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:11.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:11.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:11.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:11.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:11.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:11.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:11.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:11.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[29344]: pgmap v411: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 962 B/s rd, 0 B/s wr, 965 op/s; 10167/240750 objects degraded (4.223%); 22 MiB/s, 344 objects/s recovering 2024-03-12T03:42:11.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[29344]: 1.6 scrub starts 2024-03-12T03:42:11.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[29344]: 1.6 scrub ok 2024-03-12T03:42:11.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[29344]: osdmap e247: 8 total, 8 up, 8 in 2024-03-12T03:42:11.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:42:11.785 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[29344]: 5.b scrub starts 2024-03-12T03:42:11.785 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[29344]: 5.b scrub ok 2024-03-12T03:42:11.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[39949]: pgmap v411: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 962 B/s rd, 0 B/s wr, 965 op/s; 10167/240750 objects degraded (4.223%); 22 MiB/s, 344 objects/s recovering 2024-03-12T03:42:11.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[39949]: 1.6 scrub starts 2024-03-12T03:42:11.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[39949]: 1.6 scrub ok 2024-03-12T03:42:11.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[39949]: osdmap e247: 8 total, 8 up, 8 in 2024-03-12T03:42:11.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:42:11.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[39949]: 5.b scrub starts 2024-03-12T03:42:11.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-mon[39949]: 5.b scrub ok 2024-03-12T03:42:11.831 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:11.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:11.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:11.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:11.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:11 smithi173 ceph-mon[33544]: pgmap v411: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 962 B/s rd, 0 B/s wr, 965 op/s; 10167/240750 objects degraded (4.223%); 22 MiB/s, 344 objects/s recovering 2024-03-12T03:42:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:11 smithi173 ceph-mon[33544]: 1.6 scrub starts 2024-03-12T03:42:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:11 smithi173 ceph-mon[33544]: 1.6 scrub ok 2024-03-12T03:42:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:11 smithi173 ceph-mon[33544]: osdmap e247: 8 total, 8 up, 8 in 2024-03-12T03:42:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:11 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:42:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:11 smithi173 ceph-mon[33544]: 5.b scrub starts 2024-03-12T03:42:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:11 smithi173 ceph-mon[33544]: 5.b scrub ok 2024-03-12T03:42:12.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:12.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:12.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:12.043 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:42:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:42:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:42:12.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:12.157 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:42:12.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:12.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:12.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:12.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:12.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:12.394 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:12.452 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:12.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:12.521 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.6 to 1 (10000) 2024-03-12T03:42:12.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:12.602 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:12.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:12.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:12 smithi169 ceph-mon[29344]: 4.d scrub starts 2024-03-12T03:42:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:12 smithi169 ceph-mon[29344]: 4.d scrub ok 2024-03-12T03:42:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:12 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/764782070' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-12T03:42:12.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:12 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:42:12.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:12 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:42:12.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:12 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:42:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:12 smithi169 ceph-mon[39949]: 4.d scrub starts 2024-03-12T03:42:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:12 smithi169 ceph-mon[39949]: 4.d scrub ok 2024-03-12T03:42:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:12 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/764782070' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-12T03:42:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:12 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:42:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:12 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:42:12.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:12 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:42:12.704 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:12.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:12.741 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:12.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:12.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:12.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:12.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:12.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:12.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:12.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:42:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:42:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:12 smithi173 ceph-mon[33544]: 4.d scrub starts 2024-03-12T03:42:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:12 smithi173 ceph-mon[33544]: 4.d scrub ok 2024-03-12T03:42:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:12 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/764782070' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-12T03:42:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:12 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:42:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:12 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:42:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:12 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:42:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:13.015 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:13.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:13.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:13.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:13.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:13.257 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:13.309 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:13.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:13.411 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:13.473 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:13.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:13.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:13.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:13.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:13.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:13.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[29344]: pgmap v413: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 8103/225966 objects degraded (3.586%); 18 MiB/s, 283 objects/s recovering 2024-03-12T03:42:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[29344]: 3.13 scrub starts 2024-03-12T03:42:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[29344]: 3.13 scrub ok 2024-03-12T03:42:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/764782070' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-12T03:42:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[29344]: osdmap e248: 8 total, 8 up, 8 in 2024-03-12T03:42:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:13.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 8103/225966 objects degraded (3.586%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[39949]: pgmap v413: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 8103/225966 objects degraded (3.586%); 18 MiB/s, 283 objects/s recovering 2024-03-12T03:42:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[39949]: 3.13 scrub starts 2024-03-12T03:42:13.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[39949]: 3.13 scrub ok 2024-03-12T03:42:13.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/764782070' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-12T03:42:13.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[39949]: osdmap e248: 8 total, 8 up, 8 in 2024-03-12T03:42:13.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:13.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:13 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 8103/225966 objects degraded (3.586%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:13.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:13.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:13.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:13.844 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:13.938 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:13.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:13 smithi173 ceph-mon[33544]: pgmap v413: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 8103/225966 objects degraded (3.586%); 18 MiB/s, 283 objects/s recovering 2024-03-12T03:42:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:13 smithi173 ceph-mon[33544]: 3.13 scrub starts 2024-03-12T03:42:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:13 smithi173 ceph-mon[33544]: 3.13 scrub ok 2024-03-12T03:42:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:13 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/764782070' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-12T03:42:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:13 smithi173 ceph-mon[33544]: osdmap e248: 8 total, 8 up, 8 in 2024-03-12T03:42:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:13 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:13 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 8103/225966 objects degraded (3.586%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:14.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:14.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:14.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:14.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:14.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:14.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:14.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:14.307 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:14.350 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:14.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:14.424 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-12T03:42:14.450 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:14.508 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:14.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:14.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:14.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:14.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:14.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:14.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:14.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:14.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:14.883 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:14.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:14.914 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:14.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:15.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:15.053 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:15.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:15.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:15.191 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:15.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:15.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:15.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:15.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:15.426 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:15.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:15.467 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:15.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:15.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:15.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:15.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:15.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:15.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:15 smithi169 ceph-mon[29344]: pgmap v415: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.29k op/s; 8103/225966 objects degraded (3.586%); 22 MiB/s, 354 objects/s recovering 2024-03-12T03:42:15.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:15 smithi169 ceph-mon[29344]: 3.10 scrub starts 2024-03-12T03:42:15.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:15 smithi169 ceph-mon[29344]: 3.10 scrub ok 2024-03-12T03:42:15.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:15 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:42:15.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:15 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:42:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:15 smithi169 ceph-mon[39949]: pgmap v415: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.29k op/s; 8103/225966 objects degraded (3.586%); 22 MiB/s, 354 objects/s recovering 2024-03-12T03:42:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:15 smithi169 ceph-mon[39949]: 3.10 scrub starts 2024-03-12T03:42:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:15 smithi169 ceph-mon[39949]: 3.10 scrub ok 2024-03-12T03:42:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:15 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:42:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:15 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:42:15.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:15.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:15.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:15.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:15.863 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:15.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:15.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:15.965 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:15 smithi173 ceph-mon[33544]: pgmap v415: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.29k op/s; 8103/225966 objects degraded (3.586%); 22 MiB/s, 354 objects/s recovering 2024-03-12T03:42:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:15 smithi173 ceph-mon[33544]: 3.10 scrub starts 2024-03-12T03:42:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:15 smithi173 ceph-mon[33544]: 3.10 scrub ok 2024-03-12T03:42:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:15 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:42:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:15 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:42:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:15.993 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:15.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:16.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:16.128 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:16.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:16.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:16.230 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:16.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:16.271 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:16.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:16.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:16.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:16.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:16.555 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:16.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:16.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[29344]: 3.d scrub starts 2024-03-12T03:42:16.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[39949]: 3.d scrub starts 2024-03-12T03:42:16.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:16.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:16.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:16.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:16.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:16.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:16.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:16.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[29344]: 3.d scrub ok 2024-03-12T03:42:16.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[29344]: 4.a scrub starts 2024-03-12T03:42:16.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[29344]: 4.a scrub ok 2024-03-12T03:42:16.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[29344]: osdmap e249: 8 total, 8 up, 8 in 2024-03-12T03:42:16.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:42:16.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:16.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[39949]: 3.d scrub ok 2024-03-12T03:42:16.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[39949]: 4.a scrub starts 2024-03-12T03:42:16.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[39949]: 4.a scrub ok 2024-03-12T03:42:16.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[39949]: osdmap e249: 8 total, 8 up, 8 in 2024-03-12T03:42:16.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:42:16.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:16 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:16.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:16.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:16 smithi173 ceph-mon[33544]: 3.d scrub starts 2024-03-12T03:42:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:16 smithi173 ceph-mon[33544]: 3.d scrub ok 2024-03-12T03:42:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:16 smithi173 ceph-mon[33544]: 4.a scrub starts 2024-03-12T03:42:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:16 smithi173 ceph-mon[33544]: 4.a scrub ok 2024-03-12T03:42:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:16 smithi173 ceph-mon[33544]: osdmap e249: 8 total, 8 up, 8 in 2024-03-12T03:42:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:16 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:42:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:16 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:17.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:17.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:17.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:17.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:17.229 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:17.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:17.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:17.331 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:17.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:17.370 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:17.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:17.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:17.506 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:17.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:17.609 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:17.646 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:17.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:17.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:17.679 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:17.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:17.797 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:17.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:17.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:17.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:17 smithi169 ceph-mon[29344]: pgmap v417: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 941 op/s; 8103/223830 objects degraded (3.620%); 14 MiB/s, 231 objects/s recovering 2024-03-12T03:42:17.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:17 smithi169 ceph-mon[39949]: pgmap v417: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 941 op/s; 8103/223830 objects degraded (3.620%); 14 MiB/s, 231 objects/s recovering 2024-03-12T03:42:17.900 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:17.939 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:17.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:17 smithi173 ceph-mon[33544]: pgmap v417: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 941 op/s; 8103/223830 objects degraded (3.620%); 14 MiB/s, 231 objects/s recovering 2024-03-12T03:42:18.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:18.083 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:18.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:18.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:18.226 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:18.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:18.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:18.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:18.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:18.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:18.498 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:18.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:18.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:18.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:18.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:18.734 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:18.776 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:18.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:18.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:18.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:18.920 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:18.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:18.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:18 smithi169 ceph-mon[29344]: 1.2 scrub starts 2024-03-12T03:42:18.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:18 smithi169 ceph-mon[29344]: 1.2 scrub ok 2024-03-12T03:42:18.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:18 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 8103/223830 objects degraded (3.620%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:18.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:18 smithi169 ceph-mon[39949]: 1.2 scrub starts 2024-03-12T03:42:18.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:18 smithi169 ceph-mon[39949]: 1.2 scrub ok 2024-03-12T03:42:18.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:18 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 8103/223830 objects degraded (3.620%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:18 smithi173 ceph-mon[33544]: 1.2 scrub starts 2024-03-12T03:42:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:18 smithi173 ceph-mon[33544]: 1.2 scrub ok 2024-03-12T03:42:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:18 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 8103/223830 objects degraded (3.620%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:19.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:19.067 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:19.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:19.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:19.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:19.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:19.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:19.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:19.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:19.356 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:19.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:19.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:19.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:19.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:19.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:19.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:19.641 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:19.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:19.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:19.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:19.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:19.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:19.879 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:19 smithi169 ceph-mon[29344]: pgmap v418: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.37k op/s; 6269/210072 objects degraded (2.984%); 23 MiB/s, 363 objects/s recovering 2024-03-12T03:42:19.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:19 smithi169 ceph-mon[29344]: 5.6 scrub starts 2024-03-12T03:42:19.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:19 smithi169 ceph-mon[29344]: 5.6 scrub ok 2024-03-12T03:42:19.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:19 smithi169 ceph-mon[29344]: 3.1a scrub starts 2024-03-12T03:42:19.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:19 smithi169 ceph-mon[29344]: 3.1a scrub ok 2024-03-12T03:42:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:19 smithi169 ceph-mon[39949]: pgmap v418: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.37k op/s; 6269/210072 objects degraded (2.984%); 23 MiB/s, 363 objects/s recovering 2024-03-12T03:42:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:19 smithi169 ceph-mon[39949]: 5.6 scrub starts 2024-03-12T03:42:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:19 smithi169 ceph-mon[39949]: 5.6 scrub ok 2024-03-12T03:42:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:19 smithi169 ceph-mon[39949]: 3.1a scrub starts 2024-03-12T03:42:19.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:19 smithi169 ceph-mon[39949]: 3.1a scrub ok 2024-03-12T03:42:19.928 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:19.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:19 smithi173 ceph-mon[33544]: pgmap v418: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.37k op/s; 6269/210072 objects degraded (2.984%); 23 MiB/s, 363 objects/s recovering 2024-03-12T03:42:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:19 smithi173 ceph-mon[33544]: 5.6 scrub starts 2024-03-12T03:42:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:19 smithi173 ceph-mon[33544]: 5.6 scrub ok 2024-03-12T03:42:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:19 smithi173 ceph-mon[33544]: 3.1a scrub starts 2024-03-12T03:42:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:19 smithi173 ceph-mon[33544]: 3.1a scrub ok 2024-03-12T03:42:20.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:20.071 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:20.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:20.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:20.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:20.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:20.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:20.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:20.340 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:20.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:20.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:20.482 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:20.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:20.583 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:20.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:20.622 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:20.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:20.632 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:42:20.723 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:20.763 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:20.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:20.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:20.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:20.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:20.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:20 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1850477488' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:42:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:20 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1850477488' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:42:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:20 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1850477488' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:42:21.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:21.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:21.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:21.095 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:21.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:21.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:21.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:21.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:21.170 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:42:21.170 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:42:21.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:21.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:21.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:21.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:21.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:21.431 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:21.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:21.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:21.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:21.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:21.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:21.707 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:21.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:21.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:21.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:21.842 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:21.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:21.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:21.877 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[29344]: pgmap v419: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 747 op/s; 6269/208026 objects degraded (3.014%); 10 MiB/s, 164 objects/s recovering 2024-03-12T03:42:21.877 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[29344]: 6.5 scrub starts 2024-03-12T03:42:21.877 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[29344]: 6.5 scrub ok 2024-03-12T03:42:21.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[29344]: 6.1e scrub starts 2024-03-12T03:42:21.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[29344]: 6.1e scrub ok 2024-03-12T03:42:21.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:42:21.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1850477488' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:42:21.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[29344]: osdmap e250: 8 total, 8 up, 8 in 2024-03-12T03:42:21.878 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:42:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:42:21.878 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[39949]: pgmap v419: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 747 op/s; 6269/208026 objects degraded (3.014%); 10 MiB/s, 164 objects/s recovering 2024-03-12T03:42:21.878 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[39949]: 6.5 scrub starts 2024-03-12T03:42:21.878 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[39949]: 6.5 scrub ok 2024-03-12T03:42:21.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[39949]: 6.1e scrub starts 2024-03-12T03:42:21.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[39949]: 6.1e scrub ok 2024-03-12T03:42:21.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:42:21.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1850477488' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:42:21.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:21 smithi169 ceph-mon[39949]: osdmap e250: 8 total, 8 up, 8 in 2024-03-12T03:42:21.943 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:21.985 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:21.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:21 smithi173 ceph-mon[33544]: pgmap v419: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 747 op/s; 6269/208026 objects degraded (3.014%); 10 MiB/s, 164 objects/s recovering 2024-03-12T03:42:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:21 smithi173 ceph-mon[33544]: 6.5 scrub starts 2024-03-12T03:42:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:21 smithi173 ceph-mon[33544]: 6.5 scrub ok 2024-03-12T03:42:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:21 smithi173 ceph-mon[33544]: 6.1e scrub starts 2024-03-12T03:42:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:21 smithi173 ceph-mon[33544]: 6.1e scrub ok 2024-03-12T03:42:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:21 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:42:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:21 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1850477488' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:42:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:21 smithi173 ceph-mon[33544]: osdmap e250: 8 total, 8 up, 8 in 2024-03-12T03:42:22.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:22.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:22.128 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:22.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:22.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:22.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:22.265 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:22.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:22.367 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:22.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:22.402 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:22.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:22.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:22.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:22.544 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:22.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:22.645 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:22.685 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:22.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:22.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:22.786 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:22.816 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:22.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:22.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:22.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:22.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:22 smithi169 ceph-mon[29344]: 3.1 deep-scrub starts 2024-03-12T03:42:22.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:22 smithi169 ceph-mon[29344]: 3.1 deep-scrub ok 2024-03-12T03:42:22.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:22 smithi169 ceph-mon[39949]: 3.1 deep-scrub starts 2024-03-12T03:42:22.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:22 smithi169 ceph-mon[39949]: 3.1 deep-scrub ok 2024-03-12T03:42:22.954 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:22.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:22.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:42:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:42:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:22 smithi173 ceph-mon[33544]: 3.1 deep-scrub starts 2024-03-12T03:42:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:22 smithi173 ceph-mon[33544]: 3.1 deep-scrub ok 2024-03-12T03:42:23.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:23.087 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:23.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:23.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:23.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:23.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:23.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:23.368 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:23.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:23.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:23.505 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:23.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:23.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:23.649 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:23.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:23.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:23.735 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-12T03:42:23.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:23.792 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:23.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:23.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:23 smithi169 ceph-mon[29344]: pgmap v421: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 5547/197985 objects degraded (2.802%); 15 MiB/s, 241 objects/s recovering 2024-03-12T03:42:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:23 smithi169 ceph-mon[29344]: 3.11 scrub starts 2024-03-12T03:42:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:23 smithi169 ceph-mon[29344]: 3.11 scrub ok 2024-03-12T03:42:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:23 smithi169 ceph-mon[29344]: 8.5 deep-scrub starts 2024-03-12T03:42:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:23 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:23 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 5547/197985 objects degraded (2.802%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:23 smithi169 ceph-mon[39949]: pgmap v421: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 5547/197985 objects degraded (2.802%); 15 MiB/s, 241 objects/s recovering 2024-03-12T03:42:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:23 smithi169 ceph-mon[39949]: 3.11 scrub starts 2024-03-12T03:42:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:23 smithi169 ceph-mon[39949]: 3.11 scrub ok 2024-03-12T03:42:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:23 smithi169 ceph-mon[39949]: 8.5 deep-scrub starts 2024-03-12T03:42:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:23 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:23 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 5547/197985 objects degraded (2.802%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:23.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:23.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:23.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:23.929 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:42:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:23 smithi173 ceph-mon[33544]: pgmap v421: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 5547/197985 objects degraded (2.802%); 15 MiB/s, 241 objects/s recovering 2024-03-12T03:42:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:23 smithi173 ceph-mon[33544]: 3.11 scrub starts 2024-03-12T03:42:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:23 smithi173 ceph-mon[33544]: 3.11 scrub ok 2024-03-12T03:42:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:23 smithi173 ceph-mon[33544]: 8.5 deep-scrub starts 2024-03-12T03:42:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:23 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:23 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 5547/197985 objects degraded (2.802%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:24.031 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:24.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:24.077 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:24.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:24.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:24.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:24.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:24.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:24.333 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:42:24.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:24.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:24.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:24.378 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:42:24.416 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:42:24.450 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:24.453 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:42:24.487 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:24.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:24.490 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:42:24.540 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:42:24.578 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:42:24.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:24.621 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:42:24.624 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:24.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:24.632 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:42:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:42:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:42:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:42:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:42:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:42:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:42:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:42:24.726 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:24.762 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:24.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:24.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:24.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:24.898 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:24.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:24 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:42:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:24 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:42:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:24 smithi169 ceph-mon[29344]: osdmap e251: 8 total, 8 up, 8 in 2024-03-12T03:42:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:24 smithi169 ceph-mon[29344]: 5.8 scrub starts 2024-03-12T03:42:24.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:24 smithi169 ceph-mon[29344]: 5.8 scrub ok 2024-03-12T03:42:24.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:24 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1456737933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:42:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:24 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:42:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:24 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:42:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:24 smithi169 ceph-mon[39949]: osdmap e251: 8 total, 8 up, 8 in 2024-03-12T03:42:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:24 smithi169 ceph-mon[39949]: 5.8 scrub starts 2024-03-12T03:42:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:24 smithi169 ceph-mon[39949]: 5.8 scrub ok 2024-03-12T03:42:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:24 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1456737933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:42:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:24 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:42:24.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:24 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:42:24.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:24 smithi173 ceph-mon[33544]: osdmap e251: 8 total, 8 up, 8 in 2024-03-12T03:42:24.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:24 smithi173 ceph-mon[33544]: 5.8 scrub starts 2024-03-12T03:42:24.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:24 smithi173 ceph-mon[33544]: 5.8 scrub ok 2024-03-12T03:42:24.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:24 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1456737933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:42:25.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:25.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:25.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:25.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:25.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:25.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:25.283 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:25.316 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:25.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:25.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:25.455 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:25.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:25.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:25.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:25.572 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:25.577 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:25.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:25.588 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:25.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:25.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:25.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:25.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:25.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:25.869 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:25.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:25.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:25.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:25 smithi169 ceph-mon[29344]: pgmap v423: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.30k op/s; 4289/192489 objects degraded (2.228%); 22 MiB/s, 352 objects/s recovering 2024-03-12T03:42:25.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:25 smithi169 ceph-mon[39949]: pgmap v423: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.30k op/s; 4289/192489 objects degraded (2.228%); 22 MiB/s, 352 objects/s recovering 2024-03-12T03:42:25.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:25.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:25 smithi173 ceph-mon[33544]: pgmap v423: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.30k op/s; 4289/192489 objects degraded (2.228%); 22 MiB/s, 352 objects/s recovering 2024-03-12T03:42:26.004 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:26.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:26.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:26.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:26.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:26.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:26.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:26.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:26.288 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:26.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:26.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:26.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:26.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:26.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:26.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:26.528 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:26.565 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:26.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:26.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:26.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:26.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:26.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:26.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:26.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:26.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:26.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:26.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:26.977 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:26.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:27.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:27.117 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:27.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:27.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:27.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:27.253 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:27.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:27.354 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:27.383 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:42:27.389 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:27.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:27.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:27.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:27.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:27.525 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:27.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:27.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:27.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:27.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:27.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:27.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:27.809 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:27.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:27.910 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:27.918 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:27 smithi169 ceph-mon[29344]: pgmap v424: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 822 op/s; 4289/190341 objects degraded (2.253%); 12 MiB/s, 188 objects/s recovering 2024-03-12T03:42:27.918 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:27 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:42:27.918 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:27 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:42:27.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:27 smithi169 ceph-mon[39949]: pgmap v424: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 822 op/s; 4289/190341 objects degraded (2.253%); 12 MiB/s, 188 objects/s recovering 2024-03-12T03:42:27.918 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:27 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:42:27.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:27 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:42:27.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:27.944 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:27.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:27 smithi173 ceph-mon[33544]: pgmap v424: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 822 op/s; 4289/190341 objects degraded (2.253%); 12 MiB/s, 188 objects/s recovering 2024-03-12T03:42:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:27 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:42:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:27 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:42:28.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:28.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:28.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:28.176 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:28.212 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:28.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:28.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:28.344 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:28.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:28.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:28.472 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:28.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:28.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:28.601 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:28.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:28.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:28.687 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.7 to 1 (10000) 2024-03-12T03:42:28.703 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:28.741 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:28.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:28.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:28.885 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:28.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:28.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:28.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:28.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:28 smithi173 ceph-mon[33544]: osdmap e252: 8 total, 8 up, 8 in 2024-03-12T03:42:28.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:28 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 4289/190341 objects degraded (2.253%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:28.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:28 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1530624385' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:42:28.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:28 smithi173 ceph-mon[33544]: 8.c scrub starts 2024-03-12T03:42:29.024 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:29.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:29.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:29.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:29.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:29.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:28 smithi169 ceph-mon[29344]: osdmap e252: 8 total, 8 up, 8 in 2024-03-12T03:42:29.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:28 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 4289/190341 objects degraded (2.253%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:28 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1530624385' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:42:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:28 smithi169 ceph-mon[29344]: 8.c scrub starts 2024-03-12T03:42:29.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:28 smithi169 ceph-mon[39949]: osdmap e252: 8 total, 8 up, 8 in 2024-03-12T03:42:29.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:28 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 4289/190341 objects degraded (2.253%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:29.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:28 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1530624385' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:42:29.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:28 smithi169 ceph-mon[39949]: 8.c scrub starts 2024-03-12T03:42:29.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:29.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:29.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:29.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:29.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:29.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:29.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-12T03:42:29.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:29.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:29.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:29.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:29.699 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:29.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:29.709 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:42:29.800 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:29.828 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:29.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[29344]: pgmap v426: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded+remapped+backfilling, 154 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.42k op/s; 1799/176850 objects degraded (1.017%); 32 MiB/s, 509 objects/s recovering 2024-03-12T03:42:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[29344]: 1.9 scrub starts 2024-03-12T03:42:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[29344]: 1.9 scrub ok 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1530624385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[29344]: osdmap e253: 8 total, 8 up, 8 in 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[29344]: 1.1 scrub starts 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[29344]: 1.1 scrub ok 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[29344]: 8.c scrub ok 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3262850363' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[39949]: pgmap v426: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded+remapped+backfilling, 154 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.42k op/s; 1799/176850 objects degraded (1.017%); 32 MiB/s, 509 objects/s recovering 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[39949]: 1.9 scrub starts 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[39949]: 1.9 scrub ok 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1530624385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[39949]: osdmap e253: 8 total, 8 up, 8 in 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[39949]: 1.1 scrub starts 2024-03-12T03:42:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[39949]: 1.1 scrub ok 2024-03-12T03:42:29.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[39949]: 8.c scrub ok 2024-03-12T03:42:29.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3262850363' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:42:29.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:29 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:42:29.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:29.956 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:29.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:29.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:29 smithi173 ceph-mon[33544]: pgmap v426: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded+remapped+backfilling, 154 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.42k op/s; 1799/176850 objects degraded (1.017%); 32 MiB/s, 509 objects/s recovering 2024-03-12T03:42:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:29 smithi173 ceph-mon[33544]: 1.9 scrub starts 2024-03-12T03:42:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:29 smithi173 ceph-mon[33544]: 1.9 scrub ok 2024-03-12T03:42:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:29 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1530624385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-12T03:42:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:29 smithi173 ceph-mon[33544]: osdmap e253: 8 total, 8 up, 8 in 2024-03-12T03:42:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:29 smithi173 ceph-mon[33544]: 1.1 scrub starts 2024-03-12T03:42:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:29 smithi173 ceph-mon[33544]: 1.1 scrub ok 2024-03-12T03:42:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:29 smithi173 ceph-mon[33544]: 8.c scrub ok 2024-03-12T03:42:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:29 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3262850363' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:42:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:29 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:42:30.057 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:30.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:30.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:30.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:30.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:30.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:30.247 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:30.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:30.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:30.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:30.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:30.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:30.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:30.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:30.569 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:30.592 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:30.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:30.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:30.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:30.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:30.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:30.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:30.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:30.859 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:30.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:30.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:30.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:30.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:30 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:42:30.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:30 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:42:30.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:30 smithi173 ceph-mon[33544]: osdmap e254: 8 total, 8 up, 8 in 2024-03-12T03:42:30.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:30 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:30.991 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:30 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:42:30.991 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:30 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:42:30.991 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:30 smithi169 ceph-mon[29344]: osdmap e254: 8 total, 8 up, 8 in 2024-03-12T03:42:30.991 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:30 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:30.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:30 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:42:30.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:30 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:42:30.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:30 smithi169 ceph-mon[39949]: osdmap e254: 8 total, 8 up, 8 in 2024-03-12T03:42:30.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:30 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:31.087 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:31.111 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:31.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:31.213 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:31.265 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:31.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:31.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:31.367 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:31.393 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:31.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:31.436 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:42:31.436 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:42:31.436 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-03-12T03:42:31.436 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:42:31.494 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:31.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:31.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:31.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:31.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:31.645 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:31.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:31.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:31.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:31.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:31.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:31.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:31.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:31.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:31.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:31 smithi173 ceph-mon[33544]: pgmap v429: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded+remapped+backfilling, 154 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 959 op/s; 1799/175233 objects degraded (1.027%); 23 MiB/s, 370 objects/s recovering 2024-03-12T03:42:31.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:31 smithi173 ceph-mon[33544]: osdmap e255: 8 total, 8 up, 8 in 2024-03-12T03:42:32.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:32.051 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:32.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:32.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:32.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:32.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:31 smithi169 ceph-mon[29344]: pgmap v429: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded+remapped+backfilling, 154 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 959 op/s; 1799/175233 objects degraded (1.027%); 23 MiB/s, 370 objects/s recovering 2024-03-12T03:42:32.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:31 smithi169 ceph-mon[29344]: osdmap e255: 8 total, 8 up, 8 in 2024-03-12T03:42:32.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:31 smithi169 ceph-mon[39949]: pgmap v429: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded+remapped+backfilling, 154 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 959 op/s; 1799/175233 objects degraded (1.027%); 23 MiB/s, 370 objects/s recovering 2024-03-12T03:42:32.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:31 smithi169 ceph-mon[39949]: osdmap e255: 8 total, 8 up, 8 in 2024-03-12T03:42:32.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:42:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:42:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:42:32.187 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:32.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:32.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:32.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:32.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:32.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:32.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:32.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-12T03:42:32.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:32.469 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:32.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:32.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:32.617 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:32.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:32.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:32.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:32.755 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:32.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:32.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:32.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:32.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:32.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:32.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:32.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:32 smithi173 ceph-mon[33544]: osdmap e256: 8 total, 8 up, 8 in 2024-03-12T03:42:32.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:42:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:42:32.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:33.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:33.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:33.081 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:32 smithi169 ceph-mon[39949]: osdmap e256: 8 total, 8 up, 8 in 2024-03-12T03:42:33.081 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:32 smithi169 ceph-mon[29344]: osdmap e256: 8 total, 8 up, 8 in 2024-03-12T03:42:33.118 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:33.135 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:33.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:33.200 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:33.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:33.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:33.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:33.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:33.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:33.471 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:33.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:33.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:33.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:33.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:33.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:33.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:33.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:33.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:33 smithi169 ceph-mon[29344]: pgmap v432: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+undersized+degraded+remapped+backfilling, 155 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 605 op/s; 1641/169596 objects degraded (0.968%); 2.4 MiB/s, 38 objects/s recovering 2024-03-12T03:42:33.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:33 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:33.816 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:33 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 1641/169596 objects degraded (0.968%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:33.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:33 smithi169 ceph-mon[39949]: pgmap v432: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+undersized+degraded+remapped+backfilling, 155 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 605 op/s; 1641/169596 objects degraded (0.968%); 2.4 MiB/s, 38 objects/s recovering 2024-03-12T03:42:33.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:33 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:33.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:33 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 1641/169596 objects degraded (0.968%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:33.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:33.875 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:33.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:33.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:33 smithi173 ceph-mon[33544]: pgmap v432: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+undersized+degraded+remapped+backfilling, 155 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 605 op/s; 1641/169596 objects degraded (0.968%); 2.4 MiB/s, 38 objects/s recovering 2024-03-12T03:42:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:33 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:33 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 1641/169596 objects degraded (0.968%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-12T03:42:34.018 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:34.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:34.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:34.158 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:34.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:34.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:34.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:34.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:34.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:34.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:34.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:34.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:34.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:34.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:34.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:34.569 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:34.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:34.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:34.603 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:42:34.624 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:42:34.662 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:42:34.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:34.702 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:42:34.709 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:34.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:34.744 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:42:34.786 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:42:34.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:34.826 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:42:34.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:34.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:34.861 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:42:34.863 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:42:34.863 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:42:34.863 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:42:34.863 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:42:34.864 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:42:34.864 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:42:34.864 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:42:34.864 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:42:34.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:34.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:34.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:34.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:34 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2475541562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:42:35.013 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:34 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2475541562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:42:35.013 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:34 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2475541562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:42:35.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:35.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:35.119 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:35.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:35.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:35.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:35.259 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:35.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:35.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:35.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:35.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:35.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:35.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:35.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:35.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:42:35.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:35.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:35.698 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:35.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:35.800 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:35.836 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:35.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:35.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:35.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:35 smithi169 ceph-mon[39949]: pgmap v433: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 778 B/s rd, 0 B/s wr, 596 op/s; 1039/167442 objects degraded (0.621%); 8.8 MiB/s, 141 objects/s recovering 2024-03-12T03:42:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:35 smithi169 ceph-mon[29344]: pgmap v433: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 778 B/s rd, 0 B/s wr, 596 op/s; 1039/167442 objects degraded (0.621%); 8.8 MiB/s, 141 objects/s recovering 2024-03-12T03:42:35.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:35.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:35.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:35 smithi173 ceph-mon[33544]: pgmap v433: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 778 B/s rd, 0 B/s wr, 596 op/s; 1039/167442 objects degraded (0.621%); 8.8 MiB/s, 141 objects/s recovering 2024-03-12T03:42:36.080 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:36.118 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:36.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:36.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:36.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:36.266 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:36.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:36.360 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:36.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:36.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:36.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:36.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:36.540 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:36.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:36.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:36.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:36.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:36.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:36.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:36.812 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:36.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:36.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:36.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:36 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:42:36.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:36 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:42:36.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:36 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:42:36.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:36 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:42:36.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:36.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:36.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:37.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:37.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:37.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:37.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:37.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:37.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:36 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:42:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:36 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:42:37.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:37.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:37.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:37.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:37.503 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:37.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:37.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:37.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:37.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:37.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:37.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:37.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:42:37.742 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:37.791 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:37.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:37.892 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:37.936 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:37.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:38.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:38.078 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:38.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:38.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:37 smithi169 ceph-mon[29344]: pgmap v434: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 544 op/s; 17 MiB/s, 279 objects/s recovering 2024-03-12T03:42:38.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:37 smithi169 ceph-mon[29344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1039/167442 objects degraded (0.621%), 1 pg degraded, 1 pg undersized) 2024-03-12T03:42:38.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:37 smithi169 ceph-mon[29344]: osdmap e257: 8 total, 8 up, 8 in 2024-03-12T03:42:38.137 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:37 smithi169 ceph-mon[39949]: pgmap v434: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 544 op/s; 17 MiB/s, 279 objects/s recovering 2024-03-12T03:42:38.137 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:37 smithi169 ceph-mon[39949]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1039/167442 objects degraded (0.621%), 1 pg degraded, 1 pg undersized) 2024-03-12T03:42:38.137 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:37 smithi169 ceph-mon[39949]: osdmap e257: 8 total, 8 up, 8 in 2024-03-12T03:42:38.180 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:38.212 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:38.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:37 smithi173 ceph-mon[33544]: pgmap v434: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 544 op/s; 17 MiB/s, 279 objects/s recovering 2024-03-12T03:42:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:37 smithi173 ceph-mon[33544]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1039/167442 objects degraded (0.621%), 1 pg degraded, 1 pg undersized) 2024-03-12T03:42:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:37 smithi173 ceph-mon[33544]: osdmap e257: 8 total, 8 up, 8 in 2024-03-12T03:42:38.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:38.351 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:38.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:38.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:38.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:38.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:38.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:38.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:38.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:38.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:38.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:38.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:38.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:38.766 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:38.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:38.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:38.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:38.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:39.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:39.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:39.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:39.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:39.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:39.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:39.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:39.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:39.316 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:39.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:39.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:39.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:39.454 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:39.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:39.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:39.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:39.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:39.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:39.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:39.719 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:39.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:39.821 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:39.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:39.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:39.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:39.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:39.995 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:39.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:40.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:40.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:40.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:40.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:39 smithi169 ceph-mon[29344]: pgmap v436: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 15 MiB/s, 241 objects/s recovering 2024-03-12T03:42:40.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:39 smithi169 ceph-mon[39949]: pgmap v436: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 15 MiB/s, 241 objects/s recovering 2024-03-12T03:42:40.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:40.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:39 smithi173 ceph-mon[33544]: pgmap v436: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 15 MiB/s, 241 objects/s recovering 2024-03-12T03:42:40.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:40.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:40.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:40.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:40.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:40.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:40.529 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:40.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:40.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:40.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:40.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:40.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:40.806 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:40.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:40.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:40.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:40.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:41.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:41.066 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:41.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:41.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:41.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:40 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:42:41.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:40 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:42:41.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:40 smithi169 ceph-mon[29344]: 8.5 deep-scrub ok 2024-03-12T03:42:41.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:40 smithi169 ceph-mon[29344]: osdmap e258: 8 total, 8 up, 8 in 2024-03-12T03:42:41.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:40 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:42:41.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:40 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:42:41.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:40 smithi169 ceph-mon[39949]: 8.5 deep-scrub ok 2024-03-12T03:42:41.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:40 smithi169 ceph-mon[39949]: osdmap e258: 8 total, 8 up, 8 in 2024-03-12T03:42:41.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:41.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:40 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:42:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:40 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:42:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:40 smithi173 ceph-mon[33544]: 8.5 deep-scrub ok 2024-03-12T03:42:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:40 smithi173 ceph-mon[33544]: osdmap e258: 8 total, 8 up, 8 in 2024-03-12T03:42:41.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:41.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:41.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:41.439 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:41.463 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:41.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:41.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:41.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:41.603 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:41.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:41.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:41.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:41.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:41.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:41.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:41.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:41.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:41.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:41.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:42.013 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:42.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:42.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:42.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:42.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:42.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:41 smithi169 ceph-mon[29344]: pgmap v438: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 823 op/s; 12 MiB/s, 199 objects/s recovering 2024-03-12T03:42:42.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:41 smithi169 ceph-mon[39949]: pgmap v438: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 823 op/s; 12 MiB/s, 199 objects/s recovering 2024-03-12T03:42:42.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:42:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:42:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:42:42.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:41 smithi173 ceph-mon[33544]: pgmap v438: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 823 op/s; 12 MiB/s, 199 objects/s recovering 2024-03-12T03:42:42.250 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:42.289 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:42.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:42.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:42.433 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:42.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:42.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:42.567 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:42.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:42.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:42.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:42.708 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:42.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:42.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:42.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:42.847 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:42.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:42.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:42.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:42.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:42.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:42.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:42:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:42:43.083 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:43.119 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:43.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:43.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:43.245 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:43.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:43.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:43.380 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:43.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:43.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:43.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:43.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:43.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:43.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:43.653 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:43.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:43.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:43.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:43.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:43.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:43.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:43.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:43.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:43 smithi169 ceph-mon[29344]: pgmap v439: 158 pgs: 158 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.12k op/s; 7.8 MiB/s, 125 objects/s recovering 2024-03-12T03:42:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:43 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:43.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:43 smithi169 ceph-mon[39949]: pgmap v439: 158 pgs: 158 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.12k op/s; 7.8 MiB/s, 125 objects/s recovering 2024-03-12T03:42:43.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:43 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:44.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:44.058 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:44.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:44.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:44.160 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:44.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:44.209 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:44.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:44.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:43 smithi173 ceph-mon[33544]: pgmap v439: 158 pgs: 158 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.12k op/s; 7.8 MiB/s, 125 objects/s recovering 2024-03-12T03:42:44.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:43 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:44.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:44.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:44.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:44.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:44.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:44.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:44.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:44.616 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:44.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:44.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:44.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:44.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:44.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:44.789 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.2 to 1 (10000) 2024-03-12T03:42:44.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:44.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:44.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:44.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:45.008 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:45.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:45.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:45.076 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:44 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1770565166' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:42:45.076 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:44 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:42:45.077 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:44 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1770565166' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:42:45.077 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:44 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:42:45.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:45.150 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:45.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:45.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:44 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1770565166' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:42:45.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:44 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:42:45.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:45.287 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:45.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:45.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:45.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:45.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:45.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:45.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:45.550 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:45.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:45.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:45.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:45.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:45.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:45.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:45.829 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:45.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:45.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:45.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:45.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:45.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:45.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:46.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:45 smithi169 ceph-mon[29344]: pgmap v440: 158 pgs: 158 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-12T03:42:46.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:45 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-12T03:42:46.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:45 smithi169 ceph-mon[29344]: osdmap e259: 8 total, 8 up, 8 in 2024-03-12T03:42:46.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:45 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:42:46.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:45 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:46.053 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:45 smithi169 ceph-mon[39949]: pgmap v440: 158 pgs: 158 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-12T03:42:46.053 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:45 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-12T03:42:46.053 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:45 smithi169 ceph-mon[39949]: osdmap e259: 8 total, 8 up, 8 in 2024-03-12T03:42:46.053 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:45 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:42:46.053 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:45 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:46.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:46.111 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:46.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:46.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:46.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:45 smithi173 ceph-mon[33544]: pgmap v440: 158 pgs: 158 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-12T03:42:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:45 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-12T03:42:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:45 smithi173 ceph-mon[33544]: osdmap e259: 8 total, 8 up, 8 in 2024-03-12T03:42:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:45 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:42:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:45 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:42:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:46.266 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:46.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:46.269 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:42:46.269 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:42:46.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:46.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:46.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:46.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:46.544 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:46.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:46.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:46.690 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:46.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:46.711 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-12T03:42:46.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:42:46.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:46.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:46.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:46.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:46.986 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:46.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:47.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:47.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:47.155 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:47.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:47.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:47.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:47.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:47.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:47.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:47.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:47.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:47.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:47.566 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:47.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:47.667 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:47.714 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:47.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:47.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:47.822 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:42:47.877 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:47.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:47.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:47.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-12T03:42:47.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:48.017 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:48.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:48.103 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:48.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:48.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:48.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:47 smithi169 ceph-mon[29344]: pgmap v442: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 828 op/s 2024-03-12T03:42:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:47 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:42:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:47 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:42:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:47 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1831460147' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:42:48.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:47 smithi169 ceph-mon[39949]: pgmap v442: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 828 op/s 2024-03-12T03:42:48.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:47 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:42:48.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:47 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:42:48.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:47 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1831460147' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:42:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:47 smithi173 ceph-mon[33544]: pgmap v442: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 828 op/s 2024-03-12T03:42:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:47 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:42:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:47 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:42:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:47 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1831460147' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:42:48.248 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:48.303 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:48.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:48.404 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:48.445 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:48.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:48.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:48.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:48.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:48.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:48.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:48.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:48.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:48.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:48.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:48.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:48.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:48.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:48.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:49.078 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:48 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:42:49.078 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:48 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1831460147' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:42:49.078 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:48 smithi169 ceph-mon[29344]: osdmap e260: 8 total, 8 up, 8 in 2024-03-12T03:42:49.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:48 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:42:49.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:48 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1831460147' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:42:49.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:48 smithi169 ceph-mon[39949]: osdmap e260: 8 total, 8 up, 8 in 2024-03-12T03:42:49.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:49.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:49.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:49.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:49.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:48 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:42:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:48 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1831460147' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:42:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:48 smithi173 ceph-mon[33544]: osdmap e260: 8 total, 8 up, 8 in 2024-03-12T03:42:49.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:49.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:49.357 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:49.409 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:49.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:49.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:49.548 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:49.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:49.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:49.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:49.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:49.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:49.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:49.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:49.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:49.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:49.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:49.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:50.044 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:50.045 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:50.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:50.121 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:50.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:50.153 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:49 smithi169 ceph-mon[29344]: pgmap v444: 158 pgs: 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-12T03:42:50.154 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:49 smithi169 ceph-mon[39949]: pgmap v444: 158 pgs: 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-12T03:42:50.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:50.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:49 smithi173 ceph-mon[33544]: pgmap v444: 158 pgs: 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-12T03:42:50.269 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:50.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:50.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:50.404 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:50.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:50.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:50.553 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:50.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:50.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:50.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:42:50.655 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:50.717 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:50.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:50.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:50.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:50.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:50.892 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:42:50.938 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:42:50.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:50.977 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:42:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:50.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:50.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:51.017 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:42:51.054 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:42:51.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:51.101 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:42:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:51.141 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:51.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:51.152 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:42:51.198 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:42:51.203 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:42:51.203 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:42:51.203 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:42:51.203 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:42:51.203 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:42:51.203 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:42:51.203 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:42:51.203 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:42:51.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:51.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:51.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:51.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:51.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:51.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:51.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:51.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:51.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:51.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:51.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:51.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:51.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:51.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:51.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:51.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:51.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:51.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:51.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:52.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:52.097 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:52.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:52.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:51 smithi169 ceph-mon[29344]: pgmap v445: 158 pgs: 158 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-12T03:42:52.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:51 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2467428628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:42:52.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:42:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:42:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:42:52.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:51 smithi169 ceph-mon[39949]: pgmap v445: 158 pgs: 158 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-12T03:42:52.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:51 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2467428628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:42:52.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:51 smithi173 ceph-mon[33544]: pgmap v445: 158 pgs: 158 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-12T03:42:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:51 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2467428628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:42:52.242 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:52.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:52.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:52.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:52.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:52.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:52.517 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:52.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:52.619 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:52.659 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:52.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:52.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:52.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:52.805 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:52.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:52.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:52.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:52.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:52.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:52.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:53.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:53.072 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:53.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:53.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:53.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:53.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:53.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:53.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:53.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:42:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:42:53.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:53.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:53.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:53.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:53.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:53.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:53.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:53.586 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:53.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:53.648 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:53.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:53.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:53.793 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:53.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:53.891 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:53 smithi169 ceph-mon[29344]: pgmap v446: 158 pgs: 158 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-12T03:42:53.891 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:53 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:53.891 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:53 smithi169 ceph-mon[39949]: pgmap v446: 158 pgs: 158 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-12T03:42:53.891 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:53 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:53.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:53.937 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:53.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:53.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:54.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:54.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:54.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:54.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:54.164 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:54.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:54.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:54.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:54.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:53 smithi173 ceph-mon[33544]: pgmap v446: 158 pgs: 158 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-12T03:42:54.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:53 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:54.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:54.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:54.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:54.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:54.476 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:54.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:54.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:54.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:54.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:54.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:42:54.670 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:42:54.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:54.733 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:42:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:42:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14445.0:2 3.0 3:00000000::::head [pgnls start_epoch 67] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e67)", 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:33:21.103943+0000", 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 573.62300922500003, 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46950572800000001, 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14445", 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/3748937004", 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 2 2024-03-12T03:42:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103943+0000", 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103943+0000", 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103945+0000", 2024-03-12T03:42:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5710000000000001e-06 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103951+0000", 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8420000000000001e-06 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103961+0000", 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0139000000000001e-05 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.103972+0000", 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1579e-05 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.104099+0000", 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000126329 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.104106+0000", 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7100000000000007e-06 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.573181+0000", 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46907466599999997 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.573220+0000", 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9076999999999999e-05 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.573449+0000", 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000228815 2024-03-12T03:42:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14436.0:4 3.0 3:00000000::::head [pgnls start_epoch 67] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e67)", 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:33:21.109874+0000", 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 573.61707798800001, 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46447891499999999, 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14436", 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1557333783", 2024-03-12T03:42:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109874+0000", 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109874+0000", 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109876+0000", 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.694e-06 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109882+0000", 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.716e-06 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109894+0000", 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1758e-05 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109908+0000", 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4017e-05 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109959+0000", 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1094000000000002e-05 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.109966+0000", 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5780000000000001e-06 2024-03-12T03:42:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.574147+0000", 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46418114100000002 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.574184+0000", 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6148e-05 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:33:21.574353+0000", 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016976900000000001 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155689 7.b 7:d77d3091:::benchmark_data_smithi169_79822_object38272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.729513+0000", 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 301.997439456, 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.199297788, 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155689 2024-03-12T03:42:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729513+0000", 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729513+0000", 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729514+0000", 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4300000000000001e-06 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729519+0000", 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2549999999999999e-06 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729527+0000", 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1410000000000005e-06 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729533+0000", 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0410000000000003e-06 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.790125+0000", 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.060591712999999998 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.790253+0000", 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000128052 2024-03-12T03:42:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.790394+0000", 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000141812 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.798494+0000", 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080992140000000004 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.798540+0000", 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6567999999999999e-05 2024-03-12T03:42:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928691+0000", 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.130150604 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928746+0000", 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5340000000000002e-05 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928811+0000", 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4617999999999997e-05 2024-03-12T03:42:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155695 7.b 7:d612ca62:::benchmark_data_smithi169_79822_object38278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.732453+0000", 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 301.994499615, 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19647717000000001, 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:42:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155695 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732453+0000", 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732453+0000", 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732454+0000", 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.671e-06 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732461+0000", 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8179999999999998e-06 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732477+0000", 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6169999999999999e-05 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732486+0000", 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3790000000000006e-06 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.795096+0000", 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062610618000000007 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.795213+0000", 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011686199999999999 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:42:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.795338+0000", 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012426200000000001 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.798582+0000", 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032448910000000002 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.805288+0000", 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067054109999999997 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928826+0000", 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12353776499999999 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928877+0000", 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.147e-05 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928930+0000", 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2852999999999997e-05 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211058 8.7 8:efebe503:::benchmark_data_smithi169_106492_object77575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.681215+0000", 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.045737663000001, 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0584678489999999, 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211058 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681215+0000", 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681215+0000", 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681228+0000", 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3385e-05 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681233+0000", 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5700000000000003e-06 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681451+0000", 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000218364 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681459+0000", 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5290000000000003e-06 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.683093+0000", 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001634052 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.683194+0000", 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010169400000000001 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.683333+0000", 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013846600000000001 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.688997+0000", 2024-03-12T03:42:54.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056637600000000003 2024-03-12T03:42:54.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.697167+0000", 2024-03-12T03:42:54.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.008170198 2024-03-12T03:42:54.761 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.761 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.761 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.761 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710173+0000", 2024-03-12T03:42:54.761 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013006172999999999 2024-03-12T03:42:54.761 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.761 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.761 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.761 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710182+0000", 2024-03-12T03:42:54.761 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4239999999999999e-06 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.739635+0000", 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0294528089999999 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.739683+0000", 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7425000000000001e-05 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211062 8.7 8:e1bda5df:::benchmark_data_smithi169_106492_object77579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.685338+0000", 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.041614711000001, 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.054851787, 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211062 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685338+0000", 2024-03-12T03:42:54.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685338+0000", 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685339+0000", 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3909999999999999e-06 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685345+0000", 2024-03-12T03:42:54.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8769999999999998e-06 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685361+0000", 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5602000000000001e-05 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685369+0000", 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3720000000000005e-06 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686032+0000", 2024-03-12T03:42:54.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00066344899999999998 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686176+0000", 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000143435 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686310+0000", 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013403300000000001 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.766 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.689003+0000", 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026933310000000002 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.700621+0000", 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011617689 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710236+0000", 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0096148210000000008 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710244+0000", 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2730000000000002e-06 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740142+0000", 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.029898373 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740189+0000", 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7141000000000001e-05 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.768 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211063 8.7 8:e4c4cdd4:::benchmark_data_smithi169_106492_object77580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.685985+0000", 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.040967754, 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0542865020000001, 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211063 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.769 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685985+0000", 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685985+0000", 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685985+0000", 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4e-07 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685991+0000", 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0719999999999999e-06 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686001+0000", 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0467e-05 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686009+0000", 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.503e-06 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.689034+0000", 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003025659 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.689186+0000", 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000152027 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.689335+0000", 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014890899999999999 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.691991+0000", 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026562019999999999 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.701569+0000", 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0095778640000000002 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710278+0000", 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087084380000000006 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710285+0000", 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7600000000000002e-06 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740215+0000", 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0299298880000001 2024-03-12T03:42:54.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740271+0000", 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5872999999999999e-05 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211068 8.7 8:e27717b5:::benchmark_data_smithi169_106492_object77585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.693940+0000", 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.033011986000002, 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0464507999999999, 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.775 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211068 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693940+0000", 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693940+0000", 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693942+0000", 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.567e-06 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693948+0000", 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2949999999999999e-06 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693964+0000", 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5559000000000001e-05 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693973+0000", 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0480000000000004e-06 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.694033+0000", 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9882e-05 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.694152+0000", 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011884599999999999 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.694277+0000", 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012499899999999999 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.697558+0000", 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032812539999999999 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.779 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.701613+0000", 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040555299999999999 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710314+0000", 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087006870000000004 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710320+0000", 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3690000000000003e-06 2024-03-12T03:42:54.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740325+0000", 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0300041929999999 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740391+0000", 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6570999999999997e-05 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211087 8.7 8:e3d7750f:::benchmark_data_smithi169_106492_object77604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.732661+0000", 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.994290918000001, 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.015391752, 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211087 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.782 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732661+0000", 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732661+0000", 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732663+0000", 2024-03-12T03:42:54.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6509999999999999e-06 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732670+0000", 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.562e-06 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732685+0000", 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.543e-05 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732695+0000", 2024-03-12T03:42:54.784 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4369999999999996e-06 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732912+0000", 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000217536 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732932+0000", 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9851999999999999e-05 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740436+0000", 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.007504204 2024-03-12T03:42:54.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740552+0000", 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115425 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740681+0000", 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000129512 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.747888+0000", 2024-03-12T03:42:54.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072064850000000003 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.747948+0000", 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0829000000000003e-05 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.747962+0000", 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3424999999999999e-05 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748006+0000", 2024-03-12T03:42:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:54.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:54.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:54.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3905000000000001e-05 2024-03-12T03:42:54.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.789 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.789 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748053+0000", 2024-03-12T03:42:54.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7499e-05 2024-03-12T03:42:54.789 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.789 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.789 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211134 8.7 8:eaa19edb:::benchmark_data_smithi169_106492_object77651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.783398+0000", 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.943554296999999, 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96989582799999996, 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211134 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.790 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783398+0000", 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783398+0000", 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783400+0000", 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6080000000000001e-06 2024-03-12T03:42:54.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783405+0000", 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3249999999999998e-06 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783420+0000", 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5267000000000001e-05 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783429+0000", 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5690000000000005e-06 2024-03-12T03:42:54.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783496+0000", 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7441999999999996e-05 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783510+0000", 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4178e-05 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.743998+0000", 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96048739500000002 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.744100+0000", 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101648 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.744209+0000", 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010987700000000001 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748122+0000", 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0039126300000000003 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.750470+0000", 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023481650000000001 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753230+0000", 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002759662 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753263+0000", 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3636000000000001e-05 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753294+0000", 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0426e-05 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211149 8.7 8:e1de3961:::benchmark_data_smithi169_106492_object77666:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.803392+0000", 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.923560040000002, 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.95420632900000002, 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.796 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211149 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803392+0000", 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803392+0000", 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803394+0000", 2024-03-12T03:42:54.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4899999999999999e-06 2024-03-12T03:42:54.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803400+0000", 2024-03-12T03:42:54.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5250000000000002e-06 2024-03-12T03:42:54.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803413+0000", 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2622e-05 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803421+0000", 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9310000000000008e-06 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803476+0000", 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5510999999999998e-05 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803483+0000", 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6989999999999997e-06 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748175+0000", 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.94469182799999996 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748276+0000", 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100855 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748387+0000", 2024-03-12T03:42:54.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011091600000000001 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753338+0000", 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049511879999999996 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753348+0000", 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6150000000000003e-06 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757502+0000", 2024-03-12T03:42:54.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004154952 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757557+0000", 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4391000000000002e-05 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757599+0000", 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1805999999999999e-05 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211151 8.7 8:e778f11b:::benchmark_data_smithi169_106492_object77668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.804839+0000", 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.922113769999999, 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.95295800399999997, 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211151 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804839+0000", 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804839+0000", 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804839+0000", 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4799999999999997e-07 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804844+0000", 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.899e-06 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804859+0000", 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4983e-05 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804867+0000", 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5610000000000001e-06 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804920+0000", 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2599000000000002e-05 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804956+0000", 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6235000000000001e-05 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.750501+0000", 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.94554544200000001 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.750600+0000", 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8881000000000004e-05 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.750743+0000", 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014310700000000001 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757648+0000", 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069051119999999997 2024-03-12T03:42:54.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757696+0000", 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8081999999999998e-05 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757721+0000", 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4800999999999999e-05 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757762+0000", 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0550999999999999e-05 2024-03-12T03:42:54.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757797+0000", 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4903000000000001e-05 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211165 8.7 8:e0ccb79f:::benchmark_data_smithi169_106492_object77682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.852948+0000", 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.874004088, 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.90957021500000002, 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.809 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211165 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852948+0000", 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852948+0000", 2024-03-12T03:42:54.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852949+0000", 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0899999999999999e-06 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852955+0000", 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.519e-06 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852968+0000", 2024-03-12T03:42:54.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2690999999999999e-05 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852976+0000", 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9750000000000007e-06 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.853032+0000", 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6852999999999999e-05 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.853043+0000", 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0497e-05 2024-03-12T03:42:54.812 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753375+0000", 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.90033189899999999 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753473+0000", 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8413999999999997e-05 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753540+0000", 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6861999999999998e-05 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.813 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.814 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.814 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757805+0000", 2024-03-12T03:42:54.814 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042649519999999998 2024-03-12T03:42:54.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.814 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.814 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.814 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757850+0000", 2024-03-12T03:42:54.814 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5198999999999999e-05 2024-03-12T03:42:54.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.814 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.814 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.814 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762430+0000", 2024-03-12T03:42:54.815 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045799810000000003 2024-03-12T03:42:54.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762477+0000", 2024-03-12T03:42:54.815 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7261e-05 2024-03-12T03:42:54.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762519+0000", 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1022000000000001e-05 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211182 8.7 8:e254d1c7:::benchmark_data_smithi169_106492_object77699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.870732+0000", 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.856220153999999, 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89188010699999998, 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.816 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211182 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870732+0000", 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870732+0000", 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870734+0000", 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3179999999999999e-06 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870738+0000", 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9309999999999999e-06 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870750+0000", 2024-03-12T03:42:54.818 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1039999999999999e-05 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870755+0000", 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9070000000000004e-06 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870804+0000", 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8819999999999997e-05 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870811+0000", 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1790000000000002e-06 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.819 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.755458+0000", 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88464690400000001 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.755563+0000", 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000104603 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.755655+0000", 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2262999999999994e-05 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.760061+0000", 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044056720000000002 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762393+0000", 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023321420000000002 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762551+0000", 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000157558 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762585+0000", 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4313999999999997e-05 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762612+0000", 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7455999999999999e-05 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211190 8.7 8:e2ab686f:::benchmark_data_smithi169_106492_object77707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:42:54.822 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.880970+0000", 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.845982208999999, 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89412791400000002, 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211190 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880970+0000", 2024-03-12T03:42:54.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880970+0000", 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880972+0000", 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.491e-06 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880978+0000", 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4860000000000002e-06 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880996+0000", 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7357999999999999e-05 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.881005+0000", 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1230000000000006e-06 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.881061+0000", 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6594000000000001e-05 2024-03-12T03:42:54.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.881071+0000", 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0067e-05 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757895+0000", 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.87682368799999999 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757991+0000", 2024-03-12T03:42:54.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6284000000000002e-05 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.758109+0000", 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117265 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762403+0000", 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042943770000000003 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762648+0000", 2024-03-12T03:42:54.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00024558700000000002 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.775012+0000", 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012363733 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.775060+0000", 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7663000000000001e-05 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.775098+0000", 2024-03-12T03:42:54.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8198000000000003e-05 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211197 8.7 8:e6e0739f:::benchmark_data_smithi169_106492_object77714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.890205+0000", 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.836747398, 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88828141800000004, 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.829 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211197 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890205+0000", 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890205+0000", 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890206+0000", 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.61e-07 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890209+0000", 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7299999999999999e-06 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890216+0000", 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8040000000000004e-06 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890221+0000", 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9200000000000003e-06 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890270+0000", 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8677999999999997e-05 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890278+0000", 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1249999999999993e-06 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.760085+0000", 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.869807471 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.760173+0000", 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7177999999999998e-05 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.760276+0000", 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010333299999999999 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762620+0000", 2024-03-12T03:42:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023441180000000001 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.766983+0000", 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043625510000000001 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778399+0000", 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011416608999999999 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778446+0000", 2024-03-12T03:42:54.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6796999999999999e-05 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778486+0000", 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0442999999999999e-05 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211199 8.7 8:e601f7fc:::benchmark_data_smithi169_106492_object77716:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.893694+0000", 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.833258740000002, 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88494738100000003, 2024-03-12T03:42:54.835 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211199 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893694+0000", 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893694+0000", 2024-03-12T03:42:54.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893695+0000", 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.173e-06 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893700+0000", 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3380000000000004e-06 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893713+0000", 2024-03-12T03:42:54.837 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2649e-05 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893721+0000", 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7689999999999996e-06 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893767+0000", 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5949999999999999e-05 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893776+0000", 2024-03-12T03:42:54.838 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0680000000000003e-06 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762678+0000", 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.86890198600000002 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762763+0000", 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5915999999999995e-05 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762862+0000", 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8690999999999994e-05 2024-03-12T03:42:54.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.767024+0000", 2024-03-12T03:42:54.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041618280000000002 2024-03-12T03:42:54.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771291+0000", 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042668070000000001 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778547+0000", 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072558479999999996 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778601+0000", 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4703000000000001e-05 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778641+0000", 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9654999999999999e-05 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211211 8.7 8:e437348b:::benchmark_data_smithi169_106492_object77728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.933217+0000", 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.793735418000001, 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.84557843200000005, 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.842 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211211 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933217+0000", 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933217+0000", 2024-03-12T03:42:54.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933218+0000", 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1030000000000001e-06 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933223+0000", 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6909999999999996e-06 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933232+0000", 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8740000000000001e-06 2024-03-12T03:42:54.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933237+0000", 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6790000000000002e-06 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933283+0000", 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5688e-05 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933290+0000", 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.007e-06 2024-03-12T03:42:54.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.764859+0000", 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.83156881400000005 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.764956+0000", 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7408e-05 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.846 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.765060+0000", 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103928 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771299+0000", 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006238609 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771351+0000", 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.257e-05 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778700+0000", 2024-03-12T03:42:54.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0073482479999999999 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778758+0000", 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8440999999999998e-05 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778795+0000", 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7372000000000001e-05 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211219 8.7 8:ec912678:::benchmark_data_smithi169_106492_object77736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.959590+0000", 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.767361971, 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81933082300000004, 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211219 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959590+0000", 2024-03-12T03:42:54.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959590+0000", 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959592+0000", 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.156e-06 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959596+0000", 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6450000000000004e-06 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959608+0000", 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1902000000000001e-05 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959616+0000", 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0779999999999996e-06 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.851 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959674+0000", 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7806000000000002e-05 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959685+0000", 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0841e-05 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.767051+0000", 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80736609000000004 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.767150+0000", 2024-03-12T03:42:54.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9516000000000001e-05 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.767246+0000", 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5678000000000001e-05 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771306+0000", 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040594740000000004 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.775173+0000", 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038675329999999998 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778846+0000", 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036730550000000002 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778887+0000", 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1253e-05 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778921+0000", 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3796e-05 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211222 8.7 8:e61e560a:::benchmark_data_smithi169_106492_object77739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.969003+0000", 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.757949131, 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81401453999999995, 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:42:54.855 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211222 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969003+0000", 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969003+0000", 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969005+0000", 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2640000000000001e-06 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969009+0000", 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3549999999999998e-06 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969019+0000", 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0091e-05 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969026+0000", 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4869999999999996e-06 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969072+0000", 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5597999999999999e-05 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969078+0000", 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.003e-06 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.769181+0000", 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80010249300000003 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.769269+0000", 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8134999999999996e-05 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.769375+0000", 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000106086 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771421+0000", 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0020459520000000002 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778959+0000", 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075385310000000002 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.782940+0000", 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003980946 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.782981+0000", 2024-03-12T03:42:54.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1287e-05 2024-03-12T03:42:54.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:42:54.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:42:54.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:42:54.861 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.783018+0000", 2024-03-12T03:42:54.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6312000000000001e-05 2024-03-12T03:42:54.861 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.861 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.861 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.861 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:42:54.861 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:42:54.861 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:42:54.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:54.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:54.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:55.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:55.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:55.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:55.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:55.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:55.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:55.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:55.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:55.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:55.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:55.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:55.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:55.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:55.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:55.593 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:55.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:55.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:55.638 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:42:55.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:55.742 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:55.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:55.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:55.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:55.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:55.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:55.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:55 smithi169 ceph-mon[29344]: pgmap v447: 158 pgs: 158 active+clean; 2.0 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1004 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-12T03:42:55.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:55 smithi169 ceph-mon[29344]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:42:55.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:55 smithi169 ceph-mon[39949]: pgmap v447: 158 pgs: 158 active+clean; 2.0 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1004 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-12T03:42:55.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:55 smithi169 ceph-mon[39949]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:42:55.932 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-12T03:42:55.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:56.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:56.028 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:56.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:56.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:56.176 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:56.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:56.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:55 smithi173 ceph-mon[33544]: pgmap v447: 158 pgs: 158 active+clean; 2.0 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1004 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-12T03:42:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:55 smithi173 ceph-mon[33544]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:42:56.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:56.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:56.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:56.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:56.464 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:56.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:56.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:56.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:56.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:56.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:56.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:56.739 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:56.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:56.840 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:56.882 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:56.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:56.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:57.020 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:57.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:57.036 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:56 smithi169 ceph-mon[29344]: pgmap v448: 158 pgs: 158 active+clean; 2.0 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-12T03:42:57.040 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:56 smithi169 ceph-mon[39949]: pgmap v448: 158 pgs: 158 active+clean; 2.0 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-12T03:42:57.086 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2024-03-12T03:42:53.953051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2176969574,"num_objects":33443,"num_object_clones":0,"num_object_copies":100329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33443,"num_whiteouts":0,"num_read":1579,"num_read_kb":1308,"num_write":234392,"num_write_kb":8542986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34324,"num_bytes_recovered":2243267396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82420,"ondisk_log_size":82420,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9449580,"kb_used_data":6510312,"kb_used_omap":18,"kb_used_meta":2938989,"kb_avail":740511636,"statfs":{"total":767960285184,"available":758283915264,"internally_reserved":0,"allocated":6666559488,"data_stored":6663235904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19339,"internal_metadata":3009524853},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-712572928,"num_objects":-10873,"num_object_clones":0,"num_object_copies":-32619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10873,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":10873,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.178157"},"pg_stats":[{"pgid":"8.0","version":"260'14287","reported_seq":22668,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:49.616856+0000","last_change":"2024-03-12T03:42:31.749803+0000","last_active":"2024-03-12T03:42:49.616856+0000","last_peered":"2024-03-12T03:42:49.616856+0000","last_clean":"2024-03-12T03:42:49.616856+0000","last_became_active":"2024-03-12T03:42:31.713116+0000","last_became_peered":"2024-03-12T03:42:31.713116+0000","last_unstale":"2024-03-12T03:42:49.616856+0000","last_undegraded":"2024-03-12T03:42:49.616856+0000","last_fullsized":"2024-03-12T03:42:49.616856+0000","mapping_epoch":255,"log_start":"238'9798","ondisk_log_start":"238'9798","created":198,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4489,"log_dups_size":0,"ondisk_log_size":4489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140705792,"num_objects":2147,"num_object_clones":0,"num_object_copies":6441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14287,"num_write_kb":525888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4123,"num_bytes_recovered":270204928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":733,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.824919+0000","last_change":"2024-03-12T03:42:28.162856+0000","last_active":"2024-03-12T03:42:47.824919+0000","last_peered":"2024-03-12T03:42:47.824919+0000","last_clean":"2024-03-12T03:42:47.824919+0000","last_became_active":"2024-03-12T03:40:46.529101+0000","last_became_peered":"2024-03-12T03:40:46.529101+0000","last_unstale":"2024-03-12T03:42:47.824919+0000","last_undegraded":"2024-03-12T03:42:47.824919+0000","last_fullsized":"2024-03-12T03:42:47.824919+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:28.162735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:28.162735+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:49.132807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":445,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.815823+0000","last_change":"2024-03-12T03:41:02.239526+0000","last_active":"2024-03-12T03:42:47.815823+0000","last_peered":"2024-03-12T03:42:47.815823+0000","last_clean":"2024-03-12T03:42:47.815823+0000","last_became_active":"2024-03-12T03:40:46.518953+0000","last_became_peered":"2024-03-12T03:40:46.518953+0000","last_unstale":"2024-03-12T03:42:47.815823+0000","last_undegraded":"2024-03-12T03:42:47.815823+0000","last_fullsized":"2024-03-12T03:42:47.815823+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-12T03:41:02.239384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:41:02.239384+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.434802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":329,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.831575+0000","last_change":"2024-03-12T03:40:46.528095+0000","last_active":"2024-03-12T03:42:47.831575+0000","last_peered":"2024-03-12T03:42:47.831575+0000","last_clean":"2024-03-12T03:42:47.831575+0000","last_became_active":"2024-03-12T03:40:46.523062+0000","last_became_peered":"2024-03-12T03:40:46.523062+0000","last_unstale":"2024-03-12T03:42:47.831575+0000","last_undegraded":"2024-03-12T03:42:47.831575+0000","last_fullsized":"2024-03-12T03:42:47.831575+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.981378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.981378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":187,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.834239+0000","last_change":"2024-03-12T03:40:18.939478+0000","last_active":"2024-03-12T03:42:47.834239+0000","last_peered":"2024-03-12T03:42:47.834239+0000","last_clean":"2024-03-12T03:42:47.834239+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:42:47.834239+0000","last_undegraded":"2024-03-12T03:42:47.834239+0000","last_fullsized":"2024-03-12T03:42:47.834239+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:40:18.939349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:40:18.939349+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.353113+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.816262+0000","last_change":"2024-03-12T03:42:08.287146+0000","last_active":"2024-03-12T03:42:47.816262+0000","last_peered":"2024-03-12T03:42:47.816262+0000","last_clean":"2024-03-12T03:42:47.816262+0000","last_became_active":"2024-03-12T03:39:33.265665+0000","last_became_peered":"2024-03-12T03:39:33.265665+0000","last_unstale":"2024-03-12T03:42:47.816262+0000","last_undegraded":"2024-03-12T03:42:47.816262+0000","last_fullsized":"2024-03-12T03:42:47.816262+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:08.287006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:08.287006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:10.630966+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"260'14675","reported_seq":14810,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:52.012470+0000","last_change":"2024-03-12T03:39:34.336674+0000","last_active":"2024-03-12T03:42:52.012470+0000","last_peered":"2024-03-12T03:42:52.012470+0000","last_clean":"2024-03-12T03:42:52.012470+0000","last_became_active":"2024-03-12T03:39:34.336345+0000","last_became_peered":"2024-03-12T03:39:34.336345+0000","last_unstale":"2024-03-12T03:42:52.012470+0000","last_undegraded":"2024-03-12T03:42:52.012470+0000","last_fullsized":"2024-03-12T03:42:52.012470+0000","mapping_epoch":198,"log_start":"237'9900","ondisk_log_start":"237'9900","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132710400,"num_objects":2025,"num_object_clones":0,"num_object_copies":6075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14675,"num_write_kb":534400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":505,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.818719+0000","last_change":"2024-03-12T03:42:11.397167+0000","last_active":"2024-03-12T03:42:47.818719+0000","last_peered":"2024-03-12T03:42:47.818719+0000","last_clean":"2024-03-12T03:42:47.818719+0000","last_became_active":"2024-03-12T03:39:33.268652+0000","last_became_peered":"2024-03-12T03:39:33.268652+0000","last_unstale":"2024-03-12T03:42:47.818719+0000","last_undegraded":"2024-03-12T03:42:47.818719+0000","last_fullsized":"2024-03-12T03:42:47.818719+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:42:11.397028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:42:11.397028+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:20.606187+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":432,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.820129+0000","last_change":"2024-03-12T03:41:04.011554+0000","last_active":"2024-03-12T03:42:47.820129+0000","last_peered":"2024-03-12T03:42:47.820129+0000","last_clean":"2024-03-12T03:42:47.820129+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:42:47.820129+0000","last_undegraded":"2024-03-12T03:42:47.820129+0000","last_fullsized":"2024-03-12T03:42:47.820129+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:04.011430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:41:04.011430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":425,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.818794+0000","last_change":"2024-03-12T03:39:34.824294+0000","last_active":"2024-03-12T03:42:47.818794+0000","last_peered":"2024-03-12T03:42:47.818794+0000","last_clean":"2024-03-12T03:42:47.818794+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:42:47.818794+0000","last_undegraded":"2024-03-12T03:42:47.818794+0000","last_fullsized":"2024-03-12T03:42:47.818794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:39:34.824210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.844210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":161,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.837440+0000","last_change":"2024-03-12T03:42:07.272333+0000","last_active":"2024-03-12T03:42:47.837440+0000","last_peered":"2024-03-12T03:42:47.837440+0000","last_clean":"2024-03-12T03:42:47.837440+0000","last_became_active":"2024-03-12T03:40:46.532405+0000","last_became_peered":"2024-03-12T03:40:46.532405+0000","last_unstale":"2024-03-12T03:42:47.837440+0000","last_undegraded":"2024-03-12T03:42:47.837440+0000","last_fullsized":"2024-03-12T03:42:47.837440+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:07.272232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:07.272232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:31.724663+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":575,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.824367+0000","last_change":"2024-03-12T03:41:03.399874+0000","last_active":"2024-03-12T03:42:47.824367+0000","last_peered":"2024-03-12T03:42:47.824367+0000","last_clean":"2024-03-12T03:42:47.824367+0000","last_became_active":"2024-03-12T03:40:46.524423+0000","last_became_peered":"2024-03-12T03:40:46.524423+0000","last_unstale":"2024-03-12T03:42:47.824367+0000","last_undegraded":"2024-03-12T03:42:47.824367+0000","last_fullsized":"2024-03-12T03:42:47.824367+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:41:03.399737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:41:03.399737+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661400+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"260'14556","reported_seq":14771,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:50.932227+0000","last_change":"2024-03-12T03:40:58.061897+0000","last_active":"2024-03-12T03:42:50.932227+0000","last_peered":"2024-03-12T03:42:50.932227+0000","last_clean":"2024-03-12T03:42:50.932227+0000","last_became_active":"2024-03-12T03:39:34.249390+0000","last_became_peered":"2024-03-12T03:39:34.249390+0000","last_unstale":"2024-03-12T03:42:50.932227+0000","last_undegraded":"2024-03-12T03:42:50.932227+0000","last_fullsized":"2024-03-12T03:42:50.932227+0000","mapping_epoch":198,"log_start":"238'10000","ondisk_log_start":"238'10000","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'7205","last_scrub_stamp":"2024-03-12T03:40:58.061634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:40:58.061634+0000","objects_scrubbed":7150,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.434767+0000","scrub_duration":1751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134479872,"num_objects":2052,"num_object_clones":0,"num_object_copies":6156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14556,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"95'23","reported_seq":466,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.816071+0000","last_change":"2024-03-12T03:41:18.286718+0000","last_active":"2024-03-12T03:42:47.816071+0000","last_peered":"2024-03-12T03:42:47.816071+0000","last_clean":"2024-03-12T03:42:47.816071+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:42:47.816071+0000","last_undegraded":"2024-03-12T03:42:47.816071+0000","last_fullsized":"2024-03-12T03:42:47.816071+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'23","last_scrub_stamp":"2024-03-12T03:41:18.286562+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:41:18.286562+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.435079+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":437,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.814768+0000","last_change":"2024-03-12T03:40:32.220387+0000","last_active":"2024-03-12T03:42:47.814768+0000","last_peered":"2024-03-12T03:42:47.814768+0000","last_clean":"2024-03-12T03:42:47.814768+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:42:47.814768+0000","last_undegraded":"2024-03-12T03:42:47.814768+0000","last_fullsized":"2024-03-12T03:42:47.814768+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:32.220274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:40:32.220274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.434694+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":417,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823719+0000","last_change":"2024-03-12T03:39:38.743222+0000","last_active":"2024-03-12T03:42:47.823719+0000","last_peered":"2024-03-12T03:42:47.823719+0000","last_clean":"2024-03-12T03:42:47.823719+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:42:47.823719+0000","last_undegraded":"2024-03-12T03:42:47.823719+0000","last_fullsized":"2024-03-12T03:42:47.823719+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.743128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.743128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":339,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.830217+0000","last_change":"2024-03-12T03:40:46.521186+0000","last_active":"2024-03-12T03:42:47.830217+0000","last_peered":"2024-03-12T03:42:47.830217+0000","last_clean":"2024-03-12T03:42:47.830217+0000","last_became_active":"2024-03-12T03:40:46.515751+0000","last_became_peered":"2024-03-12T03:40:46.515751+0000","last_unstale":"2024-03-12T03:42:47.830217+0000","last_undegraded":"2024-03-12T03:42:47.830217+0000","last_fullsized":"2024-03-12T03:42:47.830217+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:13.114417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.114417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"260'14768","reported_seq":14903,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:52.013684+0000","last_change":"2024-03-12T03:39:34.251603+0000","last_active":"2024-03-12T03:42:52.013684+0000","last_peered":"2024-03-12T03:42:52.013684+0000","last_clean":"2024-03-12T03:42:52.013684+0000","last_became_active":"2024-03-12T03:39:34.251359+0000","last_became_peered":"2024-03-12T03:39:34.251359+0000","last_unstale":"2024-03-12T03:42:52.013684+0000","last_undegraded":"2024-03-12T03:42:52.013684+0000","last_fullsized":"2024-03-12T03:42:52.013684+0000","mapping_epoch":198,"log_start":"238'10000","ondisk_log_start":"238'10000","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138280960,"num_objects":2110,"num_object_clones":0,"num_object_copies":6330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14768,"num_write_kb":540096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":558,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.822792+0000","last_change":"2024-03-12T03:41:04.383619+0000","last_active":"2024-03-12T03:42:47.822792+0000","last_peered":"2024-03-12T03:42:47.822792+0000","last_clean":"2024-03-12T03:42:47.822792+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:42:47.822792+0000","last_undegraded":"2024-03-12T03:42:47.822792+0000","last_fullsized":"2024-03-12T03:42:47.822792+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:41:04.383505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:41:04.383505+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":240,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.819608+0000","last_change":"2024-03-12T03:40:46.531652+0000","last_active":"2024-03-12T03:42:47.819608+0000","last_peered":"2024-03-12T03:42:47.819608+0000","last_clean":"2024-03-12T03:42:47.819608+0000","last_became_active":"2024-03-12T03:40:46.531312+0000","last_became_peered":"2024-03-12T03:40:46.531312+0000","last_unstale":"2024-03-12T03:42:47.819608+0000","last_undegraded":"2024-03-12T03:42:47.819608+0000","last_fullsized":"2024-03-12T03:42:47.819608+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:38:33.555893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:33.555893+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":329,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.831841+0000","last_change":"2024-03-12T03:40:46.528324+0000","last_active":"2024-03-12T03:42:47.831841+0000","last_peered":"2024-03-12T03:42:47.831841+0000","last_clean":"2024-03-12T03:42:47.831841+0000","last_became_active":"2024-03-12T03:40:46.528019+0000","last_became_peered":"2024-03-12T03:40:46.528019+0000","last_unstale":"2024-03-12T03:42:47.831841+0000","last_undegraded":"2024-03-12T03:42:47.831841+0000","last_fullsized":"2024-03-12T03:42:47.831841+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:13.030426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:13.030426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":547,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.816158+0000","last_change":"2024-03-12T03:42:04.241287+0000","last_active":"2024-03-12T03:42:47.816158+0000","last_peered":"2024-03-12T03:42:47.816158+0000","last_clean":"2024-03-12T03:42:47.816158+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:42:47.816158+0000","last_undegraded":"2024-03-12T03:42:47.816158+0000","last_fullsized":"2024-03-12T03:42:47.816158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:42:04.241174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:04.241174+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:21.104873+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":405,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823568+0000","last_change":"2024-03-12T03:39:18.785112+0000","last_active":"2024-03-12T03:42:47.823568+0000","last_peered":"2024-03-12T03:42:47.823568+0000","last_clean":"2024-03-12T03:42:47.823568+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:42:47.823568+0000","last_undegraded":"2024-03-12T03:42:47.823568+0000","last_fullsized":"2024-03-12T03:42:47.823568+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:18.785001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:18.785001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661128+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"260'14700","reported_seq":29186,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:51.550435+0000","last_change":"2024-03-12T03:42:29.138644+0000","last_active":"2024-03-12T03:42:51.550435+0000","last_peered":"2024-03-12T03:42:51.550435+0000","last_clean":"2024-03-12T03:42:51.550435+0000","last_became_active":"2024-03-12T03:41:54.317227+0000","last_became_peered":"2024-03-12T03:41:54.317227+0000","last_unstale":"2024-03-12T03:42:51.550435+0000","last_undegraded":"2024-03-12T03:42:51.550435+0000","last_fullsized":"2024-03-12T03:42:51.550435+0000","mapping_epoch":241,"log_start":"236'9200","ondisk_log_start":"236'9200","created":198,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'13236","last_scrub_stamp":"2024-03-12T03:42:29.138471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:42:29.138471+0000","objects_scrubbed":3466,"log_size":5500,"log_dups_size":0,"ondisk_log_size":5500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:58.712288+0000","scrub_duration":808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129236992,"num_objects":1972,"num_object_clones":0,"num_object_copies":5916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14700,"num_write_kb":533504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7147,"num_bytes_recovered":468385792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":426,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.814677+0000","last_change":"2024-03-12T03:41:01.229746+0000","last_active":"2024-03-12T03:42:47.814677+0000","last_peered":"2024-03-12T03:42:47.814677+0000","last_clean":"2024-03-12T03:42:47.814677+0000","last_became_active":"2024-03-12T03:40:46.517997+0000","last_became_peered":"2024-03-12T03:40:46.517997+0000","last_unstale":"2024-03-12T03:42:47.814677+0000","last_undegraded":"2024-03-12T03:42:47.814677+0000","last_fullsized":"2024-03-12T03:42:47.814677+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:01.229635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:01.229635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.434912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":404,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.831393+0000","last_change":"2024-03-12T03:40:46.532639+0000","last_active":"2024-03-12T03:42:47.831393+0000","last_peered":"2024-03-12T03:42:47.831393+0000","last_clean":"2024-03-12T03:42:47.831393+0000","last_became_active":"2024-03-12T03:40:46.529798+0000","last_became_peered":"2024-03-12T03:40:46.529798+0000","last_unstale":"2024-03-12T03:42:47.831393+0000","last_undegraded":"2024-03-12T03:42:47.831393+0000","last_fullsized":"2024-03-12T03:42:47.831393+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-03-12T03:36:06.083052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.083052+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":37,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":333,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.829887+0000","last_change":"2024-03-12T03:40:46.536598+0000","last_active":"2024-03-12T03:42:47.829887+0000","last_peered":"2024-03-12T03:42:47.829887+0000","last_clean":"2024-03-12T03:42:47.829887+0000","last_became_active":"2024-03-12T03:40:46.534086+0000","last_became_peered":"2024-03-12T03:40:46.534086+0000","last_unstale":"2024-03-12T03:42:47.829887+0000","last_undegraded":"2024-03-12T03:42:47.829887+0000","last_fullsized":"2024-03-12T03:42:47.829887+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:41.918538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:41.918538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":442,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.814860+0000","last_change":"2024-03-12T03:41:23.258063+0000","last_active":"2024-03-12T03:42:47.814860+0000","last_peered":"2024-03-12T03:42:47.814860+0000","last_clean":"2024-03-12T03:42:47.814860+0000","last_became_active":"2024-03-12T03:40:46.518478+0000","last_became_peered":"2024-03-12T03:40:46.518478+0000","last_unstale":"2024-03-12T03:42:47.814860+0000","last_undegraded":"2024-03-12T03:42:47.814860+0000","last_fullsized":"2024-03-12T03:42:47.814860+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:23.257942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:23.257942+0000","last_clean_scrub_stamp":"2024-03-12T03:41:23.257942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.435109+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":558,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.816078+0000","last_change":"2024-03-12T03:40:17.351410+0000","last_active":"2024-03-12T03:42:47.816078+0000","last_peered":"2024-03-12T03:42:47.816078+0000","last_clean":"2024-03-12T03:42:47.816078+0000","last_became_active":"2024-03-12T03:39:36.251411+0000","last_became_peered":"2024-03-12T03:39:36.251411+0000","last_unstale":"2024-03-12T03:42:47.816078+0000","last_undegraded":"2024-03-12T03:42:47.816078+0000","last_fullsized":"2024-03-12T03:42:47.816078+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:40:17.351282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:40:17.351282+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.434664+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"260'14761","reported_seq":14897,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:51.015470+0000","last_change":"2024-03-12T03:39:34.336488+0000","last_active":"2024-03-12T03:42:51.015470+0000","last_peered":"2024-03-12T03:42:51.015470+0000","last_clean":"2024-03-12T03:42:51.015470+0000","last_became_active":"2024-03-12T03:39:34.336130+0000","last_became_peered":"2024-03-12T03:39:34.336130+0000","last_unstale":"2024-03-12T03:42:51.015470+0000","last_undegraded":"2024-03-12T03:42:51.015470+0000","last_fullsized":"2024-03-12T03:42:51.015470+0000","mapping_epoch":198,"log_start":"236'9700","ondisk_log_start":"236'9700","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":5061,"log_dups_size":0,"ondisk_log_size":5061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:54.575207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143523864,"num_objects":2191,"num_object_clones":0,"num_object_copies":6573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14761,"num_write_kb":542401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"223'10","reported_seq":486,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.831188+0000","last_change":"2024-03-12T03:40:46.523417+0000","last_active":"2024-03-12T03:42:47.831188+0000","last_peered":"2024-03-12T03:42:47.831188+0000","last_clean":"2024-03-12T03:42:47.831188+0000","last_became_active":"2024-03-12T03:40:46.521005+0000","last_became_peered":"2024-03-12T03:40:46.521005+0000","last_unstale":"2024-03-12T03:42:47.831188+0000","last_undegraded":"2024-03-12T03:42:47.831188+0000","last_fullsized":"2024-03-12T03:42:47.831188+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:39:37.039925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:37.039925+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"200'28","reported_seq":470,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.814959+0000","last_change":"2024-03-12T03:42:02.272123+0000","last_active":"2024-03-12T03:42:47.814959+0000","last_peered":"2024-03-12T03:42:47.814959+0000","last_clean":"2024-03-12T03:42:47.814959+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:42:47.814959+0000","last_undegraded":"2024-03-12T03:42:47.814959+0000","last_fullsized":"2024-03-12T03:42:47.814959+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'28","last_scrub_stamp":"2024-03-12T03:42:02.271980+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:42:02.271980+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:25.698870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":426,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823511+0000","last_change":"2024-03-12T03:40:25.553659+0000","last_active":"2024-03-12T03:42:47.823511+0000","last_peered":"2024-03-12T03:42:47.823511+0000","last_clean":"2024-03-12T03:42:47.823511+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:42:47.823511+0000","last_undegraded":"2024-03-12T03:42:47.823511+0000","last_fullsized":"2024-03-12T03:42:47.823511+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:40:25.553541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:40:25.553541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.512140+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825651+0000","last_change":"2024-03-12T03:42:04.017533+0000","last_active":"2024-03-12T03:42:47.825651+0000","last_peered":"2024-03-12T03:42:47.825651+0000","last_clean":"2024-03-12T03:42:47.825651+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:42:47.825651+0000","last_undegraded":"2024-03-12T03:42:47.825651+0000","last_fullsized":"2024-03-12T03:42:47.825651+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:04.017409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:04.017409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:22.499259+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":503,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.829951+0000","last_change":"2024-03-12T03:40:46.523103+0000","last_active":"2024-03-12T03:42:47.829951+0000","last_peered":"2024-03-12T03:42:47.829951+0000","last_clean":"2024-03-12T03:42:47.829951+0000","last_became_active":"2024-03-12T03:40:46.515554+0000","last_became_peered":"2024-03-12T03:40:46.515554+0000","last_unstale":"2024-03-12T03:42:47.829951+0000","last_undegraded":"2024-03-12T03:42:47.829951+0000","last_fullsized":"2024-03-12T03:42:47.829951+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:38:15.587624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:38:15.587624+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.450774+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"260'14556","reported_seq":14691,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:51.516504+0000","last_change":"2024-03-12T03:39:34.337083+0000","last_active":"2024-03-12T03:42:51.516504+0000","last_peered":"2024-03-12T03:42:51.516504+0000","last_clean":"2024-03-12T03:42:51.516504+0000","last_became_active":"2024-03-12T03:39:34.336765+0000","last_became_peered":"2024-03-12T03:39:34.336765+0000","last_unstale":"2024-03-12T03:42:51.516504+0000","last_undegraded":"2024-03-12T03:42:51.516504+0000","last_fullsized":"2024-03-12T03:42:51.516504+0000","mapping_epoch":198,"log_start":"236'9100","ondisk_log_start":"236'9100","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":5456,"log_dups_size":0,"ondisk_log_size":5456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130547712,"num_objects":1992,"num_object_clones":0,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14556,"num_write_kb":529536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":539,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.821460+0000","last_change":"2024-03-12T03:39:23.514494+0000","last_active":"2024-03-12T03:42:47.821460+0000","last_peered":"2024-03-12T03:42:47.821460+0000","last_clean":"2024-03-12T03:42:47.821460+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:42:47.821460+0000","last_undegraded":"2024-03-12T03:42:47.821460+0000","last_fullsized":"2024-03-12T03:42:47.821460+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:39:23.514401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:23.514401+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.511917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":456,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.826412+0000","last_change":"2024-03-12T03:39:30.805994+0000","last_active":"2024-03-12T03:42:47.826412+0000","last_peered":"2024-03-12T03:42:47.826412+0000","last_clean":"2024-03-12T03:42:47.826412+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:42:47.826412+0000","last_undegraded":"2024-03-12T03:42:47.826412+0000","last_fullsized":"2024-03-12T03:42:47.826412+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:39:30.805899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:30.805899+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.844182+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":446,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823625+0000","last_change":"2024-03-12T03:40:46.520387+0000","last_active":"2024-03-12T03:42:47.823625+0000","last_peered":"2024-03-12T03:42:47.823625+0000","last_clean":"2024-03-12T03:42:47.823625+0000","last_became_active":"2024-03-12T03:40:46.520076+0000","last_became_peered":"2024-03-12T03:40:46.520076+0000","last_unstale":"2024-03-12T03:42:47.823625+0000","last_undegraded":"2024-03-12T03:42:47.823625+0000","last_fullsized":"2024-03-12T03:42:47.823625+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:34.674570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:39:34.674570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":434,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.822089+0000","last_change":"2024-03-12T03:40:46.529186+0000","last_active":"2024-03-12T03:42:47.822089+0000","last_peered":"2024-03-12T03:42:47.822089+0000","last_clean":"2024-03-12T03:42:47.822089+0000","last_became_active":"2024-03-12T03:40:46.528691+0000","last_became_peered":"2024-03-12T03:40:46.528691+0000","last_unstale":"2024-03-12T03:42:47.822089+0000","last_undegraded":"2024-03-12T03:42:47.822089+0000","last_fullsized":"2024-03-12T03:42:47.822089+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:40.532991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:40.532991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.511969+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":555,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.820394+0000","last_change":"2024-03-12T03:40:57.984669+0000","last_active":"2024-03-12T03:42:47.820394+0000","last_peered":"2024-03-12T03:42:47.820394+0000","last_clean":"2024-03-12T03:42:47.820394+0000","last_became_active":"2024-03-12T03:40:46.532266+0000","last_became_peered":"2024-03-12T03:40:46.532266+0000","last_unstale":"2024-03-12T03:42:47.820394+0000","last_undegraded":"2024-03-12T03:42:47.820394+0000","last_fullsized":"2024-03-12T03:42:47.820394+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:40:57.984538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:40:57.984538+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.f","version":"260'14418","reported_seq":28462,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:49.608744+0000","last_change":"2024-03-12T03:41:55.356589+0000","last_active":"2024-03-12T03:42:49.608744+0000","last_peered":"2024-03-12T03:42:49.608744+0000","last_clean":"2024-03-12T03:42:49.608744+0000","last_became_active":"2024-03-12T03:41:55.318802+0000","last_became_peered":"2024-03-12T03:41:55.318802+0000","last_unstale":"2024-03-12T03:42:49.608744+0000","last_undegraded":"2024-03-12T03:42:49.608744+0000","last_fullsized":"2024-03-12T03:42:49.608744+0000","mapping_epoch":242,"log_start":"237'9875","ondisk_log_start":"237'9875","created":198,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143917056,"num_objects":2196,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14418,"num_write_kb":531648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6957,"num_bytes_recovered":455933952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":432,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.820303+0000","last_change":"2024-03-12T03:42:02.263983+0000","last_active":"2024-03-12T03:42:47.820303+0000","last_peered":"2024-03-12T03:42:47.820303+0000","last_clean":"2024-03-12T03:42:47.820303+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:42:47.820303+0000","last_undegraded":"2024-03-12T03:42:47.820303+0000","last_fullsized":"2024-03-12T03:42:47.820303+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:02.263808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:42:02.263808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:22.666688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":469,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.821928+0000","last_change":"2024-03-12T03:39:36.550428+0000","last_active":"2024-03-12T03:42:47.821928+0000","last_peered":"2024-03-12T03:42:47.821928+0000","last_clean":"2024-03-12T03:42:47.821928+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:42:47.821928+0000","last_undegraded":"2024-03-12T03:42:47.821928+0000","last_fullsized":"2024-03-12T03:42:47.821928+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:39:36.550291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:36.550291+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.512028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":453,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.818669+0000","last_change":"2024-03-12T03:40:32.631452+0000","last_active":"2024-03-12T03:42:47.818669+0000","last_peered":"2024-03-12T03:42:47.818669+0000","last_clean":"2024-03-12T03:42:47.818669+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:42:47.818669+0000","last_undegraded":"2024-03-12T03:42:47.818669+0000","last_fullsized":"2024-03-12T03:42:47.818669+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:40:32.631336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.844447+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.818406+0000","last_change":"2024-03-12T03:39:39.850426+0000","last_active":"2024-03-12T03:42:47.818406+0000","last_peered":"2024-03-12T03:42:47.818406+0000","last_clean":"2024-03-12T03:42:47.818406+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:42:47.818406+0000","last_undegraded":"2024-03-12T03:42:47.818406+0000","last_fullsized":"2024-03-12T03:42:47.818406+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:39.850300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:39.850300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.844305+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":563,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.820498+0000","last_change":"2024-03-12T03:42:10.356344+0000","last_active":"2024-03-12T03:42:47.820498+0000","last_peered":"2024-03-12T03:42:47.820498+0000","last_clean":"2024-03-12T03:42:47.820498+0000","last_became_active":"2024-03-12T03:40:46.513921+0000","last_became_peered":"2024-03-12T03:40:46.513921+0000","last_unstale":"2024-03-12T03:42:47.820498+0000","last_undegraded":"2024-03-12T03:42:47.820498+0000","last_fullsized":"2024-03-12T03:42:47.820498+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:10.356260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:10.356260+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:13.781974+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"260'14651","reported_seq":14786,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:51.984322+0000","last_change":"2024-03-12T03:39:34.339192+0000","last_active":"2024-03-12T03:42:51.984322+0000","last_peered":"2024-03-12T03:42:51.984322+0000","last_clean":"2024-03-12T03:42:51.984322+0000","last_became_active":"2024-03-12T03:39:34.338678+0000","last_became_peered":"2024-03-12T03:39:34.338678+0000","last_unstale":"2024-03-12T03:42:51.984322+0000","last_undegraded":"2024-03-12T03:42:51.984322+0000","last_fullsized":"2024-03-12T03:42:51.984322+0000","mapping_epoch":198,"log_start":"237'9900","ondisk_log_start":"237'9900","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132972544,"num_objects":2029,"num_object_clones":0,"num_object_copies":6087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14651,"num_write_kb":533760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":417,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823797+0000","last_change":"2024-03-12T03:40:22.791790+0000","last_active":"2024-03-12T03:42:47.823797+0000","last_peered":"2024-03-12T03:42:47.823797+0000","last_clean":"2024-03-12T03:42:47.823797+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:42:47.823797+0000","last_undegraded":"2024-03-12T03:42:47.823797+0000","last_fullsized":"2024-03-12T03:42:47.823797+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:22.791656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:40:22.791656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661306+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":479,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823425+0000","last_change":"2024-03-12T03:39:35.570165+0000","last_active":"2024-03-12T03:42:47.823425+0000","last_peered":"2024-03-12T03:42:47.823425+0000","last_clean":"2024-03-12T03:42:47.823425+0000","last_became_active":"2024-03-12T03:39:33.277868+0000","last_became_peered":"2024-03-12T03:39:33.277868+0000","last_unstale":"2024-03-12T03:42:47.823425+0000","last_undegraded":"2024-03-12T03:42:47.823425+0000","last_fullsized":"2024-03-12T03:42:47.823425+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:39:35.570046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:35.570046+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.511825+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1008,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:48.959320+0000","last_change":"2024-03-12T03:41:59.023320+0000","last_active":"2024-03-12T03:42:48.959320+0000","last_peered":"2024-03-12T03:42:48.959320+0000","last_clean":"2024-03-12T03:42:48.959320+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:42:48.959320+0000","last_undegraded":"2024-03-12T03:42:48.959320+0000","last_fullsized":"2024-03-12T03:42:48.959320+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:41:59.023235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:41:59.023235+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:18.542939+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":563,"num_read_kb":558,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":434,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825093+0000","last_change":"2024-03-12T03:42:21.445884+0000","last_active":"2024-03-12T03:42:47.825093+0000","last_peered":"2024-03-12T03:42:47.825093+0000","last_clean":"2024-03-12T03:42:47.825093+0000","last_became_active":"2024-03-12T03:40:46.530227+0000","last_became_peered":"2024-03-12T03:40:46.530227+0000","last_unstale":"2024-03-12T03:42:47.825093+0000","last_undegraded":"2024-03-12T03:42:47.825093+0000","last_fullsized":"2024-03-12T03:42:47.825093+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:42:21.445443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:21.629219+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":430,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.817801+0000","last_change":"2024-03-12T03:40:46.524410+0000","last_active":"2024-03-12T03:42:47.817801+0000","last_peered":"2024-03-12T03:42:47.817801+0000","last_clean":"2024-03-12T03:42:47.817801+0000","last_became_active":"2024-03-12T03:40:46.524067+0000","last_became_peered":"2024-03-12T03:40:46.524067+0000","last_unstale":"2024-03-12T03:42:47.817801+0000","last_undegraded":"2024-03-12T03:42:47.817801+0000","last_fullsized":"2024-03-12T03:42:47.817801+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:31.807586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:31.807586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.841872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":541,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.819722+0000","last_change":"2024-03-12T03:39:24.675700+0000","last_active":"2024-03-12T03:42:47.819722+0000","last_peered":"2024-03-12T03:42:47.819722+0000","last_clean":"2024-03-12T03:42:47.819722+0000","last_became_active":"2024-03-12T03:38:08.891939+0000","last_became_peered":"2024-03-12T03:38:08.891939+0000","last_unstale":"2024-03-12T03:42:47.819722+0000","last_undegraded":"2024-03-12T03:42:47.819722+0000","last_fullsized":"2024-03-12T03:42:47.819722+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:39:24.675620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:39:24.675620+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.844085+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"260'14814","reported_seq":15031,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:51.158383+0000","last_change":"2024-03-12T03:41:01.317901+0000","last_active":"2024-03-12T03:42:51.158383+0000","last_peered":"2024-03-12T03:42:51.158383+0000","last_clean":"2024-03-12T03:42:51.158383+0000","last_became_active":"2024-03-12T03:39:34.249591+0000","last_became_peered":"2024-03-12T03:39:34.249591+0000","last_unstale":"2024-03-12T03:42:51.158383+0000","last_undegraded":"2024-03-12T03:42:51.158383+0000","last_fullsized":"2024-03-12T03:42:51.158383+0000","mapping_epoch":198,"log_start":"235'9100","ondisk_log_start":"235'9100","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'7543","last_scrub_stamp":"2024-03-12T03:41:01.317614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:41:01.317614+0000","objects_scrubbed":7464,"log_size":5714,"log_dups_size":0,"ondisk_log_size":5714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:54.523868+0000","scrub_duration":2316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137101312,"num_objects":2092,"num_object_clones":0,"num_object_copies":6276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14814,"num_write_kb":540992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":429,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825852+0000","last_change":"2024-03-12T03:41:08.176405+0000","last_active":"2024-03-12T03:42:47.825852+0000","last_peered":"2024-03-12T03:42:47.825852+0000","last_clean":"2024-03-12T03:42:47.825852+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:42:47.825852+0000","last_undegraded":"2024-03-12T03:42:47.825852+0000","last_fullsized":"2024-03-12T03:42:47.825852+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:08.176270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:08.176270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":564,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.820725+0000","last_change":"2024-03-12T03:40:59.996399+0000","last_active":"2024-03-12T03:42:47.820725+0000","last_peered":"2024-03-12T03:42:47.820725+0000","last_clean":"2024-03-12T03:42:47.820725+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:42:47.820725+0000","last_undegraded":"2024-03-12T03:42:47.820725+0000","last_fullsized":"2024-03-12T03:42:47.820725+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:40:59.996258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:59.996258+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":330,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.829933+0000","last_change":"2024-03-12T03:40:46.520753+0000","last_active":"2024-03-12T03:42:47.829933+0000","last_peered":"2024-03-12T03:42:47.829933+0000","last_clean":"2024-03-12T03:42:47.829933+0000","last_became_active":"2024-03-12T03:40:46.516086+0000","last_became_peered":"2024-03-12T03:40:46.516086+0000","last_unstale":"2024-03-12T03:42:47.829933+0000","last_undegraded":"2024-03-12T03:42:47.829933+0000","last_fullsized":"2024-03-12T03:42:47.829933+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:14.996357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:14.996357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":690,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825028+0000","last_change":"2024-03-12T03:41:00.109498+0000","last_active":"2024-03-12T03:42:47.825028+0000","last_peered":"2024-03-12T03:42:47.825028+0000","last_clean":"2024-03-12T03:42:47.825028+0000","last_became_active":"2024-03-12T03:38:03.852040+0000","last_became_peered":"2024-03-12T03:38:03.852040+0000","last_unstale":"2024-03-12T03:42:47.825028+0000","last_undegraded":"2024-03-12T03:42:47.825028+0000","last_fullsized":"2024-03-12T03:42:47.825028+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:41:00.109285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:41:00.109285+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":422,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823181+0000","last_change":"2024-03-12T03:41:00.436835+0000","last_active":"2024-03-12T03:42:47.823181+0000","last_peered":"2024-03-12T03:42:47.823181+0000","last_clean":"2024-03-12T03:42:47.823181+0000","last_became_active":"2024-03-12T03:40:46.529802+0000","last_became_peered":"2024-03-12T03:40:46.529802+0000","last_unstale":"2024-03-12T03:42:47.823181+0000","last_undegraded":"2024-03-12T03:42:47.823181+0000","last_fullsized":"2024-03-12T03:42:47.823181+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:00.436718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:00.436718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661384+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"260'14283","reported_seq":27997,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:49.620591+0000","last_change":"2024-03-12T03:41:59.404791+0000","last_active":"2024-03-12T03:42:49.620591+0000","last_peered":"2024-03-12T03:42:49.620591+0000","last_clean":"2024-03-12T03:42:49.620591+0000","last_became_active":"2024-03-12T03:41:59.378669+0000","last_became_peered":"2024-03-12T03:41:59.378669+0000","last_unstale":"2024-03-12T03:42:49.620591+0000","last_undegraded":"2024-03-12T03:42:49.620591+0000","last_fullsized":"2024-03-12T03:42:49.620591+0000","mapping_epoch":245,"log_start":"237'9711","ondisk_log_start":"237'9711","created":198,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4572,"log_dups_size":0,"ondisk_log_size":4572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143851520,"num_objects":2195,"num_object_clones":0,"num_object_copies":6585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14283,"num_write_kb":527296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6791,"num_bytes_recovered":445054976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":152,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.833876+0000","last_change":"2024-03-12T03:42:18.272709+0000","last_active":"2024-03-12T03:42:47.833876+0000","last_peered":"2024-03-12T03:42:47.833876+0000","last_clean":"2024-03-12T03:42:47.833876+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:42:47.833876+0000","last_undegraded":"2024-03-12T03:42:47.833876+0000","last_fullsized":"2024-03-12T03:42:47.833876+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:18.272591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:18.272591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:33.114854+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":468,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.819418+0000","last_change":"2024-03-12T03:40:18.873418+0000","last_active":"2024-03-12T03:42:47.819418+0000","last_peered":"2024-03-12T03:42:47.819418+0000","last_clean":"2024-03-12T03:42:47.819418+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:42:47.819418+0000","last_undegraded":"2024-03-12T03:42:47.819418+0000","last_fullsized":"2024-03-12T03:42:47.819418+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-12T03:40:18.873281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:18.873281+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.844370+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":153,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.841371+0000","last_change":"2024-03-12T03:42:01.190049+0000","last_active":"2024-03-12T03:42:47.841371+0000","last_peered":"2024-03-12T03:42:47.841371+0000","last_clean":"2024-03-12T03:42:47.841371+0000","last_became_active":"2024-03-12T03:40:46.517899+0000","last_became_peered":"2024-03-12T03:40:46.517899+0000","last_unstale":"2024-03-12T03:42:47.841371+0000","last_undegraded":"2024-03-12T03:42:47.841371+0000","last_fullsized":"2024-03-12T03:42:47.841371+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:01.189842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:42:01.189842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:29.372176+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":445,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.820058+0000","last_change":"2024-03-12T03:40:22.866720+0000","last_active":"2024-03-12T03:42:47.820058+0000","last_peered":"2024-03-12T03:42:47.820058+0000","last_clean":"2024-03-12T03:42:47.820058+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:42:47.820058+0000","last_undegraded":"2024-03-12T03:42:47.820058+0000","last_fullsized":"2024-03-12T03:42:47.820058+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:22.866591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:38:36.598179+0000","last_clean_scrub_stamp":"2024-03-12T03:40:22.866591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.844400+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":432,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.826903+0000","last_change":"2024-03-12T03:42:20.203864+0000","last_active":"2024-03-12T03:42:47.826903+0000","last_peered":"2024-03-12T03:42:47.826903+0000","last_clean":"2024-03-12T03:42:47.826903+0000","last_became_active":"2024-03-12T03:40:46.518250+0000","last_became_peered":"2024-03-12T03:40:46.518250+0000","last_unstale":"2024-03-12T03:42:47.826903+0000","last_undegraded":"2024-03-12T03:42:47.826903+0000","last_fullsized":"2024-03-12T03:42:47.826903+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:42:20.203740+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:42:20.203740+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:31.324197+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":706,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825466+0000","last_change":"2024-03-12T03:42:17.149648+0000","last_active":"2024-03-12T03:42:47.825466+0000","last_peered":"2024-03-12T03:42:47.825466+0000","last_clean":"2024-03-12T03:42:47.825466+0000","last_became_active":"2024-03-12T03:38:08.893327+0000","last_became_peered":"2024-03-12T03:38:08.893327+0000","last_unstale":"2024-03-12T03:42:47.825466+0000","last_undegraded":"2024-03-12T03:42:47.825466+0000","last_fullsized":"2024-03-12T03:42:47.825466+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:17.149524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:17.149524+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:32.117071+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"260'14584","reported_seq":24087,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:50.987530+0000","last_change":"2024-03-12T03:42:28.730631+0000","last_active":"2024-03-12T03:42:50.987530+0000","last_peered":"2024-03-12T03:42:50.987530+0000","last_clean":"2024-03-12T03:42:50.987530+0000","last_became_active":"2024-03-12T03:42:28.694184+0000","last_became_peered":"2024-03-12T03:42:28.694184+0000","last_unstale":"2024-03-12T03:42:50.987530+0000","last_undegraded":"2024-03-12T03:42:50.987530+0000","last_fullsized":"2024-03-12T03:42:50.987530+0000","mapping_epoch":252,"log_start":"236'9500","ondisk_log_start":"236'9500","created":198,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:54.575239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135266304,"num_objects":2064,"num_object_clones":0,"num_object_copies":6192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14584,"num_write_kb":532736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4675,"num_bytes_recovered":306380800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":417,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.821648+0000","last_change":"2024-03-12T03:39:16.577317+0000","last_active":"2024-03-12T03:42:47.821648+0000","last_peered":"2024-03-12T03:42:47.821648+0000","last_clean":"2024-03-12T03:42:47.821648+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:42:47.821648+0000","last_undegraded":"2024-03-12T03:42:47.821648+0000","last_fullsized":"2024-03-12T03:42:47.821648+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:16.577211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:16.577211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.511888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"223'39","reported_seq":596,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823311+0000","last_change":"2024-03-12T03:40:46.525110+0000","last_active":"2024-03-12T03:42:47.823311+0000","last_peered":"2024-03-12T03:42:47.823311+0000","last_clean":"2024-03-12T03:42:47.823311+0000","last_became_active":"2024-03-12T03:40:46.524632+0000","last_became_peered":"2024-03-12T03:40:46.524632+0000","last_unstale":"2024-03-12T03:42:47.823311+0000","last_undegraded":"2024-03-12T03:42:47.823311+0000","last_fullsized":"2024-03-12T03:42:47.823311+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'36","last_scrub_stamp":"2024-03-12T03:38:39.557592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:39.557592+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.815567+0000","last_change":"2024-03-12T03:41:26.257807+0000","last_active":"2024-03-12T03:42:47.815567+0000","last_peered":"2024-03-12T03:42:47.815567+0000","last_clean":"2024-03-12T03:42:47.815567+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:42:47.815567+0000","last_undegraded":"2024-03-12T03:42:47.815567+0000","last_fullsized":"2024-03-12T03:42:47.815567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:26.257679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:41:26.257679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.435137+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":436,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823054+0000","last_change":"2024-03-12T03:40:16.791859+0000","last_active":"2024-03-12T03:42:47.823054+0000","last_peered":"2024-03-12T03:42:47.823054+0000","last_clean":"2024-03-12T03:42:47.823054+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:42:47.823054+0000","last_undegraded":"2024-03-12T03:42:47.823054+0000","last_fullsized":"2024-03-12T03:42:47.823054+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:40:16.791364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:40:16.791364+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661292+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":372,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.831457+0000","last_change":"2024-03-12T03:40:46.520723+0000","last_active":"2024-03-12T03:42:47.831457+0000","last_peered":"2024-03-12T03:42:47.831457+0000","last_clean":"2024-03-12T03:42:47.831457+0000","last_became_active":"2024-03-12T03:40:46.517562+0000","last_became_peered":"2024-03-12T03:40:46.517562+0000","last_unstale":"2024-03-12T03:42:47.831457+0000","last_undegraded":"2024-03-12T03:42:47.831457+0000","last_fullsized":"2024-03-12T03:42:47.831457+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:39.031166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:39.031166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.496029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":558,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.822734+0000","last_change":"2024-03-12T03:42:28.720781+0000","last_active":"2024-03-12T03:42:47.822734+0000","last_peered":"2024-03-12T03:42:47.822734+0000","last_clean":"2024-03-12T03:42:47.822734+0000","last_became_active":"2024-03-12T03:39:36.252545+0000","last_became_peered":"2024-03-12T03:39:36.252545+0000","last_unstale":"2024-03-12T03:42:47.822734+0000","last_undegraded":"2024-03-12T03:42:47.822734+0000","last_fullsized":"2024-03-12T03:42:47.822734+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:28.720248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:28.720248+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:50.426883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"260'14667","reported_seq":23868,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:52.001021+0000","last_change":"2024-03-12T03:42:30.739974+0000","last_active":"2024-03-12T03:42:52.001021+0000","last_peered":"2024-03-12T03:42:52.001021+0000","last_clean":"2024-03-12T03:42:52.001021+0000","last_became_active":"2024-03-12T03:42:30.713416+0000","last_became_peered":"2024-03-12T03:42:30.713416+0000","last_unstale":"2024-03-12T03:42:52.001021+0000","last_undegraded":"2024-03-12T03:42:52.001021+0000","last_fullsized":"2024-03-12T03:42:52.001021+0000","mapping_epoch":254,"log_start":"238'9900","ondisk_log_start":"238'9900","created":198,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4767,"log_dups_size":0,"ondisk_log_size":4767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130875392,"num_objects":1997,"num_object_clones":0,"num_object_copies":5991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14667,"num_write_kb":533248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4526,"num_bytes_recovered":296615936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"111'20","reported_seq":454,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.821626+0000","last_change":"2024-03-12T03:39:39.540698+0000","last_active":"2024-03-12T03:42:47.821626+0000","last_peered":"2024-03-12T03:42:47.821626+0000","last_clean":"2024-03-12T03:42:47.821626+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:42:47.821626+0000","last_undegraded":"2024-03-12T03:42:47.821626+0000","last_fullsized":"2024-03-12T03:42:47.821626+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'20","last_scrub_stamp":"2024-03-12T03:39:39.540585+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:39:39.540585+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.512086+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":434,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823350+0000","last_change":"2024-03-12T03:40:59.344158+0000","last_active":"2024-03-12T03:42:47.823350+0000","last_peered":"2024-03-12T03:42:47.823350+0000","last_clean":"2024-03-12T03:42:47.823350+0000","last_became_active":"2024-03-12T03:40:46.524544+0000","last_became_peered":"2024-03-12T03:40:46.524544+0000","last_unstale":"2024-03-12T03:42:47.823350+0000","last_undegraded":"2024-03-12T03:42:47.823350+0000","last_fullsized":"2024-03-12T03:42:47.823350+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:59.344058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:40:59.344058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661415+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":153,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.838479+0000","last_change":"2024-03-12T03:42:03.191515+0000","last_active":"2024-03-12T03:42:47.838479+0000","last_peered":"2024-03-12T03:42:47.838479+0000","last_clean":"2024-03-12T03:42:47.838479+0000","last_became_active":"2024-03-12T03:40:46.521976+0000","last_became_peered":"2024-03-12T03:40:46.521976+0000","last_unstale":"2024-03-12T03:42:47.838479+0000","last_undegraded":"2024-03-12T03:42:47.838479+0000","last_fullsized":"2024-03-12T03:42:47.838479+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:03.191398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:03.191398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:21.532703+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":425,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825296+0000","last_change":"2024-03-12T03:40:19.489022+0000","last_active":"2024-03-12T03:42:47.825296+0000","last_peered":"2024-03-12T03:42:47.825296+0000","last_clean":"2024-03-12T03:42:47.825296+0000","last_became_active":"2024-03-12T03:39:33.271016+0000","last_became_peered":"2024-03-12T03:39:33.271016+0000","last_unstale":"2024-03-12T03:42:47.825296+0000","last_undegraded":"2024-03-12T03:42:47.825296+0000","last_fullsized":"2024-03-12T03:42:47.825296+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:19.488908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:40:19.488908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.512111+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"260'14661","reported_seq":14796,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:51.550168+0000","last_change":"2024-03-12T03:39:34.250643+0000","last_active":"2024-03-12T03:42:51.550168+0000","last_peered":"2024-03-12T03:42:51.550168+0000","last_clean":"2024-03-12T03:42:51.550168+0000","last_became_active":"2024-03-12T03:39:34.250450+0000","last_became_peered":"2024-03-12T03:39:34.250450+0000","last_unstale":"2024-03-12T03:42:51.550168+0000","last_undegraded":"2024-03-12T03:42:51.550168+0000","last_fullsized":"2024-03-12T03:42:51.550168+0000","mapping_epoch":198,"log_start":"236'9200","ondisk_log_start":"236'9200","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":5461,"log_dups_size":0,"ondisk_log_size":5461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131792896,"num_objects":2011,"num_object_clones":0,"num_object_copies":6033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14661,"num_write_kb":533504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"195'35","reported_seq":489,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.820288+0000","last_change":"2024-03-12T03:42:15.396790+0000","last_active":"2024-03-12T03:42:47.820288+0000","last_peered":"2024-03-12T03:42:47.820288+0000","last_clean":"2024-03-12T03:42:47.820288+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:42:47.820288+0000","last_undegraded":"2024-03-12T03:42:47.820288+0000","last_fullsized":"2024-03-12T03:42:47.820288+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'35","last_scrub_stamp":"2024-03-12T03:42:15.396647+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:42:15.396647+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:31.094343+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.826850+0000","last_change":"2024-03-12T03:42:15.159053+0000","last_active":"2024-03-12T03:42:47.826850+0000","last_peered":"2024-03-12T03:42:47.826850+0000","last_clean":"2024-03-12T03:42:47.826850+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:42:47.826850+0000","last_undegraded":"2024-03-12T03:42:47.826850+0000","last_fullsized":"2024-03-12T03:42:47.826850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:15.158916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:15.158916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:41.586588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":144,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.834338+0000","last_change":"2024-03-12T03:42:11.155750+0000","last_active":"2024-03-12T03:42:47.834338+0000","last_peered":"2024-03-12T03:42:47.834338+0000","last_clean":"2024-03-12T03:42:47.834338+0000","last_became_active":"2024-03-12T03:39:33.254395+0000","last_became_peered":"2024-03-12T03:39:33.254395+0000","last_unstale":"2024-03-12T03:42:47.834338+0000","last_undegraded":"2024-03-12T03:42:47.834338+0000","last_fullsized":"2024-03-12T03:42:47.834338+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:11.155639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:11.155639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:38.130653+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":434,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.824894+0000","last_change":"2024-03-12T03:41:23.042150+0000","last_active":"2024-03-12T03:42:47.824894+0000","last_peered":"2024-03-12T03:42:47.824894+0000","last_clean":"2024-03-12T03:42:47.824894+0000","last_became_active":"2024-03-12T03:40:46.521431+0000","last_became_peered":"2024-03-12T03:40:46.521431+0000","last_unstale":"2024-03-12T03:42:47.824894+0000","last_undegraded":"2024-03-12T03:42:47.824894+0000","last_fullsized":"2024-03-12T03:42:47.824894+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:23.042030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:23.042030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"260'18230","reported_seq":18608,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:51.128372+0000","last_change":"2024-03-12T03:42:39.697501+0000","last_active":"2024-03-12T03:42:51.128372+0000","last_peered":"2024-03-12T03:42:51.128372+0000","last_clean":"2024-03-12T03:42:51.128372+0000","last_became_active":"2024-03-12T03:39:34.250817+0000","last_became_peered":"2024-03-12T03:39:34.250817+0000","last_unstale":"2024-03-12T03:42:51.128372+0000","last_undegraded":"2024-03-12T03:42:51.128372+0000","last_fullsized":"2024-03-12T03:42:51.128372+0000","mapping_epoch":198,"log_start":"249'12500","ondisk_log_start":"249'12500","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'17360","last_scrub_stamp":"2024-03-12T03:42:39.697349+0000","last_deep_scrub":"257'17360","last_deep_scrub_stamp":"2024-03-12T03:42:39.697349+0000","last_clean_scrub_stamp":"2024-03-12T03:42:39.697349+0000","objects_scrubbed":3503,"log_size":5730,"log_dups_size":0,"ondisk_log_size":5730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":18,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:55.604835+0000","scrub_duration":17365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137560064,"num_objects":2099,"num_object_clones":0,"num_object_copies":6297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14727,"num_write_kb":538432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":152,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.833880+0000","last_change":"2024-03-12T03:42:24.169808+0000","last_active":"2024-03-12T03:42:47.833880+0000","last_peered":"2024-03-12T03:42:47.833880+0000","last_clean":"2024-03-12T03:42:47.833880+0000","last_became_active":"2024-03-12T03:39:33.278326+0000","last_became_peered":"2024-03-12T03:39:33.278326+0000","last_unstale":"2024-03-12T03:42:47.833880+0000","last_undegraded":"2024-03-12T03:42:47.833880+0000","last_fullsized":"2024-03-12T03:42:47.833880+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:24.169673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:24.169673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:50.702592+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":477,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.815117+0000","last_change":"2024-03-12T03:41:15.312988+0000","last_active":"2024-03-12T03:42:47.815117+0000","last_peered":"2024-03-12T03:42:47.815117+0000","last_clean":"2024-03-12T03:42:47.815117+0000","last_became_active":"2024-03-12T03:40:46.518005+0000","last_became_peered":"2024-03-12T03:40:46.518005+0000","last_unstale":"2024-03-12T03:42:47.815117+0000","last_undegraded":"2024-03-12T03:42:47.815117+0000","last_fullsized":"2024-03-12T03:42:47.815117+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-03-12T03:41:15.312809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:41:15.312809+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.434847+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.826211+0000","last_change":"2024-03-12T03:42:09.072702+0000","last_active":"2024-03-12T03:42:47.826211+0000","last_peered":"2024-03-12T03:42:47.826211+0000","last_clean":"2024-03-12T03:42:47.826211+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:42:47.826211+0000","last_undegraded":"2024-03-12T03:42:47.826211+0000","last_fullsized":"2024-03-12T03:42:47.826211+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:09.072495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:42:09.072495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:15.928821+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":287,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.831327+0000","last_change":"2024-03-12T03:40:46.528204+0000","last_active":"2024-03-12T03:42:47.831327+0000","last_peered":"2024-03-12T03:42:47.831327+0000","last_clean":"2024-03-12T03:42:47.831327+0000","last_became_active":"2024-03-12T03:40:46.525594+0000","last_became_peered":"2024-03-12T03:40:46.525594+0000","last_unstale":"2024-03-12T03:42:47.831327+0000","last_undegraded":"2024-03-12T03:42:47.831327+0000","last_fullsized":"2024-03-12T03:42:47.831327+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:18.651418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:18.651418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"260'14640","reported_seq":14775,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:50.988429+0000","last_change":"2024-03-12T03:39:34.336572+0000","last_active":"2024-03-12T03:42:50.988429+0000","last_peered":"2024-03-12T03:42:50.988429+0000","last_clean":"2024-03-12T03:42:50.988429+0000","last_became_active":"2024-03-12T03:39:34.336260+0000","last_became_peered":"2024-03-12T03:39:34.336260+0000","last_unstale":"2024-03-12T03:42:50.988429+0000","last_undegraded":"2024-03-12T03:42:50.988429+0000","last_fullsized":"2024-03-12T03:42:50.988429+0000","mapping_epoch":198,"log_start":"236'9600","ondisk_log_start":"236'9600","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:54.575162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133824512,"num_objects":2042,"num_object_clones":0,"num_object_copies":6126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14640,"num_write_kb":533824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":539,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825337+0000","last_change":"2024-03-12T03:41:17.027767+0000","last_active":"2024-03-12T03:42:47.825337+0000","last_peered":"2024-03-12T03:42:47.825337+0000","last_clean":"2024-03-12T03:42:47.825337+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:42:47.825337+0000","last_undegraded":"2024-03-12T03:42:47.825337+0000","last_fullsized":"2024-03-12T03:42:47.825337+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:41:17.027646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:17.027646+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"90'36","reported_seq":479,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.824243+0000","last_change":"2024-03-12T03:40:14.776197+0000","last_active":"2024-03-12T03:42:47.824243+0000","last_peered":"2024-03-12T03:42:47.824243+0000","last_clean":"2024-03-12T03:42:47.824243+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:42:47.824243+0000","last_undegraded":"2024-03-12T03:42:47.824243+0000","last_fullsized":"2024-03-12T03:42:47.824243+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:40:14.776063+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:40:14.776063+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661242+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":437,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825637+0000","last_change":"2024-03-12T03:41:20.054595+0000","last_active":"2024-03-12T03:42:47.825637+0000","last_peered":"2024-03-12T03:42:47.825637+0000","last_clean":"2024-03-12T03:42:47.825637+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:42:47.825637+0000","last_undegraded":"2024-03-12T03:42:47.825637+0000","last_fullsized":"2024-03-12T03:42:47.825637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:41:20.054461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:41:20.054461+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":413,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.822958+0000","last_change":"2024-03-12T03:40:28.767260+0000","last_active":"2024-03-12T03:42:47.822958+0000","last_peered":"2024-03-12T03:42:47.822958+0000","last_clean":"2024-03-12T03:42:47.822958+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:42:47.822958+0000","last_undegraded":"2024-03-12T03:42:47.822958+0000","last_fullsized":"2024-03-12T03:42:47.822958+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:28.767140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:40:28.767140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661322+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":344,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.829996+0000","last_change":"2024-03-12T03:40:46.535232+0000","last_active":"2024-03-12T03:42:47.829996+0000","last_peered":"2024-03-12T03:42:47.829996+0000","last_clean":"2024-03-12T03:42:47.829996+0000","last_became_active":"2024-03-12T03:40:46.528004+0000","last_became_peered":"2024-03-12T03:40:46.528004+0000","last_unstale":"2024-03-12T03:42:47.829996+0000","last_undegraded":"2024-03-12T03:42:47.829996+0000","last_fullsized":"2024-03-12T03:42:47.829996+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:35:00.079152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.079152+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":440,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.819853+0000","last_change":"2024-03-12T03:42:14.426265+0000","last_active":"2024-03-12T03:42:47.819853+0000","last_peered":"2024-03-12T03:42:47.819853+0000","last_clean":"2024-03-12T03:42:47.819853+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:42:47.819853+0000","last_undegraded":"2024-03-12T03:42:47.819853+0000","last_fullsized":"2024-03-12T03:42:47.819853+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:14.426145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:14.426145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:44.257789+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":425,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.822938+0000","last_change":"2024-03-12T03:41:01.417925+0000","last_active":"2024-03-12T03:42:47.822938+0000","last_peered":"2024-03-12T03:42:47.822938+0000","last_clean":"2024-03-12T03:42:47.822938+0000","last_became_active":"2024-03-12T03:40:46.516086+0000","last_became_peered":"2024-03-12T03:40:46.516086+0000","last_unstale":"2024-03-12T03:42:47.822938+0000","last_undegraded":"2024-03-12T03:42:47.822938+0000","last_fullsized":"2024-03-12T03:42:47.822938+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:01.417809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:01.417809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":420,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825590+0000","last_change":"2024-03-12T03:42:02.044839+0000","last_active":"2024-03-12T03:42:47.825590+0000","last_peered":"2024-03-12T03:42:47.825590+0000","last_clean":"2024-03-12T03:42:47.825590+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:42:47.825590+0000","last_undegraded":"2024-03-12T03:42:47.825590+0000","last_fullsized":"2024-03-12T03:42:47.825590+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:02.044703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:02.044703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:14.327829+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":461,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823707+0000","last_change":"2024-03-12T03:40:46.519477+0000","last_active":"2024-03-12T03:42:47.823707+0000","last_peered":"2024-03-12T03:42:47.823707+0000","last_clean":"2024-03-12T03:42:47.823707+0000","last_became_active":"2024-03-12T03:40:46.518652+0000","last_became_peered":"2024-03-12T03:40:46.518652+0000","last_unstale":"2024-03-12T03:42:47.823707+0000","last_undegraded":"2024-03-12T03:42:47.823707+0000","last_fullsized":"2024-03-12T03:42:47.823707+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:39:14.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:14.564162+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.500238+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825153+0000","last_change":"2024-03-12T03:42:22.126915+0000","last_active":"2024-03-12T03:42:47.825153+0000","last_peered":"2024-03-12T03:42:47.825153+0000","last_clean":"2024-03-12T03:42:47.825153+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:42:47.825153+0000","last_undegraded":"2024-03-12T03:42:47.825153+0000","last_fullsized":"2024-03-12T03:42:47.825153+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:22.126792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:22.126792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:51.530630+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":319,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.830701+0000","last_change":"2024-03-12T03:40:46.535824+0000","last_active":"2024-03-12T03:42:47.830701+0000","last_peered":"2024-03-12T03:42:47.830701+0000","last_clean":"2024-03-12T03:42:47.830701+0000","last_became_active":"2024-03-12T03:40:46.533278+0000","last_became_peered":"2024-03-12T03:40:46.533278+0000","last_unstale":"2024-03-12T03:42:47.830701+0000","last_undegraded":"2024-03-12T03:42:47.830701+0000","last_fullsized":"2024-03-12T03:42:47.830701+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:35:04.007252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":144,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.834509+0000","last_change":"2024-03-12T03:40:19.915322+0000","last_active":"2024-03-12T03:42:47.834509+0000","last_peered":"2024-03-12T03:42:47.834509+0000","last_clean":"2024-03-12T03:42:47.834509+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:42:47.834509+0000","last_undegraded":"2024-03-12T03:42:47.834509+0000","last_fullsized":"2024-03-12T03:42:47.834509+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:19.915088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:40:19.915088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.353143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":465,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823766+0000","last_change":"2024-03-12T03:40:58.312735+0000","last_active":"2024-03-12T03:42:47.823766+0000","last_peered":"2024-03-12T03:42:47.823766+0000","last_clean":"2024-03-12T03:42:47.823766+0000","last_became_active":"2024-03-12T03:40:46.520200+0000","last_became_peered":"2024-03-12T03:40:46.520200+0000","last_unstale":"2024-03-12T03:42:47.823766+0000","last_undegraded":"2024-03-12T03:42:47.823766+0000","last_fullsized":"2024-03-12T03:42:47.823766+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:40:58.312602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:58.312602+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661369+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":434,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.822146+0000","last_change":"2024-03-12T03:42:00.266626+0000","last_active":"2024-03-12T03:42:47.822146+0000","last_peered":"2024-03-12T03:42:47.822146+0000","last_clean":"2024-03-12T03:42:47.822146+0000","last_became_active":"2024-03-12T03:40:46.528869+0000","last_became_peered":"2024-03-12T03:40:46.528869+0000","last_unstale":"2024-03-12T03:42:47.822146+0000","last_undegraded":"2024-03-12T03:42:47.822146+0000","last_fullsized":"2024-03-12T03:42:47.822146+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:00.266512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:42:00.266512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:27.852547+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"223'12","reported_seq":500,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.830530+0000","last_change":"2024-03-12T03:40:46.525685+0000","last_active":"2024-03-12T03:42:47.830530+0000","last_peered":"2024-03-12T03:42:47.830530+0000","last_clean":"2024-03-12T03:42:47.830530+0000","last_became_active":"2024-03-12T03:40:46.522898+0000","last_became_peered":"2024-03-12T03:40:46.522898+0000","last_unstale":"2024-03-12T03:42:47.830530+0000","last_undegraded":"2024-03-12T03:42:47.830530+0000","last_fullsized":"2024-03-12T03:42:47.830530+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:35:08.006375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:08.006375+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":432,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.815944+0000","last_change":"2024-03-12T03:41:19.377731+0000","last_active":"2024-03-12T03:42:47.815944+0000","last_peered":"2024-03-12T03:42:47.815944+0000","last_clean":"2024-03-12T03:42:47.815944+0000","last_became_active":"2024-03-12T03:39:33.265499+0000","last_became_peered":"2024-03-12T03:39:33.265499+0000","last_unstale":"2024-03-12T03:42:47.815944+0000","last_undegraded":"2024-03-12T03:42:47.815944+0000","last_fullsized":"2024-03-12T03:42:47.815944+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:19.377599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:19.377599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.435165+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":384,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.830113+0000","last_change":"2024-03-12T03:40:46.522496+0000","last_active":"2024-03-12T03:42:47.830113+0000","last_peered":"2024-03-12T03:42:47.830113+0000","last_clean":"2024-03-12T03:42:47.830113+0000","last_became_active":"2024-03-12T03:40:46.517123+0000","last_became_peered":"2024-03-12T03:40:46.517123+0000","last_unstale":"2024-03-12T03:42:47.830113+0000","last_undegraded":"2024-03-12T03:42:47.830113+0000","last_fullsized":"2024-03-12T03:42:47.830113+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-03-12T03:35:58.079645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.079645+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":459,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825739+0000","last_change":"2024-03-12T03:42:12.114036+0000","last_active":"2024-03-12T03:42:47.825739+0000","last_peered":"2024-03-12T03:42:47.825739+0000","last_clean":"2024-03-12T03:42:47.825739+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:42:47.825739+0000","last_undegraded":"2024-03-12T03:42:47.825739+0000","last_fullsized":"2024-03-12T03:42:47.825739+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:42:12.113938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:12.113938+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:21.487403+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":538,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823696+0000","last_change":"2024-03-12T03:40:15.773578+0000","last_active":"2024-03-12T03:42:47.823696+0000","last_peered":"2024-03-12T03:42:47.823696+0000","last_clean":"2024-03-12T03:42:47.823696+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:42:47.823696+0000","last_undegraded":"2024-03-12T03:42:47.823696+0000","last_fullsized":"2024-03-12T03:42:47.823696+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:40:15.773432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:40:15.773432+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661260+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":421,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823338+0000","last_change":"2024-03-12T03:42:10.355334+0000","last_active":"2024-03-12T03:42:47.823338+0000","last_peered":"2024-03-12T03:42:47.823338+0000","last_clean":"2024-03-12T03:42:47.823338+0000","last_became_active":"2024-03-12T03:40:46.519320+0000","last_became_peered":"2024-03-12T03:40:46.519320+0000","last_unstale":"2024-03-12T03:42:47.823338+0000","last_undegraded":"2024-03-12T03:42:47.823338+0000","last_fullsized":"2024-03-12T03:42:47.823338+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:10.355198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:10.355198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:31.051878+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":448,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.815976+0000","last_change":"2024-03-12T03:41:21.258124+0000","last_active":"2024-03-12T03:42:47.815976+0000","last_peered":"2024-03-12T03:42:47.815976+0000","last_clean":"2024-03-12T03:42:47.815976+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:42:47.815976+0000","last_undegraded":"2024-03-12T03:42:47.815976+0000","last_fullsized":"2024-03-12T03:42:47.815976+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:41:21.257991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:41:21.257991+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.435015+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.815716+0000","last_change":"2024-03-12T03:41:13.286000+0000","last_active":"2024-03-12T03:42:47.815716+0000","last_peered":"2024-03-12T03:42:47.815716+0000","last_clean":"2024-03-12T03:42:47.815716+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:42:47.815716+0000","last_undegraded":"2024-03-12T03:42:47.815716+0000","last_fullsized":"2024-03-12T03:42:47.815716+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:13.285572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:38:34.264717+0000","last_clean_scrub_stamp":"2024-03-12T03:41:13.285572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.435045+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":434,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.826358+0000","last_change":"2024-03-12T03:40:46.529405+0000","last_active":"2024-03-12T03:42:47.826358+0000","last_peered":"2024-03-12T03:42:47.826358+0000","last_clean":"2024-03-12T03:42:47.826358+0000","last_became_active":"2024-03-12T03:40:46.529091+0000","last_became_peered":"2024-03-12T03:40:46.529091+0000","last_unstale":"2024-03-12T03:42:47.826358+0000","last_undegraded":"2024-03-12T03:42:47.826358+0000","last_fullsized":"2024-03-12T03:42:47.826358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:35.850271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:35.850271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":337,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.829527+0000","last_change":"2024-03-12T03:40:46.533090+0000","last_active":"2024-03-12T03:42:47.829527+0000","last_peered":"2024-03-12T03:42:47.829527+0000","last_clean":"2024-03-12T03:42:47.829527+0000","last_became_active":"2024-03-12T03:40:46.532335+0000","last_became_peered":"2024-03-12T03:40:46.532335+0000","last_unstale":"2024-03-12T03:42:47.829527+0000","last_undegraded":"2024-03-12T03:42:47.829527+0000","last_fullsized":"2024-03-12T03:42:47.829527+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.075897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.075897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":477,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.826510+0000","last_change":"2024-03-12T03:40:17.848428+0000","last_active":"2024-03-12T03:42:47.826510+0000","last_peered":"2024-03-12T03:42:47.826510+0000","last_clean":"2024-03-12T03:42:47.826510+0000","last_became_active":"2024-03-12T03:39:33.296245+0000","last_became_peered":"2024-03-12T03:39:33.296245+0000","last_unstale":"2024-03-12T03:42:47.826510+0000","last_undegraded":"2024-03-12T03:42:47.826510+0000","last_fullsized":"2024-03-12T03:42:47.826510+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-12T03:40:17.848303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:17.848303+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.844339+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.826758+0000","last_change":"2024-03-12T03:41:18.030224+0000","last_active":"2024-03-12T03:42:47.826758+0000","last_peered":"2024-03-12T03:42:47.826758+0000","last_clean":"2024-03-12T03:42:47.826758+0000","last_became_active":"2024-03-12T03:40:46.518227+0000","last_became_peered":"2024-03-12T03:40:46.518227+0000","last_unstale":"2024-03-12T03:42:47.826758+0000","last_undegraded":"2024-03-12T03:42:47.826758+0000","last_fullsized":"2024-03-12T03:42:47.826758+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:41:18.030146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":349,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.831500+0000","last_change":"2024-03-12T03:40:46.528355+0000","last_active":"2024-03-12T03:42:47.831500+0000","last_peered":"2024-03-12T03:42:47.831500+0000","last_clean":"2024-03-12T03:42:47.831500+0000","last_became_active":"2024-03-12T03:40:46.525702+0000","last_became_peered":"2024-03-12T03:40:46.525702+0000","last_unstale":"2024-03-12T03:42:47.831500+0000","last_undegraded":"2024-03-12T03:42:47.831500+0000","last_fullsized":"2024-03-12T03:42:47.831500+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:15.808887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:15.808887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":413,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.821991+0000","last_change":"2024-03-12T03:39:38.537429+0000","last_active":"2024-03-12T03:42:47.821991+0000","last_peered":"2024-03-12T03:42:47.821991+0000","last_clean":"2024-03-12T03:42:47.821991+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:42:47.821991+0000","last_undegraded":"2024-03-12T03:42:47.821991+0000","last_fullsized":"2024-03-12T03:42:47.821991+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.537304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.500271+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"74'18","reported_seq":355,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.829715+0000","last_change":"2024-03-12T03:40:46.531983+0000","last_active":"2024-03-12T03:42:47.829715+0000","last_peered":"2024-03-12T03:42:47.829715+0000","last_clean":"2024-03-12T03:42:47.829715+0000","last_became_active":"2024-03-12T03:40:46.528402+0000","last_became_peered":"2024-03-12T03:40:46.528402+0000","last_unstale":"2024-03-12T03:42:47.829715+0000","last_undegraded":"2024-03-12T03:42:47.829715+0000","last_fullsized":"2024-03-12T03:42:47.829715+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:35:02.079224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:02.079224+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":424,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823426+0000","last_change":"2024-03-12T03:39:33.687282+0000","last_active":"2024-03-12T03:42:47.823426+0000","last_peered":"2024-03-12T03:42:47.823426+0000","last_clean":"2024-03-12T03:42:47.823426+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:42:47.823426+0000","last_undegraded":"2024-03-12T03:42:47.823426+0000","last_fullsized":"2024-03-12T03:42:47.823426+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.687198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":417,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825319+0000","last_change":"2024-03-12T03:41:16.087106+0000","last_active":"2024-03-12T03:42:47.825319+0000","last_peered":"2024-03-12T03:42:47.825319+0000","last_clean":"2024-03-12T03:42:47.825319+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:42:47.825319+0000","last_undegraded":"2024-03-12T03:42:47.825319+0000","last_fullsized":"2024-03-12T03:42:47.825319+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:16.086881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:16.086881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":372,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.831651+0000","last_change":"2024-03-12T03:40:46.529912+0000","last_active":"2024-03-12T03:42:47.831651+0000","last_peered":"2024-03-12T03:42:47.831651+0000","last_clean":"2024-03-12T03:42:47.831651+0000","last_became_active":"2024-03-12T03:40:46.525946+0000","last_became_peered":"2024-03-12T03:40:46.525946+0000","last_unstale":"2024-03-12T03:42:47.831651+0000","last_undegraded":"2024-03-12T03:42:47.831651+0000","last_fullsized":"2024-03-12T03:42:47.831651+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"90'6","reported_seq":333,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.830288+0000","last_change":"2024-03-12T03:40:46.521647+0000","last_active":"2024-03-12T03:42:47.830288+0000","last_peered":"2024-03-12T03:42:47.830288+0000","last_clean":"2024-03-12T03:42:47.830288+0000","last_became_active":"2024-03-12T03:40:46.517264+0000","last_became_peered":"2024-03-12T03:40:46.517264+0000","last_unstale":"2024-03-12T03:42:47.830288+0000","last_undegraded":"2024-03-12T03:42:47.830288+0000","last_fullsized":"2024-03-12T03:42:47.830288+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:38:38.943094+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:38:38.943094+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.824696+0000","last_change":"2024-03-12T03:40:46.535973+0000","last_active":"2024-03-12T03:42:47.824696+0000","last_peered":"2024-03-12T03:42:47.824696+0000","last_clean":"2024-03-12T03:42:47.824696+0000","last_became_active":"2024-03-12T03:40:46.535659+0000","last_became_peered":"2024-03-12T03:40:46.535659+0000","last_unstale":"2024-03-12T03:42:47.824696+0000","last_undegraded":"2024-03-12T03:42:47.824696+0000","last_fullsized":"2024-03-12T03:42:47.824696+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:19.511683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:39:19.511683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.511943+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":432,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.820117+0000","last_change":"2024-03-12T03:42:06.327482+0000","last_active":"2024-03-12T03:42:47.820117+0000","last_peered":"2024-03-12T03:42:47.820117+0000","last_clean":"2024-03-12T03:42:47.820117+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:42:47.820117+0000","last_undegraded":"2024-03-12T03:42:47.820117+0000","last_fullsized":"2024-03-12T03:42:47.820117+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:06.327362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:06.327362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:28.064769+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":425,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825557+0000","last_change":"2024-03-12T03:41:25.018344+0000","last_active":"2024-03-12T03:42:47.825557+0000","last_peered":"2024-03-12T03:42:47.825557+0000","last_clean":"2024-03-12T03:42:47.825557+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:42:47.825557+0000","last_undegraded":"2024-03-12T03:42:47.825557+0000","last_fullsized":"2024-03-12T03:42:47.825557+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:25.018224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:25.018224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"94'23","reported_seq":465,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.820498+0000","last_change":"2024-03-12T03:41:13.133503+0000","last_active":"2024-03-12T03:42:47.820498+0000","last_peered":"2024-03-12T03:42:47.820498+0000","last_clean":"2024-03-12T03:42:47.820498+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:42:47.820498+0000","last_undegraded":"2024-03-12T03:42:47.820498+0000","last_fullsized":"2024-03-12T03:42:47.820498+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'23","last_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:41:13.133173+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":365,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.830450+0000","last_change":"2024-03-12T03:40:46.525930+0000","last_active":"2024-03-12T03:42:47.830450+0000","last_peered":"2024-03-12T03:42:47.830450+0000","last_clean":"2024-03-12T03:42:47.830450+0000","last_became_active":"2024-03-12T03:40:46.522763+0000","last_became_peered":"2024-03-12T03:40:46.522763+0000","last_unstale":"2024-03-12T03:42:47.830450+0000","last_undegraded":"2024-03-12T03:42:47.830450+0000","last_fullsized":"2024-03-12T03:42:47.830450+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:25.092313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.092313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":429,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823849+0000","last_change":"2024-03-12T03:40:21.384617+0000","last_active":"2024-03-12T03:42:47.823849+0000","last_peered":"2024-03-12T03:42:47.823849+0000","last_clean":"2024-03-12T03:42:47.823849+0000","last_became_active":"2024-03-12T03:39:33.276435+0000","last_became_peered":"2024-03-12T03:39:33.276435+0000","last_unstale":"2024-03-12T03:42:47.823849+0000","last_undegraded":"2024-03-12T03:42:47.823849+0000","last_fullsized":"2024-03-12T03:42:47.823849+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:21.384531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:40:21.384531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.512205+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":419,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.818471+0000","last_change":"2024-03-12T03:39:33.820164+0000","last_active":"2024-03-12T03:42:47.818471+0000","last_peered":"2024-03-12T03:42:47.818471+0000","last_clean":"2024-03-12T03:42:47.818471+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:42:47.818471+0000","last_undegraded":"2024-03-12T03:42:47.818471+0000","last_fullsized":"2024-03-12T03:42:47.818471+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.819657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.819657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.844274+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'17","reported_seq":458,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823613+0000","last_change":"2024-03-12T03:40:46.519746+0000","last_active":"2024-03-12T03:42:47.823613+0000","last_peered":"2024-03-12T03:42:47.823613+0000","last_clean":"2024-03-12T03:42:47.823613+0000","last_became_active":"2024-03-12T03:40:46.519129+0000","last_became_peered":"2024-03-12T03:40:46.519129+0000","last_unstale":"2024-03-12T03:42:47.823613+0000","last_undegraded":"2024-03-12T03:42:47.823613+0000","last_fullsized":"2024-03-12T03:42:47.823613+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:39:37.572971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:37.572971+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.512057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":434,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.818240+0000","last_change":"2024-03-12T03:40:46.531173+0000","last_active":"2024-03-12T03:42:47.818240+0000","last_peered":"2024-03-12T03:42:47.818240+0000","last_clean":"2024-03-12T03:42:47.818240+0000","last_became_active":"2024-03-12T03:40:46.530644+0000","last_became_peered":"2024-03-12T03:40:46.530644+0000","last_unstale":"2024-03-12T03:42:47.818240+0000","last_undegraded":"2024-03-12T03:42:47.818240+0000","last_fullsized":"2024-03-12T03:42:47.818240+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:38:35.588969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:35.588969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.841846+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":549,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.820433+0000","last_change":"2024-03-12T03:41:09.084115+0000","last_active":"2024-03-12T03:42:47.820433+0000","last_peered":"2024-03-12T03:42:47.820433+0000","last_clean":"2024-03-12T03:42:47.820433+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:42:47.820433+0000","last_undegraded":"2024-03-12T03:42:47.820433+0000","last_fullsized":"2024-03-12T03:42:47.820433+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:41:09.083949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:09.083949+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":441,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825811+0000","last_change":"2024-03-12T03:41:26.041453+0000","last_active":"2024-03-12T03:42:47.825811+0000","last_peered":"2024-03-12T03:42:47.825811+0000","last_clean":"2024-03-12T03:42:47.825811+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:42:47.825811+0000","last_undegraded":"2024-03-12T03:42:47.825811+0000","last_fullsized":"2024-03-12T03:42:47.825811+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:41:26.041306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:26.041306+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":390,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.830196+0000","last_change":"2024-03-12T03:40:46.520870+0000","last_active":"2024-03-12T03:42:47.830196+0000","last_peered":"2024-03-12T03:42:47.830196+0000","last_clean":"2024-03-12T03:42:47.830196+0000","last_became_active":"2024-03-12T03:40:46.519620+0000","last_became_peered":"2024-03-12T03:40:46.519620+0000","last_unstale":"2024-03-12T03:42:47.830196+0000","last_undegraded":"2024-03-12T03:42:47.830196+0000","last_fullsized":"2024-03-12T03:42:47.830196+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:36:28.133261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.133261+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":452,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.816001+0000","last_change":"2024-03-12T03:42:18.499466+0000","last_active":"2024-03-12T03:42:47.816001+0000","last_peered":"2024-03-12T03:42:47.816001+0000","last_clean":"2024-03-12T03:42:47.816001+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:42:47.816001+0000","last_undegraded":"2024-03-12T03:42:47.816001+0000","last_fullsized":"2024-03-12T03:42:47.816001+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:18.499342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:18.499342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:45.872849+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":445,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.816059+0000","last_change":"2024-03-12T03:41:08.289828+0000","last_active":"2024-03-12T03:42:47.816059+0000","last_peered":"2024-03-12T03:42:47.816059+0000","last_clean":"2024-03-12T03:42:47.816059+0000","last_became_active":"2024-03-12T03:40:46.521420+0000","last_became_peered":"2024-03-12T03:40:46.521420+0000","last_unstale":"2024-03-12T03:42:47.816059+0000","last_undegraded":"2024-03-12T03:42:47.816059+0000","last_fullsized":"2024-03-12T03:42:47.816059+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:08.289718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:08.289718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.434980+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":367,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.830942+0000","last_change":"2024-03-12T03:40:46.536596+0000","last_active":"2024-03-12T03:42:47.830942+0000","last_peered":"2024-03-12T03:42:47.830942+0000","last_clean":"2024-03-12T03:42:47.830942+0000","last_became_active":"2024-03-12T03:40:46.535011+0000","last_became_peered":"2024-03-12T03:40:46.535011+0000","last_unstale":"2024-03-12T03:42:47.830942+0000","last_undegraded":"2024-03-12T03:42:47.830942+0000","last_fullsized":"2024-03-12T03:42:47.830942+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:16.588551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:16.588551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":169,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.834127+0000","last_change":"2024-03-12T03:42:04.171633+0000","last_active":"2024-03-12T03:42:47.834127+0000","last_peered":"2024-03-12T03:42:47.834127+0000","last_clean":"2024-03-12T03:42:47.834127+0000","last_became_active":"2024-03-12T03:40:46.519814+0000","last_became_peered":"2024-03-12T03:40:46.519814+0000","last_unstale":"2024-03-12T03:42:47.834127+0000","last_undegraded":"2024-03-12T03:42:47.834127+0000","last_fullsized":"2024-03-12T03:42:47.834127+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:42:04.171114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:42:04.171114+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:25.994710+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":424,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.821838+0000","last_change":"2024-03-12T03:39:28.536556+0000","last_active":"2024-03-12T03:42:47.821838+0000","last_peered":"2024-03-12T03:42:47.821838+0000","last_clean":"2024-03-12T03:42:47.821838+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:42:47.821838+0000","last_undegraded":"2024-03-12T03:42:47.821838+0000","last_fullsized":"2024-03-12T03:42:47.821838+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:28.536478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:39:28.536478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.511862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":423,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.818331+0000","last_change":"2024-03-12T03:39:20.647378+0000","last_active":"2024-03-12T03:42:47.818331+0000","last_peered":"2024-03-12T03:42:47.818331+0000","last_clean":"2024-03-12T03:42:47.818331+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:42:47.818331+0000","last_undegraded":"2024-03-12T03:42:47.818331+0000","last_fullsized":"2024-03-12T03:42:47.818331+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:20.647294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:20.647294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.844051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":421,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.816203+0000","last_change":"2024-03-12T03:42:20.532434+0000","last_active":"2024-03-12T03:42:47.816203+0000","last_peered":"2024-03-12T03:42:47.816203+0000","last_clean":"2024-03-12T03:42:47.816203+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:42:47.816203+0000","last_undegraded":"2024-03-12T03:42:47.816203+0000","last_fullsized":"2024-03-12T03:42:47.816203+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:20.532318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:20.532318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:29.117306+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"86'14","reported_seq":490,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.815667+0000","last_change":"2024-03-12T03:41:04.284204+0000","last_active":"2024-03-12T03:42:47.815667+0000","last_peered":"2024-03-12T03:42:47.815667+0000","last_clean":"2024-03-12T03:42:47.815667+0000","last_became_active":"2024-03-12T03:40:46.528979+0000","last_became_peered":"2024-03-12T03:40:46.528979+0000","last_unstale":"2024-03-12T03:42:47.815667+0000","last_undegraded":"2024-03-12T03:42:47.815667+0000","last_fullsized":"2024-03-12T03:42:47.815667+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:41:04.284062+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:41:04.284062+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.434730+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":424,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823503+0000","last_change":"2024-03-12T03:40:17.773177+0000","last_active":"2024-03-12T03:42:47.823503+0000","last_peered":"2024-03-12T03:42:47.823503+0000","last_clean":"2024-03-12T03:42:47.823503+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:42:47.823503+0000","last_undegraded":"2024-03-12T03:42:47.823503+0000","last_fullsized":"2024-03-12T03:42:47.823503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:17.772938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:40:17.772938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661275+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":416,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.825368+0000","last_change":"2024-03-12T03:41:24.020443+0000","last_active":"2024-03-12T03:42:47.825368+0000","last_peered":"2024-03-12T03:42:47.825368+0000","last_clean":"2024-03-12T03:42:47.825368+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:42:47.825368+0000","last_undegraded":"2024-03-12T03:42:47.825368+0000","last_fullsized":"2024-03-12T03:42:47.825368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:24.020327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.805859+0000","last_clean_scrub_stamp":"2024-03-12T03:41:24.020327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":421,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.822789+0000","last_change":"2024-03-12T03:41:15.648366+0000","last_active":"2024-03-12T03:42:47.822789+0000","last_peered":"2024-03-12T03:42:47.822789+0000","last_clean":"2024-03-12T03:42:47.822789+0000","last_became_active":"2024-03-12T03:40:46.529681+0000","last_became_peered":"2024-03-12T03:40:46.529681+0000","last_unstale":"2024-03-12T03:42:47.822789+0000","last_undegraded":"2024-03-12T03:42:47.822789+0000","last_fullsized":"2024-03-12T03:42:47.822789+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:15.648255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:15.648255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.659154+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'21","reported_seq":469,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.815251+0000","last_change":"2024-03-12T03:40:58.313751+0000","last_active":"2024-03-12T03:42:47.815251+0000","last_peered":"2024-03-12T03:42:47.815251+0000","last_clean":"2024-03-12T03:42:47.815251+0000","last_became_active":"2024-03-12T03:40:46.518368+0000","last_became_peered":"2024-03-12T03:40:46.518368+0000","last_unstale":"2024-03-12T03:42:47.815251+0000","last_undegraded":"2024-03-12T03:42:47.815251+0000","last_fullsized":"2024-03-12T03:42:47.815251+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:40:58.313611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:58.313611+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.434883+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":424,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823670+0000","last_change":"2024-03-12T03:40:25.773470+0000","last_active":"2024-03-12T03:42:47.823670+0000","last_peered":"2024-03-12T03:42:47.823670+0000","last_clean":"2024-03-12T03:42:47.823670+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:42:47.823670+0000","last_undegraded":"2024-03-12T03:42:47.823670+0000","last_fullsized":"2024-03-12T03:42:47.823670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:25.773348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:40:25.773348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661337+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":416,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.823054+0000","last_change":"2024-03-12T03:40:29.743363+0000","last_active":"2024-03-12T03:42:47.823054+0000","last_peered":"2024-03-12T03:42:47.823054+0000","last_clean":"2024-03-12T03:42:47.823054+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:42:47.823054+0000","last_undegraded":"2024-03-12T03:42:47.823054+0000","last_fullsized":"2024-03-12T03:42:47.823054+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:29.743231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:40:29.743231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:48.661352+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":412,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.822239+0000","last_change":"2024-03-12T03:40:31.512991+0000","last_active":"2024-03-12T03:42:47.822239+0000","last_peered":"2024-03-12T03:42:47.822239+0000","last_clean":"2024-03-12T03:42:47.822239+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:42:47.822239+0000","last_undegraded":"2024-03-12T03:42:47.822239+0000","last_fullsized":"2024-03-12T03:42:47.822239+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:31.512880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:40:31.512880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.512166+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"94'50","reported_seq":510,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.829212+0000","last_change":"2024-03-12T03:40:46.519002+0000","last_active":"2024-03-12T03:42:47.829212+0000","last_peered":"2024-03-12T03:42:47.829212+0000","last_clean":"2024-03-12T03:42:47.829212+0000","last_became_active":"2024-03-12T03:40:46.516177+0000","last_became_peered":"2024-03-12T03:40:46.516177+0000","last_unstale":"2024-03-12T03:42:47.829212+0000","last_undegraded":"2024-03-12T03:42:47.829212+0000","last_fullsized":"2024-03-12T03:42:47.829212+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:35:05.042030+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":392,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.831653+0000","last_change":"2024-03-12T03:40:46.513612+0000","last_active":"2024-03-12T03:42:47.831653+0000","last_peered":"2024-03-12T03:42:47.831653+0000","last_clean":"2024-03-12T03:42:47.831653+0000","last_became_active":"2024-03-12T03:40:46.513197+0000","last_became_peered":"2024-03-12T03:40:46.513197+0000","last_unstale":"2024-03-12T03:42:47.831653+0000","last_undegraded":"2024-03-12T03:42:47.831653+0000","last_fullsized":"2024-03-12T03:42:47.831653+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:26.898782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:39:26.898782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":424,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.814550+0000","last_change":"2024-03-12T03:42:09.304630+0000","last_active":"2024-03-12T03:42:47.814550+0000","last_peered":"2024-03-12T03:42:47.814550+0000","last_clean":"2024-03-12T03:42:47.814550+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:42:47.814550+0000","last_undegraded":"2024-03-12T03:42:47.814550+0000","last_fullsized":"2024-03-12T03:42:47.814550+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:09.304528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:42:09.304528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:23.606291+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":343,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.831238+0000","last_change":"2024-03-12T03:40:46.535015+0000","last_active":"2024-03-12T03:42:47.831238+0000","last_peered":"2024-03-12T03:42:47.831238+0000","last_clean":"2024-03-12T03:42:47.831238+0000","last_became_active":"2024-03-12T03:40:46.526044+0000","last_became_peered":"2024-03-12T03:40:46.526044+0000","last_unstale":"2024-03-12T03:42:47.831238+0000","last_undegraded":"2024-03-12T03:42:47.831238+0000","last_fullsized":"2024-03-12T03:42:47.831238+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.236997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.236997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.495546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"218'20","reported_seq":453,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.814528+0000","last_change":"2024-03-12T03:40:29.325294+0000","last_active":"2024-03-12T03:42:47.814528+0000","last_peered":"2024-03-12T03:42:47.814528+0000","last_clean":"2024-03-12T03:42:47.814528+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:42:47.814528+0000","last_undegraded":"2024-03-12T03:42:47.814528+0000","last_fullsized":"2024-03-12T03:42:47.814528+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'20","last_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:40:29.325128+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.434622+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":437,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.824537+0000","last_change":"2024-03-12T03:40:27.572921+0000","last_active":"2024-03-12T03:42:47.824537+0000","last_peered":"2024-03-12T03:42:47.824537+0000","last_clean":"2024-03-12T03:42:47.824537+0000","last_became_active":"2024-03-12T03:39:33.276119+0000","last_became_peered":"2024-03-12T03:39:33.276119+0000","last_unstale":"2024-03-12T03:42:47.824537+0000","last_undegraded":"2024-03-12T03:42:47.824537+0000","last_fullsized":"2024-03-12T03:42:47.824537+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:27.572788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:28.208044+0000","last_clean_scrub_stamp":"2024-03-12T03:40:27.572788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:49.512232+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":432,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.818325+0000","last_change":"2024-03-12T03:40:30.676492+0000","last_active":"2024-03-12T03:42:47.818325+0000","last_peered":"2024-03-12T03:42:47.818325+0000","last_clean":"2024-03-12T03:42:47.818325+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:42:47.818325+0000","last_undegraded":"2024-03-12T03:42:47.818325+0000","last_fullsized":"2024-03-12T03:42:47.818325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:30.676367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:26.625326+0000","last_clean_scrub_stamp":"2024-03-12T03:40:30.676367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:51.844430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":420,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-12T03:42:47.814779+0000","last_change":"2024-03-12T03:41:25.309091+0000","last_active":"2024-03-12T03:42:47.814779+0000","last_peered":"2024-03-12T03:42:47.814779+0000","last_clean":"2024-03-12T03:42:47.814779+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:42:47.814779+0000","last_undegraded":"2024-03-12T03:42:47.814779+0000","last_fullsized":"2024-03-12T03:42:47.814779+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:25.308974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.063092+0000","last_clean_scrub_stamp":"2024-03-12T03:41:25.308974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:42:52.434941+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2176647192,"num_objects":33214,"num_object_clones":0,"num_object_copies":99642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33214,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233748,"num_write_kb":8542721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34219,"num_bytes_recovered":2242576384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6529626112,"data_stored":6529613896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80267,"ondisk_log_size":80267,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":563,"num_read_kb":558,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":686386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":980,"num_read_kb":705,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":2064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3734,"internal_metadata":0},"log_size":657,"ondisk_log_size":657,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070583,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1058640,"kb_used_data":684552,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92686512,"statfs":{"total":95995035648,"available":94910988288,"internally_reserved":0,"allocated":700981248,"data_stored":700563309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":383056330},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332220,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1046708,"kb_used_data":686508,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92698444,"statfs":{"total":95995035648,"available":94923206656,"internally_reserved":0,"allocated":702984192,"data_stored":702592938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":368835014},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574997,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1337828,"kb_used_data":915100,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":92407324,"statfs":{"total":95995035648,"available":94625099776,"internally_reserved":0,"allocated":937062400,"data_stored":936623844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":432798149},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738522,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1193104,"kb_used_data":800964,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92552048,"statfs":{"total":95995035648,"available":94773297152,"internally_reserved":0,"allocated":820187136,"data_stored":819772523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411471,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1235768,"kb_used_data":821424,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92509384,"statfs":{"total":95995035648,"available":94729609216,"internally_reserved":0,"allocated":841138176,"data_stored":840744407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":222,"seq":953482739738,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1105376,"kb_used_data":836184,"kb_used_omap":7,"kb_used_meta":269112,"kb_avail":92639776,"statfs":{"total":95995035648,"available":94863130624,"internally_reserved":0,"allocated":856252416,"data_stored":855817977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8180,"internal_metadata":275570700},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953607,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":983312,"kb_used_data":673096,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92761840,"statfs":{"total":95995035648,"available":94988124160,"internally_reserved":0,"allocated":689250304,"data_stored":688831881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659265,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1488844,"kb_used_data":1092484,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92256308,"statfs":{"total":95995035648,"available":94470459392,"internally_reserved":0,"allocated":1118703616,"data_stored":1118289025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3734,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":793968640,"data_stored":793968640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":926810112,"data_stored":926810112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":813240320,"data_stored":813236248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":849412096,"data_stored":849412096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":677969920,"data_stored":677969920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1098846208,"data_stored":1098842136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":685441024,"data_stored":685441024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":683937792,"data_stored":683933720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:42:57.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:42:57.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:57.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:57.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:56 smithi173 ceph-mon[33544]: pgmap v448: 158 pgs: 158 active+clean; 2.0 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-12T03:42:57.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:57.303 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:57.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:57.404 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:57.457 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:57.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:57.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:57.559 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:57.600 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:57.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:57.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:57.690 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:57.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:57.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:57.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:57.737 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:42:57.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:57.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:57.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:42:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:58.008 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:58.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:42:58.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:58.146 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:58.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:58.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:58.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:58.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:58.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:58.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:58.380 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:42:58.428 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:58.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:42:58.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:58.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:58.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:58.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:42:58.703 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:58.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:42:58.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:42:58.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:58.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:58.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:42:58.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:58.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:58.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:58.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:59.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:42:59.109 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:59.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:42:59.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:42:59 smithi169 ceph-mon[29344]: pgmap v449: 158 pgs: 158 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1009 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-12T03:42:59.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:42:59 smithi169 ceph-mon[39949]: pgmap v449: 158 pgs: 158 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1009 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-12T03:42:59.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:59.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:59.254 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:59.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:59.355 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:42:59.401 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:59.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:42:59.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:42:59 smithi173 ceph-mon[33544]: pgmap v449: 158 pgs: 158 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1009 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-12T03:42:59.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:59.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:59.564 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:59.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:42:59.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:59.699 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:59.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:59.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:42:59.833 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:42:59.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:59.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:42:59.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:42:59.972 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:42:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:42:59.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:00.042 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:43:00.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:00.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:00.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:00.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:43:00.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:43:00.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/785464431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:43:00.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-12T03:43:00.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-12T03:43:00.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-12T03:43:00.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-12T03:43:00.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:43:00.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:43:00.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/785464431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:43:00.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-12T03:43:00.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-12T03:43:00.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-12T03:43:00.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:00 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-12T03:43:00.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:00.245 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:00.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:00.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:00.334 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:43:00.334 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:43:00.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:00.373 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:43:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:00.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:00.412 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:43:00.447 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:43:00.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:00 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:43:00.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:00 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:43:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:00 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/785464431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:43:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-12T03:43:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:00 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-12T03:43:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-12T03:43:00.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:00 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-12T03:43:00.487 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:43:00.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:00.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:00.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:00.529 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:43:00.580 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:43:00.589 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:43:00.589 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:43:00.590 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:43:00.590 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:43:00.590 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:43:00.590 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:43:00.590 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:43:00.590 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:43:00.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:00.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:00.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:00.750 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:00.788 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:00.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:00.820 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:00.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:00.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:00.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:01.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:01.074 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:01.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:01.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[29344]: pgmap v450: 158 pgs: 158 active+clean; 1.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-12T03:43:01.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:43:01.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/785464431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:43:01.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-12T03:43:01.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-12T03:43:01.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[29344]: osdmap e261: 8 total, 8 up, 8 in 2024-03-12T03:43:01.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2598864335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:01.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:01.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[29344]: osdmap e262: 8 total, 8 up, 8 in 2024-03-12T03:43:01.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[39949]: pgmap v450: 158 pgs: 158 active+clean; 1.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-12T03:43:01.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:43:01.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/785464431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:43:01.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-12T03:43:01.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-12T03:43:01.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[39949]: osdmap e261: 8 total, 8 up, 8 in 2024-03-12T03:43:01.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2598864335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:01.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:01.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-mon[39949]: osdmap e262: 8 total, 8 up, 8 in 2024-03-12T03:43:01.175 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:43:01.201 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:01.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:43:01.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:01.338 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:01.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:01.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:01.440 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:01.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:01.479 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:01.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:01 smithi173 ceph-mon[33544]: pgmap v450: 158 pgs: 158 active+clean; 1.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-12T03:43:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:01 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:43:01.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:01 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/785464431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:43:01.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:01 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-12T03:43:01.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:01 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-12T03:43:01.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:01 smithi173 ceph-mon[33544]: osdmap e261: 8 total, 8 up, 8 in 2024-03-12T03:43:01.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:01 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2598864335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:01.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:01 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:01.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:01 smithi173 ceph-mon[33544]: osdmap e262: 8 total, 8 up, 8 in 2024-03-12T03:43:01.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:01.617 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:01.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:01.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:01.719 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:01.761 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:01.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:01.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:01.907 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:01.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:01.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:02.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:02.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:02.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:02.082 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:02 smithi169 ceph-mon[29344]: 4.19 scrub starts 2024-03-12T03:43:02.082 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:02 smithi169 ceph-mon[29344]: 4.19 scrub ok 2024-03-12T03:43:02.082 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:02 smithi169 ceph-mon[29344]: 4.f scrub starts 2024-03-12T03:43:02.082 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:02 smithi169 ceph-mon[29344]: 4.f scrub ok 2024-03-12T03:43:02.083 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:02 smithi169 ceph-mon[39949]: 4.19 scrub starts 2024-03-12T03:43:02.083 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:02 smithi169 ceph-mon[39949]: 4.19 scrub ok 2024-03-12T03:43:02.083 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:02 smithi169 ceph-mon[39949]: 4.f scrub starts 2024-03-12T03:43:02.083 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:02 smithi169 ceph-mon[39949]: 4.f scrub ok 2024-03-12T03:43:02.083 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:43:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:43:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:43:02.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:02.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:02.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:02.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:02.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:02.316 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:02.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:02.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:02.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:02.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:02.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:02 smithi173 ceph-mon[33544]: 4.19 scrub starts 2024-03-12T03:43:02.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:02 smithi173 ceph-mon[33544]: 4.19 scrub ok 2024-03-12T03:43:02.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:02 smithi173 ceph-mon[33544]: 4.f scrub starts 2024-03-12T03:43:02.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:02 smithi173 ceph-mon[33544]: 4.f scrub ok 2024-03-12T03:43:02.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:02.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:02.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:02.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:02.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:02.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:02.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:02.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:02.813 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:43:02.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:02.851 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:02.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:43:02.952 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:43:02.993 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:02.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:43:03.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:43:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:43:03.095 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:43:03.142 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:03.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:03.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:43:03.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:03.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:03.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:03.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:03.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:03.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[39949]: 5.d scrub starts 2024-03-12T03:43:03.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[39949]: 5.d scrub ok 2024-03-12T03:43:03.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[39949]: pgmap v453: 158 pgs: 158 active+clean; 1.4 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 13 B/s, 0 objects/s recovering 2024-03-12T03:43:03.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[39949]: 1.7 scrub starts 2024-03-12T03:43:03.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[39949]: 1.7 scrub ok 2024-03-12T03:43:03.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[39949]: 4.18 scrub starts 2024-03-12T03:43:03.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[39949]: 4.18 scrub ok 2024-03-12T03:43:03.404 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:03.404 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[29344]: 5.d scrub starts 2024-03-12T03:43:03.404 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[29344]: 5.d scrub ok 2024-03-12T03:43:03.404 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[29344]: pgmap v453: 158 pgs: 158 active+clean; 1.4 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 13 B/s, 0 objects/s recovering 2024-03-12T03:43:03.404 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[29344]: 1.7 scrub starts 2024-03-12T03:43:03.404 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[29344]: 1.7 scrub ok 2024-03-12T03:43:03.404 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[29344]: 4.18 scrub starts 2024-03-12T03:43:03.404 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[29344]: 4.18 scrub ok 2024-03-12T03:43:03.404 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:03 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:03.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:03.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:03.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:03 smithi173 ceph-mon[33544]: 5.d scrub starts 2024-03-12T03:43:03.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:03 smithi173 ceph-mon[33544]: 5.d scrub ok 2024-03-12T03:43:03.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:03 smithi173 ceph-mon[33544]: pgmap v453: 158 pgs: 158 active+clean; 1.4 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 13 B/s, 0 objects/s recovering 2024-03-12T03:43:03.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:03 smithi173 ceph-mon[33544]: 1.7 scrub starts 2024-03-12T03:43:03.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:03 smithi173 ceph-mon[33544]: 1.7 scrub ok 2024-03-12T03:43:03.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:03 smithi173 ceph-mon[33544]: 4.18 scrub starts 2024-03-12T03:43:03.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:03 smithi173 ceph-mon[33544]: 4.18 scrub ok 2024-03-12T03:43:03.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:03 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:03.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:03.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:03.548 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:03.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:03.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:43:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:03.690 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:03.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:43:03.791 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:43:03.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:03.827 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:03.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:43:03.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:03.947 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:43:03.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:03.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:04.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:04.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:04.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:04.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:04.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:04.218 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:04.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:04.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:04.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:04.345 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:04 smithi169 ceph-mon[39949]: 1.4 scrub starts 2024-03-12T03:43:04.345 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:04 smithi169 ceph-mon[39949]: 1.4 scrub ok 2024-03-12T03:43:04.346 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:04 smithi169 ceph-mon[39949]: 6.14 scrub starts 2024-03-12T03:43:04.346 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:04 smithi169 ceph-mon[39949]: 6.14 scrub ok 2024-03-12T03:43:04.346 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:04 smithi169 ceph-mon[39949]: 4.1f scrub starts 2024-03-12T03:43:04.346 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:04 smithi169 ceph-mon[39949]: 4.1f scrub ok 2024-03-12T03:43:04.346 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:04 smithi169 ceph-mon[29344]: 1.4 scrub starts 2024-03-12T03:43:04.346 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:04 smithi169 ceph-mon[29344]: 1.4 scrub ok 2024-03-12T03:43:04.346 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:04 smithi169 ceph-mon[29344]: 6.14 scrub starts 2024-03-12T03:43:04.346 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:04 smithi169 ceph-mon[29344]: 6.14 scrub ok 2024-03-12T03:43:04.346 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:04 smithi169 ceph-mon[29344]: 4.1f scrub starts 2024-03-12T03:43:04.346 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:04 smithi169 ceph-mon[29344]: 4.1f scrub ok 2024-03-12T03:43:04.368 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:04.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:04.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:04.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:04.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:04 smithi173 ceph-mon[33544]: 1.4 scrub starts 2024-03-12T03:43:04.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:04 smithi173 ceph-mon[33544]: 1.4 scrub ok 2024-03-12T03:43:04.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:04 smithi173 ceph-mon[33544]: 6.14 scrub starts 2024-03-12T03:43:04.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:04 smithi173 ceph-mon[33544]: 6.14 scrub ok 2024-03-12T03:43:04.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:04 smithi173 ceph-mon[33544]: 4.1f scrub starts 2024-03-12T03:43:04.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:04 smithi173 ceph-mon[33544]: 4.1f scrub ok 2024-03-12T03:43:04.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:04.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:04.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:04.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:04.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:04.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:04.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:04.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:04.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:04.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:04.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:43:04.919 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:04.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:04.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:43:05.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:05.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:05.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:05.071 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:43:05.157 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:05.214 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:05.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:05.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:05.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:05.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:05.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:05.355 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:05 smithi169 ceph-mon[39949]: 4.16 scrub starts 2024-03-12T03:43:05.355 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:05 smithi169 ceph-mon[39949]: 4.16 scrub ok 2024-03-12T03:43:05.355 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:05 smithi169 ceph-mon[39949]: pgmap v454: 158 pgs: 158 active+clean; 1.3 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 13 B/s, 0 objects/s recovering 2024-03-12T03:43:05.355 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:05 smithi169 ceph-mon[39949]: 1.5 scrub starts 2024-03-12T03:43:05.355 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:05 smithi169 ceph-mon[39949]: 1.5 scrub ok 2024-03-12T03:43:05.357 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:05 smithi169 ceph-mon[29344]: 4.16 scrub starts 2024-03-12T03:43:05.357 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:05 smithi169 ceph-mon[29344]: 4.16 scrub ok 2024-03-12T03:43:05.357 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:05 smithi169 ceph-mon[29344]: pgmap v454: 158 pgs: 158 active+clean; 1.3 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 13 B/s, 0 objects/s recovering 2024-03-12T03:43:05.357 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:05 smithi169 ceph-mon[29344]: 1.5 scrub starts 2024-03-12T03:43:05.357 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:05 smithi169 ceph-mon[29344]: 1.5 scrub ok 2024-03-12T03:43:05.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:05.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:05 smithi173 ceph-mon[33544]: 4.16 scrub starts 2024-03-12T03:43:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:05 smithi173 ceph-mon[33544]: 4.16 scrub ok 2024-03-12T03:43:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:05 smithi173 ceph-mon[33544]: pgmap v454: 158 pgs: 158 active+clean; 1.3 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 13 B/s, 0 objects/s recovering 2024-03-12T03:43:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:05 smithi173 ceph-mon[33544]: 1.5 scrub starts 2024-03-12T03:43:05.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:05 smithi173 ceph-mon[33544]: 1.5 scrub ok 2024-03-12T03:43:05.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:05.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:05.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:05.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:05.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:05.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:05.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:05.719 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:05.760 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:05.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:05.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:05.909 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:05.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:06.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:06.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:06.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:06.067 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:43:06.067 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:43:06.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:06.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:06.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:06.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:06.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:06 smithi169 ceph-mon[29344]: 6.10 scrub starts 2024-03-12T03:43:06.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:06 smithi169 ceph-mon[29344]: 6.10 scrub ok 2024-03-12T03:43:06.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:06 smithi169 ceph-mon[29344]: 1.0 scrub starts 2024-03-12T03:43:06.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:06 smithi169 ceph-mon[29344]: 1.0 scrub ok 2024-03-12T03:43:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:06 smithi169 ceph-mon[39949]: 6.10 scrub starts 2024-03-12T03:43:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:06 smithi169 ceph-mon[39949]: 6.10 scrub ok 2024-03-12T03:43:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:06 smithi169 ceph-mon[39949]: 1.0 scrub starts 2024-03-12T03:43:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:06 smithi169 ceph-mon[39949]: 1.0 scrub ok 2024-03-12T03:43:06.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:06.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:43:06.362 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:43:06.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:06.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:06.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:06.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:06 smithi173 ceph-mon[33544]: 6.10 scrub starts 2024-03-12T03:43:06.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:06 smithi173 ceph-mon[33544]: 6.10 scrub ok 2024-03-12T03:43:06.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:06 smithi173 ceph-mon[33544]: 1.0 scrub starts 2024-03-12T03:43:06.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:06 smithi173 ceph-mon[33544]: 1.0 scrub ok 2024-03-12T03:43:06.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:06.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:06.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:06.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:06.707 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:06.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:06.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:06.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:06.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:06.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:06.973 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:06.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:07.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:07.119 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:07.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:07.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:07.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:07.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:07.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:07.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:43:07.389 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:07.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:07.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:43:07.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[29344]: 5.17 scrub starts 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[29344]: 5.17 scrub ok 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[29344]: 8.e scrub starts 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[29344]: pgmap v455: 158 pgs: 158 active+clean; 1.2 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 0 B/s, 1 objects/s recovering 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[29344]: 8.e scrub ok 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[29344]: from='client.25094 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[29344]: 4.4 scrub starts 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[29344]: 4.4 scrub ok 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[39949]: 5.17 scrub starts 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[39949]: 5.17 scrub ok 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[39949]: 8.e scrub starts 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[39949]: pgmap v455: 158 pgs: 158 active+clean; 1.2 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 0 B/s, 1 objects/s recovering 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[39949]: 8.e scrub ok 2024-03-12T03:43:07.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[39949]: from='client.25094 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:43:07.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[39949]: 4.4 scrub starts 2024-03-12T03:43:07.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:07 smithi169 ceph-mon[39949]: 4.4 scrub ok 2024-03-12T03:43:07.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:07 smithi173 ceph-mon[33544]: 5.17 scrub starts 2024-03-12T03:43:07.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:07 smithi173 ceph-mon[33544]: 5.17 scrub ok 2024-03-12T03:43:07.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:07 smithi173 ceph-mon[33544]: 8.e scrub starts 2024-03-12T03:43:07.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:07 smithi173 ceph-mon[33544]: pgmap v455: 158 pgs: 158 active+clean; 1.2 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 0 B/s, 1 objects/s recovering 2024-03-12T03:43:07.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:07 smithi173 ceph-mon[33544]: 8.e scrub ok 2024-03-12T03:43:07.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:07 smithi173 ceph-mon[33544]: from='client.25094 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:43:07.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:07 smithi173 ceph-mon[33544]: 4.4 scrub starts 2024-03-12T03:43:07.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:07 smithi173 ceph-mon[33544]: 4.4 scrub ok 2024-03-12T03:43:07.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-12T03:43:07.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:07.525 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:07.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:07.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:07.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:07.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:07.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:07.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:07.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:07.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:07.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:07.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:07.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:07.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:08.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:08.059 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:08.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:08.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:08.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:08.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:08.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:08.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:08.320 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:08.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:08.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:08.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:08 smithi169 ceph-mon[29344]: 4.11 scrub starts 2024-03-12T03:43:08.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:08 smithi169 ceph-mon[29344]: 4.11 scrub ok 2024-03-12T03:43:08.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:08 smithi169 ceph-mon[39949]: 4.11 scrub starts 2024-03-12T03:43:08.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:08 smithi169 ceph-mon[39949]: 4.11 scrub ok 2024-03-12T03:43:08.458 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:08.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:08.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:08.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:08 smithi173 ceph-mon[33544]: 4.11 scrub starts 2024-03-12T03:43:08.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:08 smithi173 ceph-mon[33544]: 4.11 scrub ok 2024-03-12T03:43:08.560 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:08.597 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:08.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:08.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:08.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:43:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:08.736 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:08.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:43:08.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:08.861 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:08.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:08.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:08.993 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:08.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:09.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:09.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:09.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:09.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:09.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:09.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:09.360 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:09.394 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:09.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:09.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[29344]: 3.2 scrub starts 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[29344]: 3.2 scrub ok 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[29344]: 3.19 scrub starts 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[29344]: 3.19 scrub ok 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[29344]: pgmap v456: 158 pgs: 158 active+clean; 945 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s; 0 B/s, 1 objects/s recovering 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[29344]: 3.15 scrub starts 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[29344]: 3.15 scrub ok 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[29344]: 4.5 scrub starts 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[29344]: 4.5 scrub ok 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[39949]: 3.2 scrub starts 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[39949]: 3.2 scrub ok 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[39949]: 3.19 scrub starts 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[39949]: 3.19 scrub ok 2024-03-12T03:43:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[39949]: pgmap v456: 158 pgs: 158 active+clean; 945 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s; 0 B/s, 1 objects/s recovering 2024-03-12T03:43:09.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[39949]: 3.15 scrub starts 2024-03-12T03:43:09.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[39949]: 3.15 scrub ok 2024-03-12T03:43:09.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[39949]: 4.5 scrub starts 2024-03-12T03:43:09.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:09 smithi169 ceph-mon[39949]: 4.5 scrub ok 2024-03-12T03:43:09.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:09 smithi173 ceph-mon[33544]: 3.2 scrub starts 2024-03-12T03:43:09.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:09 smithi173 ceph-mon[33544]: 3.2 scrub ok 2024-03-12T03:43:09.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:09 smithi173 ceph-mon[33544]: 3.19 scrub starts 2024-03-12T03:43:09.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:09 smithi173 ceph-mon[33544]: 3.19 scrub ok 2024-03-12T03:43:09.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:09 smithi173 ceph-mon[33544]: pgmap v456: 158 pgs: 158 active+clean; 945 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s; 0 B/s, 1 objects/s recovering 2024-03-12T03:43:09.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:09 smithi173 ceph-mon[33544]: 3.15 scrub starts 2024-03-12T03:43:09.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:09 smithi173 ceph-mon[33544]: 3.15 scrub ok 2024-03-12T03:43:09.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:09 smithi173 ceph-mon[33544]: 4.5 scrub starts 2024-03-12T03:43:09.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:09 smithi173 ceph-mon[33544]: 4.5 scrub ok 2024-03-12T03:43:09.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:09.533 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:09.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:09.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:09.635 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:43:09.676 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:09.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:09.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:43:09.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:09.812 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:09.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:09.914 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:43:09.951 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:09.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:43:10.052 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:10.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:10.080 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:10.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:10.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:10.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:10.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:10.310 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:43:10.333 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:10.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:43:10.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:10 smithi169 ceph-mon[29344]: 3.8 scrub starts 2024-03-12T03:43:10.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:10 smithi169 ceph-mon[29344]: 3.8 scrub ok 2024-03-12T03:43:10.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:10 smithi169 ceph-mon[29344]: 3.a scrub starts 2024-03-12T03:43:10.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:10 smithi169 ceph-mon[29344]: 3.a scrub ok 2024-03-12T03:43:10.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:10 smithi169 ceph-mon[29344]: 3.9 scrub starts 2024-03-12T03:43:10.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:10 smithi169 ceph-mon[29344]: 3.9 scrub ok 2024-03-12T03:43:10.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:10 smithi169 ceph-mon[39949]: 3.8 scrub starts 2024-03-12T03:43:10.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:10 smithi169 ceph-mon[39949]: 3.8 scrub ok 2024-03-12T03:43:10.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:10 smithi169 ceph-mon[39949]: 3.a scrub starts 2024-03-12T03:43:10.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:10 smithi169 ceph-mon[39949]: 3.a scrub ok 2024-03-12T03:43:10.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:10 smithi169 ceph-mon[39949]: 3.9 scrub starts 2024-03-12T03:43:10.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:10 smithi169 ceph-mon[39949]: 3.9 scrub ok 2024-03-12T03:43:10.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:10.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:10.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:10.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:10 smithi173 ceph-mon[33544]: 3.8 scrub starts 2024-03-12T03:43:10.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:10 smithi173 ceph-mon[33544]: 3.8 scrub ok 2024-03-12T03:43:10.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:10 smithi173 ceph-mon[33544]: 3.a scrub starts 2024-03-12T03:43:10.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:10 smithi173 ceph-mon[33544]: 3.a scrub ok 2024-03-12T03:43:10.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:10 smithi173 ceph-mon[33544]: 3.9 scrub starts 2024-03-12T03:43:10.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:10 smithi173 ceph-mon[33544]: 3.9 scrub ok 2024-03-12T03:43:10.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:10.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:10.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:10.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:10.742 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:10.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:10.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:10.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:10.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:10.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:10.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:10.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:11.015 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:11.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:11.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:11.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:11.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:11.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:11.290 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:11.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:11.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:11.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:11 smithi169 ceph-mon[29344]: 4.6 scrub starts 2024-03-12T03:43:11.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:11 smithi169 ceph-mon[29344]: 4.6 scrub ok 2024-03-12T03:43:11.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:11 smithi169 ceph-mon[29344]: pgmap v457: 158 pgs: 158 active+clean; 877 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.19k op/s; 0 B/s, 1 objects/s recovering 2024-03-12T03:43:11.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:11 smithi169 ceph-mon[29344]: 4.1b scrub starts 2024-03-12T03:43:11.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:11 smithi169 ceph-mon[29344]: 4.1b scrub ok 2024-03-12T03:43:11.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:11 smithi169 ceph-mon[39949]: 4.6 scrub starts 2024-03-12T03:43:11.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:11 smithi169 ceph-mon[39949]: 4.6 scrub ok 2024-03-12T03:43:11.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:11 smithi169 ceph-mon[39949]: pgmap v457: 158 pgs: 158 active+clean; 877 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.19k op/s; 0 B/s, 1 objects/s recovering 2024-03-12T03:43:11.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:11 smithi169 ceph-mon[39949]: 4.1b scrub starts 2024-03-12T03:43:11.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:11 smithi169 ceph-mon[39949]: 4.1b scrub ok 2024-03-12T03:43:11.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:11.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:11.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:11.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:11 smithi173 ceph-mon[33544]: 4.6 scrub starts 2024-03-12T03:43:11.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:11 smithi173 ceph-mon[33544]: 4.6 scrub ok 2024-03-12T03:43:11.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:11 smithi173 ceph-mon[33544]: pgmap v457: 158 pgs: 158 active+clean; 877 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.19k op/s; 0 B/s, 1 objects/s recovering 2024-03-12T03:43:11.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:11 smithi173 ceph-mon[33544]: 4.1b scrub starts 2024-03-12T03:43:11.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:11 smithi173 ceph-mon[33544]: 4.1b scrub ok 2024-03-12T03:43:11.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:11.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:11.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:11.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:11.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:11.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:11.793 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:11.794 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:11.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:11.843 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-03-12T03:43:05.962635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1298143680,"num_objects":20030,"num_object_clones":0,"num_object_copies":60090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20030,"num_whiteouts":0,"num_read":1576,"num_read_kb":1296,"num_write":247790,"num_write_kb":8542766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34334,"num_bytes_recovered":2243106496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82522,"ondisk_log_size":82522,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6817508,"kb_used_data":3858624,"kb_used_omap":18,"kb_used_meta":2958445,"kb_avail":743143708,"statfs":{"total":767960285184,"available":760979156992,"internally_reserved":0,"allocated":3951230976,"data_stored":3947841290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19339,"internal_metadata":3029447797},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-473158054,"num_objects":-7223,"num_object_clones":0,"num_object_copies":-21669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7223,"num_whiteouts":0,"num_read":-8,"num_read_kb":-17,"num_write":7208,"num_write_kb":-220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":-160900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.008600"},"pg_stats":[{"pgid":"8.0","version":"262'15416","reported_seq":23801,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:04.618960+0000","last_change":"2024-03-12T03:42:31.749803+0000","last_active":"2024-03-12T03:43:04.618960+0000","last_peered":"2024-03-12T03:43:04.618960+0000","last_clean":"2024-03-12T03:43:04.618960+0000","last_became_active":"2024-03-12T03:42:31.713116+0000","last_became_peered":"2024-03-12T03:42:31.713116+0000","last_unstale":"2024-03-12T03:43:04.618960+0000","last_undegraded":"2024-03-12T03:43:04.618960+0000","last_fullsized":"2024-03-12T03:43:04.618960+0000","mapping_epoch":255,"log_start":"244'10898","ondisk_log_start":"244'10898","created":198,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4518,"log_dups_size":0,"ondisk_log_size":4518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66715648,"num_objects":1018,"num_object_clones":0,"num_object_copies":3054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15416,"num_write_kb":525888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4123,"num_bytes_recovered":270204928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":737,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.029678+0000","last_change":"2024-03-12T03:42:28.162856+0000","last_active":"2024-03-12T03:43:01.029678+0000","last_peered":"2024-03-12T03:43:01.029678+0000","last_clean":"2024-03-12T03:43:01.029678+0000","last_became_active":"2024-03-12T03:40:46.529101+0000","last_became_peered":"2024-03-12T03:40:46.529101+0000","last_unstale":"2024-03-12T03:43:01.029678+0000","last_undegraded":"2024-03-12T03:43:01.029678+0000","last_fullsized":"2024-03-12T03:43:01.029678+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:28.162735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:28.162735+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:49.132807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":447,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042681+0000","last_change":"2024-03-12T03:41:02.239526+0000","last_active":"2024-03-12T03:43:00.042681+0000","last_peered":"2024-03-12T03:43:00.042681+0000","last_clean":"2024-03-12T03:43:00.042681+0000","last_became_active":"2024-03-12T03:40:46.518953+0000","last_became_peered":"2024-03-12T03:40:46.518953+0000","last_unstale":"2024-03-12T03:43:00.042681+0000","last_undegraded":"2024-03-12T03:43:00.042681+0000","last_fullsized":"2024-03-12T03:43:00.042681+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-12T03:41:02.239384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:41:02.239384+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":334,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.035802+0000","last_change":"2024-03-12T03:40:46.528095+0000","last_active":"2024-03-12T03:43:01.035802+0000","last_peered":"2024-03-12T03:43:01.035802+0000","last_clean":"2024-03-12T03:43:01.035802+0000","last_became_active":"2024-03-12T03:40:46.523062+0000","last_became_peered":"2024-03-12T03:40:46.523062+0000","last_unstale":"2024-03-12T03:43:01.035802+0000","last_undegraded":"2024-03-12T03:43:01.035802+0000","last_fullsized":"2024-03-12T03:43:01.035802+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.981378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.981378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":201,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.350465+0000","last_change":"2024-03-12T03:43:01.350508+0000","last_active":"2024-03-12T03:43:01.350465+0000","last_peered":"2024-03-12T03:43:01.350465+0000","last_clean":"2024-03-12T03:43:01.350465+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:43:01.350465+0000","last_undegraded":"2024-03-12T03:43:01.350465+0000","last_fullsized":"2024-03-12T03:43:01.350465+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:43:01.350377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:01.350377+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:30.689072+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":435,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.043934+0000","last_change":"2024-03-12T03:42:08.287146+0000","last_active":"2024-03-12T03:43:00.043934+0000","last_peered":"2024-03-12T03:43:00.043934+0000","last_clean":"2024-03-12T03:43:00.043934+0000","last_became_active":"2024-03-12T03:39:33.265665+0000","last_became_peered":"2024-03-12T03:39:33.265665+0000","last_unstale":"2024-03-12T03:43:00.043934+0000","last_undegraded":"2024-03-12T03:43:00.043934+0000","last_fullsized":"2024-03-12T03:43:00.043934+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:08.287006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:08.287006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:10.630966+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"262'15466","reported_seq":15605,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:02.007607+0000","last_change":"2024-03-12T03:39:34.336674+0000","last_active":"2024-03-12T03:43:02.007607+0000","last_peered":"2024-03-12T03:43:02.007607+0000","last_clean":"2024-03-12T03:43:02.007607+0000","last_became_active":"2024-03-12T03:39:34.336345+0000","last_became_peered":"2024-03-12T03:39:34.336345+0000","last_unstale":"2024-03-12T03:43:02.007607+0000","last_undegraded":"2024-03-12T03:43:02.007607+0000","last_fullsized":"2024-03-12T03:43:02.007607+0000","mapping_epoch":198,"log_start":"240'10600","ondisk_log_start":"240'10600","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80871424,"num_objects":1234,"num_object_clones":0,"num_object_copies":3702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15466,"num_write_kb":534400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":507,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.043129+0000","last_change":"2024-03-12T03:42:11.397167+0000","last_active":"2024-03-12T03:43:00.043129+0000","last_peered":"2024-03-12T03:43:00.043129+0000","last_clean":"2024-03-12T03:43:00.043129+0000","last_became_active":"2024-03-12T03:39:33.268652+0000","last_became_peered":"2024-03-12T03:39:33.268652+0000","last_unstale":"2024-03-12T03:43:00.043129+0000","last_undegraded":"2024-03-12T03:43:00.043129+0000","last_fullsized":"2024-03-12T03:43:00.043129+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:42:11.397028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:42:11.397028+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:20.606187+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":436,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.030124+0000","last_change":"2024-03-12T03:41:04.011554+0000","last_active":"2024-03-12T03:43:01.030124+0000","last_peered":"2024-03-12T03:43:01.030124+0000","last_clean":"2024-03-12T03:43:01.030124+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:43:01.030124+0000","last_undegraded":"2024-03-12T03:43:01.030124+0000","last_fullsized":"2024-03-12T03:43:01.030124+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:04.011430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:41:04.011430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":429,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.034258+0000","last_change":"2024-03-12T03:39:34.824294+0000","last_active":"2024-03-12T03:43:01.034258+0000","last_peered":"2024-03-12T03:43:01.034258+0000","last_clean":"2024-03-12T03:43:01.034258+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:43:01.034258+0000","last_undegraded":"2024-03-12T03:43:01.034258+0000","last_fullsized":"2024-03-12T03:43:01.034258+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:39:34.824210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":165,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.033528+0000","last_change":"2024-03-12T03:42:07.272333+0000","last_active":"2024-03-12T03:43:01.033528+0000","last_peered":"2024-03-12T03:43:01.033528+0000","last_clean":"2024-03-12T03:43:01.033528+0000","last_became_active":"2024-03-12T03:40:46.532405+0000","last_became_peered":"2024-03-12T03:40:46.532405+0000","last_unstale":"2024-03-12T03:43:01.033528+0000","last_undegraded":"2024-03-12T03:43:01.033528+0000","last_fullsized":"2024-03-12T03:43:01.033528+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:07.272232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:07.272232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:31.724663+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":579,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032998+0000","last_change":"2024-03-12T03:41:03.399874+0000","last_active":"2024-03-12T03:43:01.032998+0000","last_peered":"2024-03-12T03:43:01.032998+0000","last_clean":"2024-03-12T03:43:01.032998+0000","last_became_active":"2024-03-12T03:40:46.524423+0000","last_became_peered":"2024-03-12T03:40:46.524423+0000","last_unstale":"2024-03-12T03:43:01.032998+0000","last_undegraded":"2024-03-12T03:43:01.032998+0000","last_fullsized":"2024-03-12T03:43:01.032998+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:41:03.399737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:41:03.399737+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.417005+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"261'15296","reported_seq":15513,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.919869+0000","last_change":"2024-03-12T03:40:58.061897+0000","last_active":"2024-03-12T03:43:00.919869+0000","last_peered":"2024-03-12T03:43:00.919869+0000","last_clean":"2024-03-12T03:43:00.919869+0000","last_became_active":"2024-03-12T03:39:34.249390+0000","last_became_peered":"2024-03-12T03:39:34.249390+0000","last_unstale":"2024-03-12T03:43:00.919869+0000","last_undegraded":"2024-03-12T03:43:00.919869+0000","last_fullsized":"2024-03-12T03:43:00.919869+0000","mapping_epoch":198,"log_start":"240'10700","ondisk_log_start":"240'10700","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'7205","last_scrub_stamp":"2024-03-12T03:40:58.061634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:40:58.061634+0000","objects_scrubbed":7150,"log_size":4596,"log_dups_size":0,"ondisk_log_size":4596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85983232,"num_objects":1312,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15296,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"95'23","reported_seq":468,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.043871+0000","last_change":"2024-03-12T03:41:18.286718+0000","last_active":"2024-03-12T03:43:00.043871+0000","last_peered":"2024-03-12T03:43:00.043871+0000","last_clean":"2024-03-12T03:43:00.043871+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:43:00.043871+0000","last_undegraded":"2024-03-12T03:43:00.043871+0000","last_fullsized":"2024-03-12T03:43:00.043871+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'23","last_scrub_stamp":"2024-03-12T03:41:18.286562+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:41:18.286562+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560361+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":439,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042120+0000","last_change":"2024-03-12T03:40:32.220387+0000","last_active":"2024-03-12T03:43:00.042120+0000","last_peered":"2024-03-12T03:43:00.042120+0000","last_clean":"2024-03-12T03:43:00.042120+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:43:00.042120+0000","last_undegraded":"2024-03-12T03:43:00.042120+0000","last_fullsized":"2024-03-12T03:43:00.042120+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:32.220274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:40:32.220274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560010+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":421,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.028419+0000","last_change":"2024-03-12T03:39:38.743222+0000","last_active":"2024-03-12T03:43:01.028419+0000","last_peered":"2024-03-12T03:43:01.028419+0000","last_clean":"2024-03-12T03:43:01.028419+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:43:01.028419+0000","last_undegraded":"2024-03-12T03:43:01.028419+0000","last_fullsized":"2024-03-12T03:43:01.028419+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.743128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.743128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":344,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.036638+0000","last_change":"2024-03-12T03:40:46.521186+0000","last_active":"2024-03-12T03:43:01.036638+0000","last_peered":"2024-03-12T03:43:01.036638+0000","last_clean":"2024-03-12T03:43:01.036638+0000","last_became_active":"2024-03-12T03:40:46.515751+0000","last_became_peered":"2024-03-12T03:40:46.515751+0000","last_unstale":"2024-03-12T03:43:01.036638+0000","last_undegraded":"2024-03-12T03:43:01.036638+0000","last_fullsized":"2024-03-12T03:43:01.036638+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:13.114417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.114417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"262'15556","reported_seq":15695,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:02.001919+0000","last_change":"2024-03-12T03:39:34.251603+0000","last_active":"2024-03-12T03:43:02.001919+0000","last_peered":"2024-03-12T03:43:02.001919+0000","last_clean":"2024-03-12T03:43:02.001919+0000","last_became_active":"2024-03-12T03:39:34.251359+0000","last_became_peered":"2024-03-12T03:39:34.251359+0000","last_unstale":"2024-03-12T03:43:02.001919+0000","last_undegraded":"2024-03-12T03:43:02.001919+0000","last_fullsized":"2024-03-12T03:43:02.001919+0000","mapping_epoch":198,"log_start":"240'10800","ondisk_log_start":"240'10800","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4756,"log_dups_size":0,"ondisk_log_size":4756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86638592,"num_objects":1322,"num_object_clones":0,"num_object_copies":3966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15556,"num_write_kb":540096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":562,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.027993+0000","last_change":"2024-03-12T03:41:04.383619+0000","last_active":"2024-03-12T03:43:01.027993+0000","last_peered":"2024-03-12T03:43:01.027993+0000","last_clean":"2024-03-12T03:43:01.027993+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:43:01.027993+0000","last_undegraded":"2024-03-12T03:43:01.027993+0000","last_fullsized":"2024-03-12T03:43:01.027993+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:41:04.383505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:41:04.383505+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.417052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":252,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.034370+0000","last_change":"2024-03-12T03:43:00.715506+0000","last_active":"2024-03-12T03:43:01.034370+0000","last_peered":"2024-03-12T03:43:01.034370+0000","last_clean":"2024-03-12T03:43:01.034370+0000","last_became_active":"2024-03-12T03:40:46.531312+0000","last_became_peered":"2024-03-12T03:40:46.531312+0000","last_unstale":"2024-03-12T03:43:01.034370+0000","last_undegraded":"2024-03-12T03:43:01.034370+0000","last_fullsized":"2024-03-12T03:43:01.034370+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:43:00.715225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.715225+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:18.234070+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":334,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.038298+0000","last_change":"2024-03-12T03:40:46.528324+0000","last_active":"2024-03-12T03:43:01.038298+0000","last_peered":"2024-03-12T03:43:01.038298+0000","last_clean":"2024-03-12T03:43:01.038298+0000","last_became_active":"2024-03-12T03:40:46.528019+0000","last_became_peered":"2024-03-12T03:40:46.528019+0000","last_unstale":"2024-03-12T03:43:01.038298+0000","last_undegraded":"2024-03-12T03:43:01.038298+0000","last_fullsized":"2024-03-12T03:43:01.038298+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:13.030426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:13.030426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":551,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.043890+0000","last_change":"2024-03-12T03:42:04.241287+0000","last_active":"2024-03-12T03:43:00.043890+0000","last_peered":"2024-03-12T03:43:00.043890+0000","last_clean":"2024-03-12T03:43:00.043890+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:43:00.043890+0000","last_undegraded":"2024-03-12T03:43:00.043890+0000","last_fullsized":"2024-03-12T03:43:00.043890+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:42:04.241174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:04.241174+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:21.104873+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":409,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.027911+0000","last_change":"2024-03-12T03:39:18.785112+0000","last_active":"2024-03-12T03:43:01.027911+0000","last_peered":"2024-03-12T03:43:01.027911+0000","last_clean":"2024-03-12T03:43:01.027911+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:43:01.027911+0000","last_undegraded":"2024-03-12T03:43:01.027911+0000","last_fullsized":"2024-03-12T03:43:01.027911+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:18.785001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:18.785001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416702+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"262'15462","reported_seq":29952,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.542862+0000","last_change":"2024-03-12T03:42:29.138644+0000","last_active":"2024-03-12T03:43:01.542862+0000","last_peered":"2024-03-12T03:43:01.542862+0000","last_clean":"2024-03-12T03:43:01.542862+0000","last_became_active":"2024-03-12T03:41:54.317227+0000","last_became_peered":"2024-03-12T03:41:54.317227+0000","last_unstale":"2024-03-12T03:43:01.542862+0000","last_undegraded":"2024-03-12T03:43:01.542862+0000","last_fullsized":"2024-03-12T03:43:01.542862+0000","mapping_epoch":241,"log_start":"238'10000","ondisk_log_start":"238'10000","created":198,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'13236","last_scrub_stamp":"2024-03-12T03:42:29.138471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:42:29.138471+0000","objects_scrubbed":3466,"log_size":5462,"log_dups_size":0,"ondisk_log_size":5462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:58.712288+0000","scrub_duration":808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79298560,"num_objects":1210,"num_object_clones":0,"num_object_copies":3630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15462,"num_write_kb":533504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7147,"num_bytes_recovered":468385792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":428,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042094+0000","last_change":"2024-03-12T03:41:01.229746+0000","last_active":"2024-03-12T03:43:00.042094+0000","last_peered":"2024-03-12T03:43:00.042094+0000","last_clean":"2024-03-12T03:43:00.042094+0000","last_became_active":"2024-03-12T03:40:46.517997+0000","last_became_peered":"2024-03-12T03:40:46.517997+0000","last_unstale":"2024-03-12T03:43:00.042094+0000","last_undegraded":"2024-03-12T03:43:00.042094+0000","last_fullsized":"2024-03-12T03:43:00.042094+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:01.229635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:01.229635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":408,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.038375+0000","last_change":"2024-03-12T03:40:46.532639+0000","last_active":"2024-03-12T03:43:01.038375+0000","last_peered":"2024-03-12T03:43:01.038375+0000","last_clean":"2024-03-12T03:43:01.038375+0000","last_became_active":"2024-03-12T03:40:46.529798+0000","last_became_peered":"2024-03-12T03:40:46.529798+0000","last_unstale":"2024-03-12T03:43:01.038375+0000","last_undegraded":"2024-03-12T03:43:01.038375+0000","last_fullsized":"2024-03-12T03:43:01.038375+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-03-12T03:36:06.083052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.083052+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":37,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":338,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.037044+0000","last_change":"2024-03-12T03:40:46.536598+0000","last_active":"2024-03-12T03:43:01.037044+0000","last_peered":"2024-03-12T03:43:01.037044+0000","last_clean":"2024-03-12T03:43:01.037044+0000","last_became_active":"2024-03-12T03:40:46.534086+0000","last_became_peered":"2024-03-12T03:40:46.534086+0000","last_unstale":"2024-03-12T03:43:01.037044+0000","last_undegraded":"2024-03-12T03:43:01.037044+0000","last_fullsized":"2024-03-12T03:43:01.037044+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:41.918538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:41.918538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":444,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042476+0000","last_change":"2024-03-12T03:41:23.258063+0000","last_active":"2024-03-12T03:43:00.042476+0000","last_peered":"2024-03-12T03:43:00.042476+0000","last_clean":"2024-03-12T03:43:00.042476+0000","last_became_active":"2024-03-12T03:40:46.518478+0000","last_became_peered":"2024-03-12T03:40:46.518478+0000","last_unstale":"2024-03-12T03:43:00.042476+0000","last_undegraded":"2024-03-12T03:43:00.042476+0000","last_fullsized":"2024-03-12T03:43:00.042476+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:23.257942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:23.257942+0000","last_clean_scrub_stamp":"2024-03-12T03:41:23.257942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560390+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":560,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042640+0000","last_change":"2024-03-12T03:40:17.351410+0000","last_active":"2024-03-12T03:43:00.042640+0000","last_peered":"2024-03-12T03:43:00.042640+0000","last_clean":"2024-03-12T03:43:00.042640+0000","last_became_active":"2024-03-12T03:39:36.251411+0000","last_became_peered":"2024-03-12T03:39:36.251411+0000","last_unstale":"2024-03-12T03:43:00.042640+0000","last_undegraded":"2024-03-12T03:43:00.042640+0000","last_fullsized":"2024-03-12T03:43:00.042640+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:40:17.351282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:40:17.351282+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.559980+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"262'15579","reported_seq":15719,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.027859+0000","last_change":"2024-03-12T03:39:34.336488+0000","last_active":"2024-03-12T03:43:01.027859+0000","last_peered":"2024-03-12T03:43:01.027859+0000","last_clean":"2024-03-12T03:43:01.027859+0000","last_became_active":"2024-03-12T03:39:34.336130+0000","last_became_peered":"2024-03-12T03:39:34.336130+0000","last_unstale":"2024-03-12T03:43:01.027859+0000","last_undegraded":"2024-03-12T03:43:01.027859+0000","last_fullsized":"2024-03-12T03:43:01.027859+0000","mapping_epoch":198,"log_start":"240'10574","ondisk_log_start":"240'10574","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89915416,"num_objects":1373,"num_object_clones":0,"num_object_copies":4119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15579,"num_write_kb":542401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"223'10","reported_seq":493,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.038254+0000","last_change":"2024-03-12T03:40:46.523417+0000","last_active":"2024-03-12T03:43:01.038254+0000","last_peered":"2024-03-12T03:43:01.038254+0000","last_clean":"2024-03-12T03:43:01.038254+0000","last_became_active":"2024-03-12T03:40:46.521005+0000","last_became_peered":"2024-03-12T03:40:46.521005+0000","last_unstale":"2024-03-12T03:43:01.038254+0000","last_undegraded":"2024-03-12T03:43:01.038254+0000","last_fullsized":"2024-03-12T03:43:01.038254+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:39:37.039925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:37.039925+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"200'28","reported_seq":472,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042228+0000","last_change":"2024-03-12T03:42:02.272123+0000","last_active":"2024-03-12T03:43:00.042228+0000","last_peered":"2024-03-12T03:43:00.042228+0000","last_clean":"2024-03-12T03:43:00.042228+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:43:00.042228+0000","last_undegraded":"2024-03-12T03:43:00.042228+0000","last_fullsized":"2024-03-12T03:43:00.042228+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'28","last_scrub_stamp":"2024-03-12T03:42:02.271980+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:42:02.271980+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:25.698870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":430,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.038920+0000","last_change":"2024-03-12T03:40:25.553659+0000","last_active":"2024-03-12T03:43:01.038920+0000","last_peered":"2024-03-12T03:43:01.038920+0000","last_clean":"2024-03-12T03:43:01.038920+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:43:01.038920+0000","last_undegraded":"2024-03-12T03:43:01.038920+0000","last_fullsized":"2024-03-12T03:43:01.038920+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:40:25.553541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:40:25.553541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.642029+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":437,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.030768+0000","last_change":"2024-03-12T03:42:04.017533+0000","last_active":"2024-03-12T03:43:01.030768+0000","last_peered":"2024-03-12T03:43:01.030768+0000","last_clean":"2024-03-12T03:43:01.030768+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:43:01.030768+0000","last_undegraded":"2024-03-12T03:43:01.030768+0000","last_fullsized":"2024-03-12T03:43:01.030768+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:04.017409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:04.017409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:22.499259+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":515,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:02.443223+0000","last_change":"2024-03-12T03:43:02.443358+0000","last_active":"2024-03-12T03:43:02.443223+0000","last_peered":"2024-03-12T03:43:02.443223+0000","last_clean":"2024-03-12T03:43:02.443223+0000","last_became_active":"2024-03-12T03:40:46.515554+0000","last_became_peered":"2024-03-12T03:40:46.515554+0000","last_unstale":"2024-03-12T03:43:02.443223+0000","last_undegraded":"2024-03-12T03:43:02.443223+0000","last_fullsized":"2024-03-12T03:43:02.443223+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:02.442717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:02.442717+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:05.251660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"262'15336","reported_seq":15475,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.563611+0000","last_change":"2024-03-12T03:39:34.337083+0000","last_active":"2024-03-12T03:43:01.563611+0000","last_peered":"2024-03-12T03:43:01.563611+0000","last_clean":"2024-03-12T03:43:01.563611+0000","last_became_active":"2024-03-12T03:39:34.336765+0000","last_became_peered":"2024-03-12T03:39:34.336765+0000","last_unstale":"2024-03-12T03:43:01.563611+0000","last_undegraded":"2024-03-12T03:43:01.563611+0000","last_fullsized":"2024-03-12T03:43:01.563611+0000","mapping_epoch":198,"log_start":"237'9800","ondisk_log_start":"237'9800","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":5536,"log_dups_size":0,"ondisk_log_size":5536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79429632,"num_objects":1212,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15336,"num_write_kb":529536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":545,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.038035+0000","last_change":"2024-03-12T03:39:23.514494+0000","last_active":"2024-03-12T03:43:01.038035+0000","last_peered":"2024-03-12T03:43:01.038035+0000","last_clean":"2024-03-12T03:43:01.038035+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:43:01.038035+0000","last_undegraded":"2024-03-12T03:43:01.038035+0000","last_fullsized":"2024-03-12T03:43:01.038035+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:39:23.514401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:23.514401+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":460,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.034887+0000","last_change":"2024-03-12T03:39:30.805994+0000","last_active":"2024-03-12T03:43:01.034887+0000","last_peered":"2024-03-12T03:43:01.034887+0000","last_clean":"2024-03-12T03:43:01.034887+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:43:01.034887+0000","last_undegraded":"2024-03-12T03:43:01.034887+0000","last_fullsized":"2024-03-12T03:43:01.034887+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:39:30.805899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:30.805899+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":450,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.027977+0000","last_change":"2024-03-12T03:40:46.520387+0000","last_active":"2024-03-12T03:43:01.027977+0000","last_peered":"2024-03-12T03:43:01.027977+0000","last_clean":"2024-03-12T03:43:01.027977+0000","last_became_active":"2024-03-12T03:40:46.520076+0000","last_became_peered":"2024-03-12T03:40:46.520076+0000","last_unstale":"2024-03-12T03:43:01.027977+0000","last_undegraded":"2024-03-12T03:43:01.027977+0000","last_fullsized":"2024-03-12T03:43:01.027977+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:34.674570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:39:34.674570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":438,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.037343+0000","last_change":"2024-03-12T03:40:46.529186+0000","last_active":"2024-03-12T03:43:01.037343+0000","last_peered":"2024-03-12T03:43:01.037343+0000","last_clean":"2024-03-12T03:43:01.037343+0000","last_became_active":"2024-03-12T03:40:46.528691+0000","last_became_peered":"2024-03-12T03:40:46.528691+0000","last_unstale":"2024-03-12T03:43:01.037343+0000","last_undegraded":"2024-03-12T03:43:01.037343+0000","last_fullsized":"2024-03-12T03:43:01.037343+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:40.532991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:40.532991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641844+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":559,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.033617+0000","last_change":"2024-03-12T03:40:57.984669+0000","last_active":"2024-03-12T03:43:01.033617+0000","last_peered":"2024-03-12T03:43:01.033617+0000","last_clean":"2024-03-12T03:43:01.033617+0000","last_became_active":"2024-03-12T03:40:46.532266+0000","last_became_peered":"2024-03-12T03:40:46.532266+0000","last_unstale":"2024-03-12T03:43:01.033617+0000","last_undegraded":"2024-03-12T03:43:01.033617+0000","last_fullsized":"2024-03-12T03:43:01.033617+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:40:57.984538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:40:57.984538+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.f","version":"262'15529","reported_seq":29578,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:04.652478+0000","last_change":"2024-03-12T03:41:55.356589+0000","last_active":"2024-03-12T03:43:04.652478+0000","last_peered":"2024-03-12T03:43:04.652478+0000","last_clean":"2024-03-12T03:43:04.652478+0000","last_became_active":"2024-03-12T03:41:55.318802+0000","last_became_peered":"2024-03-12T03:41:55.318802+0000","last_unstale":"2024-03-12T03:43:04.652478+0000","last_undegraded":"2024-03-12T03:43:04.652478+0000","last_fullsized":"2024-03-12T03:43:04.652478+0000","mapping_epoch":242,"log_start":"243'10975","ondisk_log_start":"243'10975","created":198,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4554,"log_dups_size":0,"ondisk_log_size":4554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71106560,"num_objects":1085,"num_object_clones":0,"num_object_copies":3255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15529,"num_write_kb":531648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6957,"num_bytes_recovered":455933952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":436,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.030216+0000","last_change":"2024-03-12T03:42:02.263983+0000","last_active":"2024-03-12T03:43:01.030216+0000","last_peered":"2024-03-12T03:43:01.030216+0000","last_clean":"2024-03-12T03:43:01.030216+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:43:01.030216+0000","last_undegraded":"2024-03-12T03:43:01.030216+0000","last_fullsized":"2024-03-12T03:43:01.030216+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:02.263808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:42:02.263808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:22.666688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":473,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.038474+0000","last_change":"2024-03-12T03:39:36.550428+0000","last_active":"2024-03-12T03:43:01.038474+0000","last_peered":"2024-03-12T03:43:01.038474+0000","last_clean":"2024-03-12T03:43:01.038474+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:43:01.038474+0000","last_undegraded":"2024-03-12T03:43:01.038474+0000","last_fullsized":"2024-03-12T03:43:01.038474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:39:36.550291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:36.550291+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641907+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":457,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.034054+0000","last_change":"2024-03-12T03:40:32.631452+0000","last_active":"2024-03-12T03:43:01.034054+0000","last_peered":"2024-03-12T03:43:01.034054+0000","last_clean":"2024-03-12T03:43:01.034054+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:43:01.034054+0000","last_undegraded":"2024-03-12T03:43:01.034054+0000","last_fullsized":"2024-03-12T03:43:01.034054+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:40:32.631336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":437,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.035187+0000","last_change":"2024-03-12T03:39:39.850426+0000","last_active":"2024-03-12T03:43:01.035187+0000","last_peered":"2024-03-12T03:43:01.035187+0000","last_clean":"2024-03-12T03:43:01.035187+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:43:01.035187+0000","last_undegraded":"2024-03-12T03:43:01.035187+0000","last_fullsized":"2024-03-12T03:43:01.035187+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:39.850300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:39.850300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":567,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.031003+0000","last_change":"2024-03-12T03:42:10.356344+0000","last_active":"2024-03-12T03:43:01.031003+0000","last_peered":"2024-03-12T03:43:01.031003+0000","last_clean":"2024-03-12T03:43:01.031003+0000","last_became_active":"2024-03-12T03:40:46.513921+0000","last_became_peered":"2024-03-12T03:40:46.513921+0000","last_unstale":"2024-03-12T03:43:01.031003+0000","last_undegraded":"2024-03-12T03:43:01.031003+0000","last_fullsized":"2024-03-12T03:43:01.031003+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:10.356260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:10.356260+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:13.781974+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"262'15426","reported_seq":15565,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:02.010883+0000","last_change":"2024-03-12T03:39:34.339192+0000","last_active":"2024-03-12T03:43:02.010883+0000","last_peered":"2024-03-12T03:43:02.010883+0000","last_clean":"2024-03-12T03:43:02.010883+0000","last_became_active":"2024-03-12T03:39:34.338678+0000","last_became_peered":"2024-03-12T03:39:34.338678+0000","last_unstale":"2024-03-12T03:43:02.010883+0000","last_undegraded":"2024-03-12T03:43:02.010883+0000","last_fullsized":"2024-03-12T03:43:02.010883+0000","mapping_epoch":198,"log_start":"240'10600","ondisk_log_start":"240'10600","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4826,"log_dups_size":0,"ondisk_log_size":4826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82182144,"num_objects":1254,"num_object_clones":0,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15426,"num_write_kb":533760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":421,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.028475+0000","last_change":"2024-03-12T03:40:22.791790+0000","last_active":"2024-03-12T03:43:01.028475+0000","last_peered":"2024-03-12T03:43:01.028475+0000","last_clean":"2024-03-12T03:43:01.028475+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:43:01.028475+0000","last_undegraded":"2024-03-12T03:43:01.028475+0000","last_fullsized":"2024-03-12T03:43:01.028475+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:22.791656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:40:22.791656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416896+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":483,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.038819+0000","last_change":"2024-03-12T03:39:35.570165+0000","last_active":"2024-03-12T03:43:01.038819+0000","last_peered":"2024-03-12T03:43:01.038819+0000","last_clean":"2024-03-12T03:43:01.038819+0000","last_became_active":"2024-03-12T03:39:33.277868+0000","last_became_peered":"2024-03-12T03:39:33.277868+0000","last_unstale":"2024-03-12T03:43:01.038819+0000","last_undegraded":"2024-03-12T03:43:01.038819+0000","last_fullsized":"2024-03-12T03:43:01.038819+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:39:35.570046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:35.570046+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1027,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:03.978739+0000","last_change":"2024-03-12T03:41:59.023320+0000","last_active":"2024-03-12T03:43:03.978739+0000","last_peered":"2024-03-12T03:43:03.978739+0000","last_clean":"2024-03-12T03:43:03.978739+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:43:03.978739+0000","last_undegraded":"2024-03-12T03:43:03.978739+0000","last_fullsized":"2024-03-12T03:43:03.978739+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:41:59.023235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:41:59.023235+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:18.542939+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":578,"num_read_kb":573,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":438,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.042155+0000","last_change":"2024-03-12T03:42:21.445884+0000","last_active":"2024-03-12T03:43:01.042155+0000","last_peered":"2024-03-12T03:43:01.042155+0000","last_clean":"2024-03-12T03:43:01.042155+0000","last_became_active":"2024-03-12T03:40:46.530227+0000","last_became_peered":"2024-03-12T03:40:46.530227+0000","last_unstale":"2024-03-12T03:43:01.042155+0000","last_undegraded":"2024-03-12T03:43:01.042155+0000","last_fullsized":"2024-03-12T03:43:01.042155+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:42:21.445443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:21.629219+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":434,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.033873+0000","last_change":"2024-03-12T03:40:46.524410+0000","last_active":"2024-03-12T03:43:01.033873+0000","last_peered":"2024-03-12T03:43:01.033873+0000","last_clean":"2024-03-12T03:43:01.033873+0000","last_became_active":"2024-03-12T03:40:46.524067+0000","last_became_peered":"2024-03-12T03:40:46.524067+0000","last_unstale":"2024-03-12T03:43:01.033873+0000","last_undegraded":"2024-03-12T03:43:01.033873+0000","last_fullsized":"2024-03-12T03:43:01.033873+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:31.807586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:31.807586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":545,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.036254+0000","last_change":"2024-03-12T03:39:24.675700+0000","last_active":"2024-03-12T03:43:01.036254+0000","last_peered":"2024-03-12T03:43:01.036254+0000","last_clean":"2024-03-12T03:43:01.036254+0000","last_became_active":"2024-03-12T03:38:08.891939+0000","last_became_peered":"2024-03-12T03:38:08.891939+0000","last_unstale":"2024-03-12T03:43:01.036254+0000","last_undegraded":"2024-03-12T03:43:01.036254+0000","last_fullsized":"2024-03-12T03:43:01.036254+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:39:24.675620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:39:24.675620+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"262'15583","reported_seq":15804,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.190925+0000","last_change":"2024-03-12T03:41:01.317901+0000","last_active":"2024-03-12T03:43:01.190925+0000","last_peered":"2024-03-12T03:43:01.190925+0000","last_clean":"2024-03-12T03:43:01.190925+0000","last_became_active":"2024-03-12T03:39:34.249591+0000","last_became_peered":"2024-03-12T03:39:34.249591+0000","last_unstale":"2024-03-12T03:43:01.190925+0000","last_undegraded":"2024-03-12T03:43:01.190925+0000","last_fullsized":"2024-03-12T03:43:01.190925+0000","mapping_epoch":198,"log_start":"236'9900","ondisk_log_start":"236'9900","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'7543","last_scrub_stamp":"2024-03-12T03:41:01.317614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:41:01.317614+0000","objects_scrubbed":7464,"log_size":5683,"log_dups_size":0,"ondisk_log_size":5683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86704128,"num_objects":1323,"num_object_clones":0,"num_object_copies":3969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15583,"num_write_kb":540992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":433,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.031228+0000","last_change":"2024-03-12T03:41:08.176405+0000","last_active":"2024-03-12T03:43:01.031228+0000","last_peered":"2024-03-12T03:43:01.031228+0000","last_clean":"2024-03-12T03:43:01.031228+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:43:01.031228+0000","last_undegraded":"2024-03-12T03:43:01.031228+0000","last_fullsized":"2024-03-12T03:43:01.031228+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:08.176270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:08.176270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:04.266464+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":570,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.029929+0000","last_change":"2024-03-12T03:40:59.996399+0000","last_active":"2024-03-12T03:43:01.029929+0000","last_peered":"2024-03-12T03:43:01.029929+0000","last_clean":"2024-03-12T03:43:01.029929+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:43:01.029929+0000","last_undegraded":"2024-03-12T03:43:01.029929+0000","last_fullsized":"2024-03-12T03:43:01.029929+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:40:59.996258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:59.996258+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":334,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032097+0000","last_change":"2024-03-12T03:40:46.520753+0000","last_active":"2024-03-12T03:43:01.032097+0000","last_peered":"2024-03-12T03:43:01.032097+0000","last_clean":"2024-03-12T03:43:01.032097+0000","last_became_active":"2024-03-12T03:40:46.516086+0000","last_became_peered":"2024-03-12T03:40:46.516086+0000","last_unstale":"2024-03-12T03:43:01.032097+0000","last_undegraded":"2024-03-12T03:43:01.032097+0000","last_fullsized":"2024-03-12T03:43:01.032097+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:14.996357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:14.996357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":702,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:05.222953+0000","last_change":"2024-03-12T03:43:05.222995+0000","last_active":"2024-03-12T03:43:05.222953+0000","last_peered":"2024-03-12T03:43:05.222953+0000","last_clean":"2024-03-12T03:43:05.222953+0000","last_became_active":"2024-03-12T03:38:03.852040+0000","last_became_peered":"2024-03-12T03:38:03.852040+0000","last_unstale":"2024-03-12T03:43:05.222953+0000","last_undegraded":"2024-03-12T03:43:05.222953+0000","last_fullsized":"2024-03-12T03:43:05.222953+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:05.222876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:05.222876+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:11.274990+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":426,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.028109+0000","last_change":"2024-03-12T03:41:00.436835+0000","last_active":"2024-03-12T03:43:01.028109+0000","last_peered":"2024-03-12T03:43:01.028109+0000","last_clean":"2024-03-12T03:43:01.028109+0000","last_became_active":"2024-03-12T03:40:46.529802+0000","last_became_peered":"2024-03-12T03:40:46.529802+0000","last_unstale":"2024-03-12T03:43:01.028109+0000","last_undegraded":"2024-03-12T03:43:01.028109+0000","last_fullsized":"2024-03-12T03:43:01.028109+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:00.436718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:00.436718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416988+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"262'15422","reported_seq":29141,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:04.629064+0000","last_change":"2024-03-12T03:41:59.404791+0000","last_active":"2024-03-12T03:43:04.629064+0000","last_peered":"2024-03-12T03:43:04.629064+0000","last_clean":"2024-03-12T03:43:04.629064+0000","last_became_active":"2024-03-12T03:41:59.378669+0000","last_became_peered":"2024-03-12T03:41:59.378669+0000","last_unstale":"2024-03-12T03:43:04.629064+0000","last_undegraded":"2024-03-12T03:43:04.629064+0000","last_fullsized":"2024-03-12T03:43:04.629064+0000","mapping_epoch":245,"log_start":"244'10911","ondisk_log_start":"244'10911","created":198,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4511,"log_dups_size":0,"ondisk_log_size":4511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69206016,"num_objects":1056,"num_object_clones":0,"num_object_copies":3168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15422,"num_write_kb":527296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6791,"num_bytes_recovered":445054976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":156,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032277+0000","last_change":"2024-03-12T03:42:18.272709+0000","last_active":"2024-03-12T03:43:01.032277+0000","last_peered":"2024-03-12T03:43:01.032277+0000","last_clean":"2024-03-12T03:43:01.032277+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:43:01.032277+0000","last_undegraded":"2024-03-12T03:43:01.032277+0000","last_fullsized":"2024-03-12T03:43:01.032277+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:18.272591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:18.272591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:33.114854+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":472,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.034217+0000","last_change":"2024-03-12T03:40:18.873418+0000","last_active":"2024-03-12T03:43:01.034217+0000","last_peered":"2024-03-12T03:43:01.034217+0000","last_clean":"2024-03-12T03:43:01.034217+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:43:01.034217+0000","last_undegraded":"2024-03-12T03:43:01.034217+0000","last_fullsized":"2024-03-12T03:43:01.034217+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-12T03:40:18.873281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:18.873281+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":157,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.033276+0000","last_change":"2024-03-12T03:42:01.190049+0000","last_active":"2024-03-12T03:43:01.033276+0000","last_peered":"2024-03-12T03:43:01.033276+0000","last_clean":"2024-03-12T03:43:01.033276+0000","last_became_active":"2024-03-12T03:40:46.517899+0000","last_became_peered":"2024-03-12T03:40:46.517899+0000","last_unstale":"2024-03-12T03:43:01.033276+0000","last_undegraded":"2024-03-12T03:43:01.033276+0000","last_fullsized":"2024-03-12T03:43:01.033276+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:01.189842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:42:01.189842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:29.372176+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":449,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.035583+0000","last_change":"2024-03-12T03:40:22.866720+0000","last_active":"2024-03-12T03:43:01.035583+0000","last_peered":"2024-03-12T03:43:01.035583+0000","last_clean":"2024-03-12T03:43:01.035583+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:43:01.035583+0000","last_undegraded":"2024-03-12T03:43:01.035583+0000","last_fullsized":"2024-03-12T03:43:01.035583+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:22.866591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:38:36.598179+0000","last_clean_scrub_stamp":"2024-03-12T03:40:22.866591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":436,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.031987+0000","last_change":"2024-03-12T03:42:20.203864+0000","last_active":"2024-03-12T03:43:01.031987+0000","last_peered":"2024-03-12T03:43:01.031987+0000","last_clean":"2024-03-12T03:43:01.031987+0000","last_became_active":"2024-03-12T03:40:46.518250+0000","last_became_peered":"2024-03-12T03:40:46.518250+0000","last_unstale":"2024-03-12T03:43:01.031987+0000","last_undegraded":"2024-03-12T03:43:01.031987+0000","last_fullsized":"2024-03-12T03:43:01.031987+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:42:20.203740+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:42:20.203740+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:31.324197+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":710,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.031726+0000","last_change":"2024-03-12T03:42:17.149648+0000","last_active":"2024-03-12T03:43:01.031726+0000","last_peered":"2024-03-12T03:43:01.031726+0000","last_clean":"2024-03-12T03:43:01.031726+0000","last_became_active":"2024-03-12T03:38:08.893327+0000","last_became_peered":"2024-03-12T03:38:08.893327+0000","last_unstale":"2024-03-12T03:43:01.031726+0000","last_undegraded":"2024-03-12T03:43:01.031726+0000","last_fullsized":"2024-03-12T03:43:01.031726+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:17.149524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:17.149524+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:32.117071+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"262'15333","reported_seq":24840,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.046273+0000","last_change":"2024-03-12T03:42:28.730631+0000","last_active":"2024-03-12T03:43:01.046273+0000","last_peered":"2024-03-12T03:43:01.046273+0000","last_clean":"2024-03-12T03:43:01.046273+0000","last_became_active":"2024-03-12T03:42:28.694184+0000","last_became_peered":"2024-03-12T03:42:28.694184+0000","last_unstale":"2024-03-12T03:43:01.046273+0000","last_undegraded":"2024-03-12T03:43:01.046273+0000","last_fullsized":"2024-03-12T03:43:01.046273+0000","mapping_epoch":252,"log_start":"240'10338","ondisk_log_start":"240'10338","created":198,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4995,"log_dups_size":0,"ondisk_log_size":4995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86179840,"num_objects":1315,"num_object_clones":0,"num_object_copies":3945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15333,"num_write_kb":532736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4675,"num_bytes_recovered":306380800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":421,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.038775+0000","last_change":"2024-03-12T03:39:16.577317+0000","last_active":"2024-03-12T03:43:01.038775+0000","last_peered":"2024-03-12T03:43:01.038775+0000","last_clean":"2024-03-12T03:43:01.038775+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:43:01.038775+0000","last_undegraded":"2024-03-12T03:43:01.038775+0000","last_fullsized":"2024-03-12T03:43:01.038775+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:16.577211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:16.577211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"223'39","reported_seq":602,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032523+0000","last_change":"2024-03-12T03:40:46.525110+0000","last_active":"2024-03-12T03:43:01.032523+0000","last_peered":"2024-03-12T03:43:01.032523+0000","last_clean":"2024-03-12T03:43:01.032523+0000","last_became_active":"2024-03-12T03:40:46.524632+0000","last_became_peered":"2024-03-12T03:40:46.524632+0000","last_unstale":"2024-03-12T03:43:01.032523+0000","last_undegraded":"2024-03-12T03:43:01.032523+0000","last_fullsized":"2024-03-12T03:43:01.032523+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'36","last_scrub_stamp":"2024-03-12T03:38:39.557592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:38:39.557592+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":435,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042535+0000","last_change":"2024-03-12T03:41:26.257807+0000","last_active":"2024-03-12T03:43:00.042535+0000","last_peered":"2024-03-12T03:43:00.042535+0000","last_clean":"2024-03-12T03:43:00.042535+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:43:00.042535+0000","last_undegraded":"2024-03-12T03:43:00.042535+0000","last_fullsized":"2024-03-12T03:43:00.042535+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:26.257679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:41:26.257679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":440,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.028316+0000","last_change":"2024-03-12T03:40:16.791859+0000","last_active":"2024-03-12T03:43:01.028316+0000","last_peered":"2024-03-12T03:43:01.028316+0000","last_clean":"2024-03-12T03:43:01.028316+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:43:01.028316+0000","last_undegraded":"2024-03-12T03:43:01.028316+0000","last_fullsized":"2024-03-12T03:43:01.028316+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:40:16.791364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:40:16.791364+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416880+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":377,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032824+0000","last_change":"2024-03-12T03:40:46.520723+0000","last_active":"2024-03-12T03:43:01.032824+0000","last_peered":"2024-03-12T03:43:01.032824+0000","last_clean":"2024-03-12T03:43:01.032824+0000","last_became_active":"2024-03-12T03:40:46.517562+0000","last_became_peered":"2024-03-12T03:40:46.517562+0000","last_unstale":"2024-03-12T03:43:01.032824+0000","last_undegraded":"2024-03-12T03:43:01.032824+0000","last_fullsized":"2024-03-12T03:43:01.032824+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:39.031166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:39.031166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":562,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.027495+0000","last_change":"2024-03-12T03:42:28.720781+0000","last_active":"2024-03-12T03:43:01.027495+0000","last_peered":"2024-03-12T03:43:01.027495+0000","last_clean":"2024-03-12T03:43:01.027495+0000","last_became_active":"2024-03-12T03:39:36.252545+0000","last_became_peered":"2024-03-12T03:39:36.252545+0000","last_unstale":"2024-03-12T03:43:01.027495+0000","last_undegraded":"2024-03-12T03:43:01.027495+0000","last_fullsized":"2024-03-12T03:43:01.027495+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:28.720248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:28.720248+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:50.426883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"262'15428","reported_seq":24633,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:02.009776+0000","last_change":"2024-03-12T03:42:30.739974+0000","last_active":"2024-03-12T03:43:02.009776+0000","last_peered":"2024-03-12T03:43:02.009776+0000","last_clean":"2024-03-12T03:43:02.009776+0000","last_became_active":"2024-03-12T03:42:30.713416+0000","last_became_peered":"2024-03-12T03:42:30.713416+0000","last_unstale":"2024-03-12T03:43:02.009776+0000","last_undegraded":"2024-03-12T03:43:02.009776+0000","last_fullsized":"2024-03-12T03:43:02.009776+0000","mapping_epoch":254,"log_start":"240'10600","ondisk_log_start":"240'10600","created":198,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4828,"log_dups_size":0,"ondisk_log_size":4828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81002496,"num_objects":1236,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15428,"num_write_kb":533248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4526,"num_bytes_recovered":296615936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"111'20","reported_seq":458,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.038182+0000","last_change":"2024-03-12T03:39:39.540698+0000","last_active":"2024-03-12T03:43:01.038182+0000","last_peered":"2024-03-12T03:43:01.038182+0000","last_clean":"2024-03-12T03:43:01.038182+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:43:01.038182+0000","last_undegraded":"2024-03-12T03:43:01.038182+0000","last_fullsized":"2024-03-12T03:43:01.038182+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'20","last_scrub_stamp":"2024-03-12T03:39:39.540585+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:39:39.540585+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641970+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":438,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032812+0000","last_change":"2024-03-12T03:40:59.344158+0000","last_active":"2024-03-12T03:43:01.032812+0000","last_peered":"2024-03-12T03:43:01.032812+0000","last_clean":"2024-03-12T03:43:01.032812+0000","last_became_active":"2024-03-12T03:40:46.524544+0000","last_became_peered":"2024-03-12T03:40:46.524544+0000","last_unstale":"2024-03-12T03:43:01.032812+0000","last_undegraded":"2024-03-12T03:43:01.032812+0000","last_fullsized":"2024-03-12T03:43:01.032812+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:59.344058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:40:59.344058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.417020+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":157,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032920+0000","last_change":"2024-03-12T03:42:03.191515+0000","last_active":"2024-03-12T03:43:01.032920+0000","last_peered":"2024-03-12T03:43:01.032920+0000","last_clean":"2024-03-12T03:43:01.032920+0000","last_became_active":"2024-03-12T03:40:46.521976+0000","last_became_peered":"2024-03-12T03:40:46.521976+0000","last_unstale":"2024-03-12T03:43:01.032920+0000","last_undegraded":"2024-03-12T03:43:01.032920+0000","last_fullsized":"2024-03-12T03:43:01.032920+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:03.191398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:03.191398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:21.532703+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":429,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.042366+0000","last_change":"2024-03-12T03:40:19.489022+0000","last_active":"2024-03-12T03:43:01.042366+0000","last_peered":"2024-03-12T03:43:01.042366+0000","last_clean":"2024-03-12T03:43:01.042366+0000","last_became_active":"2024-03-12T03:39:33.271016+0000","last_became_peered":"2024-03-12T03:39:33.271016+0000","last_unstale":"2024-03-12T03:43:01.042366+0000","last_undegraded":"2024-03-12T03:43:01.042366+0000","last_fullsized":"2024-03-12T03:43:01.042366+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:19.488908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:40:19.488908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641998+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"262'15439","reported_seq":15578,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.542006+0000","last_change":"2024-03-12T03:39:34.250643+0000","last_active":"2024-03-12T03:43:01.542006+0000","last_peered":"2024-03-12T03:43:01.542006+0000","last_clean":"2024-03-12T03:43:01.542006+0000","last_became_active":"2024-03-12T03:39:34.250450+0000","last_became_peered":"2024-03-12T03:39:34.250450+0000","last_unstale":"2024-03-12T03:43:01.542006+0000","last_undegraded":"2024-03-12T03:43:01.542006+0000","last_fullsized":"2024-03-12T03:43:01.542006+0000","mapping_epoch":198,"log_start":"238'9900","ondisk_log_start":"238'9900","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":5539,"log_dups_size":0,"ondisk_log_size":5539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80805888,"num_objects":1233,"num_object_clones":0,"num_object_copies":3699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15439,"num_write_kb":533504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"195'35","reported_seq":493,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.030549+0000","last_change":"2024-03-12T03:42:15.396790+0000","last_active":"2024-03-12T03:43:01.030549+0000","last_peered":"2024-03-12T03:43:01.030549+0000","last_clean":"2024-03-12T03:43:01.030549+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:43:01.030549+0000","last_undegraded":"2024-03-12T03:43:01.030549+0000","last_fullsized":"2024-03-12T03:43:01.030549+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'35","last_scrub_stamp":"2024-03-12T03:42:15.396647+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:42:15.396647+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:31.094343+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":437,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.031511+0000","last_change":"2024-03-12T03:42:15.159053+0000","last_active":"2024-03-12T03:43:01.031511+0000","last_peered":"2024-03-12T03:43:01.031511+0000","last_clean":"2024-03-12T03:43:01.031511+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:43:01.031511+0000","last_undegraded":"2024-03-12T03:43:01.031511+0000","last_fullsized":"2024-03-12T03:43:01.031511+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:15.158916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:15.158916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:41.586588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":148,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032281+0000","last_change":"2024-03-12T03:42:11.155750+0000","last_active":"2024-03-12T03:43:01.032281+0000","last_peered":"2024-03-12T03:43:01.032281+0000","last_clean":"2024-03-12T03:43:01.032281+0000","last_became_active":"2024-03-12T03:39:33.254395+0000","last_became_peered":"2024-03-12T03:39:33.254395+0000","last_unstale":"2024-03-12T03:43:01.032281+0000","last_undegraded":"2024-03-12T03:43:01.032281+0000","last_fullsized":"2024-03-12T03:43:01.032281+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:11.155639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:11.155639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:38.130653+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":438,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.029902+0000","last_change":"2024-03-12T03:41:23.042150+0000","last_active":"2024-03-12T03:43:01.029902+0000","last_peered":"2024-03-12T03:43:01.029902+0000","last_clean":"2024-03-12T03:43:01.029902+0000","last_became_active":"2024-03-12T03:40:46.521431+0000","last_became_peered":"2024-03-12T03:40:46.521431+0000","last_unstale":"2024-03-12T03:43:01.029902+0000","last_undegraded":"2024-03-12T03:43:01.029902+0000","last_fullsized":"2024-03-12T03:43:01.029902+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:23.042030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:23.042030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:04.266382+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"262'19002","reported_seq":19384,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.158377+0000","last_change":"2024-03-12T03:42:39.697501+0000","last_active":"2024-03-12T03:43:01.158377+0000","last_peered":"2024-03-12T03:43:01.158377+0000","last_clean":"2024-03-12T03:43:01.158377+0000","last_became_active":"2024-03-12T03:39:34.250817+0000","last_became_peered":"2024-03-12T03:39:34.250817+0000","last_unstale":"2024-03-12T03:43:01.158377+0000","last_undegraded":"2024-03-12T03:43:01.158377+0000","last_fullsized":"2024-03-12T03:43:01.158377+0000","mapping_epoch":198,"log_start":"251'13300","ondisk_log_start":"251'13300","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'17360","last_scrub_stamp":"2024-03-12T03:42:39.697349+0000","last_deep_scrub":"257'17360","last_deep_scrub_stamp":"2024-03-12T03:42:39.697349+0000","last_clean_scrub_stamp":"2024-03-12T03:42:39.697349+0000","objects_scrubbed":3503,"log_size":5702,"log_dups_size":0,"ondisk_log_size":5702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":18,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:55.604835+0000","scrub_duration":17365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86966272,"num_objects":1327,"num_object_clones":0,"num_object_copies":3981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15499,"num_write_kb":538432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":156,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032051+0000","last_change":"2024-03-12T03:42:24.169808+0000","last_active":"2024-03-12T03:43:01.032051+0000","last_peered":"2024-03-12T03:43:01.032051+0000","last_clean":"2024-03-12T03:43:01.032051+0000","last_became_active":"2024-03-12T03:39:33.278326+0000","last_became_peered":"2024-03-12T03:39:33.278326+0000","last_unstale":"2024-03-12T03:43:01.032051+0000","last_undegraded":"2024-03-12T03:43:01.032051+0000","last_fullsized":"2024-03-12T03:43:01.032051+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:24.169673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:24.169673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:50.702592+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":479,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042283+0000","last_change":"2024-03-12T03:41:15.312988+0000","last_active":"2024-03-12T03:43:00.042283+0000","last_peered":"2024-03-12T03:43:00.042283+0000","last_clean":"2024-03-12T03:43:00.042283+0000","last_became_active":"2024-03-12T03:40:46.518005+0000","last_became_peered":"2024-03-12T03:40:46.518005+0000","last_unstale":"2024-03-12T03:43:00.042283+0000","last_undegraded":"2024-03-12T03:43:00.042283+0000","last_fullsized":"2024-03-12T03:43:00.042283+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-03-12T03:41:15.312809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:41:15.312809+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560144+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":437,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.031418+0000","last_change":"2024-03-12T03:42:09.072702+0000","last_active":"2024-03-12T03:43:01.031418+0000","last_peered":"2024-03-12T03:43:01.031418+0000","last_clean":"2024-03-12T03:43:01.031418+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:43:01.031418+0000","last_undegraded":"2024-03-12T03:43:01.031418+0000","last_fullsized":"2024-03-12T03:43:01.031418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:09.072495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:42:09.072495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:15.928821+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":292,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032178+0000","last_change":"2024-03-12T03:40:46.528204+0000","last_active":"2024-03-12T03:43:01.032178+0000","last_peered":"2024-03-12T03:43:01.032178+0000","last_clean":"2024-03-12T03:43:01.032178+0000","last_became_active":"2024-03-12T03:40:46.525594+0000","last_became_peered":"2024-03-12T03:40:46.525594+0000","last_unstale":"2024-03-12T03:43:01.032178+0000","last_undegraded":"2024-03-12T03:43:01.032178+0000","last_fullsized":"2024-03-12T03:43:01.032178+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:18.651418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:18.651418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"262'15392","reported_seq":15531,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.055098+0000","last_change":"2024-03-12T03:39:34.336572+0000","last_active":"2024-03-12T03:43:01.055098+0000","last_peered":"2024-03-12T03:43:01.055098+0000","last_clean":"2024-03-12T03:43:01.055098+0000","last_became_active":"2024-03-12T03:39:34.336260+0000","last_became_peered":"2024-03-12T03:39:34.336260+0000","last_unstale":"2024-03-12T03:43:01.055098+0000","last_undegraded":"2024-03-12T03:43:01.055098+0000","last_fullsized":"2024-03-12T03:43:01.055098+0000","mapping_epoch":198,"log_start":"240'10400","ondisk_log_start":"240'10400","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.235465+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.235465+0000","objects_scrubbed":0,"log_size":4992,"log_dups_size":0,"ondisk_log_size":4992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84541440,"num_objects":1290,"num_object_clones":0,"num_object_copies":3870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15392,"num_write_kb":533824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":546,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.030943+0000","last_change":"2024-03-12T03:41:17.027767+0000","last_active":"2024-03-12T03:43:01.030943+0000","last_peered":"2024-03-12T03:43:01.030943+0000","last_clean":"2024-03-12T03:43:01.030943+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:43:01.030943+0000","last_undegraded":"2024-03-12T03:43:01.030943+0000","last_fullsized":"2024-03-12T03:43:01.030943+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:41:17.027646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:17.027646+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:04.266437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"90'36","reported_seq":483,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032544+0000","last_change":"2024-03-12T03:40:14.776197+0000","last_active":"2024-03-12T03:43:01.032544+0000","last_peered":"2024-03-12T03:43:01.032544+0000","last_clean":"2024-03-12T03:43:01.032544+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:43:01.032544+0000","last_undegraded":"2024-03-12T03:43:01.032544+0000","last_fullsized":"2024-03-12T03:43:01.032544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:40:14.776063+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:40:14.776063+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416822+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":441,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.031368+0000","last_change":"2024-03-12T03:41:20.054595+0000","last_active":"2024-03-12T03:43:01.031368+0000","last_peered":"2024-03-12T03:43:01.031368+0000","last_clean":"2024-03-12T03:43:01.031368+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:43:01.031368+0000","last_undegraded":"2024-03-12T03:43:01.031368+0000","last_fullsized":"2024-03-12T03:43:01.031368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:41:20.054461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:41:20.054461+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:04.266552+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":417,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.028138+0000","last_change":"2024-03-12T03:40:28.767260+0000","last_active":"2024-03-12T03:43:01.028138+0000","last_peered":"2024-03-12T03:43:01.028138+0000","last_clean":"2024-03-12T03:43:01.028138+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:43:01.028138+0000","last_undegraded":"2024-03-12T03:43:01.028138+0000","last_fullsized":"2024-03-12T03:43:01.028138+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:28.767140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:40:28.767140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416913+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":348,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.037128+0000","last_change":"2024-03-12T03:40:46.535232+0000","last_active":"2024-03-12T03:43:01.037128+0000","last_peered":"2024-03-12T03:43:01.037128+0000","last_clean":"2024-03-12T03:43:01.037128+0000","last_became_active":"2024-03-12T03:40:46.528004+0000","last_became_peered":"2024-03-12T03:40:46.528004+0000","last_unstale":"2024-03-12T03:43:01.037128+0000","last_undegraded":"2024-03-12T03:43:01.037128+0000","last_fullsized":"2024-03-12T03:43:01.037128+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:35:00.079152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:00.079152+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":444,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.029708+0000","last_change":"2024-03-12T03:42:14.426265+0000","last_active":"2024-03-12T03:43:01.029708+0000","last_peered":"2024-03-12T03:43:01.029708+0000","last_clean":"2024-03-12T03:43:01.029708+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:43:01.029708+0000","last_undegraded":"2024-03-12T03:43:01.029708+0000","last_fullsized":"2024-03-12T03:43:01.029708+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:14.426145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:14.426145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:44.257789+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":429,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.027575+0000","last_change":"2024-03-12T03:41:01.417925+0000","last_active":"2024-03-12T03:43:01.027575+0000","last_peered":"2024-03-12T03:43:01.027575+0000","last_clean":"2024-03-12T03:43:01.027575+0000","last_became_active":"2024-03-12T03:40:46.516086+0000","last_became_peered":"2024-03-12T03:40:46.516086+0000","last_unstale":"2024-03-12T03:43:01.027575+0000","last_undegraded":"2024-03-12T03:43:01.027575+0000","last_fullsized":"2024-03-12T03:43:01.027575+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:01.417809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:01.417809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.417036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":425,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.031190+0000","last_change":"2024-03-12T03:42:02.044839+0000","last_active":"2024-03-12T03:43:01.031190+0000","last_peered":"2024-03-12T03:43:01.031190+0000","last_clean":"2024-03-12T03:43:01.031190+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:43:01.031190+0000","last_undegraded":"2024-03-12T03:43:01.031190+0000","last_fullsized":"2024-03-12T03:43:01.031190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:02.044703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:02.044703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:14.327829+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":465,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.041770+0000","last_change":"2024-03-12T03:40:46.519477+0000","last_active":"2024-03-12T03:43:01.041770+0000","last_peered":"2024-03-12T03:43:01.041770+0000","last_clean":"2024-03-12T03:43:01.041770+0000","last_became_active":"2024-03-12T03:40:46.518652+0000","last_became_peered":"2024-03-12T03:40:46.518652+0000","last_unstale":"2024-03-12T03:43:01.041770+0000","last_undegraded":"2024-03-12T03:43:01.041770+0000","last_fullsized":"2024-03-12T03:43:01.041770+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:39:14.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:14.564162+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":437,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.030677+0000","last_change":"2024-03-12T03:42:22.126915+0000","last_active":"2024-03-12T03:43:01.030677+0000","last_peered":"2024-03-12T03:43:01.030677+0000","last_clean":"2024-03-12T03:43:01.030677+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:43:01.030677+0000","last_undegraded":"2024-03-12T03:43:01.030677+0000","last_fullsized":"2024-03-12T03:43:01.030677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:22.126792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:22.126792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:51.530630+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":323,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.037378+0000","last_change":"2024-03-12T03:40:46.535824+0000","last_active":"2024-03-12T03:43:01.037378+0000","last_peered":"2024-03-12T03:43:01.037378+0000","last_clean":"2024-03-12T03:43:01.037378+0000","last_became_active":"2024-03-12T03:40:46.533278+0000","last_became_peered":"2024-03-12T03:40:46.533278+0000","last_unstale":"2024-03-12T03:43:01.037378+0000","last_undegraded":"2024-03-12T03:43:01.037378+0000","last_fullsized":"2024-03-12T03:43:01.037378+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:35:04.007252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":148,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.039837+0000","last_change":"2024-03-12T03:40:19.915322+0000","last_active":"2024-03-12T03:43:01.039837+0000","last_peered":"2024-03-12T03:43:01.039837+0000","last_clean":"2024-03-12T03:43:01.039837+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:43:01.039837+0000","last_undegraded":"2024-03-12T03:43:01.039837+0000","last_fullsized":"2024-03-12T03:43:01.039837+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:19.915088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:40:19.915088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":469,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.028302+0000","last_change":"2024-03-12T03:40:58.312735+0000","last_active":"2024-03-12T03:43:01.028302+0000","last_peered":"2024-03-12T03:43:01.028302+0000","last_clean":"2024-03-12T03:43:01.028302+0000","last_became_active":"2024-03-12T03:40:46.520200+0000","last_became_peered":"2024-03-12T03:40:46.520200+0000","last_unstale":"2024-03-12T03:43:01.028302+0000","last_undegraded":"2024-03-12T03:43:01.028302+0000","last_fullsized":"2024-03-12T03:43:01.028302+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:40:58.312602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:58.312602+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416971+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":438,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.041006+0000","last_change":"2024-03-12T03:42:00.266626+0000","last_active":"2024-03-12T03:43:01.041006+0000","last_peered":"2024-03-12T03:43:01.041006+0000","last_clean":"2024-03-12T03:43:01.041006+0000","last_became_active":"2024-03-12T03:40:46.528869+0000","last_became_peered":"2024-03-12T03:40:46.528869+0000","last_unstale":"2024-03-12T03:43:01.041006+0000","last_undegraded":"2024-03-12T03:43:01.041006+0000","last_fullsized":"2024-03-12T03:43:01.041006+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:00.266512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:42:00.266512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:27.852547+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"223'12","reported_seq":508,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.739815+0000","last_change":"2024-03-12T03:40:46.525685+0000","last_active":"2024-03-12T03:43:01.739815+0000","last_peered":"2024-03-12T03:43:01.739815+0000","last_clean":"2024-03-12T03:43:01.739815+0000","last_became_active":"2024-03-12T03:40:46.522898+0000","last_became_peered":"2024-03-12T03:40:46.522898+0000","last_unstale":"2024-03-12T03:43:01.739815+0000","last_undegraded":"2024-03-12T03:43:01.739815+0000","last_fullsized":"2024-03-12T03:43:01.739815+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:35:08.006375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:35:08.006375+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":434,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042979+0000","last_change":"2024-03-12T03:41:19.377731+0000","last_active":"2024-03-12T03:43:00.042979+0000","last_peered":"2024-03-12T03:43:00.042979+0000","last_clean":"2024-03-12T03:43:00.042979+0000","last_became_active":"2024-03-12T03:39:33.265499+0000","last_became_peered":"2024-03-12T03:39:33.265499+0000","last_unstale":"2024-03-12T03:43:00.042979+0000","last_undegraded":"2024-03-12T03:43:00.042979+0000","last_fullsized":"2024-03-12T03:43:00.042979+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:19.377599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:19.377599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560448+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":411,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.110315+0000","last_change":"2024-03-12T03:43:01.104577+0000","last_active":"2024-03-12T03:43:01.110315+0000","last_peered":"2024-03-12T03:43:01.110315+0000","last_clean":"2024-03-12T03:43:01.110315+0000","last_became_active":"2024-03-12T03:43:01.055157+0000","last_became_peered":"2024-03-12T03:43:01.055157+0000","last_unstale":"2024-03-12T03:43:01.110315+0000","last_undegraded":"2024-03-12T03:43:01.110315+0000","last_fullsized":"2024-03-12T03:43:01.110315+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-03-12T03:35:58.079645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.079645+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":463,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.030882+0000","last_change":"2024-03-12T03:42:12.114036+0000","last_active":"2024-03-12T03:43:01.030882+0000","last_peered":"2024-03-12T03:43:01.030882+0000","last_clean":"2024-03-12T03:43:01.030882+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:43:01.030882+0000","last_undegraded":"2024-03-12T03:43:01.030882+0000","last_fullsized":"2024-03-12T03:43:01.030882+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:42:12.113938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:12.113938+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:21.487403+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":544,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.028153+0000","last_change":"2024-03-12T03:40:15.773578+0000","last_active":"2024-03-12T03:43:01.028153+0000","last_peered":"2024-03-12T03:43:01.028153+0000","last_clean":"2024-03-12T03:43:01.028153+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:43:01.028153+0000","last_undegraded":"2024-03-12T03:43:01.028153+0000","last_fullsized":"2024-03-12T03:43:01.028153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:40:15.773432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:40:15.773432+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416846+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":425,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.038728+0000","last_change":"2024-03-12T03:42:10.355334+0000","last_active":"2024-03-12T03:43:01.038728+0000","last_peered":"2024-03-12T03:43:01.038728+0000","last_clean":"2024-03-12T03:43:01.038728+0000","last_became_active":"2024-03-12T03:40:46.519320+0000","last_became_peered":"2024-03-12T03:40:46.519320+0000","last_unstale":"2024-03-12T03:43:01.038728+0000","last_undegraded":"2024-03-12T03:43:01.038728+0000","last_fullsized":"2024-03-12T03:43:01.038728+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:10.355198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:10.355198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:31.051878+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":450,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042855+0000","last_change":"2024-03-12T03:41:21.258124+0000","last_active":"2024-03-12T03:43:00.042855+0000","last_peered":"2024-03-12T03:43:00.042855+0000","last_clean":"2024-03-12T03:43:00.042855+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:43:00.042855+0000","last_undegraded":"2024-03-12T03:43:00.042855+0000","last_fullsized":"2024-03-12T03:43:00.042855+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:41:21.257991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:41:21.257991+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560297+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":435,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042542+0000","last_change":"2024-03-12T03:41:13.286000+0000","last_active":"2024-03-12T03:43:00.042542+0000","last_peered":"2024-03-12T03:43:00.042542+0000","last_clean":"2024-03-12T03:43:00.042542+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:43:00.042542+0000","last_undegraded":"2024-03-12T03:43:00.042542+0000","last_fullsized":"2024-03-12T03:43:00.042542+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:13.285572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:38:34.264717+0000","last_clean_scrub_stamp":"2024-03-12T03:41:13.285572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560327+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":438,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.034790+0000","last_change":"2024-03-12T03:40:46.529405+0000","last_active":"2024-03-12T03:43:01.034790+0000","last_peered":"2024-03-12T03:43:01.034790+0000","last_clean":"2024-03-12T03:43:01.034790+0000","last_became_active":"2024-03-12T03:40:46.529091+0000","last_became_peered":"2024-03-12T03:40:46.529091+0000","last_unstale":"2024-03-12T03:43:01.034790+0000","last_undegraded":"2024-03-12T03:43:01.034790+0000","last_fullsized":"2024-03-12T03:43:01.034790+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:35.850271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:35.850271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":342,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032400+0000","last_change":"2024-03-12T03:40:46.533090+0000","last_active":"2024-03-12T03:43:01.032400+0000","last_peered":"2024-03-12T03:43:01.032400+0000","last_clean":"2024-03-12T03:43:01.032400+0000","last_became_active":"2024-03-12T03:40:46.532335+0000","last_became_peered":"2024-03-12T03:40:46.532335+0000","last_unstale":"2024-03-12T03:43:01.032400+0000","last_undegraded":"2024-03-12T03:43:01.032400+0000","last_fullsized":"2024-03-12T03:43:01.032400+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.075897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.075897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":21,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.048913+0000","last_change":"2024-03-12T03:43:01.048588+0000","last_active":"2024-03-12T03:43:01.048913+0000","last_peered":"2024-03-12T03:43:01.048913+0000","last_clean":"2024-03-12T03:43:01.048913+0000","last_became_active":"2024-03-12T03:43:01.037348+0000","last_became_peered":"2024-03-12T03:43:01.037348+0000","last_unstale":"2024-03-12T03:43:01.048913+0000","last_undegraded":"2024-03-12T03:43:01.048913+0000","last_fullsized":"2024-03-12T03:43:01.048913+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-12T03:40:17.848303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:17.848303+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":438,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.031302+0000","last_change":"2024-03-12T03:41:18.030224+0000","last_active":"2024-03-12T03:43:01.031302+0000","last_peered":"2024-03-12T03:43:01.031302+0000","last_clean":"2024-03-12T03:43:01.031302+0000","last_became_active":"2024-03-12T03:40:46.518227+0000","last_became_peered":"2024-03-12T03:40:46.518227+0000","last_unstale":"2024-03-12T03:43:01.031302+0000","last_undegraded":"2024-03-12T03:43:01.031302+0000","last_fullsized":"2024-03-12T03:43:01.031302+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:41:18.030146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:04.266328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":354,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.035753+0000","last_change":"2024-03-12T03:40:46.528355+0000","last_active":"2024-03-12T03:43:01.035753+0000","last_peered":"2024-03-12T03:43:01.035753+0000","last_clean":"2024-03-12T03:43:01.035753+0000","last_became_active":"2024-03-12T03:40:46.525702+0000","last_became_peered":"2024-03-12T03:40:46.525702+0000","last_unstale":"2024-03-12T03:43:01.035753+0000","last_undegraded":"2024-03-12T03:43:01.035753+0000","last_fullsized":"2024-03-12T03:43:01.035753+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:15.808887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:15.808887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":417,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.037237+0000","last_change":"2024-03-12T03:39:38.537429+0000","last_active":"2024-03-12T03:43:01.037237+0000","last_peered":"2024-03-12T03:43:01.037237+0000","last_clean":"2024-03-12T03:43:01.037237+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:43:01.037237+0000","last_undegraded":"2024-03-12T03:43:01.037237+0000","last_fullsized":"2024-03-12T03:43:01.037237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.537304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641627+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"74'18","reported_seq":360,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032587+0000","last_change":"2024-03-12T03:40:46.531983+0000","last_active":"2024-03-12T03:43:01.032587+0000","last_peered":"2024-03-12T03:43:01.032587+0000","last_clean":"2024-03-12T03:43:01.032587+0000","last_became_active":"2024-03-12T03:40:46.528402+0000","last_became_peered":"2024-03-12T03:40:46.528402+0000","last_unstale":"2024-03-12T03:43:01.032587+0000","last_undegraded":"2024-03-12T03:43:01.032587+0000","last_fullsized":"2024-03-12T03:43:01.032587+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:35:02.079224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:02.079224+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":429,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.027702+0000","last_change":"2024-03-12T03:39:33.687282+0000","last_active":"2024-03-12T03:43:01.027702+0000","last_peered":"2024-03-12T03:43:01.027702+0000","last_clean":"2024-03-12T03:43:01.027702+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:43:01.027702+0000","last_undegraded":"2024-03-12T03:43:01.027702+0000","last_fullsized":"2024-03-12T03:43:01.027702+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.687198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":421,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.030317+0000","last_change":"2024-03-12T03:41:16.087106+0000","last_active":"2024-03-12T03:43:01.030317+0000","last_peered":"2024-03-12T03:43:01.030317+0000","last_clean":"2024-03-12T03:43:01.030317+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:43:01.030317+0000","last_undegraded":"2024-03-12T03:43:01.030317+0000","last_fullsized":"2024-03-12T03:43:01.030317+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:16.086881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:16.086881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:04.266356+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":377,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.035945+0000","last_change":"2024-03-12T03:40:46.529912+0000","last_active":"2024-03-12T03:43:01.035945+0000","last_peered":"2024-03-12T03:43:01.035945+0000","last_clean":"2024-03-12T03:43:01.035945+0000","last_became_active":"2024-03-12T03:40:46.525946+0000","last_became_peered":"2024-03-12T03:40:46.525946+0000","last_unstale":"2024-03-12T03:43:01.035945+0000","last_undegraded":"2024-03-12T03:43:01.035945+0000","last_fullsized":"2024-03-12T03:43:01.035945+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"90'6","reported_seq":338,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.037826+0000","last_change":"2024-03-12T03:40:46.521647+0000","last_active":"2024-03-12T03:43:01.037826+0000","last_peered":"2024-03-12T03:43:01.037826+0000","last_clean":"2024-03-12T03:43:01.037826+0000","last_became_active":"2024-03-12T03:40:46.517264+0000","last_became_peered":"2024-03-12T03:40:46.517264+0000","last_unstale":"2024-03-12T03:43:01.037826+0000","last_undegraded":"2024-03-12T03:43:01.037826+0000","last_fullsized":"2024-03-12T03:43:01.037826+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:38:38.943094+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:38:38.943094+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":437,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.040233+0000","last_change":"2024-03-12T03:40:46.535973+0000","last_active":"2024-03-12T03:43:01.040233+0000","last_peered":"2024-03-12T03:43:01.040233+0000","last_clean":"2024-03-12T03:43:01.040233+0000","last_became_active":"2024-03-12T03:40:46.535659+0000","last_became_peered":"2024-03-12T03:40:46.535659+0000","last_unstale":"2024-03-12T03:43:01.040233+0000","last_undegraded":"2024-03-12T03:43:01.040233+0000","last_fullsized":"2024-03-12T03:43:01.040233+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:19.511683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:39:19.511683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":436,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.030526+0000","last_change":"2024-03-12T03:42:06.327482+0000","last_active":"2024-03-12T03:43:01.030526+0000","last_peered":"2024-03-12T03:43:01.030526+0000","last_clean":"2024-03-12T03:43:01.030526+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:43:01.030526+0000","last_undegraded":"2024-03-12T03:43:01.030526+0000","last_fullsized":"2024-03-12T03:43:01.030526+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:06.327362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:06.327362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:28.064769+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":429,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.031831+0000","last_change":"2024-03-12T03:41:25.018344+0000","last_active":"2024-03-12T03:43:01.031831+0000","last_peered":"2024-03-12T03:43:01.031831+0000","last_clean":"2024-03-12T03:43:01.031831+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:43:01.031831+0000","last_undegraded":"2024-03-12T03:43:01.031831+0000","last_fullsized":"2024-03-12T03:43:01.031831+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:25.018224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:25.018224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:04.266490+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"94'23","reported_seq":469,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.033863+0000","last_change":"2024-03-12T03:41:13.133503+0000","last_active":"2024-03-12T03:43:01.033863+0000","last_peered":"2024-03-12T03:43:01.033863+0000","last_clean":"2024-03-12T03:43:01.033863+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:43:01.033863+0000","last_undegraded":"2024-03-12T03:43:01.033863+0000","last_fullsized":"2024-03-12T03:43:01.033863+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'23","last_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:41:13.133173+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":369,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.037935+0000","last_change":"2024-03-12T03:40:46.525930+0000","last_active":"2024-03-12T03:43:01.037935+0000","last_peered":"2024-03-12T03:43:01.037935+0000","last_clean":"2024-03-12T03:43:01.037935+0000","last_became_active":"2024-03-12T03:40:46.522763+0000","last_became_peered":"2024-03-12T03:40:46.522763+0000","last_unstale":"2024-03-12T03:43:01.037935+0000","last_undegraded":"2024-03-12T03:43:01.037935+0000","last_fullsized":"2024-03-12T03:43:01.037935+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:25.092313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.092313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":433,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.041947+0000","last_change":"2024-03-12T03:40:21.384617+0000","last_active":"2024-03-12T03:43:01.041947+0000","last_peered":"2024-03-12T03:43:01.041947+0000","last_clean":"2024-03-12T03:43:01.041947+0000","last_became_active":"2024-03-12T03:39:33.276435+0000","last_became_peered":"2024-03-12T03:39:33.276435+0000","last_unstale":"2024-03-12T03:43:01.041947+0000","last_undegraded":"2024-03-12T03:43:01.041947+0000","last_fullsized":"2024-03-12T03:43:01.041947+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:21.384531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:40:21.384531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.642085+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":423,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.034422+0000","last_change":"2024-03-12T03:39:33.820164+0000","last_active":"2024-03-12T03:43:01.034422+0000","last_peered":"2024-03-12T03:43:01.034422+0000","last_clean":"2024-03-12T03:43:01.034422+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:43:01.034422+0000","last_undegraded":"2024-03-12T03:43:01.034422+0000","last_fullsized":"2024-03-12T03:43:01.034422+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:33.819657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:33.819657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'17","reported_seq":462,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.041653+0000","last_change":"2024-03-12T03:40:46.519746+0000","last_active":"2024-03-12T03:43:01.041653+0000","last_peered":"2024-03-12T03:43:01.041653+0000","last_clean":"2024-03-12T03:43:01.041653+0000","last_became_active":"2024-03-12T03:40:46.519129+0000","last_became_peered":"2024-03-12T03:40:46.519129+0000","last_unstale":"2024-03-12T03:43:01.041653+0000","last_undegraded":"2024-03-12T03:43:01.041653+0000","last_fullsized":"2024-03-12T03:43:01.041653+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:39:37.572971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:37.572971+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":438,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.035020+0000","last_change":"2024-03-12T03:40:46.531173+0000","last_active":"2024-03-12T03:43:01.035020+0000","last_peered":"2024-03-12T03:43:01.035020+0000","last_clean":"2024-03-12T03:43:01.035020+0000","last_became_active":"2024-03-12T03:40:46.530644+0000","last_became_peered":"2024-03-12T03:40:46.530644+0000","last_unstale":"2024-03-12T03:43:01.035020+0000","last_undegraded":"2024-03-12T03:43:01.035020+0000","last_fullsized":"2024-03-12T03:43:01.035020+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:38:35.588969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:35.588969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":555,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.033745+0000","last_change":"2024-03-12T03:41:09.084115+0000","last_active":"2024-03-12T03:43:01.033745+0000","last_peered":"2024-03-12T03:43:01.033745+0000","last_clean":"2024-03-12T03:43:01.033745+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:43:01.033745+0000","last_undegraded":"2024-03-12T03:43:01.033745+0000","last_fullsized":"2024-03-12T03:43:01.033745+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:41:09.083949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:09.083949+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":445,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.031110+0000","last_change":"2024-03-12T03:41:26.041453+0000","last_active":"2024-03-12T03:43:01.031110+0000","last_peered":"2024-03-12T03:43:01.031110+0000","last_clean":"2024-03-12T03:43:01.031110+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:43:01.031110+0000","last_undegraded":"2024-03-12T03:43:01.031110+0000","last_fullsized":"2024-03-12T03:43:01.031110+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:41:26.041306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:26.041306+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:04.266521+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":394,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032727+0000","last_change":"2024-03-12T03:40:46.520870+0000","last_active":"2024-03-12T03:43:01.032727+0000","last_peered":"2024-03-12T03:43:01.032727+0000","last_clean":"2024-03-12T03:43:01.032727+0000","last_became_active":"2024-03-12T03:40:46.519620+0000","last_became_peered":"2024-03-12T03:40:46.519620+0000","last_unstale":"2024-03-12T03:43:01.032727+0000","last_undegraded":"2024-03-12T03:43:01.032727+0000","last_fullsized":"2024-03-12T03:43:01.032727+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:36:28.133261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.133261+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":454,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042672+0000","last_change":"2024-03-12T03:42:18.499466+0000","last_active":"2024-03-12T03:43:00.042672+0000","last_peered":"2024-03-12T03:43:00.042672+0000","last_clean":"2024-03-12T03:43:00.042672+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:43:00.042672+0000","last_undegraded":"2024-03-12T03:43:00.042672+0000","last_fullsized":"2024-03-12T03:43:00.042672+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:18.499342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:18.499342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:45.872849+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":447,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042769+0000","last_change":"2024-03-12T03:41:08.289828+0000","last_active":"2024-03-12T03:43:00.042769+0000","last_peered":"2024-03-12T03:43:00.042769+0000","last_clean":"2024-03-12T03:43:00.042769+0000","last_became_active":"2024-03-12T03:40:46.521420+0000","last_became_peered":"2024-03-12T03:40:46.521420+0000","last_unstale":"2024-03-12T03:43:00.042769+0000","last_undegraded":"2024-03-12T03:43:00.042769+0000","last_fullsized":"2024-03-12T03:43:00.042769+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:08.289718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:08.289718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560263+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":371,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.037662+0000","last_change":"2024-03-12T03:40:46.536596+0000","last_active":"2024-03-12T03:43:01.037662+0000","last_peered":"2024-03-12T03:43:01.037662+0000","last_clean":"2024-03-12T03:43:01.037662+0000","last_became_active":"2024-03-12T03:40:46.535011+0000","last_became_peered":"2024-03-12T03:40:46.535011+0000","last_unstale":"2024-03-12T03:43:01.037662+0000","last_undegraded":"2024-03-12T03:43:01.037662+0000","last_fullsized":"2024-03-12T03:43:01.037662+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:16.588551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:16.588551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":173,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.039623+0000","last_change":"2024-03-12T03:42:04.171633+0000","last_active":"2024-03-12T03:43:01.039623+0000","last_peered":"2024-03-12T03:43:01.039623+0000","last_clean":"2024-03-12T03:43:01.039623+0000","last_became_active":"2024-03-12T03:40:46.519814+0000","last_became_peered":"2024-03-12T03:40:46.519814+0000","last_unstale":"2024-03-12T03:43:01.039623+0000","last_undegraded":"2024-03-12T03:43:01.039623+0000","last_fullsized":"2024-03-12T03:43:01.039623+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:42:04.171114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:42:04.171114+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:25.994710+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":428,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.038383+0000","last_change":"2024-03-12T03:39:28.536556+0000","last_active":"2024-03-12T03:43:01.038383+0000","last_peered":"2024-03-12T03:43:01.038383+0000","last_clean":"2024-03-12T03:43:01.038383+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:43:01.038383+0000","last_undegraded":"2024-03-12T03:43:01.038383+0000","last_fullsized":"2024-03-12T03:43:01.038383+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:28.536478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:39:28.536478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.641720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":427,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.034294+0000","last_change":"2024-03-12T03:39:20.647378+0000","last_active":"2024-03-12T03:43:01.034294+0000","last_peered":"2024-03-12T03:43:01.034294+0000","last_clean":"2024-03-12T03:43:01.034294+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:43:01.034294+0000","last_undegraded":"2024-03-12T03:43:01.034294+0000","last_fullsized":"2024-03-12T03:43:01.034294+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:20.647294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:20.647294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":423,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.043808+0000","last_change":"2024-03-12T03:42:20.532434+0000","last_active":"2024-03-12T03:43:00.043808+0000","last_peered":"2024-03-12T03:43:00.043808+0000","last_clean":"2024-03-12T03:43:00.043808+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:43:00.043808+0000","last_undegraded":"2024-03-12T03:43:00.043808+0000","last_fullsized":"2024-03-12T03:43:00.043808+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:20.532318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:20.532318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:29.117306+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"86'14","reported_seq":492,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042625+0000","last_change":"2024-03-12T03:41:04.284204+0000","last_active":"2024-03-12T03:43:00.042625+0000","last_peered":"2024-03-12T03:43:00.042625+0000","last_clean":"2024-03-12T03:43:00.042625+0000","last_became_active":"2024-03-12T03:40:46.528979+0000","last_became_peered":"2024-03-12T03:40:46.528979+0000","last_unstale":"2024-03-12T03:43:00.042625+0000","last_undegraded":"2024-03-12T03:43:00.042625+0000","last_fullsized":"2024-03-12T03:43:00.042625+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:41:04.284062+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:41:04.284062+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560048+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":428,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.033017+0000","last_change":"2024-03-12T03:40:17.773177+0000","last_active":"2024-03-12T03:43:01.033017+0000","last_peered":"2024-03-12T03:43:01.033017+0000","last_clean":"2024-03-12T03:43:01.033017+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:43:01.033017+0000","last_undegraded":"2024-03-12T03:43:01.033017+0000","last_fullsized":"2024-03-12T03:43:01.033017+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:17.772938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:40:17.772938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416863+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":420,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.030764+0000","last_change":"2024-03-12T03:41:24.020443+0000","last_active":"2024-03-12T03:43:01.030764+0000","last_peered":"2024-03-12T03:43:01.030764+0000","last_clean":"2024-03-12T03:43:01.030764+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:43:01.030764+0000","last_undegraded":"2024-03-12T03:43:01.030764+0000","last_fullsized":"2024-03-12T03:43:01.030764+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:24.020327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.805859+0000","last_clean_scrub_stamp":"2024-03-12T03:41:24.020327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:04.266407+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":425,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.027669+0000","last_change":"2024-03-12T03:41:15.648366+0000","last_active":"2024-03-12T03:43:01.027669+0000","last_peered":"2024-03-12T03:43:01.027669+0000","last_clean":"2024-03-12T03:43:01.027669+0000","last_became_active":"2024-03-12T03:40:46.529681+0000","last_became_peered":"2024-03-12T03:40:46.529681+0000","last_unstale":"2024-03-12T03:43:01.027669+0000","last_undegraded":"2024-03-12T03:43:01.027669+0000","last_fullsized":"2024-03-12T03:43:01.027669+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:15.648255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:15.648255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.417067+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'21","reported_seq":471,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042570+0000","last_change":"2024-03-12T03:40:58.313751+0000","last_active":"2024-03-12T03:43:00.042570+0000","last_peered":"2024-03-12T03:43:00.042570+0000","last_clean":"2024-03-12T03:43:00.042570+0000","last_became_active":"2024-03-12T03:40:46.518368+0000","last_became_peered":"2024-03-12T03:40:46.518368+0000","last_unstale":"2024-03-12T03:43:00.042570+0000","last_undegraded":"2024-03-12T03:43:00.042570+0000","last_fullsized":"2024-03-12T03:43:00.042570+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:40:58.313611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:58.313611+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560178+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":428,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.028066+0000","last_change":"2024-03-12T03:40:25.773470+0000","last_active":"2024-03-12T03:43:01.028066+0000","last_peered":"2024-03-12T03:43:01.028066+0000","last_clean":"2024-03-12T03:43:01.028066+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:43:01.028066+0000","last_undegraded":"2024-03-12T03:43:01.028066+0000","last_fullsized":"2024-03-12T03:43:01.028066+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:25.773348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:40:25.773348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416929+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":420,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.028535+0000","last_change":"2024-03-12T03:40:29.743363+0000","last_active":"2024-03-12T03:43:01.028535+0000","last_peered":"2024-03-12T03:43:01.028535+0000","last_clean":"2024-03-12T03:43:01.028535+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:43:01.028535+0000","last_undegraded":"2024-03-12T03:43:01.028535+0000","last_fullsized":"2024-03-12T03:43:01.028535+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:29.743231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:40:29.743231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:01.416944+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":416,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.037490+0000","last_change":"2024-03-12T03:40:31.512991+0000","last_active":"2024-03-12T03:43:01.037490+0000","last_peered":"2024-03-12T03:43:01.037490+0000","last_clean":"2024-03-12T03:43:01.037490+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:43:01.037490+0000","last_undegraded":"2024-03-12T03:43:01.037490+0000","last_fullsized":"2024-03-12T03:43:01.037490+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:31.512880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:40:31.512880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.642056+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"94'50","reported_seq":522,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032042+0000","last_change":"2024-03-12T03:43:00.508573+0000","last_active":"2024-03-12T03:43:01.032042+0000","last_peered":"2024-03-12T03:43:01.032042+0000","last_clean":"2024-03-12T03:43:01.032042+0000","last_became_active":"2024-03-12T03:40:46.516177+0000","last_became_peered":"2024-03-12T03:40:46.516177+0000","last_unstale":"2024-03-12T03:43:01.032042+0000","last_undegraded":"2024-03-12T03:43:01.032042+0000","last_fullsized":"2024-03-12T03:43:01.032042+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:18.189827+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":396,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.035410+0000","last_change":"2024-03-12T03:40:46.513612+0000","last_active":"2024-03-12T03:43:01.035410+0000","last_peered":"2024-03-12T03:43:01.035410+0000","last_clean":"2024-03-12T03:43:01.035410+0000","last_became_active":"2024-03-12T03:40:46.513197+0000","last_became_peered":"2024-03-12T03:40:46.513197+0000","last_unstale":"2024-03-12T03:43:01.035410+0000","last_undegraded":"2024-03-12T03:43:01.035410+0000","last_fullsized":"2024-03-12T03:43:01.035410+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:26.898782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:39:26.898782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":426,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042054+0000","last_change":"2024-03-12T03:42:09.304630+0000","last_active":"2024-03-12T03:43:00.042054+0000","last_peered":"2024-03-12T03:43:00.042054+0000","last_clean":"2024-03-12T03:43:00.042054+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:43:00.042054+0000","last_undegraded":"2024-03-12T03:43:00.042054+0000","last_fullsized":"2024-03-12T03:43:00.042054+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:09.304528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:42:09.304528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:23.606291+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":347,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.032082+0000","last_change":"2024-03-12T03:40:46.535015+0000","last_active":"2024-03-12T03:43:01.032082+0000","last_peered":"2024-03-12T03:43:01.032082+0000","last_clean":"2024-03-12T03:43:01.032082+0000","last_became_active":"2024-03-12T03:40:46.526044+0000","last_became_peered":"2024-03-12T03:40:46.526044+0000","last_unstale":"2024-03-12T03:43:01.032082+0000","last_undegraded":"2024-03-12T03:43:01.032082+0000","last_fullsized":"2024-03-12T03:43:01.032082+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.236997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.236997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"218'20","reported_seq":455,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042045+0000","last_change":"2024-03-12T03:40:29.325294+0000","last_active":"2024-03-12T03:43:00.042045+0000","last_peered":"2024-03-12T03:43:00.042045+0000","last_clean":"2024-03-12T03:43:00.042045+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:43:00.042045+0000","last_undegraded":"2024-03-12T03:43:00.042045+0000","last_fullsized":"2024-03-12T03:43:00.042045+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'20","last_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:40:29.325128+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.559936+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":441,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.040076+0000","last_change":"2024-03-12T03:40:27.572921+0000","last_active":"2024-03-12T03:43:01.040076+0000","last_peered":"2024-03-12T03:43:01.040076+0000","last_clean":"2024-03-12T03:43:01.040076+0000","last_became_active":"2024-03-12T03:39:33.276119+0000","last_became_peered":"2024-03-12T03:39:33.276119+0000","last_unstale":"2024-03-12T03:43:01.040076+0000","last_undegraded":"2024-03-12T03:43:01.040076+0000","last_fullsized":"2024-03-12T03:43:01.040076+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:27.572788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:28.208044+0000","last_clean_scrub_stamp":"2024-03-12T03:40:27.572788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:02.642112+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":436,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-03-12T03:43:01.035141+0000","last_change":"2024-03-12T03:40:30.676492+0000","last_active":"2024-03-12T03:43:01.035141+0000","last_peered":"2024-03-12T03:43:01.035141+0000","last_clean":"2024-03-12T03:43:01.035141+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:43:01.035141+0000","last_undegraded":"2024-03-12T03:43:01.035141+0000","last_fullsized":"2024-03-12T03:43:01.035141+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:30.676367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:26.625326+0000","last_clean_scrub_stamp":"2024-03-12T03:40:30.676367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":422,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-12T03:43:00.042423+0000","last_change":"2024-03-12T03:41:25.309091+0000","last_active":"2024-03-12T03:43:00.042423+0000","last_peered":"2024-03-12T03:43:00.042423+0000","last_clean":"2024-03-12T03:43:00.042423+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:43:00.042423+0000","last_undegraded":"2024-03-12T03:43:00.042423+0000","last_fullsized":"2024-03-12T03:43:00.042423+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:25.308974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.063092+0000","last_clean_scrub_stamp":"2024-03-12T03:41:25.308974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:00.560235+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1297547288,"num_objects":19800,"num_object_clones":0,"num_object_copies":59400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":247162,"num_write_kb":8542721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34219,"num_bytes_recovered":2242576384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3853398016,"data_stored":3853385800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80369,"ondisk_log_size":80369,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":578,"num_read_kb":573,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":980,"num_read_kb":705,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":2174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3734,"internal_metadata":0},"log_size":657,"ondisk_log_size":657,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070586,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":700776,"kb_used_data":322528,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93044376,"statfs":{"total":95995035648,"available":95277441024,"internally_reserved":0,"allocated":330268672,"data_stored":329839929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":387316170},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332222,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":796996,"kb_used_data":436728,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92948156,"statfs":{"total":95995035648,"available":95178911744,"internally_reserved":0,"allocated":447209472,"data_stored":446812862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":368835014},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574999,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":981504,"kb_used_data":575472,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92763648,"statfs":{"total":95995035648,"available":94989975552,"internally_reserved":0,"allocated":589283328,"data_stored":588837954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738524,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":874092,"kb_used_data":481756,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92871060,"statfs":{"total":95995035648,"available":95099965440,"internally_reserved":0,"allocated":493318144,"data_stored":492892727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411473,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":918488,"kb_used_data":499976,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92826664,"statfs":{"total":95995035648,"available":95054503936,"internally_reserved":0,"allocated":511975424,"data_stored":511570851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":222,"seq":953482739741,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":703288,"kb_used_data":406320,"kb_used_omap":7,"kb_used_meta":296888,"kb_avail":93041864,"statfs":{"total":95995035648,"available":95274868736,"internally_reserved":0,"allocated":416071680,"data_stored":415626437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8180,"internal_metadata":304013324},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953609,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":736032,"kb_used_data":429972,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93009120,"statfs":{"total":95995035648,"available":95241338880,"internally_reserved":0,"allocated":440291328,"data_stored":439867549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659267,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1106332,"kb_used_data":705872,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92638820,"statfs":{"total":95995035648,"available":94862151680,"internally_reserved":0,"allocated":722812928,"data_stored":722392981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3734,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":491716608,"data_stored":491716608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":574291968,"data_stored":574291968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":509022208,"data_stored":509018136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":409927680,"data_stored":409927680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":429719552,"data_stored":429719552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":695406592,"data_stored":695402520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":430702592,"data_stored":430702592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":312610816,"data_stored":312606744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:43:11.849 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-12T03:43:11.849 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:43:11.849 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:43:11.849 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-12T03:43:11.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3 2024-03-12T03:43:11.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:11.881 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:11.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:11.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:12.027 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:12.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:12.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:12.134 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:43:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:43:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:43:12.134 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:11 smithi169 systemd[1]: Stopping Ceph osd.3 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:43:12.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:12.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:12.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:12.292 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:12.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:12.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:12.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:12 smithi169 ceph-mon[29344]: 3.1b scrub starts 2024-03-12T03:43:12.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:12 smithi169 ceph-mon[29344]: 3.1b scrub ok 2024-03-12T03:43:12.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:12 smithi169 ceph-mon[39949]: 3.1b scrub starts 2024-03-12T03:43:12.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:12 smithi169 ceph-mon[39949]: 3.1b scrub ok 2024-03-12T03:43:12.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:12.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:12.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:12 smithi173 ceph-mon[33544]: 3.1b scrub starts 2024-03-12T03:43:12.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:12 smithi173 ceph-mon[33544]: 3.1b scrub ok 2024-03-12T03:43:12.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:12.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:12.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:12.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:12.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:12.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:12.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:12.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:12.844 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:12.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:12.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:13.009 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:13.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:13.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:13.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:13.141 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:43:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:43:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:13.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:13.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:13.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:13.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:13.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:13.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:13.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:13.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[39949]: 6.4 scrub starts 2024-03-12T03:43:13.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[39949]: 6.4 scrub ok 2024-03-12T03:43:13.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[39949]: pgmap v458: 158 pgs: 158 active+clean; 673 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s; 0 B/s, 0 objects/s recovering 2024-03-12T03:43:13.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:43:13.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:43:13.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[39949]: 8.2 scrub starts 2024-03-12T03:43:13.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:43:13.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:13.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[29344]: 6.4 scrub starts 2024-03-12T03:43:13.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[29344]: 6.4 scrub ok 2024-03-12T03:43:13.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[29344]: pgmap v458: 158 pgs: 158 active+clean; 673 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s; 0 B/s, 0 objects/s recovering 2024-03-12T03:43:13.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:43:13.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:43:13.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[29344]: 8.2 scrub starts 2024-03-12T03:43:13.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:43:13.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:13 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:13 smithi173 ceph-mon[33544]: 6.4 scrub starts 2024-03-12T03:43:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:13 smithi173 ceph-mon[33544]: 6.4 scrub ok 2024-03-12T03:43:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:13 smithi173 ceph-mon[33544]: pgmap v458: 158 pgs: 158 active+clean; 673 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s; 0 B/s, 0 objects/s recovering 2024-03-12T03:43:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:13 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:43:13.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:13 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:43:13.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:13 smithi173 ceph-mon[33544]: 8.2 scrub starts 2024-03-12T03:43:13.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:13 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:43:13.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:13 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:13.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:13.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:13.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:13.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:13.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:13.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:13.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:13.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:13.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:13.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:13.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:13.905 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:13.945 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:13.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:13.964 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:14.046 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:14.083 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:14.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:14.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:14.218 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:14.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:14.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:14.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:14.363 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:14.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:14.394 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: 5.14 scrub starts 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: 5.14 scrub ok 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: 8.a scrub starts 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: 8.2 scrub ok 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: 5.10 scrub starts 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: 5.10 scrub ok 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: 8.a scrub ok 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: osdmap e263: 8 total, 8 up, 8 in 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: osdmap e264: 8 total, 8 up, 8 in 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: 8.9 scrub starts 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[39949]: 8.9 scrub ok 2024-03-12T03:43:14.395 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: 5.14 scrub starts 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: 5.14 scrub ok 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: 8.a scrub starts 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: 8.2 scrub ok 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: 5.10 scrub starts 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: 5.10 scrub ok 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: 8.a scrub ok 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: osdmap e263: 8 total, 8 up, 8 in 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: osdmap e264: 8 total, 8 up, 8 in 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: 8.9 scrub starts 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:43:14.396 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:14 smithi169 ceph-mon[29344]: 8.9 scrub ok 2024-03-12T03:43:14.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:14.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: 5.14 scrub starts 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: 5.14 scrub ok 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: 8.a scrub starts 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: 8.2 scrub ok 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: 5.10 scrub starts 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: 5.10 scrub ok 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: 8.a scrub ok 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: osdmap e263: 8 total, 8 up, 8 in 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: osdmap e264: 8 total, 8 up, 8 in 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: 8.9 scrub starts 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:43:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:14 smithi173 ceph-mon[33544]: 8.9 scrub ok 2024-03-12T03:43:14.504 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:14.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:14.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:14.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:14.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:14.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:14.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:14.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:14.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:14.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:14.874 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:14.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:14.914 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:14.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:15.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:15.055 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:15.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:15.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:15.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:15.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:15.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:15.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:15.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:15.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:15.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:15 smithi169 ceph-mon[29344]: 8.1 scrub starts 2024-03-12T03:43:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:15 smithi169 ceph-mon[29344]: pgmap v461: 158 pgs: 158 active+clean; 580 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.32k op/s 2024-03-12T03:43:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:15 smithi169 ceph-mon[29344]: 8.1 scrub ok 2024-03-12T03:43:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:15 smithi169 ceph-mon[29344]: 4.c scrub starts 2024-03-12T03:43:15.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:15 smithi169 ceph-mon[29344]: 4.c scrub ok 2024-03-12T03:43:15.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:15 smithi169 ceph-mon[39949]: 8.1 scrub starts 2024-03-12T03:43:15.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:15 smithi169 ceph-mon[39949]: pgmap v461: 158 pgs: 158 active+clean; 580 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.32k op/s 2024-03-12T03:43:15.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:15 smithi169 ceph-mon[39949]: 8.1 scrub ok 2024-03-12T03:43:15.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:15 smithi169 ceph-mon[39949]: 4.c scrub starts 2024-03-12T03:43:15.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:15 smithi169 ceph-mon[39949]: 4.c scrub ok 2024-03-12T03:43:15.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:15.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:15.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:15.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:15.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:15 smithi173 ceph-mon[33544]: 8.1 scrub starts 2024-03-12T03:43:15.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:15 smithi173 ceph-mon[33544]: pgmap v461: 158 pgs: 158 active+clean; 580 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.32k op/s 2024-03-12T03:43:15.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:15 smithi173 ceph-mon[33544]: 8.1 scrub ok 2024-03-12T03:43:15.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:15 smithi173 ceph-mon[33544]: 4.c scrub starts 2024-03-12T03:43:15.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:15 smithi173 ceph-mon[33544]: 4.c scrub ok 2024-03-12T03:43:15.560 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:15.598 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:15.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:15.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:15.729 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:15.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:15.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:15.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:15.867 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:15.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:15.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:15.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:15.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:15.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:16.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:16.146 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:16.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:16.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:16.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:16.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:16.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:16.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:16.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:16.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:16.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:16 smithi169 ceph-mon[29344]: 6.8 scrub starts 2024-03-12T03:43:16.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:16 smithi169 ceph-mon[29344]: 6.8 scrub ok 2024-03-12T03:43:16.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:16 smithi169 ceph-mon[29344]: 4.9 scrub starts 2024-03-12T03:43:16.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:16 smithi169 ceph-mon[29344]: 4.9 scrub ok 2024-03-12T03:43:16.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:16 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:16.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:16 smithi169 ceph-mon[39949]: 6.8 scrub starts 2024-03-12T03:43:16.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:16 smithi169 ceph-mon[39949]: 6.8 scrub ok 2024-03-12T03:43:16.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:16 smithi169 ceph-mon[39949]: 4.9 scrub starts 2024-03-12T03:43:16.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:16 smithi169 ceph-mon[39949]: 4.9 scrub ok 2024-03-12T03:43:16.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:16 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:16.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:16 smithi173 ceph-mon[33544]: 6.8 scrub starts 2024-03-12T03:43:16.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:16 smithi173 ceph-mon[33544]: 6.8 scrub ok 2024-03-12T03:43:16.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:16 smithi173 ceph-mon[33544]: 4.9 scrub starts 2024-03-12T03:43:16.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:16 smithi173 ceph-mon[33544]: 4.9 scrub ok 2024-03-12T03:43:16.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:16 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:16.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:16.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:16.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:16.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:16.696 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:16.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:16.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:16.842 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:16.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:16.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:16.993 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:16.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:17.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:17.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:17.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:17.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:17.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:17.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:17.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:17.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:17.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:17.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:17.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:17 smithi169 ceph-mon[29344]: pgmap v462: 158 pgs: 158 active+clean; 514 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 864 op/s 2024-03-12T03:43:17.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:17 smithi169 ceph-mon[29344]: 4.1a scrub starts 2024-03-12T03:43:17.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:17 smithi169 ceph-mon[29344]: 4.1a scrub ok 2024-03-12T03:43:17.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:17 smithi169 ceph-mon[39949]: pgmap v462: 158 pgs: 158 active+clean; 514 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 864 op/s 2024-03-12T03:43:17.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:17 smithi169 ceph-mon[39949]: 4.1a scrub starts 2024-03-12T03:43:17.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:17 smithi169 ceph-mon[39949]: 4.1a scrub ok 2024-03-12T03:43:17.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:17 smithi173 ceph-mon[33544]: pgmap v462: 158 pgs: 158 active+clean; 514 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 864 op/s 2024-03-12T03:43:17.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:17 smithi173 ceph-mon[33544]: 4.1a scrub starts 2024-03-12T03:43:17.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:17 smithi173 ceph-mon[33544]: 4.1a scrub ok 2024-03-12T03:43:17.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:17.536 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:17.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:17.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:17.637 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:17.680 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:17.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:17.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:17.817 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:17.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:17.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:17.962 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:17.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:18.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:18.094 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:18.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:18.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:18.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:18.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:18.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:18.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:18.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:18.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:18.378 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-mon[29344]: 5.1d scrub starts 2024-03-12T03:43:18.378 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-mon[29344]: 5.1d scrub ok 2024-03-12T03:43:18.378 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-mon[29344]: 5.1f scrub starts 2024-03-12T03:43:18.378 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-mon[29344]: 5.1f scrub ok 2024-03-12T03:43:18.379 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-mon[29344]: osd.3 marked itself down and dead 2024-03-12T03:43:18.379 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[113028]: 2024-03-12T03:43:18.114+0000 7fa2340ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:43:18.379 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[113028]: 2024-03-12T03:43:18.114+0000 7fa2340ea700 -1 osd.3 264 *** Got signal Terminated *** 2024-03-12T03:43:18.379 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[113028]: 2024-03-12T03:43:18.114+0000 7fa2340ea700 -1 osd.3 264 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:43:18.379 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-mon[39949]: 5.1d scrub starts 2024-03-12T03:43:18.379 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-mon[39949]: 5.1d scrub ok 2024-03-12T03:43:18.379 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-mon[39949]: 5.1f scrub starts 2024-03-12T03:43:18.379 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-mon[39949]: 5.1f scrub ok 2024-03-12T03:43:18.379 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:18 smithi169 ceph-mon[39949]: osd.3 marked itself down and dead 2024-03-12T03:43:18.466 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:18.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:18 smithi173 ceph-mon[33544]: 5.1d scrub starts 2024-03-12T03:43:18.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:18 smithi173 ceph-mon[33544]: 5.1d scrub ok 2024-03-12T03:43:18.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:18 smithi173 ceph-mon[33544]: 5.1f scrub starts 2024-03-12T03:43:18.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:18 smithi173 ceph-mon[33544]: 5.1f scrub ok 2024-03-12T03:43:18.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:18 smithi173 ceph-mon[33544]: osd.3 marked itself down and dead 2024-03-12T03:43:18.500 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:18.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:18.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:18.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:18.633 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:18.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:18.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:18.735 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:18.772 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:18.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:18.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:18.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:18.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:18.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:18.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:18.923 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:18 smithi169 podman[129271]: 2024-03-12 03:43:18.760026888 +0000 UTC m=+3.349633605 container died d630955caaecd5099827c47c69eba501c9a19f37d2a64b47df5df454feb96214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:43:19.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:19.017 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Removed 133480 objects 2024-03-12T03:43:19.017 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Clean up completed and total clean up time :123.974 2024-03-12T03:43:19.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:19.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:19.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:19.140 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:19.164 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:19.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:19.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:19.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:19.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:19.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:19.347 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:43:19.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:19.419 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:19.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:19.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[29344]: 4.17 scrub starts 2024-03-12T03:43:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[29344]: 4.17 scrub ok 2024-03-12T03:43:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[29344]: pgmap v463: 158 pgs: 158 active+clean; 216 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-12T03:43:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[29344]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T03:43:19.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[29344]: osdmap e265: 8 total, 7 up, 8 in 2024-03-12T03:43:19.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[29344]: osdmap e266: 8 total, 7 up, 8 in 2024-03-12T03:43:19.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/767151306' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:43:19.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[29344]: 6.1a scrub starts 2024-03-12T03:43:19.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[29344]: 6.1a scrub ok 2024-03-12T03:43:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[39949]: 4.17 scrub starts 2024-03-12T03:43:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[39949]: 4.17 scrub ok 2024-03-12T03:43:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[39949]: pgmap v463: 158 pgs: 158 active+clean; 216 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-12T03:43:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[39949]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T03:43:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[39949]: osdmap e265: 8 total, 7 up, 8 in 2024-03-12T03:43:19.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[39949]: osdmap e266: 8 total, 7 up, 8 in 2024-03-12T03:43:19.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/767151306' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:43:19.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[39949]: 6.1a scrub starts 2024-03-12T03:43:19.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:19 smithi169 ceph-mon[39949]: 6.1a scrub ok 2024-03-12T03:43:19.426 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:19 smithi169 podman[129271]: 2024-03-12 03:43:19.219370274 +0000 UTC m=+3.808976984 container cleanup d630955caaecd5099827c47c69eba501c9a19f37d2a64b47df5df454feb96214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-12T03:43:19.426 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:19 smithi169 bash[129271]: ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3 2024-03-12T03:43:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:19 smithi173 ceph-mon[33544]: 4.17 scrub starts 2024-03-12T03:43:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:19 smithi173 ceph-mon[33544]: 4.17 scrub ok 2024-03-12T03:43:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:19 smithi173 ceph-mon[33544]: pgmap v463: 158 pgs: 158 active+clean; 216 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-12T03:43:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:19 smithi173 ceph-mon[33544]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T03:43:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:19 smithi173 ceph-mon[33544]: osdmap e265: 8 total, 7 up, 8 in 2024-03-12T03:43:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:19 smithi173 ceph-mon[33544]: osdmap e266: 8 total, 7 up, 8 in 2024-03-12T03:43:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:19 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/767151306' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:43:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:19 smithi173 ceph-mon[33544]: 6.1a scrub starts 2024-03-12T03:43:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:19 smithi173 ceph-mon[33544]: 6.1a scrub ok 2024-03-12T03:43:19.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:19.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:19.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:19.649 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:19.665 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:43:19.666 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:43:19.666 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:43:19.666 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:43:19.666 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:43:19.666 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:43:19.666 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:43:19.666 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:43:19.666 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:43:19.666 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:43:19.666 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:43:19.667 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:43:19.667 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:43:19.667 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:43:19.667 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:43:19.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:19.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:19.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:19.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:19.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:19.887 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:19 smithi169 podman[129535]: 2024-03-12 03:43:19.555171601 +0000 UTC m=+0.873282705 container remove d630955caaecd5099827c47c69eba501c9a19f37d2a64b47df5df454feb96214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-12T03:43:19.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:19.931 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:19.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:19.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:20.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:20.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:20.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:20.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:20.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:20.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:20.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:20.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:20.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:20.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:20.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:20.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[29344]: 5.5 scrub starts 2024-03-12T03:43:20.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[29344]: 5.5 scrub ok 2024-03-12T03:43:20.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:43:20.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/767151306' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:43:20.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[29344]: osdmap e267: 8 total, 7 up, 8 in 2024-03-12T03:43:20.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3720875621' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:20.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[29344]: 1.3 scrub starts 2024-03-12T03:43:20.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[29344]: 1.3 scrub ok 2024-03-12T03:43:20.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[39949]: 5.5 scrub starts 2024-03-12T03:43:20.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[39949]: 5.5 scrub ok 2024-03-12T03:43:20.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:43:20.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/767151306' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:43:20.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[39949]: osdmap e267: 8 total, 7 up, 8 in 2024-03-12T03:43:20.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3720875621' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:20.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[39949]: 1.3 scrub starts 2024-03-12T03:43:20.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:20 smithi169 ceph-mon[39949]: 1.3 scrub ok 2024-03-12T03:43:20.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:20.442 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:20.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:20.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:20 smithi173 ceph-mon[33544]: 5.5 scrub starts 2024-03-12T03:43:20.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:20 smithi173 ceph-mon[33544]: 5.5 scrub ok 2024-03-12T03:43:20.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:20 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:43:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:20 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/767151306' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:43:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:20 smithi173 ceph-mon[33544]: osdmap e267: 8 total, 7 up, 8 in 2024-03-12T03:43:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:20 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3720875621' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:20 smithi173 ceph-mon[33544]: 1.3 scrub starts 2024-03-12T03:43:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:20 smithi173 ceph-mon[33544]: 1.3 scrub ok 2024-03-12T03:43:20.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:20.568 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:20.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:20.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:20.695 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:20.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:20.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:20.817 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:43:20.817 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:43:20.817 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-03-12T03:43:20.817 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-03-12T03:43:20.817 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:43:20.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:20.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:20.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:20.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:20.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:20.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:21.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:21.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:21.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:21.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:21.208 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:21.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:21.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:21.310 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:21.333 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:21.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:21.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:21.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:21 smithi169 ceph-mon[29344]: pgmap v467: 158 pgs: 28 stale+active+clean, 130 active+clean; 216 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 970 op/s 2024-03-12T03:43:21.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:21 smithi169 ceph-mon[29344]: 4.2 scrub starts 2024-03-12T03:43:21.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:21 smithi169 ceph-mon[29344]: 4.2 scrub ok 2024-03-12T03:43:21.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:21 smithi169 ceph-mon[39949]: pgmap v467: 158 pgs: 28 stale+active+clean, 130 active+clean; 216 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 970 op/s 2024-03-12T03:43:21.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:21 smithi169 ceph-mon[39949]: 4.2 scrub starts 2024-03-12T03:43:21.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:21 smithi169 ceph-mon[39949]: 4.2 scrub ok 2024-03-12T03:43:21.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:21.459 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:21.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:21.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:21.560 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:21.584 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:21.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:21.627 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:43:21.627 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:43:21.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:21.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:21.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:21 smithi173 ceph-mon[33544]: pgmap v467: 158 pgs: 28 stale+active+clean, 130 active+clean; 216 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 970 op/s 2024-03-12T03:43:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:21 smithi173 ceph-mon[33544]: 4.2 scrub starts 2024-03-12T03:43:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:21 smithi173 ceph-mon[33544]: 4.2 scrub ok 2024-03-12T03:43:21.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:21.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:21.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:21.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:43:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:43:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:43:21.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:21.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-12T03:43:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:21.970 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:21.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:22.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:22.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:22.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:22.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:22.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:22.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:22.248 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:22 smithi169 podman[129903]: 2024-03-12 03:43:22.064824227 +0000 UTC m=+0.018272701 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:43:22.248 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:22 smithi169 podman[129903]: 2024-03-12 03:43:22.162681546 +0000 UTC m=+0.116130011 container create d09c86f898978859d97607f6347549c06f3e30ead66014cc9ca641d5eb223c98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-12T03:43:22.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:22.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:22.358 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:22.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:22.457 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:22.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:22.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:22.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:22.570 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:43:22.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:22 smithi169 ceph-mon[29344]: 5.c scrub starts 2024-03-12T03:43:22.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:22 smithi169 ceph-mon[29344]: 5.c scrub ok 2024-03-12T03:43:22.586 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:22 smithi169 ceph-mon[39949]: 5.c scrub starts 2024-03-12T03:43:22.586 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:22 smithi169 ceph-mon[39949]: 5.c scrub ok 2024-03-12T03:43:22.586 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:22 smithi169 podman[129903]: 2024-03-12 03:43:22.32367579 +0000 UTC m=+0.277124257 container init d09c86f898978859d97607f6347549c06f3e30ead66014cc9ca641d5eb223c98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-12T03:43:22.586 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:22 smithi169 podman[129903]: 2024-03-12 03:43:22.327741019 +0000 UTC m=+0.281189487 container start d09c86f898978859d97607f6347549c06f3e30ead66014cc9ca641d5eb223c98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-03-12T03:43:22.586 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:22 smithi169 podman[129903]: 2024-03-12 03:43:22.407085939 +0000 UTC m=+0.360534420 container attach d09c86f898978859d97607f6347549c06f3e30ead66014cc9ca641d5eb223c98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-12T03:43:22.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:22.614 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:22.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:22.715 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:22 smithi173 ceph-mon[33544]: 5.c scrub starts 2024-03-12T03:43:22.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:22 smithi173 ceph-mon[33544]: 5.c scrub ok 2024-03-12T03:43:22.741 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:22.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:22.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:22.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:22.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:22.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:22.923 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:22 smithi169 podman[129903]: 2024-03-12 03:43:22.720781952 +0000 UTC m=+0.674230418 container died d09c86f898978859d97607f6347549c06f3e30ead66014cc9ca641d5eb223c98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-12T03:43:22.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:23.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:23.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:23.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:23.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:23.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:23.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:23.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:23.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:23.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:43:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:43:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:23.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:23.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:23.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:23.387 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:23.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:23.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:23.516 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:23.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:23.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:23.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:43:23.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:23.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:23.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:23.653 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:23 smithi169 ceph-mon[29344]: pgmap v468: 158 pgs: 18 active+undersized+degraded, 5 stale+active+clean, 39 active+undersized, 96 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 37 KiB/s wr, 1.27k op/s; 98/2469 objects degraded (3.969%); 26 KiB/s, 0 objects/s recovering 2024-03-12T03:43:23.653 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:23 smithi169 ceph-mon[29344]: Health check failed: Degraded data redundancy: 98/2469 objects degraded (3.969%), 18 pgs degraded (PG_DEGRADED) 2024-03-12T03:43:23.653 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:23 smithi169 ceph-mon[29344]: 4.13 scrub starts 2024-03-12T03:43:23.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:23 smithi169 ceph-mon[29344]: 4.13 scrub ok 2024-03-12T03:43:23.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:23 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:23.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:23 smithi169 ceph-mon[29344]: osdmap e268: 8 total, 7 up, 8 in 2024-03-12T03:43:23.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:23 smithi169 ceph-mon[39949]: pgmap v468: 158 pgs: 18 active+undersized+degraded, 5 stale+active+clean, 39 active+undersized, 96 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 37 KiB/s wr, 1.27k op/s; 98/2469 objects degraded (3.969%); 26 KiB/s, 0 objects/s recovering 2024-03-12T03:43:23.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:23 smithi169 ceph-mon[39949]: Health check failed: Degraded data redundancy: 98/2469 objects degraded (3.969%), 18 pgs degraded (PG_DEGRADED) 2024-03-12T03:43:23.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:23 smithi169 ceph-mon[39949]: 4.13 scrub starts 2024-03-12T03:43:23.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:23 smithi169 ceph-mon[39949]: 4.13 scrub ok 2024-03-12T03:43:23.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:23 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:23.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:23 smithi169 ceph-mon[39949]: osdmap e268: 8 total, 7 up, 8 in 2024-03-12T03:43:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:23 smithi173 ceph-mon[33544]: pgmap v468: 158 pgs: 18 active+undersized+degraded, 5 stale+active+clean, 39 active+undersized, 96 active+clean; 37 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 37 KiB/s wr, 1.27k op/s; 98/2469 objects degraded (3.969%); 26 KiB/s, 0 objects/s recovering 2024-03-12T03:43:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:23 smithi173 ceph-mon[33544]: Health check failed: Degraded data redundancy: 98/2469 objects degraded (3.969%), 18 pgs degraded (PG_DEGRADED) 2024-03-12T03:43:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:23 smithi173 ceph-mon[33544]: 4.13 scrub starts 2024-03-12T03:43:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:23 smithi173 ceph-mon[33544]: 4.13 scrub ok 2024-03-12T03:43:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:23 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:23.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:23 smithi173 ceph-mon[33544]: osdmap e268: 8 total, 7 up, 8 in 2024-03-12T03:43:23.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:23.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:23.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:23.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:23.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:23.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:23.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:23.906 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:23 smithi169 podman[130037]: 2024-03-12 03:43:23.884109726 +0000 UTC m=+1.206447138 container remove d09c86f898978859d97607f6347549c06f3e30ead66014cc9ca641d5eb223c98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-12T03:43:23.906 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:23 smithi169 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-12T03:43:23.906 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:23 smithi169 systemd[1]: Stopped Ceph osd.3 for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:43:23.906 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:43:23 smithi169 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3.service: Consumed 3min 34.699s CPU time. 2024-03-12T03:43:23.907 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3.service' 2024-03-12T03:43:23.989 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:43:23.990 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-12T03:43:23.990 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-03-12T03:43:23.990 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi169.front.sepia.ceph.com 2024-03-12T03:43:23.990 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.3 --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-03-12T03:43:24.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:24.053 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:24.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:24.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:24.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:24.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:24.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:24.285 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:24.310 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:24.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:24.375 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:24.388 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/osd.3/config 2024-03-12T03:43:24.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:24.415 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:24 smithi169 ceph-mon[29344]: 5.1a deep-scrub starts 2024-03-12T03:43:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:24 smithi169 ceph-mon[29344]: 5.1a deep-scrub ok 2024-03-12T03:43:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:24 smithi169 ceph-mon[29344]: 6.d scrub starts 2024-03-12T03:43:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:24 smithi169 ceph-mon[29344]: 6.d scrub ok 2024-03-12T03:43:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:24 smithi169 ceph-mon[29344]: 8.3 scrub starts 2024-03-12T03:43:24.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:24 smithi169 ceph-mon[29344]: 8.3 scrub ok 2024-03-12T03:43:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:24 smithi169 ceph-mon[39949]: 5.1a deep-scrub starts 2024-03-12T03:43:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:24 smithi169 ceph-mon[39949]: 5.1a deep-scrub ok 2024-03-12T03:43:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:24 smithi169 ceph-mon[39949]: 6.d scrub starts 2024-03-12T03:43:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:24 smithi169 ceph-mon[39949]: 6.d scrub ok 2024-03-12T03:43:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:24 smithi169 ceph-mon[39949]: 8.3 scrub starts 2024-03-12T03:43:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:24 smithi169 ceph-mon[39949]: 8.3 scrub ok 2024-03-12T03:43:24.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:24.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:24.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:24.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:24.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:24.567 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:24.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:24.593 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-12T03:43:24.594 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-12T03:43:24.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:24.692 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:24.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:24.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:24 smithi173 ceph-mon[33544]: 5.1a deep-scrub starts 2024-03-12T03:43:24.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:24 smithi173 ceph-mon[33544]: 5.1a deep-scrub ok 2024-03-12T03:43:24.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:24 smithi173 ceph-mon[33544]: 6.d scrub starts 2024-03-12T03:43:24.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:24 smithi173 ceph-mon[33544]: 6.d scrub ok 2024-03-12T03:43:24.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:24 smithi173 ceph-mon[33544]: 8.3 scrub starts 2024-03-12T03:43:24.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:24 smithi173 ceph-mon[33544]: 8.3 scrub ok 2024-03-12T03:43:24.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:24.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:24.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:24.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:24.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:24.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:25.052 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:25.078 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:25.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:25.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:25.205 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:25.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:25.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:25.331 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:25.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:25.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:25.456 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:25.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:25.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:25 smithi169 ceph-mon[39949]: pgmap v470: 158 pgs: 23 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 38 KiB/s wr, 602 op/s; 115/687 objects degraded (16.739%); 27 KiB/s, 0 objects/s recovering 2024-03-12T03:43:25.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:25 smithi169 ceph-mon[39949]: 5.9 deep-scrub starts 2024-03-12T03:43:25.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:25 smithi169 ceph-mon[39949]: 5.9 deep-scrub ok 2024-03-12T03:43:25.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:25 smithi169 ceph-mon[39949]: 5.3 scrub starts 2024-03-12T03:43:25.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:25 smithi169 ceph-mon[39949]: 5.3 scrub ok 2024-03-12T03:43:25.549 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:25 smithi169 ceph-mon[29344]: pgmap v470: 158 pgs: 23 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 38 KiB/s wr, 602 op/s; 115/687 objects degraded (16.739%); 27 KiB/s, 0 objects/s recovering 2024-03-12T03:43:25.549 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:25 smithi169 ceph-mon[29344]: 5.9 deep-scrub starts 2024-03-12T03:43:25.550 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:25 smithi169 ceph-mon[29344]: 5.9 deep-scrub ok 2024-03-12T03:43:25.550 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:25 smithi169 ceph-mon[29344]: 5.3 scrub starts 2024-03-12T03:43:25.550 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:25 smithi169 ceph-mon[29344]: 5.3 scrub ok 2024-03-12T03:43:25.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:25.586 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:25.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:25.688 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:25.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:25.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:25.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:25 smithi173 ceph-mon[33544]: pgmap v470: 158 pgs: 23 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 38 KiB/s wr, 602 op/s; 115/687 objects degraded (16.739%); 27 KiB/s, 0 objects/s recovering 2024-03-12T03:43:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:25 smithi173 ceph-mon[33544]: 5.9 deep-scrub starts 2024-03-12T03:43:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:25 smithi173 ceph-mon[33544]: 5.9 deep-scrub ok 2024-03-12T03:43:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:25 smithi173 ceph-mon[33544]: 5.3 scrub starts 2024-03-12T03:43:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:25 smithi173 ceph-mon[33544]: 5.3 scrub ok 2024-03-12T03:43:25.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:25.841 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:25.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:25.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:25.943 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:25.967 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:25.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:25.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:26.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:26.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:26.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:26.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:26.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:26.224 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:26.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:26.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:26.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:26.355 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:26.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:26.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:26.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:26.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:26.481 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:26.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:26.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:26.606 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:26.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:26.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:26.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:26.736 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:26.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:26.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:26.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:26.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:26.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:26.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:26.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:26.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:27.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:27.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:27.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:27.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:27.250 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:27.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:27.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:27.352 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:27.375 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:27.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:27.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:27.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:27.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:27.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:27.530 INFO:teuthology.orchestra.run.smithi169.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-12T03:43:27.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:27.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:27.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:27 smithi169 ceph-mon[29344]: pgmap v471: 158 pgs: 23 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 29 KiB/s wr, 456 op/s; 115/687 objects degraded (16.739%); 21 KiB/s, 0 objects/s recovering 2024-03-12T03:43:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:27 smithi169 ceph-mon[29344]: 5.4 scrub starts 2024-03-12T03:43:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:27 smithi169 ceph-mon[29344]: 5.4 scrub ok 2024-03-12T03:43:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:27 smithi169 ceph-mon[29344]: 8.4 deep-scrub starts 2024-03-12T03:43:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:27 smithi169 ceph-mon[29344]: 8.4 deep-scrub ok 2024-03-12T03:43:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:27 smithi169 ceph-mon[39949]: pgmap v471: 158 pgs: 23 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 29 KiB/s wr, 456 op/s; 115/687 objects degraded (16.739%); 21 KiB/s, 0 objects/s recovering 2024-03-12T03:43:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:27 smithi169 ceph-mon[39949]: 5.4 scrub starts 2024-03-12T03:43:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:27 smithi169 ceph-mon[39949]: 5.4 scrub ok 2024-03-12T03:43:27.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:27 smithi169 ceph-mon[39949]: 8.4 deep-scrub starts 2024-03-12T03:43:27.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:27 smithi169 ceph-mon[39949]: 8.4 deep-scrub ok 2024-03-12T03:43:27.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:27 smithi173 ceph-mon[33544]: pgmap v471: 158 pgs: 23 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 29 KiB/s wr, 456 op/s; 115/687 objects degraded (16.739%); 21 KiB/s, 0 objects/s recovering 2024-03-12T03:43:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:27 smithi173 ceph-mon[33544]: 5.4 scrub starts 2024-03-12T03:43:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:27 smithi173 ceph-mon[33544]: 5.4 scrub ok 2024-03-12T03:43:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:27 smithi173 ceph-mon[33544]: 8.4 deep-scrub starts 2024-03-12T03:43:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:27 smithi173 ceph-mon[33544]: 8.4 deep-scrub ok 2024-03-12T03:43:27.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:27.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:27.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:27.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:27.891 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:27.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:27.992 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:28.015 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:28.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:28.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:28.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:28.142 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:28.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:28.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:28.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:28.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:28.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:28.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:28.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:28.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:28.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:28.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[29344]: 5.2 scrub starts 2024-03-12T03:43:28.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[29344]: 5.2 scrub ok 2024-03-12T03:43:28.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[29344]: 8.6 deep-scrub starts 2024-03-12T03:43:28.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[29344]: 8.6 deep-scrub ok 2024-03-12T03:43:28.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[29344]: 3.16 scrub starts 2024-03-12T03:43:28.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[29344]: 3.16 scrub ok 2024-03-12T03:43:28.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[29344]: 6.1d scrub starts 2024-03-12T03:43:28.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[29344]: 6.1d scrub ok 2024-03-12T03:43:28.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 115/687 objects degraded (16.739%), 23 pgs degraded (PG_DEGRADED) 2024-03-12T03:43:28.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[29344]: osdmap e269: 8 total, 7 up, 8 in 2024-03-12T03:43:28.453 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[39949]: 5.2 scrub starts 2024-03-12T03:43:28.453 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[39949]: 5.2 scrub ok 2024-03-12T03:43:28.453 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[39949]: 8.6 deep-scrub starts 2024-03-12T03:43:28.453 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[39949]: 8.6 deep-scrub ok 2024-03-12T03:43:28.453 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[39949]: 3.16 scrub starts 2024-03-12T03:43:28.453 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[39949]: 3.16 scrub ok 2024-03-12T03:43:28.453 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[39949]: 6.1d scrub starts 2024-03-12T03:43:28.453 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[39949]: 6.1d scrub ok 2024-03-12T03:43:28.454 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 115/687 objects degraded (16.739%), 23 pgs degraded (PG_DEGRADED) 2024-03-12T03:43:28.454 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:28 smithi169 ceph-mon[39949]: osdmap e269: 8 total, 7 up, 8 in 2024-03-12T03:43:28.454 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:43:28.454 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:43:28.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:43:28.454 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:107724 7.3 e118/97)", 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:48.181816+0000", 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 400.267291016, 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14563426900000001, 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181816+0000", 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181816+0000", 2024-03-12T03:43:28.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181817+0000", 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8800000000000002e-07 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181844+0000", 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6902000000000001e-05 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181854+0000", 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0237999999999999e-05 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.181862+0000", 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6499999999999996e-06 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.327391+0000", 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.145529713 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.327450+0000", 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9077999999999999e-05 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.457 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.458 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:107729 7.18 7:1fba6e10:::benchmark_data_smithi169_79822_object107728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:43:28.458 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:36:48.268874+0000", 2024-03-12T03:43:28.458 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 400.18023347600001, 2024-03-12T03:43:28.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0638903, 2024-03-12T03:43:28.458 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.458 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:28.458 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:43:28.458 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:43:28.458 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:28.458 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 107729 2024-03-12T03:43:28.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.268874+0000", 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.268874+0000", 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.268877+0000", 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0819999999999999e-06 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.268912+0000", 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5632e-05 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.268927+0000", 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4759999999999999e-05 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.268941+0000", 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4099e-05 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.337241+0000", 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.068299769999999996 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:48.337249+0000", 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4409999999999993e-06 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.328599+0000", 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.99134966800000002 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.328731+0000", 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013136199999999999 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.329109+0000", 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037817299999999999 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.332209+0000", 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0030999220000000002 2024-03-12T03:43:28.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.332622+0000", 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00041314800000000001 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.332666+0000", 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4583999999999999e-05 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.332722+0000", 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5760999999999997e-05 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:36:49.332764+0000", 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1897999999999998e-05 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:123433 7.18 7:1bf673c0:::benchmark_data_smithi169_79822_object6016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:08.388510+0000", 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 380.06059759200002, 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21007616200000001, 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:28.467 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 123433 2024-03-12T03:43:28.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.468 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.388510+0000", 2024-03-12T03:43:28.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.388510+0000", 2024-03-12T03:43:28.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.388510+0000", 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6899999999999996e-07 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.388515+0000", 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5580000000000002e-06 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.388527+0000", 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2534000000000001e-05 2024-03-12T03:43:28.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.388535+0000", 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3649999999999998e-06 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.400551+0000", 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012015794999999999 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.400762+0000", 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00021125399999999999 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,6,7", 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.401167+0000", 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00040495499999999997 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.562999+0000", 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16183207199999999 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563021+0000", 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2439999999999999e-05 2024-03-12T03:43:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563069+0000", 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8158000000000002e-05 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.594649+0000", 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031579122000000001 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.594806+0000", 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015787700000000001 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.598586+0000", 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0037792630000000002 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:123452 7.18 7:18ac3f9a:::benchmark_data_smithi169_79822_object6035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:08.400278+0000", 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 380.04882938399999, 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.204641092, 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:43:28.473 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 123452 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.400278+0000", 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.400278+0000", 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.400278+0000", 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4700000000000002e-07 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.400281+0000", 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5639999999999999e-06 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.400286+0000", 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6289999999999998e-06 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.400292+0000", 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.575e-06 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405420+0000", 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051278390000000004 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405569+0000", 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014876200000000001 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,6,7", 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405906+0000", 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00033699100000000003 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563086+0000", 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15717998399999999 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563205+0000", 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000119707 2024-03-12T03:43:28.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563397+0000", 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000191993 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.598721+0000", 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.035323655000000002 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.600122+0000", 2024-03-12T03:43:28.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014006369999999999 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.604919+0000", 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0047973089999999996 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPushReply(7.18 127/122 [PushReplyOp(7:1f73a134:::benchmark_data_smithi169_79822_object10740:head)])", 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:08.401945+0000", 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 380.04716259600002, 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16489880000000001, 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.401945+0000", 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.401945+0000", 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.401945+0000", 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0400000000000002e-07 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.401949+0000", 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9809999999999997e-06 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.401955+0000", 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1750000000000002e-06 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.401961+0000", 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9649999999999996e-06 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563437+0000", 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.161476066 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.566843+0000", 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034062089999999999 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14799.0:123453 7.10 e127/122 7:0ef43bf3:::benchmark_data_smithi169_79822_object6036:head v 127'7155, mlcod=127'7153)", 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:08.404073+0000", 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 380.04503458900001, 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16400984599999999, 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.482 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.404073+0000", 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.404073+0000", 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.404073+0000", 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6400000000000003e-07 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.404078+0000", 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4959999999999998e-06 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.404084+0000", 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1399999999999997e-06 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.404088+0000", 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6249999999999998e-06 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563238+0000", 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15914983499999999 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563273+0000", 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4752999999999997e-05 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.567989+0000", 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0047158369999999996 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.568082+0000", 2024-03-12T03:43:28.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3696000000000006e-05 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.14799.0:123433 7.18 e127/122 ondisk, result = 0)", 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:08.404821+0000", 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 380.044285683, 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15820129299999999, 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.404821+0000", 2024-03-12T03:43:28.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.404821+0000", 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.404822+0000", 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2399999999999999e-07 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.404825+0000", 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.14e-06 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.404839+0000", 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4103e-05 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.404845+0000", 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6060000000000002e-06 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.562972+0000", 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15812709999999999 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.562991+0000", 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9054000000000001e-05 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563023+0000", 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1965999999999999e-05 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.14799.0:123433 7.18 e127/122 ondisk, result = 0)", 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:08.405698+0000", 2024-03-12T03:43:28.489 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 380.043409607, 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15738949399999999, 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405698+0000", 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405698+0000", 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405698+0000", 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9699999999999995e-07 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405705+0000", 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0559999999999999e-06 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405714+0000", 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9740000000000008e-06 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405721+0000", 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9330000000000004e-06 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563054+0000", 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15733285399999999 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563063+0000", 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5799999999999992e-06 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563087+0000", 2024-03-12T03:43:28.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.44e-05 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:123458 7.e 7:726a7d0a:::benchmark_data_smithi169_79822_object6041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:08.405982+0000", 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 380.043124987, 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16957428599999999, 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 123458 2024-03-12T03:43:28.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.494 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405982+0000", 2024-03-12T03:43:28.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405982+0000", 2024-03-12T03:43:28.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405983+0000", 2024-03-12T03:43:28.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7000000000000003e-07 2024-03-12T03:43:28.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405987+0000", 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0110000000000002e-06 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.405996+0000", 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4989999999999994e-06 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.406008+0000", 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2310000000000001e-05 2024-03-12T03:43:28.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.406050+0000", 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2079999999999997e-05 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.406160+0000", 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000110196 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.406561+0000", 2024-03-12T03:43:28.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00040112200000000001 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563283+0000", 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15672135300000001 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563463+0000", 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018029400000000001 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.575464+0000", 2024-03-12T03:43:28.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012001111 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.575513+0000", 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9021999999999999e-05 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.575556+0000", 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3318000000000001e-05 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.499 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.14799.0:123452 7.18 e127/122 ondisk, result = 0)", 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:08.407695+0000", 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 380.04141226600001, 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15551786000000001, 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.407695+0000", 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.407695+0000", 2024-03-12T03:43:28.500 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.407695+0000", 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0999999999999999e-07 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.407699+0000", 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4910000000000001e-06 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.562612+0000", 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15491279899999999 2024-03-12T03:43:28.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.562627+0000", 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5420000000000001e-05 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563190+0000", 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00056265700000000004 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563198+0000", 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3340000000000002e-06 2024-03-12T03:43:28.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563213+0000", 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4649e-05 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.14799.0:123452 7.18 e127/122 ondisk, result = 0)", 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:08.408866+0000", 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 380.04024087400001, 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15454095100000001, 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:43:28.503 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.408866+0000", 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.408866+0000", 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.408867+0000", 2024-03-12T03:43:28.504 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5600000000000005e-07 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.408874+0000", 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0729999999999999e-06 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.562713+0000", 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15383877500000001 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.562723+0000", 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0513000000000001e-05 2024-03-12T03:43:28.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563379+0000", 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00065594799999999995 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563390+0000", 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0805000000000001e-05 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563407+0000", 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7181e-05 2024-03-12T03:43:28.506 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.14799.0:123458 7.e e127/97 ondisk, result = 0)", 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:08.409576+0000", 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 380.03953102000003, 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15371739000000001, 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.409576+0000", 2024-03-12T03:43:28.507 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.409576+0000", 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.409577+0000", 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4099999999999999e-07 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.409580+0000", 2024-03-12T03:43:28.508 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3390000000000001e-06 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563085+0000", 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.153505375 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563101+0000", 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5866999999999999e-05 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563271+0000", 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016998999999999999 2024-03-12T03:43:28.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563277+0000", 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0920000000000003e-06 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:08.563294+0000", 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6186000000000001e-05 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.510 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:124524 7.e 7:79c85047:::benchmark_data_smithi169_79822_object7107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:09.786463+0000", 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 378.66264405099997, 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.145756409, 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 124524 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.786463+0000", 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.511 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.786463+0000", 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.786464+0000", 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2999999999999999e-07 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.786469+0000", 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7199999999999997e-06 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.786477+0000", 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6810000000000004e-06 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.786485+0000", 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3390000000000004e-06 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.786686+0000", 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020075900000000001 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.786799+0000", 2024-03-12T03:43:28.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000113344 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.786912+0000", 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000113458 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.792369+0000", 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054563499999999996 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.792393+0000", 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3822999999999999e-05 2024-03-12T03:43:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.932128+0000", 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.139735211 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.932175+0000", 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7324000000000003e-05 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.932220+0000", 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4469999999999999e-05 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.515 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:124528 7.13 7:cd339f48:::benchmark_data_smithi169_79822_object7111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:09.789128+0000", 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 378.65997882300002, 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14316630799999999, 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 124528 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.516 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789128+0000", 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789128+0000", 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789130+0000", 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3599999999999999e-06 2024-03-12T03:43:28.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789135+0000", 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6919999999999999e-06 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789147+0000", 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.168e-05 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789155+0000", 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9789999999999993e-06 2024-03-12T03:43:28.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789251+0000", 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6002999999999995e-05 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789383+0000", 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013234499999999999 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,3,4,7", 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789503+0000", 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000119862 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.793909+0000", 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044059820000000001 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.793967+0000", 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8183999999999997e-05 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.794354+0000", 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038668900000000001 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.520 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.795801+0000", 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014472599999999999 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.932087+0000", 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13628593899999999 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.932160+0000", 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2475e-05 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.932295+0000", 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013485799999999999 2024-03-12T03:43:28.521 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:124529 7.18 7:1ab82c69:::benchmark_data_smithi169_79822_object7112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:09.789174+0000", 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 378.65993288700002, 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16348067799999999, 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 124529 2024-03-12T03:43:28.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789174+0000", 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789174+0000", 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789175+0000", 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9999999999999998e-07 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789177+0000", 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4770000000000002e-06 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789183+0000", 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7060000000000001e-06 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.789188+0000", 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5010000000000004e-06 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.792679+0000", 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034910599999999998 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.792818+0000", 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013881899999999999 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,6,7", 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.793193+0000", 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037458100000000002 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.797707+0000", 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045142100000000003 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803426+0000", 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005719053 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803472+0000", 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5942000000000002e-05 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.948155+0000", 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.144682854 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.949890+0000", 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017351759999999999 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.952655+0000", 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027648989999999999 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:124538 7.18 7:1933ad47:::benchmark_data_smithi169_79822_object7121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:09.797321+0000", 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 378.65178621899997, 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15815179400000001, 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 124538 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.528 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.797321+0000", 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.797321+0000", 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.797323+0000", 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6130000000000001e-06 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.797328+0000", 2024-03-12T03:43:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2020000000000003e-06 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.797339+0000", 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1299e-05 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.797346+0000", 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1919999999999999e-06 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.797750+0000", 2024-03-12T03:43:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00040343900000000001 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.797903+0000", 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015283699999999999 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,6,7", 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.798289+0000", 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038690300000000002 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803515+0000", 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052253550000000001 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803547+0000", 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1958999999999997e-05 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803629+0000", 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2729e-05 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.952728+0000", 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.149098816 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.952792+0000", 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3414e-05 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.955473+0000", 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026810359999999999 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:124544 7.18 7:1c395009:::benchmark_data_smithi169_79822_object7127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:09.803594+0000", 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 378.64551296600001, 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15469355200000001, 2024-03-12T03:43:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 124544 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803594+0000", 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803594+0000", 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803595+0000", 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.223e-06 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803600+0000", 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.972e-06 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803610+0000", 2024-03-12T03:43:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5780000000000005e-06 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803617+0000", 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2910000000000002e-06 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803658+0000", 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0834000000000002e-05 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.803807+0000", 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014902399999999999 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,6,7", 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.804193+0000", 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038620100000000002 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.813543+0000", 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0093494700000000003 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.813572+0000", 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8926000000000001e-05 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.813606+0000", 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4276999999999997e-05 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.955534+0000", 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14192760900000001 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.955643+0000", 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109755 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:09.958288+0000", 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026443920000000002 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:124808 7.e 7:77617c82:::benchmark_data_smithi169_79822_object7391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:10.279125+0000", 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 378.16998222900003, 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14500996999999999, 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:43:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 124808 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.279125+0000", 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.279125+0000", 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.279126+0000", 2024-03-12T03:43:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8899999999999999e-07 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.279129+0000", 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7969999999999998e-06 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.279136+0000", 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8190000000000003e-06 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.279142+0000", 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9089999999999997e-06 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.282767+0000", 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036250140000000002 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.282877+0000", 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000110313 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,5", 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.282990+0000", 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112447 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.293565+0000", 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010574872000000001 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.293577+0000", 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2052000000000001e-05 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.424027+0000", 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.130450661 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.424092+0000", 2024-03-12T03:43:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4422000000000001e-05 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.424135+0000", 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3074999999999997e-05 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:124832 7.13 7:c9b4b2f2:::benchmark_data_smithi169_79822_object7415:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:10.296570+0000", 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 378.15253686400001, 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15629132900000001, 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 124832 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.296570+0000", 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.296570+0000", 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.296571+0000", 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1699999999999998e-07 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.296574+0000", 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4800000000000001e-06 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.296581+0000", 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0809999999999996e-06 2024-03-12T03:43:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.296587+0000", 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9290000000000003e-06 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.296793+0000", 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020526400000000001 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.296947+0000", 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000154649 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,3,4,7", 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.297069+0000", 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000122156 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.298531+0000", 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014619419999999999 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.298567+0000", 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5228999999999998e-05 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.299703+0000", 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001136155 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.423372+0000", 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.123669009 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.452461+0000", 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.029089634 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:43:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.452725+0000", 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00026385100000000001 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:10.452862+0000", 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013633300000000001 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPushReply(7.18 169/122 [PushReplyOp(7:19f1d56d:::benchmark_data_smithi169_79822_object98865:head)])", 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:38:30.648429+0000", 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 297.80067857699999, 2024-03-12T03:43:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14376434900000001, 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:30.648429+0000", 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:30.648429+0000", 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:30.648435+0000", 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4389999999999997e-06 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:30.648445+0000", 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8069999999999994e-06 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:30.648466+0000", 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1273000000000001e-05 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:43:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:30.648481+0000", 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.507e-05 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:30.648569+0000", 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7318999999999995e-05 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:38:30.792193+0000", 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14362444099999999 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:43:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:43:28.554 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:43:28.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:28.578 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:28.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:28.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:28.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:28.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:28 smithi173 ceph-mon[33544]: 5.2 scrub starts 2024-03-12T03:43:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:28 smithi173 ceph-mon[33544]: 5.2 scrub ok 2024-03-12T03:43:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:28 smithi173 ceph-mon[33544]: 8.6 deep-scrub starts 2024-03-12T03:43:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:28 smithi173 ceph-mon[33544]: 8.6 deep-scrub ok 2024-03-12T03:43:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:28 smithi173 ceph-mon[33544]: 3.16 scrub starts 2024-03-12T03:43:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:28 smithi173 ceph-mon[33544]: 3.16 scrub ok 2024-03-12T03:43:28.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:28 smithi173 ceph-mon[33544]: 6.1d scrub starts 2024-03-12T03:43:28.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:28 smithi173 ceph-mon[33544]: 6.1d scrub ok 2024-03-12T03:43:28.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:28 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 115/687 objects degraded (16.739%), 23 pgs degraded (PG_DEGRADED) 2024-03-12T03:43:28.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:28 smithi173 ceph-mon[33544]: osdmap e269: 8 total, 7 up, 8 in 2024-03-12T03:43:28.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:28.833 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:28.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:28.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:28.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:28.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:28.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:29.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:29.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:29.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:29.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:29.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:29.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:29.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:29.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:29.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:29.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:29.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:29.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:29.460 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:29.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:29.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-12T03:43:29.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:29.586 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:29.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:29 smithi173 ceph-mon[33544]: pgmap v472: 158 pgs: 23 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 25 KiB/s wr, 404 op/s; 115/687 objects degraded (16.739%); 18 KiB/s, 0 objects/s recovering 2024-03-12T03:43:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:29 smithi173 ceph-mon[33544]: 6.12 scrub starts 2024-03-12T03:43:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:29 smithi173 ceph-mon[33544]: 6.12 scrub ok 2024-03-12T03:43:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:29 smithi173 ceph-mon[33544]: 6.1 scrub starts 2024-03-12T03:43:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:29 smithi173 ceph-mon[33544]: 6.1 scrub ok 2024-03-12T03:43:29.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:29 smithi169 ceph-mon[39949]: pgmap v472: 158 pgs: 23 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 25 KiB/s wr, 404 op/s; 115/687 objects degraded (16.739%); 18 KiB/s, 0 objects/s recovering 2024-03-12T03:43:29.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:29 smithi169 ceph-mon[39949]: 6.12 scrub starts 2024-03-12T03:43:29.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:29 smithi169 ceph-mon[39949]: 6.12 scrub ok 2024-03-12T03:43:29.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:29 smithi169 ceph-mon[39949]: 6.1 scrub starts 2024-03-12T03:43:29.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:29 smithi169 ceph-mon[39949]: 6.1 scrub ok 2024-03-12T03:43:29.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:29 smithi169 ceph-mon[29344]: pgmap v472: 158 pgs: 23 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 25 KiB/s wr, 404 op/s; 115/687 objects degraded (16.739%); 18 KiB/s, 0 objects/s recovering 2024-03-12T03:43:29.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:29 smithi169 ceph-mon[29344]: 6.12 scrub starts 2024-03-12T03:43:29.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:29 smithi169 ceph-mon[29344]: 6.12 scrub ok 2024-03-12T03:43:29.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:29 smithi169 ceph-mon[29344]: 6.1 scrub starts 2024-03-12T03:43:29.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:29 smithi169 ceph-mon[29344]: 6.1 scrub ok 2024-03-12T03:43:29.687 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:29.694 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:29.714 INFO:teuthology.orchestra.run.smithi169.stdout:8.f 2024-03-12T03:43:29.714 INFO:teuthology.orchestra.run.smithi169.stdout:8.c 2024-03-12T03:43:29.714 INFO:teuthology.orchestra.run.smithi169.stdout:8.b 2024-03-12T03:43:29.714 INFO:teuthology.orchestra.run.smithi169.stdout:8.8 2024-03-12T03:43:29.714 INFO:teuthology.orchestra.run.smithi169.stdout:8.7 2024-03-12T03:43:29.714 INFO:teuthology.orchestra.run.smithi169.stdout:8.0 2024-03-12T03:43:29.714 INFO:teuthology.orchestra.run.smithi169.stdout:6.6 2024-03-12T03:43:29.714 INFO:teuthology.orchestra.run.smithi169.stdout:6.c 2024-03-12T03:43:29.714 INFO:teuthology.orchestra.run.smithi169.stdout:6.4 2024-03-12T03:43:29.714 INFO:teuthology.orchestra.run.smithi169.stdout:6.f 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:6.2 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:6.1f 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:6.16 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:6.7 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:6.11 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:5.a 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:5.1c 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:5.14 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:6.0 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:5.16 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:5.13 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:5.12 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:5.0 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:3.1e 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:6.5 2024-03-12T03:43:29.715 INFO:teuthology.orchestra.run.smithi169.stdout:4.16 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:1.7 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:4.1e 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:6.b 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:3.2 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:3.8 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:4.4 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:4.c 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:3.17 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:6.13 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:3.5 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:6.19 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:4.0 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:3.15 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:1.6 2024-03-12T03:43:29.716 INFO:teuthology.orchestra.run.smithi169.stdout:2.1 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:5.1 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:3.18 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:6.8 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:1.9 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:3.b 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:4.10 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:4.11 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:3.12 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:4.9 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:4.17 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:4.14 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:3.1 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:4.19 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:6.1b 2024-03-12T03:43:29.717 INFO:teuthology.orchestra.run.smithi169.stdout:3.7 2024-03-12T03:43:29.718 INFO:teuthology.orchestra.run.smithi169.stdout:3.19 2024-03-12T03:43:29.718 INFO:teuthology.orchestra.run.smithi169.stdout:1.8 2024-03-12T03:43:29.718 INFO:teuthology.orchestra.run.smithi169.stdout:3.c 2024-03-12T03:43:29.718 INFO:teuthology.orchestra.run.smithi169.stdout:4.1a 2024-03-12T03:43:29.718 INFO:teuthology.orchestra.run.smithi169.stdout:5.17 2024-03-12T03:43:29.718 INFO:teuthology.orchestra.run.smithi169.stdout:4.1b 2024-03-12T03:43:29.718 INFO:teuthology.orchestra.run.smithi169.stdout:4.15 2024-03-12T03:43:29.718 INFO:teuthology.orchestra.run.smithi169.stdout:4.1c 2024-03-12T03:43:29.718 INFO:teuthology.orchestra.run.smithi169.stdout:1.4 2024-03-12T03:43:29.718 INFO:teuthology.orchestra.run.smithi169.stdout:4.1d 2024-03-12T03:43:29.718 INFO:teuthology.orchestra.run.smithi169.stdout:4.f 2024-03-12T03:43:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:29.719 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:29.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:29.725 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:43:29.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:29.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:29.844 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:29.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:29.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:29.969 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:29.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:30.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:30.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:30.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:30.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:30.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:30.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:30.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:30.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:30.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:30.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:30.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:30.377 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:43:30.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:30 smithi169 ceph-mon[39949]: 4.7 scrub starts 2024-03-12T03:43:30.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:30 smithi169 ceph-mon[39949]: 4.7 scrub ok 2024-03-12T03:43:30.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:30 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1709540666' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:43:30.410 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:30 smithi169 ceph-mon[29344]: 4.7 scrub starts 2024-03-12T03:43:30.410 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:30 smithi169 ceph-mon[29344]: 4.7 scrub ok 2024-03-12T03:43:30.410 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:30 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1709540666' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:43:30.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:30.472 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:30.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:30.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:30 smithi173 ceph-mon[33544]: 4.7 scrub starts 2024-03-12T03:43:30.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:30 smithi173 ceph-mon[33544]: 4.7 scrub ok 2024-03-12T03:43:30.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:30 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1709540666' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:43:30.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:30.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:30.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:30.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:30.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:30.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:30.824 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:30.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:30.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:30.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:30.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:30.974 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:30.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:30.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:31.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:31.100 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:31.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:31.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:31.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:31.226 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:31.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:31.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:31.351 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:31.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:31.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:31.478 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:31.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:31.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:31 smithi173 ceph-mon[33544]: pgmap v474: 158 pgs: 23 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 74 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:31.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:31 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:43:31.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:31 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1709540666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:43:31.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:31 smithi173 ceph-mon[33544]: osdmap e270: 8 total, 7 up, 8 in 2024-03-12T03:43:31.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:31 smithi173 ceph-mon[33544]: 8.d scrub starts 2024-03-12T03:43:31.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:31 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:43:31.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:31.569 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:43:31.570 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:43:31.570 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-03-12T03:43:31.570 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:43:31.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:31.604 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:31.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:31.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[39949]: pgmap v474: 158 pgs: 23 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 74 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:43:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1709540666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:43:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[39949]: osdmap e270: 8 total, 7 up, 8 in 2024-03-12T03:43:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[39949]: 8.d scrub starts 2024-03-12T03:43:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:43:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[29344]: pgmap v474: 158 pgs: 23 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 74 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:43:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1709540666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:43:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[29344]: osdmap e270: 8 total, 7 up, 8 in 2024-03-12T03:43:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[29344]: 8.d scrub starts 2024-03-12T03:43:31.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:43:31.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:31.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:31.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:31.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:31.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:31.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:31.856 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:31.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:31.957 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:31.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:31.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:31.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:32.022 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:43:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:43:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:43:32.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.3 --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 4.1c --file /var/log/ceph/exp.4.1c.3 2024-03-12T03:43:32.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:32.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:32.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:32.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:32.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-12T03:43:32.232 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:32.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:32.258 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/osd.3/config 2024-03-12T03:43:32.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:32.357 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:32.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:32.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:32.468 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:43:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:32.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:32.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:32.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:32 smithi173 ceph-mon[33544]: 5.11 scrub starts 2024-03-12T03:43:32.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:32 smithi173 ceph-mon[33544]: 3.0 deep-scrub starts 2024-03-12T03:43:32.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:32 smithi173 ceph-mon[33544]: 3.0 deep-scrub ok 2024-03-12T03:43:32.586 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:32.612 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:32.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:32.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:32.644 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:32 smithi169 ceph-mon[39949]: 5.11 scrub starts 2024-03-12T03:43:32.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:32 smithi169 ceph-mon[39949]: 3.0 deep-scrub starts 2024-03-12T03:43:32.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:32 smithi169 ceph-mon[39949]: 3.0 deep-scrub ok 2024-03-12T03:43:32.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:32 smithi169 ceph-mon[29344]: 5.11 scrub starts 2024-03-12T03:43:32.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:32 smithi169 ceph-mon[29344]: 3.0 deep-scrub starts 2024-03-12T03:43:32.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:32 smithi169 ceph-mon[29344]: 3.0 deep-scrub ok 2024-03-12T03:43:32.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:32.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:32.840 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:32.845 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:43:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:43:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:32.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:32.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:32.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:32.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:32.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:33.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:33.117 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:33.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:33.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:33.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:33.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:33.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:33.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:33.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:33.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:33.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:33.493 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:33.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:33.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:33.558 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:33 smithi173 ceph-mon[33544]: pgmap v476: 158 pgs: 1 active+clean+scrubbing, 23 active+undersized+degraded, 44 active+undersized, 90 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:33.558 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:33 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:33.558 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:33 smithi173 ceph-mon[33544]: osdmap e271: 8 total, 7 up, 8 in 2024-03-12T03:43:33.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:33.598 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:33 smithi169 ceph-mon[29344]: pgmap v476: 158 pgs: 1 active+clean+scrubbing, 23 active+undersized+degraded, 44 active+undersized, 90 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:33.598 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:33 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:33.598 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:33 smithi169 ceph-mon[29344]: osdmap e271: 8 total, 7 up, 8 in 2024-03-12T03:43:33.599 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:33 smithi169 ceph-mon[39949]: pgmap v476: 158 pgs: 1 active+clean+scrubbing, 23 active+undersized+degraded, 44 active+undersized, 90 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:33.599 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:33 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:33.599 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:33 smithi169 ceph-mon[39949]: osdmap e271: 8 total, 7 up, 8 in 2024-03-12T03:43:33.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:43:33.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:33.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:33.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:33.650 INFO:teuthology.orchestra.run.smithi169.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-12T03:43:33.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:33.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:33.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:33.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:33.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:33.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:33.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:33.880 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:43:33.900 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:43:33.919 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:43:33.939 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:43:33.959 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:43:33.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:33.979 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:43:33.997 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:33.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:33.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:33.999 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:43:34.000 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:43:34.000 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:43:34.000 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:43:34.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:43:34.000 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:43:34.000 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:43:34.000 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:43:34.000 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:43:34.087 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:43:34.087 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:43:34.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:34.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:34.158 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:34.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:34.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:34.283 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:34.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:34.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:34.408 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:34.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:34.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:34.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:34.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:34.636 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:34.661 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:34.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:34.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:34 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2489435872' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:34.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:34 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2489435872' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:34.692 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:34 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2489435872' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:34.699 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:43:34.700 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:43:34.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-12T03:43:34.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:34.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:34.787 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:34.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:34.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:34.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:34.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:34.961 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:43:35.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:35.040 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:35.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:35.108 INFO:teuthology.orchestra.run.smithi169.stderr:Exporting 4.1c info 4.1c( v 74'15 (0'0,74'15] local-lis/les=222/223 n=7 ec=68/68 lis/c=222/222 les/c/f=223/223/0 sis=222) 2024-03-12T03:43:35.111 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:38283b28:::obj_delete_at_hint.0000000089:head# 2024-03-12T03:43:35.111 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:388bc67b:::obj_delete_at_hint.0000000022:head# 2024-03-12T03:43:35.112 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:3b93e7b4:::obj_delete_at_hint.0000000016:head# 2024-03-12T03:43:35.113 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:3d9e841d:::obj_delete_at_hint.0000000041:head# 2024-03-12T03:43:35.113 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:3e426232:gc::gc.9:head# 2024-03-12T03:43:35.113 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:3fb1de85:::obj_delete_at_hint.0000000018:head# 2024-03-12T03:43:35.113 INFO:teuthology.orchestra.run.smithi169.stderr:Read #4:3ffaa114:::obj_delete_at_hint.0000000092:head# 2024-03-12T03:43:35.113 INFO:teuthology.orchestra.run.smithi169.stderr:Export successful 2024-03-12T03:43:35.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:35.167 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:35.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:35.269 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:35.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:35.293 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:35.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:35.394 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:35.418 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:35.419 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:35.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:35.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:35.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:35.545 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:35.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:35.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:35 smithi169 ceph-mon[29344]: pgmap v478: 158 pgs: 1 active+clean+scrubbing, 23 active+undersized+degraded, 44 active+undersized, 90 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:35.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:35 smithi169 ceph-mon[39949]: pgmap v478: 158 pgs: 1 active+clean+scrubbing, 23 active+undersized+degraded, 44 active+undersized, 90 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:35.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:35.670 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:35.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:35.676 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:35 smithi173 ceph-mon[33544]: pgmap v478: 158 pgs: 1 active+clean+scrubbing, 23 active+undersized+degraded, 44 active+undersized, 90 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:35.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:35.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:35.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:35.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:35.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:35.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:36.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:36.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:36.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:36.157 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:36.181 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:36.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:36.283 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:36.307 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:36.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:36.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:36.409 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:36.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:36.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:36.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:36.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:36.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:36.570 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:43:36.570 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:43:36.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:36.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:36.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:36.701 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:43:36.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:36.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:36.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:36.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:36.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:36.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:37.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:37.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:37.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:37.163 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:37.187 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:37.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:37.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-12T03:43:37.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:37.313 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:37.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:37.403 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'unique_pool_1' removed 2024-03-12T03:43:37.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:37.443 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:37.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:37.458 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:43:37.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:37 smithi173 ceph-mon[33544]: pgmap v479: 158 pgs: 1 active+clean+scrubbing, 23 active+undersized+degraded, 44 active+undersized, 90 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:37.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:37 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1249650055' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:43:37.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:37.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:37.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:37.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:37 smithi169 ceph-mon[29344]: pgmap v479: 158 pgs: 1 active+clean+scrubbing, 23 active+undersized+degraded, 44 active+undersized, 90 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:37.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:37 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1249650055' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:43:37.585 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:37 smithi169 ceph-mon[39949]: pgmap v479: 158 pgs: 1 active+clean+scrubbing, 23 active+undersized+degraded, 44 active+undersized, 90 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:37.585 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:37 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1249650055' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:43:37.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:37.698 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:37.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:37.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:37.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:37.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:37.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.3 --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 4.1c 2024-03-12T03:43:37.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:37.950 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:37.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:37.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:38.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:38.076 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/osd.3/config 2024-03-12T03:43:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:38.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:38.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:38.178 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:38.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:38.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:38.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:38.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:38.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:38.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:38.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:38 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1249650055' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:43:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:38 smithi173 ceph-mon[33544]: osdmap e272: 8 total, 7 up, 8 in 2024-03-12T03:43:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:38 smithi173 ceph-mon[33544]: osdmap e273: 8 total, 7 up, 8 in 2024-03-12T03:43:38.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:38.567 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:38.582 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:38.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:38.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:38.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:38 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1249650055' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:43:38.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:38 smithi169 ceph-mon[29344]: osdmap e272: 8 total, 7 up, 8 in 2024-03-12T03:43:38.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:38 smithi169 ceph-mon[29344]: osdmap e273: 8 total, 7 up, 8 in 2024-03-12T03:43:38.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:38 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1249650055' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:43:38.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:38 smithi169 ceph-mon[39949]: osdmap e272: 8 total, 7 up, 8 in 2024-03-12T03:43:38.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:38 smithi169 ceph-mon[39949]: osdmap e273: 8 total, 7 up, 8 in 2024-03-12T03:43:38.683 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:38.706 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:38.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:38.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:38.832 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:38.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:38.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:38.934 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:38.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:38.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:38.993 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-12T03:43:38.994 INFO:tasks.radosbench:Beginning radosbench... 2024-03-12T03:43:38.994 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-12T03:43:38.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-12T03:43:39.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:39.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:39.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:39.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:39.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:39.253 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:43:39.253 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:43:39.311 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:39.335 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:39.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:39.411 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:43:39.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:39.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:39.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:39 smithi173 ceph-mon[33544]: pgmap v481: 142 pgs: 23 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 58 KiB/s rd, 57 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:39 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1165362350' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:43:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:39 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:43:39.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:39.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:39.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:39.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:39.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:39 smithi169 ceph-mon[39949]: pgmap v481: 142 pgs: 23 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 58 KiB/s rd, 57 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:39.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:39 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1165362350' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:43:39.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:39 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:43:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:39 smithi169 ceph-mon[29344]: pgmap v481: 142 pgs: 23 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 58 KiB/s rd, 57 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:39 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1165362350' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:43:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:39 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:43:39.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:39.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:39.715 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:39.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:39.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:39.840 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:39.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:39.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-12T03:43:39.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:39.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:39.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:40.068 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:40.093 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:40.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:40.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:40.135 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:43:40.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:40.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:40.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:40.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:40.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:40.346 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:40.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:40.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:40.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:40.473 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:40.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:40.569 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:40 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:43:40.569 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:40 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:43:40.569 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:40 smithi173 ceph-mon[33544]: osdmap e274: 8 total, 7 up, 8 in 2024-03-12T03:43:40.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:40.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:40.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:40.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:40.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:40 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:43:40.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:40 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:43:40.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:40 smithi169 ceph-mon[29344]: osdmap e274: 8 total, 7 up, 8 in 2024-03-12T03:43:40.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:40 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:43:40.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:40 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:43:40.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:40 smithi169 ceph-mon[39949]: osdmap e274: 8 total, 7 up, 8 in 2024-03-12T03:43:40.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:40.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:40.725 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:40.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:40.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:40.850 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:40.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:40.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:40.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:40.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:40.976 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:40.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:41.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:41.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:41.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:41.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:41.120 INFO:teuthology.orchestra.run.smithi169.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-12T03:43:41.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:41.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:41.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:41.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:41.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:41.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:41.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:41.410 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:41 smithi173 ceph-mon[33544]: pgmap v484: 142 pgs: 23 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:41.410 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:41 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:43:41.410 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:41 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:43:41.410 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:41 smithi173 ceph-mon[33544]: osdmap e275: 8 total, 7 up, 8 in 2024-03-12T03:43:41.410 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:41 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:43:41.410 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:41 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:43:41.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:41.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:41.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:41.509 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:41.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:41.624 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:41.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:41.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-mon[39949]: pgmap v484: 142 pgs: 23 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:41.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:43:41.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:43:41.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-mon[39949]: osdmap e275: 8 total, 7 up, 8 in 2024-03-12T03:43:41.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:43:41.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:43:41.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-mon[29344]: pgmap v484: 142 pgs: 23 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:41.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:43:41.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:43:41.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-mon[29344]: osdmap e275: 8 total, 7 up, 8 in 2024-03-12T03:43:41.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:43:41.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:43:41.675 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:43:41.675 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:43:41.726 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:41.751 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:41.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:41.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:41.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:41.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:41.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:41.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:42.003 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:42.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:42.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:42.076 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:43:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:43:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:43:42.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:42.128 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:42.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:42.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:42.254 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:42.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:42.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:42.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-12T03:43:42.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:42.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:42.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:42.518 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:42.542 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:42.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:42.588 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:43:42.627 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:43:42.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:42.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:42 smithi169 ceph-mon[29344]: osdmap e276: 8 total, 7 up, 8 in 2024-03-12T03:43:42.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:42 smithi169 ceph-mon[39949]: osdmap e276: 8 total, 7 up, 8 in 2024-03-12T03:43:42.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:42.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:42.710 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:42 smithi173 ceph-mon[33544]: osdmap e276: 8 total, 7 up, 8 in 2024-03-12T03:43:42.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:42.792 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:42.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:42.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:42.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:42.918 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:42.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:42.967 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:43:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:43:43.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:43.043 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:43.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:43.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:43.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:43.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:43.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:43.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:43.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:43.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:43.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:43.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:43.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:43.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:43.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:43.546 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:43.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:43.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:43.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:43.672 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:43.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:43.713 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:43 smithi173 ceph-mon[33544]: pgmap v487: 142 pgs: 23 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:43.713 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:43 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:43.713 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:43 smithi173 ceph-mon[33544]: osdmap e277: 8 total, 7 up, 8 in 2024-03-12T03:43:43.773 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:43.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:43 smithi169 ceph-mon[29344]: pgmap v487: 142 pgs: 23 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:43.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:43 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:43.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:43 smithi169 ceph-mon[29344]: osdmap e277: 8 total, 7 up, 8 in 2024-03-12T03:43:43.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:43 smithi169 ceph-mon[39949]: pgmap v487: 142 pgs: 23 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:43.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:43 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:43.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:43 smithi169 ceph-mon[39949]: osdmap e277: 8 total, 7 up, 8 in 2024-03-12T03:43:43.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:43.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:43.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:43.925 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:43.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:44.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:44.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:44.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:44.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:44.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:44.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:44.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:44.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:44.300 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:43:44.300 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:43:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:44.301 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:44.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:44.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:44.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:44.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:44.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:44.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:44.552 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:44.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:44.580 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:44 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:43:44.580 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:44 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:43:44.580 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:44 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/316661662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-12T03:43:44.590 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'unique_pool_2' created 2024-03-12T03:43:44.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:44.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:44.679 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:44.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:44.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:44.803 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:44.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:44.875 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:44 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:43:44.875 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:44 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:43:44.875 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:44 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/316661662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-12T03:43:44.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:44 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:43:44.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:44 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:43:44.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:44 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/316661662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-12T03:43:44.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:44.928 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:44.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:44.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-12T03:43:45.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:45.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:45.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:45.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:45.182 INFO:teuthology.orchestra.run.smithi169.stdout: marking collection for removal 2024-03-12T03:43:45.182 INFO:teuthology.orchestra.run.smithi169.stdout:setting '_remove' omap key 2024-03-12T03:43:45.182 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:45.183 INFO:teuthology.orchestra.run.smithi169.stdout:finish_remove_pgs 4.1c_head removing 4.1c 2024-03-12T03:43:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:45.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:45.185 INFO:teuthology.orchestra.run.smithi169.stdout:Remove successful 2024-03-12T03:43:45.188 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:43:45.225 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:45.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:45.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:45.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:45.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:45.434 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:45.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:45.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:45.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:45.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:45.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:45.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:45.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:45.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:45.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:45.813 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:45.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:45.825 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-12T03:43:45.889 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:45 smithi169 ceph-mon[29344]: pgmap v489: 142 pgs: 23 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:45.890 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:45 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-12T03:43:45.890 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:45 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-12T03:43:45.890 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:45 smithi169 ceph-mon[29344]: osdmap e278: 8 total, 7 up, 8 in 2024-03-12T03:43:45.890 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:45 smithi169 ceph-mon[39949]: pgmap v489: 142 pgs: 23 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:45.890 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:45 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-12T03:43:45.890 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:45 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-12T03:43:45.890 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:45 smithi169 ceph-mon[39949]: osdmap e278: 8 total, 7 up, 8 in 2024-03-12T03:43:45.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:45.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:45.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:45.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:45.943 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:45 smithi173 ceph-mon[33544]: pgmap v489: 142 pgs: 23 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:45.943 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:45 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-12T03:43:45.944 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:45 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-12T03:43:45.944 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:45 smithi173 ceph-mon[33544]: osdmap e278: 8 total, 7 up, 8 in 2024-03-12T03:43:46.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:46.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:46.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:46.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:46.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:46.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:46.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:46.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:46.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:46.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:46.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.3 --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.1c.3 2024-03-12T03:43:46.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:46.442 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:46.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:46.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:46.569 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:46.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:46.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:46.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:46 smithi169 ceph-mon[29344]: osdmap e279: 8 total, 7 up, 8 in 2024-03-12T03:43:46.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:46 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:46.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:46 smithi169 ceph-mon[29344]: osdmap e280: 8 total, 7 up, 8 in 2024-03-12T03:43:46.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:46 smithi169 ceph-mon[39949]: osdmap e279: 8 total, 7 up, 8 in 2024-03-12T03:43:46.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:46 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:46.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:46 smithi169 ceph-mon[39949]: osdmap e280: 8 total, 7 up, 8 in 2024-03-12T03:43:46.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:46.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:46.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:46.761 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:43:46.761 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:43:46.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:46.820 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:46.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:46.839 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/osd.3/config 2024-03-12T03:43:46.858 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:43:46.884 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:43:46.903 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:43:46.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:46.923 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:43:46.943 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:43:46.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:46.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:46.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:46.962 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:43:46.982 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:43:46.982 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:43:46.983 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:43:46.983 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:43:46.983 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:43:46.983 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:43:46.983 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:43:46.983 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:43:46.983 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:43:46.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:46 smithi173 ceph-mon[33544]: osdmap e279: 8 total, 7 up, 8 in 2024-03-12T03:43:46.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:46 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:46.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:46 smithi173 ceph-mon[33544]: osdmap e280: 8 total, 7 up, 8 in 2024-03-12T03:43:47.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:47.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:47.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:47.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:47.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:47.194 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:47.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:47.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:47.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:47.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:47.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:47.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:47.431 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-12T03:43:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:47.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:47.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:47.547 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:47.575 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:47.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:47.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:47.678 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:43:47.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:47.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:47 smithi173 ceph-mon[33544]: pgmap v492: 158 pgs: 12 unknown, 23 active+undersized+degraded, 40 active+undersized, 83 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:47 smithi173 ceph-mon[33544]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:43:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:47 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2361112821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:47.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:47.827 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:47.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:47 smithi169 ceph-mon[29344]: pgmap v492: 158 pgs: 12 unknown, 23 active+undersized+degraded, 40 active+undersized, 83 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:47 smithi169 ceph-mon[29344]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:43:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:47 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2361112821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:47 smithi169 ceph-mon[39949]: pgmap v492: 158 pgs: 12 unknown, 23 active+undersized+degraded, 40 active+undersized, 83 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:47 smithi169 ceph-mon[39949]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:43:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:47 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2361112821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:47.928 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:47.956 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:47.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:47.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:48.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:48.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:48.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:48.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:43:48.199 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:43:48.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:48.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:48.218 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:48.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:48.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:48.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:48.459 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:48.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:48.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:48.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:48.585 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:48.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:48.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:48.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:48.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:48.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:48.837 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:48.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:48.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:48.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:48.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:48.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:49.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:49.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:49.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:49.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:49.163 INFO:teuthology.orchestra.run.smithi169.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-12T03:43:49.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:49.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:49.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:49.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:49.298 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:43:49.298 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:43:49.298 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:43:49.298 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:43:49.298 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:43:49.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:49.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:49.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:49.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:49.414 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:49 smithi169 ceph-mon[29344]: pgmap v494: 158 pgs: 23 active+undersized+degraded, 46 active+undersized, 89 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:49.414 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:49 smithi169 ceph-mon[29344]: osdmap e281: 8 total, 7 up, 8 in 2024-03-12T03:43:49.414 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:49 smithi169 ceph-mon[39949]: pgmap v494: 158 pgs: 23 active+undersized+degraded, 46 active+undersized, 89 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:49.414 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:49 smithi169 ceph-mon[39949]: osdmap e281: 8 total, 7 up, 8 in 2024-03-12T03:43:49.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:49.464 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:49.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:49.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:49.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:49.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:49.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:49 smithi173 ceph-mon[33544]: pgmap v494: 158 pgs: 23 active+undersized+degraded, 46 active+undersized, 89 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:49.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:49 smithi173 ceph-mon[33544]: osdmap e281: 8 total, 7 up, 8 in 2024-03-12T03:43:49.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:49.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:49.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:49.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:49.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:49.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:49.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:49.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:49.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:49.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:49.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:49.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-12T03:43:50.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:50.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:50.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:50.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:50.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:50.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:43:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:50.221 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:50.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:50.270 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:43:50.301 INFO:teuthology.orchestra.run.smithi169.stdout:get_pg_num_history pg_num_history pg_num_history(e264 pg_nums {1={149=11},2={65=3},3={66=32},4={68=32},5={70=32},6={72=32},8={198=16}} deleted_pools ) 2024-03-12T03:43:50.301 INFO:teuthology.orchestra.run.smithi169.stdout:Importing pgid 4.1c 2024-03-12T03:43:50.301 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:38283b28:::obj_delete_at_hint.0000000089:head# 2024-03-12T03:43:50.301 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:43:50.301 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:388bc67b:::obj_delete_at_hint.0000000022:head# 2024-03-12T03:43:50.302 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:43:50.302 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:3b93e7b4:::obj_delete_at_hint.0000000016:head# 2024-03-12T03:43:50.302 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:43:50.302 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:3d9e841d:::obj_delete_at_hint.0000000041:head# 2024-03-12T03:43:50.302 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:43:50.302 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:3e426232:gc::gc.9:head# 2024-03-12T03:43:50.302 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:43:50.303 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:3fb1de85:::obj_delete_at_hint.0000000018:head# 2024-03-12T03:43:50.303 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:43:50.303 INFO:teuthology.orchestra.run.smithi169.stdout:Write #4:3ffaa114:::obj_delete_at_hint.0000000092:head# 2024-03-12T03:43:50.303 INFO:teuthology.orchestra.run.smithi169.stdout:snapset 0=[]:{} 2024-03-12T03:43:50.303 INFO:teuthology.orchestra.run.smithi169.stdout:write_pg epoch 246 info 4.1c( v 74'15 (0'0,74'15] local-lis/les=222/223 n=7 ec=68/68 lis/c=222/222 les/c/f=223/223/0 sis=222) 2024-03-12T03:43:50.303 INFO:teuthology.orchestra.run.smithi169.stdout:Import successful 2024-03-12T03:43:50.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:50.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:50.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:50.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:50.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:50.471 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:50.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:50.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:50.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:50.596 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:50.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:50.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:50.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:50.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:50.825 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:50.851 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:50.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:50.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:50.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:50.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:50.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:51.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:51.101 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:51.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:51.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:51.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:51.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:51.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:51.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:51.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:51.353 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:51.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:51.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:51.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:51 smithi169 ceph-mon[29344]: pgmap v496: 158 pgs: 23 active+undersized+degraded, 46 active+undersized, 89 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:51.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:51 smithi169 ceph-mon[39949]: pgmap v496: 158 pgs: 23 active+undersized+degraded, 46 active+undersized, 89 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:51.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:51.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:51.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:51.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/exp.4.1c.3 2024-03-12T03:43:51.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:51.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:51.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:51.620 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:51 smithi173 ceph-mon[33544]: pgmap v496: 158 pgs: 23 active+undersized+degraded, 46 active+undersized, 89 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:51.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:51.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:51.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:51.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:51.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:51.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:51.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:51.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:51.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:51.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:52.004 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:52.038 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:43:52.038 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:43:52.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:43:52.038 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:43:52.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.038 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107727 7.17 7:ea575e37:::benchmark_data_smithi169_79822_object107726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.269042+0000", 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 423.73031460700003, 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0759451440000001, 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107727 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269042+0000", 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269042+0000", 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269044+0000", 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0250000000000001e-06 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269080+0000", 2024-03-12T03:43:52.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5604e-05 2024-03-12T03:43:52.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269135+0000", 2024-03-12T03:43:52.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5294000000000003e-05 2024-03-12T03:43:52.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.269165+0000", 2024-03-12T03:43:52.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0079000000000001e-05 2024-03-12T03:43:52.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.292245+0000", 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.023079873000000001 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.292257+0000", 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1508999999999999e-05 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.327768+0000", 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.035511074 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for degraded object", 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.328777+0000", 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001009111 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.344893+0000", 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.016116164999999998 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.344987+0000", 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4409999999999999e-05 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:184738 7.19 7:9d3bf2f7:::benchmark_data_smithi169_79822_object67321:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.273582+0000", 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.72577490899999, 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.019321822, 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.044 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 184738 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.273582+0000", 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.273582+0000", 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.273583+0000", 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.031e-06 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.273588+0000", 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9089999999999999e-06 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.273603+0000", 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.522e-05 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.273611+0000", 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6559999999999992e-06 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.273803+0000", 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000192435 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.273811+0000", 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.926e-06 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.285197+0000", 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0113855169999999 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.285331+0000", 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013430000000000001 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.285647+0000", 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031567599999999998 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.288010+0000", 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023633510000000001 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.292754+0000", 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047436539999999999 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.292809+0000", 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5837000000000001e-05 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.292853+0000", 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3597000000000003e-05 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.292904+0000", 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0713e-05 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.050 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:184741 7.19 7:9e039d40:::benchmark_data_smithi169_79822_object67324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.274503+0000", 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.72485422400001, 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0212236139999999, 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 184741 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.274503+0000", 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.274503+0000", 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.274504+0000", 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7900000000000007e-07 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.274511+0000", 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3780000000000003e-06 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.274521+0000", 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0553e-05 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.274529+0000", 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3170000000000004e-06 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.274581+0000", 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1783000000000002e-05 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.274589+0000", 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0770000000000008e-06 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.288043+0000", 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0134548370000001 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.288155+0000", 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111722 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.288456+0000", 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030064099999999998 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.292799+0000", 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043434399999999996 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.293092+0000", 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029289499999999999 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.295617+0000", 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025248380000000002 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.295677+0000", 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9979000000000003e-05 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.295726+0000", 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9175000000000003e-05 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:184776 7.19 7:9bfb555f:::benchmark_data_smithi169_79822_object67359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.319510+0000", 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.67984626800001, 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98152070000000002, 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.057 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 184776 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.319510+0000", 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.319510+0000", 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.319511+0000", 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5499999999999997e-07 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.319517+0000", 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.22e-06 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.319528+0000", 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1079e-05 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.319537+0000", 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8060000000000006e-06 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.319625+0000", 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7583000000000005e-05 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.319634+0000", 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3470000000000005e-06 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.290196+0000", 2024-03-12T03:43:52.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97056158400000003 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.290321+0000", 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012518699999999999 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.290587+0000", 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026541800000000001 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.292970+0000", 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023833460000000002 2024-03-12T03:43:52.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.295742+0000", 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027725670000000001 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.300924+0000", 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0051812869999999997 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.300984+0000", 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0341999999999999e-05 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.301031+0000", 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7079000000000001e-05 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:184823 7.19 7:9d509b72:::benchmark_data_smithi169_79822_object67406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.369866+0000", 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.62949051599998, 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94231233700000006, 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.063 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 184823 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.369866+0000", 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.369866+0000", 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.369867+0000", 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8100000000000003e-07 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.369870+0000", 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1599999999999998e-06 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.369877+0000", 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9580000000000001e-06 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.369883+0000", 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.511e-06 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.370192+0000", 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030853199999999998 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.370201+0000", 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7339999999999997e-06 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.293138+0000", 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.922936969 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.293275+0000", 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013695499999999999 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.293610+0000", 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00033587899999999997 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.300882+0000", 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0072718039999999998 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.301133+0000", 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025098399999999999 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312106+0000", 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010973147000000001 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312139+0000", 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2753000000000003e-05 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312179+0000", 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.947e-05 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:184830 7.19 7:9d997f27:::benchmark_data_smithi169_79822_object67413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.377195+0000", 2024-03-12T03:43:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.62216125800001, 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93503756299999996, 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 184830 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.377195+0000", 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.377195+0000", 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.377197+0000", 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.361e-06 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.377202+0000", 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2050000000000001e-06 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.377214+0000", 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1986e-05 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.377221+0000", 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4000000000000003e-06 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.377276+0000", 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4669000000000001e-05 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.377283+0000", 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0550000000000002e-06 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.295772+0000", 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91848921400000005 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.295904+0000", 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000132113 2024-03-12T03:43:52.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.296243+0000", 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00033868499999999997 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.301093+0000", 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048501309999999997 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.307247+0000", 2024-03-12T03:43:52.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006153258 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312185+0000", 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049381959999999997 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312210+0000", 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5202999999999999e-05 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312233+0000", 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3087000000000001e-05 2024-03-12T03:43:52.075 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:184862 7.19 7:99252577:::benchmark_data_smithi169_79822_object67445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.412250+0000", 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.58710681000002, 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90003329799999998, 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 184862 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.076 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.412250+0000", 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.412250+0000", 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.412251+0000", 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2109999999999999e-06 2024-03-12T03:43:52.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.412256+0000", 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9760000000000003e-06 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.412266+0000", 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5130000000000002e-06 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.412272+0000", 2024-03-12T03:43:52.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7070000000000003e-06 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.412319+0000", 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6794e-05 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.412327+0000", 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2050000000000002e-06 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.298388+0000", 2024-03-12T03:43:52.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88606040699999999 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.298494+0000", 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106684 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.298824+0000", 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032952900000000001 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.301169+0000", 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023445639999999999 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.307326+0000", 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0061577000000000003 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312238+0000", 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049117299999999996 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312261+0000", 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3226000000000001e-05 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312283+0000", 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2052000000000002e-05 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:184864 7.19 7:9b7eb12d:::benchmark_data_smithi169_79822_object67447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.416409+0000", 2024-03-12T03:43:52.082 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.582947924, 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89592324999999995, 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 184864 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.416409+0000", 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.416409+0000", 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.416410+0000", 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0279999999999999e-06 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.416415+0000", 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1880000000000001e-06 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.416425+0000", 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0358e-05 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.416433+0000", 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0209999999999999e-06 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.416495+0000", 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1519999999999994e-05 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.416505+0000", 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0338e-05 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.301196+0000", 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88469030699999995 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.301295+0000", 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9371999999999997e-05 2024-03-12T03:43:52.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.301600+0000", 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030459399999999999 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.307288+0000", 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.005688032 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.307364+0000", 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6732999999999999e-05 2024-03-12T03:43:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312288+0000", 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049237389999999999 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312311+0000", 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3229000000000001e-05 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312332+0000", 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0791000000000001e-05 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.088 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:184896 7.19 7:9d45387d:::benchmark_data_smithi169_79822_object67479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.452187+0000", 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.54716968399998, 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.86502783000000005, 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 184896 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.089 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.452187+0000", 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.452187+0000", 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.452189+0000", 2024-03-12T03:43:52.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.438e-06 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.452195+0000", 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4200000000000004e-06 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.452211+0000", 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5804e-05 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.452219+0000", 2024-03-12T03:43:52.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3399999999999998e-06 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.452290+0000", 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1298999999999996e-05 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.452300+0000", 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4769999999999992e-06 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.304332+0000", 2024-03-12T03:43:52.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85203218800000002 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.304440+0000", 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010761500000000001 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.304793+0000", 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035366299999999999 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.311993+0000", 2024-03-12T03:43:52.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0071997210000000001 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312337+0000", 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000344013 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.317118+0000", 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047805449999999998 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.317172+0000", 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4456999999999997e-05 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.317215+0000", 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2849999999999998e-05 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:184942 7.19 7:9a7747d6:::benchmark_data_smithi169_79822_object67525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.510363+0000", 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.48899337799998, 2024-03-12T03:43:52.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.81423479399999998, 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 184942 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.510363+0000", 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.510363+0000", 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.510364+0000", 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0900000000000002e-07 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.510366+0000", 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5720000000000001e-06 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.510372+0000", 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5500000000000002e-06 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.510377+0000", 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9370000000000003e-06 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.511050+0000", 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00067296000000000001 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.511058+0000", 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8050000000000007e-06 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.307395+0000", 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79633692300000003 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.307502+0000", 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107314 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.307837+0000", 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00033465799999999999 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312053+0000", 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042160189999999997 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312341+0000", 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028807799999999999 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.324377+0000", 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012036296 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.324551+0000", 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000174408 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.324598+0000", 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6864999999999999e-05 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.101 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:184996 7.19 7:9da99b50:::benchmark_data_smithi169_79822_object67579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.567850+0000", 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.43150682599997, 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76220293900000002, 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 184996 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.102 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.567850+0000", 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.567850+0000", 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.567851+0000", 2024-03-12T03:43:52.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1620000000000001e-06 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.567856+0000", 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5040000000000004e-06 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.567864+0000", 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8419999999999994e-06 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.567870+0000", 2024-03-12T03:43:52.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5820000000000001e-06 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.567921+0000", 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0589e-05 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.567926+0000", 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7910000000000001e-06 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.309782+0000", 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.74185560299999997 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.309885+0000", 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103069 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.310186+0000", 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030050399999999999 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312098+0000", 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001912548 2024-03-12T03:43:52.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.324687+0000", 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012588607 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.329865+0000", 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0051785240000000003 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330003+0000", 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013747400000000001 2024-03-12T03:43:52.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330053+0000", 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0139999999999998e-05 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:185008 7.19 7:9f67ecfc:::benchmark_data_smithi169_79822_object67591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.660984+0000", 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.33837295400002, 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.66934452300000002, 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 185008 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.660984+0000", 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.660984+0000", 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.660985+0000", 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.13e-06 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.660990+0000", 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8910000000000003e-06 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.661001+0000", 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0912e-05 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.661007+0000", 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1589999999999998e-06 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.661058+0000", 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0967999999999997e-05 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.661068+0000", 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0332999999999999e-05 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312375+0000", 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65130713399999995 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312501+0000", 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012601999999999999 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.312810+0000", 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030910900000000001 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.324338+0000", 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011527272 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330106+0000", 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0057683509999999997 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330222+0000", 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115918 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330290+0000", 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7871000000000001e-05 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330328+0000", 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8454999999999997e-05 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:185037 7.19 7:990b2b8f:::benchmark_data_smithi169_79822_object67620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.114 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.697506+0000", 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.301851032, 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.63301895100000005, 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 185037 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.697506+0000", 2024-03-12T03:43:52.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.697506+0000", 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.697507+0000", 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5099999999999999e-06 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.697513+0000", 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2489999999999998e-06 2024-03-12T03:43:52.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.697528+0000", 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.488e-05 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.697537+0000", 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0559999999999994e-06 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.697590+0000", 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3034000000000001e-05 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.697599+0000", 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2279999999999998e-06 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.315010+0000", 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61741088799999999 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.315106+0000", 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6376e-05 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.315406+0000", 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029986500000000002 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.324643+0000", 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0092367040000000001 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330112+0000", 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054692630000000002 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330419+0000", 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030712699999999998 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330473+0000", 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4184999999999997e-05 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330525+0000", 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1585999999999997e-05 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:185044 7.19 7:98a39777:::benchmark_data_smithi169_79822_object67627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.707097+0000", 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.29226018999998, 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62361953999999997, 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 185044 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.707097+0000", 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.707097+0000", 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.707098+0000", 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4589999999999999e-06 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.707104+0000", 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0469999999999999e-06 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.707117+0000", 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3258000000000001e-05 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.707127+0000", 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1900000000000001e-06 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.707246+0000", 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118993 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.707255+0000", 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3070000000000008e-06 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.317249+0000", 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60999463499999995 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.317353+0000", 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103911 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.317632+0000", 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00027841299999999998 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330093+0000", 2024-03-12T03:43:52.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012461541 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330117+0000", 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3462e-05 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330625+0000", 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00050856600000000001 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330672+0000", 2024-03-12T03:43:52.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6351999999999999e-05 2024-03-12T03:43:52.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330716+0000", 2024-03-12T03:43:52.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4406000000000001e-05 2024-03-12T03:43:52.127 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.127 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.127 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.129 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:185076 7.19 7:9f0d6854:::benchmark_data_smithi169_79822_object67659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.129 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.754971+0000", 2024-03-12T03:43:52.129 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.24438583300002, 2024-03-12T03:43:52.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57841417100000003, 2024-03-12T03:43:52.129 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.129 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.129 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.129 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.129 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 185076 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.754971+0000", 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.754971+0000", 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.754972+0000", 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5239999999999999e-06 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.754978+0000", 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5049999999999998e-06 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.754990+0000", 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2031000000000001e-05 2024-03-12T03:43:52.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.754996+0000", 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.426e-06 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.755050+0000", 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3993000000000003e-05 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.755060+0000", 2024-03-12T03:43:52.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7880000000000001e-06 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.319456+0000", 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.56439600999999995 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.319570+0000", 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000113989 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.319805+0000", 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000234546 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330122+0000", 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010317154 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330171+0000", 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8946999999999997e-05 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.333260+0000", 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003089226 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.333340+0000", 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0134000000000002e-05 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.333385+0000", 2024-03-12T03:43:52.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4898000000000003e-05 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:185182 7.19 7:9ac20767:::benchmark_data_smithi169_79822_object67765:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:25.992526+0000", 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 326.00683068699999, 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34100462799999998, 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.136 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 185182 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.992526+0000", 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.992526+0000", 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.992527+0000", 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.198e-06 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.992533+0000", 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7760000000000003e-06 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.992547+0000", 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3732e-05 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.992555+0000", 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2910000000000007e-06 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.992635+0000", 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9809999999999997e-05 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:25.992646+0000", 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1055e-05 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.322134+0000", 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32948766299999999 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.322234+0000", 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9909999999999997e-05 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.322476+0000", 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024222600000000001 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330127+0000", 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0076515699999999999 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330380+0000", 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025250700000000001 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.333421+0000", 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0030415529999999998 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.333495+0000", 2024-03-12T03:43:52.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3849000000000004e-05 2024-03-12T03:43:52.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.333531+0000", 2024-03-12T03:43:52.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5488000000000003e-05 2024-03-12T03:43:52.142 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.142 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.142 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.142 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:185210 7.19 7:996b54cb:::benchmark_data_smithi169_79822_object67793:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:26.097736+0000", 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 325.9016206, 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.236650006, 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 185210 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.097736+0000", 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.097736+0000", 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.097737+0000", 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9600000000000008e-07 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.097742+0000", 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0780000000000003e-06 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.097753+0000", 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0508e-05 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.097759+0000", 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9529999999999996e-06 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.097805+0000", 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6613999999999998e-05 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.097812+0000", 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9299999999999997e-06 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.324718+0000", 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22690606699999999 2024-03-12T03:43:52.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.324825+0000", 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010621800000000001 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.325160+0000", 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000335013 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330132+0000", 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.004972565 2024-03-12T03:43:52.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.330581+0000", 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044881400000000002 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.333564+0000", 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029834190000000002 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.334345+0000", 2024-03-12T03:43:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00078032899999999996 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:26.334386+0000", 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1501999999999997e-05 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:207226 8.5 8:afc51c0f:::benchmark_data_smithi169_106492_object73743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:25.224290+0000", 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 86.775067164000006, 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18384750899999999, 2024-03-12T03:43:52.149 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 207226 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224290+0000", 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224290+0000", 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224291+0000", 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.204e-06 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224295+0000", 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.296e-06 2024-03-12T03:43:52.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224303+0000", 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0160000000000008e-06 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224311+0000", 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4599999999999997e-06 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.230995+0000", 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0066846989999999997 2024-03-12T03:43:52.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.231191+0000", 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000196202 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.231333+0000", 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000141361 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.235484+0000", 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041509989999999998 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.256368+0000", 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020884171 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.407984+0000", 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.151616267 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.408078+0000", 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3574999999999998e-05 2024-03-12T03:43:52.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.408137+0000", 2024-03-12T03:43:52.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9259000000000003e-05 2024-03-12T03:43:52.155 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.155 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.155 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.155 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:210664 8.5 8:ac07f159:::benchmark_data_smithi169_106492_object77181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:43:52.155 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:29.292061+0000", 2024-03-12T03:43:52.155 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 82.707296227, 2024-03-12T03:43:52.156 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:52.157 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:43:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:43:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:43:52.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19662268099999999, 2024-03-12T03:43:52.157 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.157 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.157 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.157 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:43:52.157 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:43:52.157 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 210664 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292061+0000", 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292061+0000", 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292062+0000", 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.483e-06 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292066+0000", 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1309999999999999e-06 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292076+0000", 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5689999999999994e-06 2024-03-12T03:43:52.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292082+0000", 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6760000000000001e-06 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.359067+0000", 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.066984848 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.359154+0000", 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7153999999999999e-05 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.359237+0000", 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3061999999999999e-05 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.365384+0000", 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006146744 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.368955+0000", 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035707500000000001 2024-03-12T03:43:52.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.488598+0000", 2024-03-12T03:43:52.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.119642817 2024-03-12T03:43:52.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.488634+0000", 2024-03-12T03:43:52.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6406999999999999e-05 2024-03-12T03:43:52.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.488683+0000", 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9039999999999998e-05 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:210936 8.5 8:a7cf8e7a:::benchmark_data_smithi169_106492_object77453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:29.560667+0000", 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 82.438690097999995, 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17761102000000001, 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:43:52.163 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 210936 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560667+0000", 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:43:52.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560667+0000", 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560668+0000", 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.201e-06 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560673+0000", 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8840000000000002e-06 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560685+0000", 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1907e-05 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560691+0000", 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4670000000000001e-06 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.626200+0000", 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.065508937000000003 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.626307+0000", 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107222 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.626416+0000", 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108634 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.632883+0000", 2024-03-12T03:43:52.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00646724 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.634125+0000", 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012422920000000001 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.738168+0000", 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104042204 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.738216+0000", 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8018e-05 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:43:52.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:43:52.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:43:52.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.738278+0000", 2024-03-12T03:43:52.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2014000000000006e-05 2024-03-12T03:43:52.169 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.169 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.169 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.169 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:43:52.169 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:43:52.169 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:43:52.180 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:52.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:52.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:52.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:52.304 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:52.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:52.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:52.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:52.430 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:52.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:52.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:52.555 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:52.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:52.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:52.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:52.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:52.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:52.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:52.809 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:52.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:52.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-12T03:43:52.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:43:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:43:52.910 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:52.935 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:52.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:53.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:53.060 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:53.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:53.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:53.100 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:43:53.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:53.186 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:53.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:53.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:53.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:53.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:53.362 INFO:teuthology.orchestra.run.smithi169.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-12T03:43:53.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:53.438 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:53.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:53.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:53.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:53 smithi173 ceph-mon[33544]: pgmap v497: 158 pgs: 23 active+undersized+degraded, 46 active+undersized, 89 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:53.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:53 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:43:53.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:53 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:43:53.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:53 smithi173 ceph-mon[33544]: osdmap e282: 8 total, 7 up, 8 in 2024-03-12T03:43:53.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:53 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1374922626' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:43:53.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:53 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:53.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:53 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1374922626' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:43:53.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:53 smithi173 ceph-mon[33544]: osdmap e283: 8 total, 7 up, 8 in 2024-03-12T03:43:53.539 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:53.565 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:53.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:53.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:53.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[29344]: pgmap v497: 158 pgs: 23 active+undersized+degraded, 46 active+undersized, 89 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:53.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:43:53.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:43:53.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[29344]: osdmap e282: 8 total, 7 up, 8 in 2024-03-12T03:43:53.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1374922626' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:43:53.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:53.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1374922626' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:43:53.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[29344]: osdmap e283: 8 total, 7 up, 8 in 2024-03-12T03:43:53.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[39949]: pgmap v497: 158 pgs: 23 active+undersized+degraded, 46 active+undersized, 89 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:53.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:43:53.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:43:53.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[39949]: osdmap e282: 8 total, 7 up, 8 in 2024-03-12T03:43:53.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1374922626' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:43:53.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:53.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1374922626' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:43:53.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:53 smithi169 ceph-mon[39949]: osdmap e283: 8 total, 7 up, 8 in 2024-03-12T03:43:53.689 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:53.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:53.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:53.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:53.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:53.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:53.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:53.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:53.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:53.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:54.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:54.065 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:54.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:54.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:54.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:54.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:54.292 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:54.317 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:54.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:54.370 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:43:54.419 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:54.440 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:54 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2736693726' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:43:54.441 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:54 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:43:54.444 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:54.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:54.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:54.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:54.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:54.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:54.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:54.577 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:54 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2736693726' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:43:54.577 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:54 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:43:54.578 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:54 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2736693726' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:43:54.578 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:54 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:43:54.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:54.687 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:43:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:43:54.687 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-03-12T03:43:54.687 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:43:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:54.696 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:54.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:54.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:54.820 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:54.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:54.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:54.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:54.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:54.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:55.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:55.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:55.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:55.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:55.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:55.198 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:55.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:55.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-12T03:43:55.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:55.314 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-12T03:43:55.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:55.325 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:55.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:55.326 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-12T03:43:55.326 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-12T03:43:55.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[29344]: pgmap v500: 158 pgs: 23 active+undersized+degraded, 46 active+undersized, 89 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:55.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[29344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:43:55.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:43:55.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:43:55.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[29344]: osdmap e284: 8 total, 7 up, 8 in 2024-03-12T03:43:55.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[29344]: 5.19 scrub starts 2024-03-12T03:43:55.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[29344]: 5.19 scrub ok 2024-03-12T03:43:55.375 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[39949]: pgmap v500: 158 pgs: 23 active+undersized+degraded, 46 active+undersized, 89 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:55.375 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[39949]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:43:55.376 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:43:55.376 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:43:55.376 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[39949]: osdmap e284: 8 total, 7 up, 8 in 2024-03-12T03:43:55.376 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[39949]: 5.19 scrub starts 2024-03-12T03:43:55.376 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:55 smithi169 ceph-mon[39949]: 5.19 scrub ok 2024-03-12T03:43:55.376 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:hints = 1 2024-03-12T03:43:55.376 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-12T03:43:55.377 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Object prefix: benchmark_data_smithi169_135229 2024-03-12T03:43:55.377 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:43:55.377 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 0 0 0 0 0 0 - 0 2024-03-12T03:43:55.427 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:55.466 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:55.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:55.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:55.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:55.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:55.603 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:55.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:55.690 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:43:55.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:55 smithi173 ceph-mon[33544]: pgmap v500: 158 pgs: 23 active+undersized+degraded, 46 active+undersized, 89 active+clean; 315 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 115/687 objects degraded (16.739%) 2024-03-12T03:43:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:55 smithi173 ceph-mon[33544]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:43:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:55 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:43:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:55 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:43:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:55 smithi173 ceph-mon[33544]: osdmap e284: 8 total, 7 up, 8 in 2024-03-12T03:43:55.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:55 smithi173 ceph-mon[33544]: 5.19 scrub starts 2024-03-12T03:43:55.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:55 smithi173 ceph-mon[33544]: 5.19 scrub ok 2024-03-12T03:43:55.755 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:55.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:55.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:55.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:55.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:55.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:56.000 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:56.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:56.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:56.140 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:56.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:56.177 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:56.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:56.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:56.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:56.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:56.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:56.376 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 1 16 4017 4001 250.049 250.062 0.00522869 0.00398619 2024-03-12T03:43:56.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:56.440 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:56 smithi169 ceph-mon[29344]: 5.f scrub starts 2024-03-12T03:43:56.440 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:56 smithi169 ceph-mon[29344]: 5.f scrub ok 2024-03-12T03:43:56.441 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:56 smithi169 ceph-mon[39949]: 5.f scrub starts 2024-03-12T03:43:56.441 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:56 smithi169 ceph-mon[39949]: 5.f scrub ok 2024-03-12T03:43:56.456 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:56.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:56.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:56.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:56 smithi173 ceph-mon[33544]: 5.f scrub starts 2024-03-12T03:43:56.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:56 smithi173 ceph-mon[33544]: 5.f scrub ok 2024-03-12T03:43:56.558 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-12T03:43:56.558 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:43:56.559 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-12T03:43:56.560 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:43:56.560 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:56.600 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:56.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:56.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:56.650 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:43:56.693 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:43:56.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:56.733 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:43:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:56.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:56.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:56.772 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:43:56.810 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:43:56.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:56.853 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:43:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:56.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:56.892 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:43:56.892 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:43:56.893 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:43:56.893 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:43:56.893 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:43:56.893 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:43:56.893 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:43:56.893 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:43:56.893 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:43:56.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:57.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:57.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:57.125 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:57.167 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:57.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:57.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:57.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:57.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:57.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:57.376 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 2 16 7533 7517 234.887 219.75 0.00343255 0.00424828 2024-03-12T03:43:57.404 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:57.413 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:57 smithi173 ceph-mon[33544]: pgmap v502: 158 pgs: 25 active+undersized+degraded, 44 active+undersized, 89 active+clean; 40 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 6.6 MiB/s wr, 106 op/s; 421/2583 objects degraded (16.299%) 2024-03-12T03:43:57.413 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:57 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 421/2583 objects degraded (16.299%), 25 pgs degraded (PG_DEGRADED) 2024-03-12T03:43:57.413 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:57 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3006088089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:57.426 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:43:57.440 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:57.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:57.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:57.580 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:57.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:57.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:57.637 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:43:57.638 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:43:57.661 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:57 smithi169 ceph-mon[29344]: pgmap v502: 158 pgs: 25 active+undersized+degraded, 44 active+undersized, 89 active+clean; 40 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 6.6 MiB/s wr, 106 op/s; 421/2583 objects degraded (16.299%) 2024-03-12T03:43:57.661 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:57 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 421/2583 objects degraded (16.299%), 25 pgs degraded (PG_DEGRADED) 2024-03-12T03:43:57.661 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:57 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3006088089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:57.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:57 smithi169 ceph-mon[39949]: pgmap v502: 158 pgs: 25 active+undersized+degraded, 44 active+undersized, 89 active+clean; 40 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 6.6 MiB/s wr, 106 op/s; 421/2583 objects degraded (16.299%) 2024-03-12T03:43:57.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:57 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 421/2583 objects degraded (16.299%), 25 pgs degraded (PG_DEGRADED) 2024-03-12T03:43:57.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:57 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3006088089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:43:57.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:57.712 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:57.716 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:43:57.717 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:43:57.735 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:57.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:57.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:57.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:57.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:57.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:57.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:57.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:58.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:58.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:58.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:58.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:43:58.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:58.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:43:58.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:43:58.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-12T03:43:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:58.357 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:58.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:43:58.376 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 3 16 10987 10971 228.544 215.875 0.00270013 0.00436751 2024-03-12T03:43:58.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:58.495 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:58.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:58.523 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:43:58.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:58.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:58 smithi169 ceph-mon[29344]: 5.11 scrub starts 2024-03-12T03:43:58.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:58 smithi169 ceph-mon[29344]: 5.11 scrub ok 2024-03-12T03:43:58.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:58 smithi169 ceph-mon[29344]: osdmap e285: 8 total, 7 up, 8 in 2024-03-12T03:43:58.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:58 smithi169 ceph-mon[39949]: 5.11 scrub starts 2024-03-12T03:43:58.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:58 smithi169 ceph-mon[39949]: 5.11 scrub ok 2024-03-12T03:43:58.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:58 smithi169 ceph-mon[39949]: osdmap e285: 8 total, 7 up, 8 in 2024-03-12T03:43:58.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:58.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:58.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:58.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:43:58.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:58 smithi173 ceph-mon[33544]: 5.11 scrub starts 2024-03-12T03:43:58.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:58 smithi173 ceph-mon[33544]: 5.11 scrub ok 2024-03-12T03:43:58.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:58 smithi173 ceph-mon[33544]: osdmap e285: 8 total, 7 up, 8 in 2024-03-12T03:43:58.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:58.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:58.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:43:58.818 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:43:58.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:58.921 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:58.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:58.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:59.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:43:59.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:43:59.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:59.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:43:59.163 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:59.199 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:59.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:59.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:59.301 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:59.336 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:59.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:59.376 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 4 16 14146 14130 220.763 197.438 0.00505616 0.00452283 2024-03-12T03:43:59.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:43:59.472 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:59.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:43:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:59 smithi173 ceph-mon[33544]: pgmap v503: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 245 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 41 MiB/s wr, 653 op/s; 2076/12447 objects degraded (16.679%) 2024-03-12T03:43:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:59 smithi173 ceph-mon[33544]: 4.a scrub starts 2024-03-12T03:43:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:59 smithi173 ceph-mon[33544]: 4.a scrub ok 2024-03-12T03:43:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:59 smithi173 ceph-mon[33544]: 3.4 scrub starts 2024-03-12T03:43:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:43:59 smithi173 ceph-mon[33544]: 3.4 scrub ok 2024-03-12T03:43:59.495 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:59 smithi169 ceph-mon[29344]: pgmap v503: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 245 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 41 MiB/s wr, 653 op/s; 2076/12447 objects degraded (16.679%) 2024-03-12T03:43:59.495 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:59 smithi169 ceph-mon[29344]: 4.a scrub starts 2024-03-12T03:43:59.495 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:59 smithi169 ceph-mon[29344]: 4.a scrub ok 2024-03-12T03:43:59.495 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:59 smithi169 ceph-mon[29344]: 3.4 scrub starts 2024-03-12T03:43:59.495 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:43:59 smithi169 ceph-mon[29344]: 3.4 scrub ok 2024-03-12T03:43:59.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:59 smithi169 ceph-mon[39949]: pgmap v503: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 245 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 41 MiB/s wr, 653 op/s; 2076/12447 objects degraded (16.679%) 2024-03-12T03:43:59.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:59 smithi169 ceph-mon[39949]: 4.a scrub starts 2024-03-12T03:43:59.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:59 smithi169 ceph-mon[39949]: 4.a scrub ok 2024-03-12T03:43:59.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:59 smithi169 ceph-mon[39949]: 3.4 scrub starts 2024-03-12T03:43:59.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:43:59 smithi169 ceph-mon[39949]: 3.4 scrub ok 2024-03-12T03:43:59.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:59.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:59.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:43:59.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:43:59.745 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:59.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:43:59.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:43:59.893 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:43:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:43:59.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:43:59.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:00.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:00.038 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:00.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:00.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:00.169 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:00.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:00.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:00.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:00.273 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:44:00.276 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:44:00.345 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:00.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:00.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:00.376 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 5 16 16856 16840 210.483 169.375 0.00539291 0.00474394 2024-03-12T03:44:00.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:00.482 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:00.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:00 smithi173 ceph-mon[33544]: 4.3 scrub starts 2024-03-12T03:44:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:00 smithi173 ceph-mon[33544]: 4.3 scrub ok 2024-03-12T03:44:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:00 smithi173 ceph-mon[33544]: 3.14 scrub starts 2024-03-12T03:44:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:00 smithi173 ceph-mon[33544]: 3.14 scrub ok 2024-03-12T03:44:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:00 smithi173 ceph-mon[33544]: 6.1c scrub starts 2024-03-12T03:44:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:00 smithi173 ceph-mon[33544]: 6.1c scrub ok 2024-03-12T03:44:00.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:00.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:00.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:00.645 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:00 smithi169 ceph-mon[29344]: 4.3 scrub starts 2024-03-12T03:44:00.646 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:00 smithi169 ceph-mon[29344]: 4.3 scrub ok 2024-03-12T03:44:00.646 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:00 smithi169 ceph-mon[29344]: 3.14 scrub starts 2024-03-12T03:44:00.646 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:00 smithi169 ceph-mon[29344]: 3.14 scrub ok 2024-03-12T03:44:00.646 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:00 smithi169 ceph-mon[29344]: 6.1c scrub starts 2024-03-12T03:44:00.646 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:00 smithi169 ceph-mon[29344]: 6.1c scrub ok 2024-03-12T03:44:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:00 smithi169 ceph-mon[39949]: 4.3 scrub starts 2024-03-12T03:44:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:00 smithi169 ceph-mon[39949]: 4.3 scrub ok 2024-03-12T03:44:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:00 smithi169 ceph-mon[39949]: 3.14 scrub starts 2024-03-12T03:44:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:00 smithi169 ceph-mon[39949]: 3.14 scrub ok 2024-03-12T03:44:00.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:00 smithi169 ceph-mon[39949]: 6.1c scrub starts 2024-03-12T03:44:00.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:00 smithi169 ceph-mon[39949]: 6.1c scrub ok 2024-03-12T03:44:00.724 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:00.754 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:00.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:00.856 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:00.887 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:00.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:00.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:01.020 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:44:01.021 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:44:01.024 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:01.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:01.126 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:01.160 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:01.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:01.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:01.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:01.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:01.376 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 6 16 19364 19348 201.525 156.75 0.00386607 0.00495635 2024-03-12T03:44:01.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:01.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:01.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:01.535 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:01.571 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:01.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:01.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:01.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:01 smithi173 ceph-mon[33544]: 4.e scrub starts 2024-03-12T03:44:01.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:01 smithi173 ceph-mon[33544]: 4.e scrub ok 2024-03-12T03:44:01.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:01 smithi173 ceph-mon[33544]: 3.f scrub starts 2024-03-12T03:44:01.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:01 smithi173 ceph-mon[33544]: 3.f scrub ok 2024-03-12T03:44:01.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:01 smithi173 ceph-mon[33544]: pgmap v505: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 245 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 770 B/s rd, 37 MiB/s wr, 590 op/s; 2076/12447 objects degraded (16.679%) 2024-03-12T03:44:01.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:01 smithi173 ceph-mon[33544]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:44:01.643 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:01 smithi173 ceph-mon[33544]: 3.10 scrub starts 2024-03-12T03:44:01.643 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:01 smithi173 ceph-mon[33544]: 3.10 scrub ok 2024-03-12T03:44:01.643 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:01 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:01.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[29344]: 4.e scrub starts 2024-03-12T03:44:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[29344]: 4.e scrub ok 2024-03-12T03:44:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[29344]: 3.f scrub starts 2024-03-12T03:44:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[29344]: 3.f scrub ok 2024-03-12T03:44:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[29344]: pgmap v505: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 245 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 770 B/s rd, 37 MiB/s wr, 590 op/s; 2076/12447 objects degraded (16.679%) 2024-03-12T03:44:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[29344]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:44:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[29344]: 3.10 scrub starts 2024-03-12T03:44:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[29344]: 3.10 scrub ok 2024-03-12T03:44:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:01.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[39949]: 4.e scrub starts 2024-03-12T03:44:01.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[39949]: 4.e scrub ok 2024-03-12T03:44:01.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[39949]: 3.f scrub starts 2024-03-12T03:44:01.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[39949]: 3.f scrub ok 2024-03-12T03:44:01.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[39949]: pgmap v505: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 245 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 770 B/s rd, 37 MiB/s wr, 590 op/s; 2076/12447 objects degraded (16.679%) 2024-03-12T03:44:01.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[39949]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:44:01.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[39949]: 3.10 scrub starts 2024-03-12T03:44:01.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[39949]: 3.10 scrub ok 2024-03-12T03:44:01.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:01.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:01.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:01.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:01.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:01.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:01.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:01.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:01.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:01.988 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:01.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:02.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:02.100 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:44:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:44:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:44:02.124 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:02.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:02.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:02.149 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-03-12T03:43:59.989933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":257223502,"num_objects":4149,"num_object_clones":0,"num_object_copies":12447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4149,"num_whiteouts":0,"num_read":2072,"num_read_kb":1801,"num_write":4850,"num_write_kb":251145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":691232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6363,"ondisk_log_size":6363,"up":405,"acting":405,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":472,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3536316,"kb_used_data":488904,"kb_used_omap":18,"kb_used_meta":3047277,"kb_avail":746424900,"statfs":{"total":767960285184,"available":764339097600,"internally_reserved":0,"allocated":500637696,"data_stored":496365722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19339,"internal_metadata":3120411765},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":256901120,"num_objects":3920,"num_object_clones":0,"num_object_copies":11760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3920,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3920,"num_write_kb":250880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.647752"},"pg_stats":[{"pgid":"9.1","version":"284'202","reported_seq":220,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:56.188437+0000","last_change":"2024-03-12T03:43:55.390953+0000","last_active":"2024-03-12T03:43:56.188437+0000","last_peered":"2024-03-12T03:43:56.188437+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.591676+0000","last_became_peered":"2024-03-12T03:43:45.591676+0000","last_unstale":"2024-03-12T03:43:56.188437+0000","last_undegraded":"2024-03-12T03:43:55.390315+0000","last_fullsized":"2024-03-12T03:43:45.580214+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:48.943427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":202,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":202,"num_write_kb":12928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":202}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":783,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.415150+0000","last_change":"2024-03-12T03:43:18.345099+0000","last_active":"2024-03-12T03:43:54.415150+0000","last_peered":"2024-03-12T03:43:54.415150+0000","last_clean":"2024-03-12T03:43:13.340455+0000","last_became_active":"2024-03-12T03:43:18.345099+0000","last_became_peered":"2024-03-12T03:43:18.345099+0000","last_unstale":"2024-03-12T03:43:54.415150+0000","last_undegraded":"2024-03-12T03:43:54.415150+0000","last_fullsized":"2024-03-12T03:43:18.338164+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:28.162735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:28.162735+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:55.523355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"269'20","reported_seq":518,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.360365+0000","last_change":"2024-03-12T03:43:18.351745+0000","last_active":"2024-03-12T03:43:54.360365+0000","last_peered":"2024-03-12T03:43:54.360365+0000","last_clean":"2024-03-12T03:43:14.546139+0000","last_became_active":"2024-03-12T03:43:18.351745+0000","last_became_peered":"2024-03-12T03:43:18.351745+0000","last_unstale":"2024-03-12T03:43:54.360365+0000","last_undegraded":"2024-03-12T03:43:18.333421+0000","last_fullsized":"2024-03-12T03:43:18.333308+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-12T03:43:14.545889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:14.545889+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:43.528094+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":290,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.415070+0000","last_change":"2024-03-12T03:43:18.339149+0000","last_active":"2024-03-12T03:43:54.415070+0000","last_peered":"2024-03-12T03:43:54.415070+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:43:18.339149+0000","last_became_peered":"2024-03-12T03:43:18.339149+0000","last_unstale":"2024-03-12T03:43:54.415070+0000","last_undegraded":"2024-03-12T03:43:54.415070+0000","last_fullsized":"2024-03-12T03:43:18.331129+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.981378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.981378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":255,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.364711+0000","last_change":"2024-03-12T03:43:01.350508+0000","last_active":"2024-03-12T03:43:54.364711+0000","last_peered":"2024-03-12T03:43:54.364711+0000","last_clean":"2024-03-12T03:43:54.364711+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:43:54.364711+0000","last_undegraded":"2024-03-12T03:43:54.364711+0000","last_fullsized":"2024-03-12T03:43:54.364711+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:43:01.350377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:01.350377+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:30.689072+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.361022+0000","last_change":"2024-03-12T03:42:08.287146+0000","last_active":"2024-03-12T03:43:54.361022+0000","last_peered":"2024-03-12T03:43:54.361022+0000","last_clean":"2024-03-12T03:43:54.361022+0000","last_became_active":"2024-03-12T03:39:33.265665+0000","last_became_peered":"2024-03-12T03:39:33.265665+0000","last_unstale":"2024-03-12T03:43:54.361022+0000","last_undegraded":"2024-03-12T03:43:54.361022+0000","last_fullsized":"2024-03-12T03:43:54.361022+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:08.287006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:08.287006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"284'404","reported_seq":425,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:57.058914+0000","last_change":"2024-03-12T03:43:45.632046+0000","last_active":"2024-03-12T03:43:57.058914+0000","last_peered":"2024-03-12T03:43:57.058914+0000","last_clean":"2024-03-12T03:43:57.058914+0000","last_became_active":"2024-03-12T03:43:45.631752+0000","last_became_peered":"2024-03-12T03:43:45.631752+0000","last_unstale":"2024-03-12T03:43:57.058914+0000","last_undegraded":"2024-03-12T03:43:57.058914+0000","last_fullsized":"2024-03-12T03:43:57.058914+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:49.076842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":404,"num_write_kb":25856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"269'31","reported_seq":573,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.360737+0000","last_change":"2024-03-12T03:42:11.397167+0000","last_active":"2024-03-12T03:43:54.360737+0000","last_peered":"2024-03-12T03:43:54.360737+0000","last_clean":"2024-03-12T03:43:54.360737+0000","last_became_active":"2024-03-12T03:39:33.268652+0000","last_became_peered":"2024-03-12T03:39:33.268652+0000","last_unstale":"2024-03-12T03:43:54.360737+0000","last_undegraded":"2024-03-12T03:43:54.360737+0000","last_fullsized":"2024-03-12T03:43:54.360737+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:42:11.397028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:42:11.397028+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":486,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414112+0000","last_change":"2024-03-12T03:43:09.441186+0000","last_active":"2024-03-12T03:43:54.414112+0000","last_peered":"2024-03-12T03:43:54.414112+0000","last_clean":"2024-03-12T03:43:54.414112+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:43:54.414112+0000","last_undegraded":"2024-03-12T03:43:54.414112+0000","last_fullsized":"2024-03-12T03:43:54.414112+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:09.441075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:43:09.441075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:34.311884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414464+0000","last_change":"2024-03-12T03:43:21.789919+0000","last_active":"2024-03-12T03:43:54.414464+0000","last_peered":"2024-03-12T03:43:54.414464+0000","last_clean":"2024-03-12T03:43:54.414464+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:43:54.414464+0000","last_undegraded":"2024-03-12T03:43:54.414464+0000","last_fullsized":"2024-03-12T03:43:54.414464+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:21.789828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:43:21.789828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.198862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":211,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.364758+0000","last_change":"2024-03-12T03:43:18.344122+0000","last_active":"2024-03-12T03:43:54.364758+0000","last_peered":"2024-03-12T03:43:54.364758+0000","last_clean":"2024-03-12T03:43:13.349376+0000","last_became_active":"2024-03-12T03:43:18.344122+0000","last_became_peered":"2024-03-12T03:43:18.344122+0000","last_unstale":"2024-03-12T03:43:54.364758+0000","last_undegraded":"2024-03-12T03:43:54.364758+0000","last_fullsized":"2024-03-12T03:43:18.337641+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:07.272232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:07.272232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":624,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.414300+0000","last_change":"2024-03-12T03:43:18.379878+0000","last_active":"2024-03-12T03:43:54.414300+0000","last_peered":"2024-03-12T03:43:54.414300+0000","last_clean":"2024-03-12T03:43:13.353305+0000","last_became_active":"2024-03-12T03:43:18.379878+0000","last_became_peered":"2024-03-12T03:43:18.379878+0000","last_unstale":"2024-03-12T03:43:54.414300+0000","last_undegraded":"2024-03-12T03:43:18.336786+0000","last_fullsized":"2024-03-12T03:43:18.336678+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:41:03.399737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:41:03.399737+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"284'179","reported_seq":198,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:55.960724+0000","last_change":"2024-03-12T03:43:55.386278+0000","last_active":"2024-03-12T03:43:55.960724+0000","last_peered":"2024-03-12T03:43:55.960724+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.592856+0000","last_became_peered":"2024-03-12T03:43:45.592856+0000","last_unstale":"2024-03-12T03:43:55.960724+0000","last_undegraded":"2024-03-12T03:43:55.380028+0000","last_fullsized":"2024-03-12T03:43:45.589896+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:14.284774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730944,"num_objects":179,"num_object_clones":0,"num_object_copies":537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":179,"num_write_kb":11456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":179}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"269'31","reported_seq":525,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.360910+0000","last_change":"2024-03-12T03:41:18.286718+0000","last_active":"2024-03-12T03:43:54.360910+0000","last_peered":"2024-03-12T03:43:54.360910+0000","last_clean":"2024-03-12T03:43:54.360910+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:43:54.360910+0000","last_undegraded":"2024-03-12T03:43:54.360910+0000","last_fullsized":"2024-03-12T03:43:54.360910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'23","last_scrub_stamp":"2024-03-12T03:41:18.286562+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:41:18.286562+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":492,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.359864+0000","last_change":"2024-03-12T03:43:09.624124+0000","last_active":"2024-03-12T03:43:54.359864+0000","last_peered":"2024-03-12T03:43:54.359864+0000","last_clean":"2024-03-12T03:43:54.359864+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:43:54.359864+0000","last_undegraded":"2024-03-12T03:43:54.359864+0000","last_fullsized":"2024-03-12T03:43:54.359864+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:09.624047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:09.624047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:15.734223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":471,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:55.569256+0000","last_change":"2024-03-12T03:43:55.569297+0000","last_active":"2024-03-12T03:43:55.569256+0000","last_peered":"2024-03-12T03:43:55.569256+0000","last_clean":"2024-03-12T03:43:55.569256+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:43:55.569256+0000","last_undegraded":"2024-03-12T03:43:55.569256+0000","last_fullsized":"2024-03-12T03:43:55.569256+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:55.569174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:55.569174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:06.654122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":299,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.413604+0000","last_change":"2024-03-12T03:43:18.353217+0000","last_active":"2024-03-12T03:43:54.413604+0000","last_peered":"2024-03-12T03:43:54.413604+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.353217+0000","last_became_peered":"2024-03-12T03:43:18.353217+0000","last_unstale":"2024-03-12T03:43:54.413604+0000","last_undegraded":"2024-03-12T03:43:54.413604+0000","last_fullsized":"2024-03-12T03:43:18.336780+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:13.114417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.114417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"284'417","reported_seq":436,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:57.047531+0000","last_change":"2024-03-12T03:43:55.387593+0000","last_active":"2024-03-12T03:43:57.047531+0000","last_peered":"2024-03-12T03:43:57.047531+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.631713+0000","last_became_peered":"2024-03-12T03:43:45.631713+0000","last_unstale":"2024-03-12T03:43:57.047531+0000","last_undegraded":"2024-03-12T03:43:55.379093+0000","last_fullsized":"2024-03-12T03:43:45.587023+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:57.946789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27328512,"num_objects":417,"num_object_clones":0,"num_object_copies":1251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":417,"num_write_kb":26688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":417}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":604,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.413896+0000","last_change":"2024-03-12T03:41:04.383619+0000","last_active":"2024-03-12T03:43:54.413896+0000","last_peered":"2024-03-12T03:43:54.413896+0000","last_clean":"2024-03-12T03:43:54.413896+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:43:54.413896+0000","last_undegraded":"2024-03-12T03:43:54.413896+0000","last_fullsized":"2024-03-12T03:43:54.413896+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:41:04.383505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:41:04.383505+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":316,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.415311+0000","last_change":"2024-03-12T03:43:18.346647+0000","last_active":"2024-03-12T03:43:54.415311+0000","last_peered":"2024-03-12T03:43:54.415311+0000","last_clean":"2024-03-12T03:43:13.349764+0000","last_became_active":"2024-03-12T03:43:18.346647+0000","last_became_peered":"2024-03-12T03:43:18.346647+0000","last_unstale":"2024-03-12T03:43:54.415311+0000","last_undegraded":"2024-03-12T03:43:18.336192+0000","last_fullsized":"2024-03-12T03:43:18.335538+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:43:00.715225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.715225+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:22.163928+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":290,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414148+0000","last_change":"2024-03-12T03:43:18.344931+0000","last_active":"2024-03-12T03:43:54.414148+0000","last_peered":"2024-03-12T03:43:54.414148+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:43:18.344931+0000","last_became_peered":"2024-03-12T03:43:18.344931+0000","last_unstale":"2024-03-12T03:43:54.414148+0000","last_undegraded":"2024-03-12T03:43:54.414148+0000","last_fullsized":"2024-03-12T03:43:18.334088+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:08.619817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:08.619817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:28.359940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":607,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.360972+0000","last_change":"2024-03-12T03:42:04.241287+0000","last_active":"2024-03-12T03:43:54.360972+0000","last_peered":"2024-03-12T03:43:54.360972+0000","last_clean":"2024-03-12T03:43:54.360972+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:43:54.360972+0000","last_undegraded":"2024-03-12T03:43:54.360972+0000","last_fullsized":"2024-03-12T03:43:54.360972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:42:04.241174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:04.241174+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":459,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.416160+0000","last_change":"2024-03-12T03:43:23.613276+0000","last_active":"2024-03-12T03:43:54.416160+0000","last_peered":"2024-03-12T03:43:54.416160+0000","last_clean":"2024-03-12T03:43:54.416160+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:43:54.416160+0000","last_undegraded":"2024-03-12T03:43:54.416160+0000","last_fullsized":"2024-03-12T03:43:54.416160+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:23.613137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:23.613137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.197845+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"284'180","reported_seq":202,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:55.959608+0000","last_change":"2024-03-12T03:43:45.597209+0000","last_active":"2024-03-12T03:43:55.959608+0000","last_peered":"2024-03-12T03:43:55.959608+0000","last_clean":"2024-03-12T03:43:55.959608+0000","last_became_active":"2024-03-12T03:43:45.596788+0000","last_became_peered":"2024-03-12T03:43:45.596788+0000","last_unstale":"2024-03-12T03:43:55.959608+0000","last_undegraded":"2024-03-12T03:43:55.959608+0000","last_fullsized":"2024-03-12T03:43:55.959608+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:50.337416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11796480,"num_objects":180,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":11520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":477,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.359857+0000","last_change":"2024-03-12T03:43:18.344476+0000","last_active":"2024-03-12T03:43:54.359857+0000","last_peered":"2024-03-12T03:43:54.359857+0000","last_clean":"2024-03-12T03:43:13.337325+0000","last_became_active":"2024-03-12T03:43:18.344476+0000","last_became_peered":"2024-03-12T03:43:18.344476+0000","last_unstale":"2024-03-12T03:43:54.359857+0000","last_undegraded":"2024-03-12T03:43:54.359857+0000","last_fullsized":"2024-03-12T03:43:18.329711+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:01.229635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:01.229635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"269'42","reported_seq":179,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.416600+0000","last_change":"2024-03-12T03:43:18.345917+0000","last_active":"2024-03-12T03:43:54.416600+0000","last_peered":"2024-03-12T03:43:54.416600+0000","last_clean":"2024-03-12T03:33:28.528154+0000","last_became_active":"2024-03-12T03:43:18.345917+0000","last_became_peered":"2024-03-12T03:43:18.345917+0000","last_unstale":"2024-03-12T03:43:54.416600+0000","last_undegraded":"2024-03-12T03:43:18.336762+0000","last_fullsized":"2024-03-12T03:43:18.336652+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-03-12T03:36:06.083052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.083052+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:54.970795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":294,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.415773+0000","last_change":"2024-03-12T03:43:18.345673+0000","last_active":"2024-03-12T03:43:54.415773+0000","last_peered":"2024-03-12T03:43:54.415773+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:43:18.345673+0000","last_became_peered":"2024-03-12T03:43:18.345673+0000","last_unstale":"2024-03-12T03:43:54.415773+0000","last_undegraded":"2024-03-12T03:43:54.415773+0000","last_fullsized":"2024-03-12T03:43:18.337055+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:41.918538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:41.918538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:54.970822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":493,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.360020+0000","last_change":"2024-03-12T03:43:18.344050+0000","last_active":"2024-03-12T03:43:54.360020+0000","last_peered":"2024-03-12T03:43:54.360020+0000","last_clean":"2024-03-12T03:43:13.338394+0000","last_became_active":"2024-03-12T03:43:18.344050+0000","last_became_peered":"2024-03-12T03:43:18.344050+0000","last_unstale":"2024-03-12T03:43:54.360020+0000","last_undegraded":"2024-03-12T03:43:54.360020+0000","last_fullsized":"2024-03-12T03:43:18.333957+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:23.257942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:23.257942+0000","last_clean_scrub_stamp":"2024-03-12T03:41:23.257942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":613,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.360266+0000","last_change":"2024-03-12T03:43:04.665501+0000","last_active":"2024-03-12T03:43:54.360266+0000","last_peered":"2024-03-12T03:43:54.360266+0000","last_clean":"2024-03-12T03:43:54.360266+0000","last_became_active":"2024-03-12T03:39:36.251411+0000","last_became_peered":"2024-03-12T03:39:36.251411+0000","last_unstale":"2024-03-12T03:43:54.360266+0000","last_undegraded":"2024-03-12T03:43:54.360266+0000","last_fullsized":"2024-03-12T03:43:54.360266+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:04.664919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:04.664919+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:12.858915+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"284'127","reported_seq":146,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:55.976669+0000","last_change":"2024-03-12T03:43:55.383487+0000","last_active":"2024-03-12T03:43:55.976669+0000","last_peered":"2024-03-12T03:43:55.976669+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.594399+0000","last_became_peered":"2024-03-12T03:43:45.594399+0000","last_unstale":"2024-03-12T03:43:55.976669+0000","last_undegraded":"2024-03-12T03:43:55.379614+0000","last_fullsized":"2024-03-12T03:43:45.586741+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:07.446575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8323072,"num_objects":127,"num_object_clones":0,"num_object_copies":381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127,"num_write_kb":8128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":127}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"266'11","reported_seq":133,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.413969+0000","last_change":"2024-03-12T03:43:18.345511+0000","last_active":"2024-03-12T03:43:54.413969+0000","last_peered":"2024-03-12T03:43:54.413969+0000","last_clean":"2024-03-12T03:33:28.353771+0000","last_became_active":"2024-03-12T03:43:18.345511+0000","last_became_peered":"2024-03-12T03:43:18.345511+0000","last_unstale":"2024-03-12T03:43:54.413969+0000","last_undegraded":"2024-03-12T03:43:18.332743+0000","last_fullsized":"2024-03-12T03:43:18.332472+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:39:37.039925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:37.039925+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:59.997080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"269'42","reported_seq":539,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.360092+0000","last_change":"2024-03-12T03:42:02.272123+0000","last_active":"2024-03-12T03:43:54.360092+0000","last_peered":"2024-03-12T03:43:54.360092+0000","last_clean":"2024-03-12T03:43:54.360092+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:43:54.360092+0000","last_undegraded":"2024-03-12T03:43:54.360092+0000","last_fullsized":"2024-03-12T03:43:54.360092+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'28","last_scrub_stamp":"2024-03-12T03:42:02.271980+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:42:02.271980+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":473,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415265+0000","last_change":"2024-03-12T03:40:25.553659+0000","last_active":"2024-03-12T03:43:54.415265+0000","last_peered":"2024-03-12T03:43:54.415265+0000","last_clean":"2024-03-12T03:43:54.415265+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:43:54.415265+0000","last_undegraded":"2024-03-12T03:43:54.415265+0000","last_fullsized":"2024-03-12T03:43:54.415265+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:40:25.553541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:40:25.553541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":481,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414736+0000","last_change":"2024-03-12T03:42:04.017533+0000","last_active":"2024-03-12T03:43:54.414736+0000","last_peered":"2024-03-12T03:43:54.414736+0000","last_clean":"2024-03-12T03:43:54.414736+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:43:54.414736+0000","last_undegraded":"2024-03-12T03:43:54.414736+0000","last_fullsized":"2024-03-12T03:43:54.414736+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:04.017409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:04.017409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:54.970949+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":384,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.413488+0000","last_change":"2024-03-12T03:43:18.347755+0000","last_active":"2024-03-12T03:43:54.413488+0000","last_peered":"2024-03-12T03:43:54.413488+0000","last_clean":"2024-03-12T03:34:01.860694+0000","last_became_active":"2024-03-12T03:43:18.347755+0000","last_became_peered":"2024-03-12T03:43:18.347755+0000","last_unstale":"2024-03-12T03:43:54.413488+0000","last_undegraded":"2024-03-12T03:43:54.413488+0000","last_fullsized":"2024-03-12T03:43:18.331155+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:02.442717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:02.442717+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:56.635399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"284'178","reported_seq":197,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:56.060118+0000","last_change":"2024-03-12T03:43:55.390924+0000","last_active":"2024-03-12T03:43:56.060118+0000","last_peered":"2024-03-12T03:43:56.060118+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.588776+0000","last_became_peered":"2024-03-12T03:43:45.588776+0000","last_unstale":"2024-03-12T03:43:56.060118+0000","last_undegraded":"2024-03-12T03:43:55.379272+0000","last_fullsized":"2024-03-12T03:43:45.585743+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:49.133021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11665408,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":178,"num_write_kb":11392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":178}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":608,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415014+0000","last_change":"2024-03-12T03:43:24.493648+0000","last_active":"2024-03-12T03:43:54.415014+0000","last_peered":"2024-03-12T03:43:54.415014+0000","last_clean":"2024-03-12T03:43:54.415014+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:43:54.415014+0000","last_undegraded":"2024-03-12T03:43:54.415014+0000","last_fullsized":"2024-03-12T03:43:54.415014+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:43:24.493559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:24.493559+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:52.783472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"269'26","reported_seq":527,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.413980+0000","last_change":"2024-03-12T03:43:20.765528+0000","last_active":"2024-03-12T03:43:54.413980+0000","last_peered":"2024-03-12T03:43:54.413980+0000","last_clean":"2024-03-12T03:43:54.413980+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:43:54.413980+0000","last_undegraded":"2024-03-12T03:43:54.413980+0000","last_fullsized":"2024-03-12T03:43:54.413980+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:43:20.765428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:20.765428+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:25.320833+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":496,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.416231+0000","last_change":"2024-03-12T03:43:18.347416+0000","last_active":"2024-03-12T03:43:54.416231+0000","last_peered":"2024-03-12T03:43:54.416231+0000","last_clean":"2024-03-12T03:43:13.353362+0000","last_became_active":"2024-03-12T03:43:18.347416+0000","last_became_peered":"2024-03-12T03:43:18.347416+0000","last_unstale":"2024-03-12T03:43:54.416231+0000","last_undegraded":"2024-03-12T03:43:54.416231+0000","last_fullsized":"2024-03-12T03:43:18.331325+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:34.674570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:39:34.674570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:56.635409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":485,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.413426+0000","last_change":"2024-03-12T03:43:18.342056+0000","last_active":"2024-03-12T03:43:54.413426+0000","last_peered":"2024-03-12T03:43:54.413426+0000","last_clean":"2024-03-12T03:43:13.330724+0000","last_became_active":"2024-03-12T03:43:18.342056+0000","last_became_peered":"2024-03-12T03:43:18.342056+0000","last_unstale":"2024-03-12T03:43:54.413426+0000","last_undegraded":"2024-03-12T03:43:54.413426+0000","last_fullsized":"2024-03-12T03:43:18.333058+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:40.532991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:40.532991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":612,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.415040+0000","last_change":"2024-03-12T03:43:18.340387+0000","last_active":"2024-03-12T03:43:54.415040+0000","last_peered":"2024-03-12T03:43:54.415040+0000","last_clean":"2024-03-12T03:43:13.348756+0000","last_became_active":"2024-03-12T03:43:18.340387+0000","last_became_peered":"2024-03-12T03:43:18.340387+0000","last_unstale":"2024-03-12T03:43:54.415040+0000","last_undegraded":"2024-03-12T03:43:54.415040+0000","last_fullsized":"2024-03-12T03:43:18.334582+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:02.551349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:02.551349+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:32.074223+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"284'146","reported_seq":168,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:55.959956+0000","last_change":"2024-03-12T03:43:45.597285+0000","last_active":"2024-03-12T03:43:55.959956+0000","last_peered":"2024-03-12T03:43:55.959956+0000","last_clean":"2024-03-12T03:43:55.959956+0000","last_became_active":"2024-03-12T03:43:45.596944+0000","last_became_peered":"2024-03-12T03:43:45.596944+0000","last_unstale":"2024-03-12T03:43:55.959956+0000","last_undegraded":"2024-03-12T03:43:55.959956+0000","last_fullsized":"2024-03-12T03:43:55.959956+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:57.787068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9568256,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":146,"num_write_kb":9344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":487,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414354+0000","last_change":"2024-03-12T03:43:27.393346+0000","last_active":"2024-03-12T03:43:54.414354+0000","last_peered":"2024-03-12T03:43:54.414354+0000","last_clean":"2024-03-12T03:43:54.414354+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:43:54.414354+0000","last_undegraded":"2024-03-12T03:43:54.414354+0000","last_fullsized":"2024-03-12T03:43:54.414354+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:27.393251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:43:27.393251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:55.551797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"269'35","reported_seq":532,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415667+0000","last_change":"2024-03-12T03:39:36.550428+0000","last_active":"2024-03-12T03:43:54.415667+0000","last_peered":"2024-03-12T03:43:54.415667+0000","last_clean":"2024-03-12T03:43:54.415667+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:43:54.415667+0000","last_undegraded":"2024-03-12T03:43:54.415667+0000","last_fullsized":"2024-03-12T03:43:54.415667+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-03-12T03:39:36.550291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:36.550291+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":501,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414405+0000","last_change":"2024-03-12T03:40:32.631452+0000","last_active":"2024-03-12T03:43:54.414405+0000","last_peered":"2024-03-12T03:43:54.414405+0000","last_clean":"2024-03-12T03:43:54.414405+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:43:54.414405+0000","last_undegraded":"2024-03-12T03:43:54.414405+0000","last_fullsized":"2024-03-12T03:43:54.414405+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:40:32.631336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":489,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.413714+0000","last_change":"2024-03-12T03:43:28.804503+0000","last_active":"2024-03-12T03:43:54.413714+0000","last_peered":"2024-03-12T03:43:54.413714+0000","last_clean":"2024-03-12T03:43:54.413714+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:43:54.413714+0000","last_undegraded":"2024-03-12T03:43:54.413714+0000","last_fullsized":"2024-03-12T03:43:54.413714+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:28.804369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:28.804369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:53.918392+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":612,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414428+0000","last_change":"2024-03-12T03:43:18.349140+0000","last_active":"2024-03-12T03:43:54.414428+0000","last_peered":"2024-03-12T03:43:54.414428+0000","last_clean":"2024-03-12T03:43:13.344628+0000","last_became_active":"2024-03-12T03:43:18.349140+0000","last_became_peered":"2024-03-12T03:43:18.349140+0000","last_unstale":"2024-03-12T03:43:54.414428+0000","last_undegraded":"2024-03-12T03:43:54.414428+0000","last_fullsized":"2024-03-12T03:43:18.335600+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:10.356260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:10.356260+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"284'279","reported_seq":301,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:56.593700+0000","last_change":"2024-03-12T03:43:45.589372+0000","last_active":"2024-03-12T03:43:56.593700+0000","last_peered":"2024-03-12T03:43:56.593700+0000","last_clean":"2024-03-12T03:43:56.593700+0000","last_became_active":"2024-03-12T03:43:45.588933+0000","last_became_peered":"2024-03-12T03:43:45.588933+0000","last_unstale":"2024-03-12T03:43:56.593700+0000","last_undegraded":"2024-03-12T03:43:56.593700+0000","last_fullsized":"2024-03-12T03:43:56.593700+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:50.687640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18284544,"num_objects":279,"num_object_clones":0,"num_object_copies":837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":279,"num_write_kb":17856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":463,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415535+0000","last_change":"2024-03-12T03:40:22.791790+0000","last_active":"2024-03-12T03:43:54.415535+0000","last_peered":"2024-03-12T03:43:54.415535+0000","last_clean":"2024-03-12T03:43:54.415535+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:43:54.415535+0000","last_undegraded":"2024-03-12T03:43:54.415535+0000","last_fullsized":"2024-03-12T03:43:54.415535+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:22.791656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:40:22.791656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"269'24","reported_seq":552,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415753+0000","last_change":"2024-03-12T03:43:09.665948+0000","last_active":"2024-03-12T03:43:54.415753+0000","last_peered":"2024-03-12T03:43:54.415753+0000","last_clean":"2024-03-12T03:43:54.415753+0000","last_became_active":"2024-03-12T03:39:33.277868+0000","last_became_peered":"2024-03-12T03:39:33.277868+0000","last_unstale":"2024-03-12T03:43:54.415753+0000","last_undegraded":"2024-03-12T03:43:54.415753+0000","last_fullsized":"2024-03-12T03:43:54.415753+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:43:09.665793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:09.665793+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:22.334225+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1121,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.416472+0000","last_change":"2024-03-12T03:41:59.023320+0000","last_active":"2024-03-12T03:43:54.416472+0000","last_peered":"2024-03-12T03:43:54.416472+0000","last_clean":"2024-03-12T03:43:54.416472+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:43:54.416472+0000","last_undegraded":"2024-03-12T03:43:54.416472+0000","last_fullsized":"2024-03-12T03:43:54.416472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-12T03:41:59.023235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:41:59.023235+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":628,"num_read_kb":623,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":485,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.413964+0000","last_change":"2024-03-12T03:43:18.338719+0000","last_active":"2024-03-12T03:43:54.413964+0000","last_peered":"2024-03-12T03:43:54.413964+0000","last_clean":"2024-03-12T03:43:13.332262+0000","last_became_active":"2024-03-12T03:43:18.338719+0000","last_became_peered":"2024-03-12T03:43:18.338719+0000","last_unstale":"2024-03-12T03:43:54.413964+0000","last_undegraded":"2024-03-12T03:43:54.413964+0000","last_fullsized":"2024-03-12T03:43:18.331120+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:42:21.445443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":489,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414332+0000","last_change":"2024-03-12T03:43:18.350023+0000","last_active":"2024-03-12T03:43:54.414332+0000","last_peered":"2024-03-12T03:43:54.414332+0000","last_clean":"2024-03-12T03:43:13.346844+0000","last_became_active":"2024-03-12T03:43:18.350023+0000","last_became_peered":"2024-03-12T03:43:18.350023+0000","last_unstale":"2024-03-12T03:43:54.414332+0000","last_undegraded":"2024-03-12T03:43:54.414332+0000","last_fullsized":"2024-03-12T03:43:18.331452+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:11.815414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:11.815414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:19.252340+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":596,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414176+0000","last_change":"2024-03-12T03:43:19.738133+0000","last_active":"2024-03-12T03:43:54.414176+0000","last_peered":"2024-03-12T03:43:54.414176+0000","last_clean":"2024-03-12T03:43:54.414176+0000","last_became_active":"2024-03-12T03:38:08.891939+0000","last_became_peered":"2024-03-12T03:38:08.891939+0000","last_unstale":"2024-03-12T03:43:54.414176+0000","last_undegraded":"2024-03-12T03:43:54.414176+0000","last_fullsized":"2024-03-12T03:43:54.414176+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:19.738048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:19.738048+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:32.339889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"284'290","reported_seq":309,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:56.590052+0000","last_change":"2024-03-12T03:43:55.393294+0000","last_active":"2024-03-12T03:43:56.590052+0000","last_peered":"2024-03-12T03:43:56.590052+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.588708+0000","last_became_peered":"2024-03-12T03:43:45.588708+0000","last_unstale":"2024-03-12T03:43:56.590052+0000","last_undegraded":"2024-03-12T03:43:55.382438+0000","last_fullsized":"2024-03-12T03:43:45.583017+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:11.359288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":290,"num_object_clones":0,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":290,"num_write_kb":18560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":290}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":485,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.416485+0000","last_change":"2024-03-12T03:43:26.022980+0000","last_active":"2024-03-12T03:43:54.416485+0000","last_peered":"2024-03-12T03:43:54.416485+0000","last_clean":"2024-03-12T03:43:54.416485+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:43:54.416485+0000","last_undegraded":"2024-03-12T03:43:54.416485+0000","last_fullsized":"2024-03-12T03:43:54.416485+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:26.022875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:26.022875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:50.896439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"269'28","reported_seq":650,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415228+0000","last_change":"2024-03-12T03:43:08.426091+0000","last_active":"2024-03-12T03:43:54.415228+0000","last_peered":"2024-03-12T03:43:54.415228+0000","last_clean":"2024-03-12T03:43:54.415228+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:43:54.415228+0000","last_undegraded":"2024-03-12T03:43:54.415228+0000","last_fullsized":"2024-03-12T03:43:54.415228+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:43:08.425934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:08.425934+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:25.334339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":289,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414093+0000","last_change":"2024-03-12T03:43:18.349427+0000","last_active":"2024-03-12T03:43:54.414093+0000","last_peered":"2024-03-12T03:43:54.414093+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:43:18.349427+0000","last_became_peered":"2024-03-12T03:43:18.349427+0000","last_unstale":"2024-03-12T03:43:54.414093+0000","last_undegraded":"2024-03-12T03:43:54.414093+0000","last_fullsized":"2024-03-12T03:43:18.334581+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:07.572642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:07.572642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:11.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":746,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.416485+0000","last_change":"2024-03-12T03:43:05.222995+0000","last_active":"2024-03-12T03:43:54.416485+0000","last_peered":"2024-03-12T03:43:54.416485+0000","last_clean":"2024-03-12T03:43:54.416485+0000","last_became_active":"2024-03-12T03:38:03.852040+0000","last_became_peered":"2024-03-12T03:38:03.852040+0000","last_unstale":"2024-03-12T03:43:54.416485+0000","last_undegraded":"2024-03-12T03:43:54.416485+0000","last_fullsized":"2024-03-12T03:43:54.416485+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:05.222876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:05.222876+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:11.274990+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":471,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414958+0000","last_change":"2024-03-12T03:43:18.345043+0000","last_active":"2024-03-12T03:43:54.414958+0000","last_peered":"2024-03-12T03:43:54.414958+0000","last_clean":"2024-03-12T03:43:13.352684+0000","last_became_active":"2024-03-12T03:43:18.345043+0000","last_became_peered":"2024-03-12T03:43:18.345043+0000","last_unstale":"2024-03-12T03:43:54.414958+0000","last_undegraded":"2024-03-12T03:43:54.414958+0000","last_fullsized":"2024-03-12T03:43:18.334020+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:00.436718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:00.436718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"284'173","reported_seq":193,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:56.065393+0000","last_change":"2024-03-12T03:43:45.595179+0000","last_active":"2024-03-12T03:43:56.065393+0000","last_peered":"2024-03-12T03:43:56.065393+0000","last_clean":"2024-03-12T03:43:56.065393+0000","last_became_active":"2024-03-12T03:43:45.594793+0000","last_became_peered":"2024-03-12T03:43:45.594793+0000","last_unstale":"2024-03-12T03:43:56.065393+0000","last_undegraded":"2024-03-12T03:43:56.065393+0000","last_fullsized":"2024-03-12T03:43:56.065393+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:54.033626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":173,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":200,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.364787+0000","last_change":"2024-03-12T03:42:18.272709+0000","last_active":"2024-03-12T03:43:54.364787+0000","last_peered":"2024-03-12T03:43:54.364787+0000","last_clean":"2024-03-12T03:43:54.364787+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:43:54.364787+0000","last_undegraded":"2024-03-12T03:43:54.364787+0000","last_fullsized":"2024-03-12T03:43:54.364787+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:18.272591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:18.272591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"269'23","reported_seq":529,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415187+0000","last_change":"2024-03-12T03:43:29.744286+0000","last_active":"2024-03-12T03:43:54.415187+0000","last_peered":"2024-03-12T03:43:54.415187+0000","last_clean":"2024-03-12T03:43:54.415187+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:43:54.415187+0000","last_undegraded":"2024-03-12T03:43:54.415187+0000","last_fullsized":"2024-03-12T03:43:54.415187+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'23","last_scrub_stamp":"2024-03-12T03:43:29.744179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:29.744179+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.298331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":204,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.364425+0000","last_change":"2024-03-12T03:43:18.343047+0000","last_active":"2024-03-12T03:43:54.364425+0000","last_peered":"2024-03-12T03:43:54.364425+0000","last_clean":"2024-03-12T03:43:13.346919+0000","last_became_active":"2024-03-12T03:43:18.343047+0000","last_became_peered":"2024-03-12T03:43:18.343047+0000","last_unstale":"2024-03-12T03:43:54.364425+0000","last_undegraded":"2024-03-12T03:43:54.364425+0000","last_fullsized":"2024-03-12T03:43:18.336453+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:01.189842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:42:01.189842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":501,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414033+0000","last_change":"2024-03-12T03:43:31.765331+0000","last_active":"2024-03-12T03:43:54.414033+0000","last_peered":"2024-03-12T03:43:54.414033+0000","last_clean":"2024-03-12T03:43:54.414033+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:43:54.414033+0000","last_undegraded":"2024-03-12T03:43:54.414033+0000","last_fullsized":"2024-03-12T03:43:54.414033+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:43:31.765257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:32.553377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":483,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.415149+0000","last_change":"2024-03-12T03:43:18.348792+0000","last_active":"2024-03-12T03:43:54.415149+0000","last_peered":"2024-03-12T03:43:54.415149+0000","last_clean":"2024-03-12T03:43:13.342858+0000","last_became_active":"2024-03-12T03:43:18.348792+0000","last_became_peered":"2024-03-12T03:43:18.348792+0000","last_unstale":"2024-03-12T03:43:54.415149+0000","last_undegraded":"2024-03-12T03:43:18.339155+0000","last_fullsized":"2024-03-12T03:43:18.339042+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:42:20.203740+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:42:20.203740+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":753,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.413918+0000","last_change":"2024-03-12T03:42:17.149648+0000","last_active":"2024-03-12T03:43:54.413918+0000","last_peered":"2024-03-12T03:43:54.413918+0000","last_clean":"2024-03-12T03:43:54.413918+0000","last_became_active":"2024-03-12T03:38:08.893327+0000","last_became_peered":"2024-03-12T03:38:08.893327+0000","last_unstale":"2024-03-12T03:43:54.413918+0000","last_undegraded":"2024-03-12T03:43:54.413918+0000","last_fullsized":"2024-03-12T03:43:54.413918+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:17.149524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:17.149524+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:54.970983+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"284'303","reported_seq":325,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:56.589460+0000","last_change":"2024-03-12T03:43:45.591852+0000","last_active":"2024-03-12T03:43:56.589460+0000","last_peered":"2024-03-12T03:43:56.589460+0000","last_clean":"2024-03-12T03:43:56.589460+0000","last_became_active":"2024-03-12T03:43:45.591646+0000","last_became_peered":"2024-03-12T03:43:45.591646+0000","last_unstale":"2024-03-12T03:43:56.589460+0000","last_undegraded":"2024-03-12T03:43:56.589460+0000","last_fullsized":"2024-03-12T03:43:56.589460+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:07.133196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":303,"num_write_kb":19392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":473,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414391+0000","last_change":"2024-03-12T03:43:18.612572+0000","last_active":"2024-03-12T03:43:54.414391+0000","last_peered":"2024-03-12T03:43:54.414391+0000","last_clean":"2024-03-12T03:43:54.414391+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:43:54.414391+0000","last_undegraded":"2024-03-12T03:43:54.414391+0000","last_fullsized":"2024-03-12T03:43:54.414391+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:18.612450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:18.612450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:44.393407+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"269'52","reported_seq":685,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.415567+0000","last_change":"2024-03-12T03:43:18.348827+0000","last_active":"2024-03-12T03:43:54.415567+0000","last_peered":"2024-03-12T03:43:54.415567+0000","last_clean":"2024-03-12T03:43:16.743398+0000","last_became_active":"2024-03-12T03:43:18.348827+0000","last_became_peered":"2024-03-12T03:43:18.348827+0000","last_unstale":"2024-03-12T03:43:54.415567+0000","last_undegraded":"2024-03-12T03:43:18.331846+0000","last_fullsized":"2024-03-12T03:43:18.331727+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'39","last_scrub_stamp":"2024-03-12T03:43:06.482430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:06.482430+0000","objects_scrubbed":11,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:26.211347+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":11}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":481,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.360130+0000","last_change":"2024-03-12T03:41:26.257807+0000","last_active":"2024-03-12T03:43:54.360130+0000","last_peered":"2024-03-12T03:43:54.360130+0000","last_clean":"2024-03-12T03:43:54.360130+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:43:54.360130+0000","last_undegraded":"2024-03-12T03:43:54.360130+0000","last_fullsized":"2024-03-12T03:43:54.360130+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:26.257679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:41:26.257679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":483,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414128+0000","last_change":"2024-03-12T03:40:16.791859+0000","last_active":"2024-03-12T03:43:54.414128+0000","last_peered":"2024-03-12T03:43:54.414128+0000","last_clean":"2024-03-12T03:43:54.414128+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:43:54.414128+0000","last_undegraded":"2024-03-12T03:43:54.414128+0000","last_fullsized":"2024-03-12T03:43:54.414128+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:40:16.791364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:40:16.791364+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":333,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.360500+0000","last_change":"2024-03-12T03:43:18.344442+0000","last_active":"2024-03-12T03:43:54.360500+0000","last_peered":"2024-03-12T03:43:54.360500+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.344442+0000","last_became_peered":"2024-03-12T03:43:18.344442+0000","last_unstale":"2024-03-12T03:43:54.360500+0000","last_undegraded":"2024-03-12T03:43:54.360500+0000","last_fullsized":"2024-03-12T03:43:18.330930+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:39.031166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:39.031166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":604,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.413761+0000","last_change":"2024-03-12T03:42:28.720781+0000","last_active":"2024-03-12T03:43:54.413761+0000","last_peered":"2024-03-12T03:43:54.413761+0000","last_clean":"2024-03-12T03:43:54.413761+0000","last_became_active":"2024-03-12T03:39:36.252545+0000","last_became_peered":"2024-03-12T03:39:36.252545+0000","last_unstale":"2024-03-12T03:43:54.413761+0000","last_undegraded":"2024-03-12T03:43:54.413761+0000","last_fullsized":"2024-03-12T03:43:54.413761+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:28.720248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:28.720248+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:56.635431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"284'176","reported_seq":198,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:56.054547+0000","last_change":"2024-03-12T03:43:45.631811+0000","last_active":"2024-03-12T03:43:56.054547+0000","last_peered":"2024-03-12T03:43:56.054547+0000","last_clean":"2024-03-12T03:43:56.054547+0000","last_became_active":"2024-03-12T03:43:45.631620+0000","last_became_peered":"2024-03-12T03:43:45.631620+0000","last_unstale":"2024-03-12T03:43:56.054547+0000","last_undegraded":"2024-03-12T03:43:56.054547+0000","last_fullsized":"2024-03-12T03:43:56.054547+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:52.062274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":176,"num_object_clones":0,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"269'26","reported_seq":510,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415379+0000","last_change":"2024-03-12T03:39:39.540698+0000","last_active":"2024-03-12T03:43:54.415379+0000","last_peered":"2024-03-12T03:43:54.415379+0000","last_clean":"2024-03-12T03:43:54.415379+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:43:54.415379+0000","last_undegraded":"2024-03-12T03:43:54.415379+0000","last_fullsized":"2024-03-12T03:43:54.415379+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'20","last_scrub_stamp":"2024-03-12T03:39:39.540585+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:39:39.540585+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":483,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.415766+0000","last_change":"2024-03-12T03:43:18.349816+0000","last_active":"2024-03-12T03:43:54.415766+0000","last_peered":"2024-03-12T03:43:54.415766+0000","last_clean":"2024-03-12T03:43:13.352139+0000","last_became_active":"2024-03-12T03:43:18.349816+0000","last_became_peered":"2024-03-12T03:43:18.349816+0000","last_unstale":"2024-03-12T03:43:54.415766+0000","last_undegraded":"2024-03-12T03:43:54.415766+0000","last_fullsized":"2024-03-12T03:43:18.332719+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:59.344058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:40:59.344058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":204,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.364476+0000","last_change":"2024-03-12T03:43:18.346068+0000","last_active":"2024-03-12T03:43:54.364476+0000","last_peered":"2024-03-12T03:43:54.364476+0000","last_clean":"2024-03-12T03:43:13.348345+0000","last_became_active":"2024-03-12T03:43:18.346068+0000","last_became_peered":"2024-03-12T03:43:18.346068+0000","last_unstale":"2024-03-12T03:43:54.364476+0000","last_undegraded":"2024-03-12T03:43:54.364476+0000","last_fullsized":"2024-03-12T03:43:18.339666+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:03.191398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:03.191398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":472,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414376+0000","last_change":"2024-03-12T03:40:19.489022+0000","last_active":"2024-03-12T03:43:54.414376+0000","last_peered":"2024-03-12T03:43:54.414376+0000","last_clean":"2024-03-12T03:43:54.414376+0000","last_became_active":"2024-03-12T03:39:33.271016+0000","last_became_peered":"2024-03-12T03:39:33.271016+0000","last_unstale":"2024-03-12T03:43:54.414376+0000","last_undegraded":"2024-03-12T03:43:54.414376+0000","last_fullsized":"2024-03-12T03:43:54.414376+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:19.488908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:40:19.488908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"284'159","reported_seq":178,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:56.067516+0000","last_change":"2024-03-12T03:43:55.403641+0000","last_active":"2024-03-12T03:43:56.067516+0000","last_peered":"2024-03-12T03:43:56.067516+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.592972+0000","last_became_peered":"2024-03-12T03:43:45.592972+0000","last_unstale":"2024-03-12T03:43:56.067516+0000","last_undegraded":"2024-03-12T03:43:55.398613+0000","last_fullsized":"2024-03-12T03:43:45.585435+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:04.063684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420224,"num_objects":159,"num_object_clones":0,"num_object_copies":477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":10176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":159}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'51","reported_seq":559,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414262+0000","last_change":"2024-03-12T03:42:15.396790+0000","last_active":"2024-03-12T03:43:54.414262+0000","last_peered":"2024-03-12T03:43:54.414262+0000","last_clean":"2024-03-12T03:43:54.414262+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:43:54.414262+0000","last_undegraded":"2024-03-12T03:43:54.414262+0000","last_fullsized":"2024-03-12T03:43:54.414262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'35","last_scrub_stamp":"2024-03-12T03:42:15.396647+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:42:15.396647+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414904+0000","last_change":"2024-03-12T03:42:15.159053+0000","last_active":"2024-03-12T03:43:54.414904+0000","last_peered":"2024-03-12T03:43:54.414904+0000","last_clean":"2024-03-12T03:43:54.414904+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:43:54.414904+0000","last_undegraded":"2024-03-12T03:43:54.414904+0000","last_fullsized":"2024-03-12T03:43:54.414904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:15.158916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:15.158916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:54.970999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":191,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.364435+0000","last_change":"2024-03-12T03:42:11.155750+0000","last_active":"2024-03-12T03:43:54.364435+0000","last_peered":"2024-03-12T03:43:54.364435+0000","last_clean":"2024-03-12T03:43:54.364435+0000","last_became_active":"2024-03-12T03:39:33.254395+0000","last_became_peered":"2024-03-12T03:39:33.254395+0000","last_unstale":"2024-03-12T03:43:54.364435+0000","last_undegraded":"2024-03-12T03:43:54.364435+0000","last_fullsized":"2024-03-12T03:43:54.364435+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:11.155639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:11.155639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:57.583996+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":492,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.416283+0000","last_change":"2024-03-12T03:43:18.339713+0000","last_active":"2024-03-12T03:43:54.416283+0000","last_peered":"2024-03-12T03:43:54.416283+0000","last_clean":"2024-03-12T03:43:15.079953+0000","last_became_active":"2024-03-12T03:43:18.339713+0000","last_became_peered":"2024-03-12T03:43:18.339713+0000","last_unstale":"2024-03-12T03:43:54.416283+0000","last_undegraded":"2024-03-12T03:43:54.416283+0000","last_fullsized":"2024-03-12T03:43:18.336066+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:15.079874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:15.079874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:19.243397+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"284'409","reported_seq":428,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:57.053344+0000","last_change":"2024-03-12T03:43:55.386671+0000","last_active":"2024-03-12T03:43:57.053344+0000","last_peered":"2024-03-12T03:43:57.053344+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.590025+0000","last_became_peered":"2024-03-12T03:43:45.590025+0000","last_unstale":"2024-03-12T03:43:57.053344+0000","last_undegraded":"2024-03-12T03:43:55.378927+0000","last_fullsized":"2024-03-12T03:43:45.586649+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:00.255405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26804224,"num_objects":409,"num_object_clones":0,"num_object_copies":1227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":409,"num_write_kb":26176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":409}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":200,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.364278+0000","last_change":"2024-03-12T03:42:24.169808+0000","last_active":"2024-03-12T03:43:54.364278+0000","last_peered":"2024-03-12T03:43:54.364278+0000","last_clean":"2024-03-12T03:43:54.364278+0000","last_became_active":"2024-03-12T03:39:33.278326+0000","last_became_peered":"2024-03-12T03:39:33.278326+0000","last_unstale":"2024-03-12T03:43:54.364278+0000","last_undegraded":"2024-03-12T03:43:54.364278+0000","last_fullsized":"2024-03-12T03:43:54.364278+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:24.169673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:24.169673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:56.608238+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"269'30","reported_seq":544,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.360386+0000","last_change":"2024-03-12T03:43:18.341247+0000","last_active":"2024-03-12T03:43:54.360386+0000","last_peered":"2024-03-12T03:43:54.360386+0000","last_clean":"2024-03-12T03:43:15.540480+0000","last_became_active":"2024-03-12T03:43:18.341247+0000","last_became_peered":"2024-03-12T03:43:18.341247+0000","last_unstale":"2024-03-12T03:43:54.360386+0000","last_undegraded":"2024-03-12T03:43:18.330226+0000","last_fullsized":"2024-03-12T03:43:18.330117+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-03-12T03:43:15.540339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:15.540339+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.594572+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415852+0000","last_change":"2024-03-12T03:42:09.072702+0000","last_active":"2024-03-12T03:43:54.415852+0000","last_peered":"2024-03-12T03:43:54.415852+0000","last_clean":"2024-03-12T03:43:54.415852+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:43:54.415852+0000","last_undegraded":"2024-03-12T03:43:54.415852+0000","last_fullsized":"2024-03-12T03:43:54.415852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:09.072495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:42:09.072495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:54.970897+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":248,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.415264+0000","last_change":"2024-03-12T03:43:18.345830+0000","last_active":"2024-03-12T03:43:54.415264+0000","last_peered":"2024-03-12T03:43:54.415264+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.345830+0000","last_became_peered":"2024-03-12T03:43:18.345830+0000","last_unstale":"2024-03-12T03:43:54.415264+0000","last_undegraded":"2024-03-12T03:43:54.415264+0000","last_fullsized":"2024-03-12T03:43:18.336109+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:18.651418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:18.651418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:54.970814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"284'298","reported_seq":320,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:56.598165+0000","last_change":"2024-03-12T03:43:45.593274+0000","last_active":"2024-03-12T03:43:56.598165+0000","last_peered":"2024-03-12T03:43:56.598165+0000","last_clean":"2024-03-12T03:43:56.598165+0000","last_became_active":"2024-03-12T03:43:45.593074+0000","last_became_peered":"2024-03-12T03:43:45.593074+0000","last_unstale":"2024-03-12T03:43:56.598165+0000","last_undegraded":"2024-03-12T03:43:56.598165+0000","last_fullsized":"2024-03-12T03:43:56.598165+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:11.927395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19529728,"num_objects":298,"num_object_clones":0,"num_object_copies":894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":298,"num_write_kb":19072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":608,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415972+0000","last_change":"2024-03-12T03:43:23.985482+0000","last_active":"2024-03-12T03:43:54.415972+0000","last_peered":"2024-03-12T03:43:54.415972+0000","last_clean":"2024-03-12T03:43:54.415972+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:43:54.415972+0000","last_undegraded":"2024-03-12T03:43:54.415972+0000","last_fullsized":"2024-03-12T03:43:54.415972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:43:23.985362+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:26.252895+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"269'48","reported_seq":543,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414092+0000","last_change":"2024-03-12T03:40:14.776197+0000","last_active":"2024-03-12T03:43:54.414092+0000","last_peered":"2024-03-12T03:43:54.414092+0000","last_clean":"2024-03-12T03:43:54.414092+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:43:54.414092+0000","last_undegraded":"2024-03-12T03:43:54.414092+0000","last_fullsized":"2024-03-12T03:43:54.414092+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'36","last_scrub_stamp":"2024-03-12T03:40:14.776063+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:40:14.776063+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":484,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.416065+0000","last_change":"2024-03-12T03:41:20.054595+0000","last_active":"2024-03-12T03:43:54.416065+0000","last_peered":"2024-03-12T03:43:54.416065+0000","last_clean":"2024-03-12T03:43:54.416065+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:43:54.416065+0000","last_undegraded":"2024-03-12T03:43:54.416065+0000","last_fullsized":"2024-03-12T03:43:54.416065+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:41:20.054461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:41:20.054461+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:54.970866+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":459,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414027+0000","last_change":"2024-03-12T03:40:28.767260+0000","last_active":"2024-03-12T03:43:54.414027+0000","last_peered":"2024-03-12T03:43:54.414027+0000","last_clean":"2024-03-12T03:43:54.414027+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:43:54.414027+0000","last_undegraded":"2024-03-12T03:43:54.414027+0000","last_fullsized":"2024-03-12T03:43:54.414027+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:28.767140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:40:28.767140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"269'18","reported_seq":135,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.415986+0000","last_change":"2024-03-12T03:43:18.350084+0000","last_active":"2024-03-12T03:43:54.415986+0000","last_peered":"2024-03-12T03:43:54.415986+0000","last_clean":"2024-03-12T03:33:28.523929+0000","last_became_active":"2024-03-12T03:43:18.350084+0000","last_became_peered":"2024-03-12T03:43:18.350084+0000","last_unstale":"2024-03-12T03:43:54.415986+0000","last_undegraded":"2024-03-12T03:43:18.336688+0000","last_fullsized":"2024-03-12T03:43:18.336568+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:43:17.648603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:17.648603+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:33.783792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":486,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.413385+0000","last_change":"2024-03-12T03:42:14.426265+0000","last_active":"2024-03-12T03:43:54.413385+0000","last_peered":"2024-03-12T03:43:54.413385+0000","last_clean":"2024-03-12T03:43:54.413385+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:43:54.413385+0000","last_undegraded":"2024-03-12T03:43:54.413385+0000","last_fullsized":"2024-03-12T03:43:54.413385+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:14.426145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:14.426145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":475,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414080+0000","last_change":"2024-03-12T03:43:18.348340+0000","last_active":"2024-03-12T03:43:54.414080+0000","last_peered":"2024-03-12T03:43:54.414080+0000","last_clean":"2024-03-12T03:43:13.352260+0000","last_became_active":"2024-03-12T03:43:18.348340+0000","last_became_peered":"2024-03-12T03:43:18.348340+0000","last_unstale":"2024-03-12T03:43:54.414080+0000","last_undegraded":"2024-03-12T03:43:54.414080+0000","last_fullsized":"2024-03-12T03:43:18.333554+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:01.417809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:01.417809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":468,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414525+0000","last_change":"2024-03-12T03:42:02.044839+0000","last_active":"2024-03-12T03:43:54.414525+0000","last_peered":"2024-03-12T03:43:54.414525+0000","last_clean":"2024-03-12T03:43:54.414525+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:43:54.414525+0000","last_undegraded":"2024-03-12T03:43:54.414525+0000","last_fullsized":"2024-03-12T03:43:54.414525+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:02.044703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:02.044703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:54.970882+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"269'27","reported_seq":529,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.415585+0000","last_change":"2024-03-12T03:43:18.358476+0000","last_active":"2024-03-12T03:43:54.415585+0000","last_peered":"2024-03-12T03:43:54.415585+0000","last_clean":"2024-03-12T03:43:13.333279+0000","last_became_active":"2024-03-12T03:43:18.358476+0000","last_became_peered":"2024-03-12T03:43:18.358476+0000","last_unstale":"2024-03-12T03:43:54.415585+0000","last_undegraded":"2024-03-12T03:43:18.343811+0000","last_fullsized":"2024-03-12T03:43:18.343694+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:43:03.583989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:03.583989+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:07.810082+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415865+0000","last_change":"2024-03-12T03:42:22.126915+0000","last_active":"2024-03-12T03:43:54.415865+0000","last_peered":"2024-03-12T03:43:54.415865+0000","last_clean":"2024-03-12T03:43:54.415865+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:43:54.415865+0000","last_undegraded":"2024-03-12T03:43:54.415865+0000","last_fullsized":"2024-03-12T03:43:54.415865+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:22.126792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:22.126792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:57.003045+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":280,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.415225+0000","last_change":"2024-03-12T03:43:18.346246+0000","last_active":"2024-03-12T03:43:54.415225+0000","last_peered":"2024-03-12T03:43:54.415225+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:43:18.346246+0000","last_became_peered":"2024-03-12T03:43:18.346246+0000","last_unstale":"2024-03-12T03:43:54.415225+0000","last_undegraded":"2024-03-12T03:43:54.415225+0000","last_fullsized":"2024-03-12T03:43:18.335155+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:05.530729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:43:05.530729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:30.419635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":199,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.364921+0000","last_change":"2024-03-12T03:43:03.293930+0000","last_active":"2024-03-12T03:43:54.364921+0000","last_peered":"2024-03-12T03:43:54.364921+0000","last_clean":"2024-03-12T03:43:54.364921+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:43:54.364921+0000","last_undegraded":"2024-03-12T03:43:54.364921+0000","last_fullsized":"2024-03-12T03:43:54.364921+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:03.293463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:03.293463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:12.312596+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"269'21","reported_seq":524,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.416479+0000","last_change":"2024-03-12T03:43:18.347815+0000","last_active":"2024-03-12T03:43:54.416479+0000","last_peered":"2024-03-12T03:43:54.416479+0000","last_clean":"2024-03-12T03:43:13.353750+0000","last_became_active":"2024-03-12T03:43:18.347815+0000","last_became_peered":"2024-03-12T03:43:18.347815+0000","last_unstale":"2024-03-12T03:43:54.416479+0000","last_undegraded":"2024-03-12T03:43:18.332425+0000","last_fullsized":"2024-03-12T03:43:18.332310+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:40:58.312602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:58.312602+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":484,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414557+0000","last_change":"2024-03-12T03:43:18.343018+0000","last_active":"2024-03-12T03:43:54.414557+0000","last_peered":"2024-03-12T03:43:54.414557+0000","last_clean":"2024-03-12T03:43:13.331412+0000","last_became_active":"2024-03-12T03:43:18.343018+0000","last_became_peered":"2024-03-12T03:43:18.343018+0000","last_unstale":"2024-03-12T03:43:54.414557+0000","last_undegraded":"2024-03-12T03:43:54.414557+0000","last_fullsized":"2024-03-12T03:43:18.331686+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:00.266512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:42:00.266512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"266'13","reported_seq":135,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.416264+0000","last_change":"2024-03-12T03:43:18.350782+0000","last_active":"2024-03-12T03:43:54.416264+0000","last_peered":"2024-03-12T03:43:54.416264+0000","last_clean":"2024-03-12T03:33:28.372811+0000","last_became_active":"2024-03-12T03:43:18.350782+0000","last_became_peered":"2024-03-12T03:43:18.350782+0000","last_unstale":"2024-03-12T03:43:54.416264+0000","last_undegraded":"2024-03-12T03:43:18.340626+0000","last_fullsized":"2024-03-12T03:43:18.340503+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'12","last_scrub_stamp":"2024-03-12T03:43:12.675070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:12.675070+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:32.314704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.360451+0000","last_change":"2024-03-12T03:41:19.377731+0000","last_active":"2024-03-12T03:43:54.360451+0000","last_peered":"2024-03-12T03:43:54.360451+0000","last_clean":"2024-03-12T03:43:54.360451+0000","last_became_active":"2024-03-12T03:39:33.265499+0000","last_became_peered":"2024-03-12T03:39:33.265499+0000","last_unstale":"2024-03-12T03:43:54.360451+0000","last_undegraded":"2024-03-12T03:43:54.360451+0000","last_fullsized":"2024-03-12T03:43:54.360451+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:19.377599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:19.377599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"269'32","reported_seq":444,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.364587+0000","last_change":"2024-03-12T03:43:18.357702+0000","last_active":"2024-03-12T03:43:54.364587+0000","last_peered":"2024-03-12T03:43:54.364587+0000","last_clean":"2024-03-12T03:42:47.830157+0000","last_became_active":"2024-03-12T03:43:18.357702+0000","last_became_peered":"2024-03-12T03:43:18.357702+0000","last_unstale":"2024-03-12T03:43:54.364587+0000","last_undegraded":"2024-03-12T03:43:18.340096+0000","last_fullsized":"2024-03-12T03:43:18.339982+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-03-12T03:35:58.079645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.079645+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"67'1","reported_seq":506,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414828+0000","last_change":"2024-03-12T03:42:12.114036+0000","last_active":"2024-03-12T03:43:54.414828+0000","last_peered":"2024-03-12T03:43:54.414828+0000","last_clean":"2024-03-12T03:43:54.414828+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:43:54.414828+0000","last_undegraded":"2024-03-12T03:43:54.414828+0000","last_fullsized":"2024-03-12T03:43:54.414828+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:42:12.113938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:12.113938+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:54.970933+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":598,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.416783+0000","last_change":"2024-03-12T03:40:15.773578+0000","last_active":"2024-03-12T03:43:54.416783+0000","last_peered":"2024-03-12T03:43:54.416783+0000","last_clean":"2024-03-12T03:43:54.416783+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:43:54.416783+0000","last_undegraded":"2024-03-12T03:43:54.416783+0000","last_fullsized":"2024-03-12T03:43:54.416783+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:40:15.773432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:40:15.773432+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":472,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414734+0000","last_change":"2024-03-12T03:43:18.357954+0000","last_active":"2024-03-12T03:43:54.414734+0000","last_peered":"2024-03-12T03:43:54.414734+0000","last_clean":"2024-03-12T03:43:13.332853+0000","last_became_active":"2024-03-12T03:43:18.357954+0000","last_became_peered":"2024-03-12T03:43:18.357954+0000","last_unstale":"2024-03-12T03:43:54.414734+0000","last_undegraded":"2024-03-12T03:43:54.414734+0000","last_fullsized":"2024-03-12T03:43:18.341976+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:10.355198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:10.355198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"269'19","reported_seq":515,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.360485+0000","last_change":"2024-03-12T03:43:22.673154+0000","last_active":"2024-03-12T03:43:54.360485+0000","last_peered":"2024-03-12T03:43:54.360485+0000","last_clean":"2024-03-12T03:43:54.360485+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:43:54.360485+0000","last_undegraded":"2024-03-12T03:43:54.360485+0000","last_fullsized":"2024-03-12T03:43:54.360485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:43:22.673042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:22.673042+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:24.371289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.360230+0000","last_change":"2024-03-12T03:41:13.286000+0000","last_active":"2024-03-12T03:43:54.360230+0000","last_peered":"2024-03-12T03:43:54.360230+0000","last_clean":"2024-03-12T03:43:54.360230+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:43:54.360230+0000","last_undegraded":"2024-03-12T03:43:54.360230+0000","last_fullsized":"2024-03-12T03:43:54.360230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:13.285572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:38:34.264717+0000","last_clean_scrub_stamp":"2024-03-12T03:41:13.285572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":484,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.413955+0000","last_change":"2024-03-12T03:43:18.347925+0000","last_active":"2024-03-12T03:43:54.413955+0000","last_peered":"2024-03-12T03:43:54.413955+0000","last_clean":"2024-03-12T03:43:13.351418+0000","last_became_active":"2024-03-12T03:43:18.347925+0000","last_became_peered":"2024-03-12T03:43:18.347925+0000","last_unstale":"2024-03-12T03:43:54.413955+0000","last_undegraded":"2024-03-12T03:43:54.413955+0000","last_fullsized":"2024-03-12T03:43:18.334927+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:35.850271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:35.850271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":298,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414148+0000","last_change":"2024-03-12T03:43:18.350340+0000","last_active":"2024-03-12T03:43:54.414148+0000","last_peered":"2024-03-12T03:43:54.414148+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.350340+0000","last_became_peered":"2024-03-12T03:43:18.350340+0000","last_unstale":"2024-03-12T03:43:54.414148+0000","last_undegraded":"2024-03-12T03:43:54.414148+0000","last_fullsized":"2024-03-12T03:43:18.332506+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.075897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.075897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"269'15","reported_seq":72,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.413774+0000","last_change":"2024-03-12T03:43:01.048588+0000","last_active":"2024-03-12T03:43:54.413774+0000","last_peered":"2024-03-12T03:43:54.413774+0000","last_clean":"2024-03-12T03:43:54.413774+0000","last_became_active":"2024-03-12T03:43:01.037348+0000","last_became_peered":"2024-03-12T03:43:01.037348+0000","last_unstale":"2024-03-12T03:43:54.413774+0000","last_undegraded":"2024-03-12T03:43:54.413774+0000","last_fullsized":"2024-03-12T03:43:54.413774+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-12T03:40:17.848303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:17.848303+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":492,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414621+0000","last_change":"2024-03-12T03:43:18.347900+0000","last_active":"2024-03-12T03:43:54.414621+0000","last_peered":"2024-03-12T03:43:54.414621+0000","last_clean":"2024-03-12T03:43:13.342510+0000","last_became_active":"2024-03-12T03:43:18.347900+0000","last_became_peered":"2024-03-12T03:43:18.347900+0000","last_unstale":"2024-03-12T03:43:54.414621+0000","last_undegraded":"2024-03-12T03:43:54.414621+0000","last_fullsized":"2024-03-12T03:43:18.337740+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:08.131514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:43:08.131514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:30.625871+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":310,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.415480+0000","last_change":"2024-03-12T03:43:18.343467+0000","last_active":"2024-03-12T03:43:54.415480+0000","last_peered":"2024-03-12T03:43:54.415480+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:43:18.343467+0000","last_became_peered":"2024-03-12T03:43:18.343467+0000","last_unstale":"2024-03-12T03:43:54.415480+0000","last_undegraded":"2024-03-12T03:43:54.415480+0000","last_fullsized":"2024-03-12T03:43:18.332412+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:15.808887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:15.808887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":468,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.413370+0000","last_change":"2024-03-12T03:43:04.613657+0000","last_active":"2024-03-12T03:43:54.413370+0000","last_peered":"2024-03-12T03:43:54.413370+0000","last_clean":"2024-03-12T03:43:54.413370+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:43:54.413370+0000","last_undegraded":"2024-03-12T03:43:54.413370+0000","last_fullsized":"2024-03-12T03:43:54.413370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:04.613384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:43:04.613384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:20.298614+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"269'26","reported_seq":149,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.415958+0000","last_change":"2024-03-12T03:43:18.353670+0000","last_active":"2024-03-12T03:43:54.415958+0000","last_peered":"2024-03-12T03:43:54.415958+0000","last_clean":"2024-03-12T03:33:28.559608+0000","last_became_active":"2024-03-12T03:43:18.353670+0000","last_became_peered":"2024-03-12T03:43:18.353670+0000","last_unstale":"2024-03-12T03:43:54.415958+0000","last_undegraded":"2024-03-12T03:43:18.338840+0000","last_fullsized":"2024-03-12T03:43:18.338735+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:43:06.549441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:06.549441+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:09.293527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":479,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414267+0000","last_change":"2024-03-12T03:43:27.499252+0000","last_active":"2024-03-12T03:43:54.414267+0000","last_peered":"2024-03-12T03:43:54.414267+0000","last_clean":"2024-03-12T03:43:54.414267+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:43:54.414267+0000","last_undegraded":"2024-03-12T03:43:54.414267+0000","last_fullsized":"2024-03-12T03:43:54.414267+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:27.499165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:43:27.499165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:37.701629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":472,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.416619+0000","last_change":"2024-03-12T03:43:13.075342+0000","last_active":"2024-03-12T03:43:54.416619+0000","last_peered":"2024-03-12T03:43:54.416619+0000","last_clean":"2024-03-12T03:43:54.416619+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:43:54.416619+0000","last_undegraded":"2024-03-12T03:43:54.416619+0000","last_fullsized":"2024-03-12T03:43:54.416619+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:13.075266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:13.075266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:22.129582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":333,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414810+0000","last_change":"2024-03-12T03:43:18.344649+0000","last_active":"2024-03-12T03:43:54.414810+0000","last_peered":"2024-03-12T03:43:54.414810+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.344649+0000","last_became_peered":"2024-03-12T03:43:18.344649+0000","last_unstale":"2024-03-12T03:43:54.414810+0000","last_undegraded":"2024-03-12T03:43:54.414810+0000","last_fullsized":"2024-03-12T03:43:18.331783+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"269'10","reported_seq":154,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.414092+0000","last_change":"2024-03-12T03:43:18.337555+0000","last_active":"2024-03-12T03:43:54.414092+0000","last_peered":"2024-03-12T03:43:54.414092+0000","last_clean":"2024-03-12T03:34:34.272872+0000","last_became_active":"2024-03-12T03:43:18.337555+0000","last_became_peered":"2024-03-12T03:43:18.337555+0000","last_unstale":"2024-03-12T03:43:54.414092+0000","last_undegraded":"2024-03-12T03:43:18.333971+0000","last_fullsized":"2024-03-12T03:43:18.333854+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:38:38.943094+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:38:38.943094+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":484,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414777+0000","last_change":"2024-03-12T03:43:18.349342+0000","last_active":"2024-03-12T03:43:54.414777+0000","last_peered":"2024-03-12T03:43:54.414777+0000","last_clean":"2024-03-12T03:43:13.332505+0000","last_became_active":"2024-03-12T03:43:18.349342+0000","last_became_peered":"2024-03-12T03:43:18.349342+0000","last_unstale":"2024-03-12T03:43:54.414777+0000","last_undegraded":"2024-03-12T03:43:54.414777+0000","last_fullsized":"2024-03-12T03:43:18.335423+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:19.511683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:39:19.511683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":481,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414928+0000","last_change":"2024-03-12T03:43:34.038641+0000","last_active":"2024-03-12T03:43:54.414928+0000","last_peered":"2024-03-12T03:43:54.414928+0000","last_clean":"2024-03-12T03:43:54.414928+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:43:54.414928+0000","last_undegraded":"2024-03-12T03:43:54.414928+0000","last_fullsized":"2024-03-12T03:43:54.414928+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:06.327362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:06.327362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414029+0000","last_change":"2024-03-12T03:43:28.028114+0000","last_active":"2024-03-12T03:43:54.414029+0000","last_peered":"2024-03-12T03:43:54.414029+0000","last_clean":"2024-03-12T03:43:54.414029+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:43:54.414029+0000","last_undegraded":"2024-03-12T03:43:54.414029+0000","last_fullsized":"2024-03-12T03:43:54.414029+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:28.028020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:28.028020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:32.032062+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"269'31","reported_seq":532,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415351+0000","last_change":"2024-03-12T03:43:03.536079+0000","last_active":"2024-03-12T03:43:54.415351+0000","last_peered":"2024-03-12T03:43:54.415351+0000","last_clean":"2024-03-12T03:43:54.415351+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:43:54.415351+0000","last_undegraded":"2024-03-12T03:43:54.415351+0000","last_fullsized":"2024-03-12T03:43:54.415351+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'23","last_scrub_stamp":"2024-03-12T03:43:03.535926+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:43:03.535926+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:19.248123+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":94,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.413479+0000","last_change":"2024-03-12T03:43:18.342446+0000","last_active":"2024-03-12T03:43:54.413479+0000","last_peered":"2024-03-12T03:43:54.413479+0000","last_clean":"2024-03-12T03:33:21.096389+0000","last_became_active":"2024-03-12T03:43:18.342446+0000","last_became_peered":"2024-03-12T03:43:18.342446+0000","last_unstale":"2024-03-12T03:43:54.413479+0000","last_undegraded":"2024-03-12T03:43:18.333444+0000","last_fullsized":"2024-03-12T03:43:18.333330+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:25.092313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.092313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":477,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415813+0000","last_change":"2024-03-12T03:40:21.384617+0000","last_active":"2024-03-12T03:43:54.415813+0000","last_peered":"2024-03-12T03:43:54.415813+0000","last_clean":"2024-03-12T03:43:54.415813+0000","last_became_active":"2024-03-12T03:39:33.276435+0000","last_became_peered":"2024-03-12T03:39:33.276435+0000","last_unstale":"2024-03-12T03:43:54.415813+0000","last_undegraded":"2024-03-12T03:43:54.415813+0000","last_fullsized":"2024-03-12T03:43:54.415813+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:21.384531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:40:21.384531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":475,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415107+0000","last_change":"2024-03-12T03:43:27.868017+0000","last_active":"2024-03-12T03:43:54.415107+0000","last_peered":"2024-03-12T03:43:54.415107+0000","last_clean":"2024-03-12T03:43:54.415107+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:43:54.415107+0000","last_undegraded":"2024-03-12T03:43:54.415107+0000","last_fullsized":"2024-03-12T03:43:54.415107+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:27.867900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:27.867900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:33.395076+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"269'27","reported_seq":524,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.415367+0000","last_change":"2024-03-12T03:43:18.358106+0000","last_active":"2024-03-12T03:43:54.415367+0000","last_peered":"2024-03-12T03:43:54.415367+0000","last_clean":"2024-03-12T03:43:13.333148+0000","last_became_active":"2024-03-12T03:43:18.358106+0000","last_became_peered":"2024-03-12T03:43:18.358106+0000","last_unstale":"2024-03-12T03:43:54.415367+0000","last_undegraded":"2024-03-12T03:43:18.343380+0000","last_fullsized":"2024-03-12T03:43:18.343203+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:39:37.572971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:37.572971+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":493,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.413493+0000","last_change":"2024-03-12T03:43:18.350788+0000","last_active":"2024-03-12T03:43:54.413493+0000","last_peered":"2024-03-12T03:43:54.413493+0000","last_clean":"2024-03-12T03:43:13.349345+0000","last_became_active":"2024-03-12T03:43:18.350788+0000","last_became_peered":"2024-03-12T03:43:18.350788+0000","last_unstale":"2024-03-12T03:43:54.413493+0000","last_undegraded":"2024-03-12T03:43:18.332266+0000","last_fullsized":"2024-03-12T03:43:18.332143+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:43:07.772205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:07.772205+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:31.743033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":617,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415250+0000","last_change":"2024-03-12T03:43:17.506932+0000","last_active":"2024-03-12T03:43:54.415250+0000","last_peered":"2024-03-12T03:43:54.415250+0000","last_clean":"2024-03-12T03:43:54.415250+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:43:54.415250+0000","last_undegraded":"2024-03-12T03:43:54.415250+0000","last_fullsized":"2024-03-12T03:43:54.415250+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:43:17.506791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:17.506791+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:38.384185+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":489,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.416321+0000","last_change":"2024-03-12T03:41:26.041453+0000","last_active":"2024-03-12T03:43:54.416321+0000","last_peered":"2024-03-12T03:43:54.416321+0000","last_clean":"2024-03-12T03:43:54.416321+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:43:54.416321+0000","last_undegraded":"2024-03-12T03:43:54.416321+0000","last_fullsized":"2024-03-12T03:43:54.416321+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:41:26.041306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:26.041306+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:54.970846+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"269'24","reported_seq":149,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.414695+0000","last_change":"2024-03-12T03:43:18.346292+0000","last_active":"2024-03-12T03:43:54.414695+0000","last_peered":"2024-03-12T03:43:54.414695+0000","last_clean":"2024-03-12T03:33:28.567700+0000","last_became_active":"2024-03-12T03:43:18.346292+0000","last_became_peered":"2024-03-12T03:43:18.346292+0000","last_unstale":"2024-03-12T03:43:54.414695+0000","last_undegraded":"2024-03-12T03:43:18.333553+0000","last_fullsized":"2024-03-12T03:43:18.333434+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:36:28.133261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.133261+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":500,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.360869+0000","last_change":"2024-03-12T03:42:18.499466+0000","last_active":"2024-03-12T03:43:54.360869+0000","last_peered":"2024-03-12T03:43:54.360869+0000","last_clean":"2024-03-12T03:43:54.360869+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:43:54.360869+0000","last_undegraded":"2024-03-12T03:43:54.360869+0000","last_fullsized":"2024-03-12T03:43:54.360869+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:18.499342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:18.499342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":496,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.360572+0000","last_change":"2024-03-12T03:43:18.351475+0000","last_active":"2024-03-12T03:43:54.360572+0000","last_peered":"2024-03-12T03:43:54.360572+0000","last_clean":"2024-03-12T03:43:13.338616+0000","last_became_active":"2024-03-12T03:43:18.351475+0000","last_became_peered":"2024-03-12T03:43:18.351475+0000","last_unstale":"2024-03-12T03:43:54.360572+0000","last_undegraded":"2024-03-12T03:43:54.360572+0000","last_fullsized":"2024-03-12T03:43:18.332508+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:08.289718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:08.289718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":326,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.415618+0000","last_change":"2024-03-12T03:43:18.349876+0000","last_active":"2024-03-12T03:43:54.415618+0000","last_peered":"2024-03-12T03:43:54.415618+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.349876+0000","last_became_peered":"2024-03-12T03:43:18.349876+0000","last_unstale":"2024-03-12T03:43:54.415618+0000","last_undegraded":"2024-03-12T03:43:54.415618+0000","last_fullsized":"2024-03-12T03:43:18.334950+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:16.588551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:16.588551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"269'27","reported_seq":237,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.364864+0000","last_change":"2024-03-12T03:43:18.338861+0000","last_active":"2024-03-12T03:43:54.364864+0000","last_peered":"2024-03-12T03:43:54.364864+0000","last_clean":"2024-03-12T03:43:13.346680+0000","last_became_active":"2024-03-12T03:43:18.338861+0000","last_became_peered":"2024-03-12T03:43:18.338861+0000","last_unstale":"2024-03-12T03:43:54.364864+0000","last_undegraded":"2024-03-12T03:43:18.337001+0000","last_fullsized":"2024-03-12T03:43:18.336879+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:42:04.171114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:42:04.171114+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415619+0000","last_change":"2024-03-12T03:43:10.670801+0000","last_active":"2024-03-12T03:43:54.415619+0000","last_peered":"2024-03-12T03:43:54.415619+0000","last_clean":"2024-03-12T03:43:54.415619+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:43:54.415619+0000","last_undegraded":"2024-03-12T03:43:54.415619+0000","last_fullsized":"2024-03-12T03:43:54.415619+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:10.670685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:10.670685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:25.442899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":479,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414906+0000","last_change":"2024-03-12T03:43:16.842242+0000","last_active":"2024-03-12T03:43:54.414906+0000","last_peered":"2024-03-12T03:43:54.414906+0000","last_clean":"2024-03-12T03:43:54.414906+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:43:54.414906+0000","last_undegraded":"2024-03-12T03:43:54.414906+0000","last_fullsized":"2024-03-12T03:43:54.414906+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:16.842126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:16.842126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.283787+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":469,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.360793+0000","last_change":"2024-03-12T03:42:20.532434+0000","last_active":"2024-03-12T03:43:54.360793+0000","last_peered":"2024-03-12T03:43:54.360793+0000","last_clean":"2024-03-12T03:43:54.360793+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:43:54.360793+0000","last_undegraded":"2024-03-12T03:43:54.360793+0000","last_fullsized":"2024-03-12T03:43:54.360793+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:20.532318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:20.532318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"269'16","reported_seq":552,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.360308+0000","last_change":"2024-03-12T03:43:18.345649+0000","last_active":"2024-03-12T03:43:54.360308+0000","last_peered":"2024-03-12T03:43:54.360308+0000","last_clean":"2024-03-12T03:43:13.343489+0000","last_became_active":"2024-03-12T03:43:18.345649+0000","last_became_peered":"2024-03-12T03:43:18.345649+0000","last_unstale":"2024-03-12T03:43:54.360308+0000","last_undegraded":"2024-03-12T03:43:18.330821+0000","last_fullsized":"2024-03-12T03:43:18.330708+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:43:10.580243+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:43:10.580243+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:19.088823+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":471,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.415895+0000","last_change":"2024-03-12T03:40:17.773177+0000","last_active":"2024-03-12T03:43:54.415895+0000","last_peered":"2024-03-12T03:43:54.415895+0000","last_clean":"2024-03-12T03:43:54.415895+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:43:54.415895+0000","last_undegraded":"2024-03-12T03:43:54.415895+0000","last_fullsized":"2024-03-12T03:43:54.415895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:17.772938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:40:17.772938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":472,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.413772+0000","last_change":"2024-03-12T03:43:22.975945+0000","last_active":"2024-03-12T03:43:54.413772+0000","last_peered":"2024-03-12T03:43:54.413772+0000","last_clean":"2024-03-12T03:43:54.413772+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:43:54.413772+0000","last_undegraded":"2024-03-12T03:43:54.413772+0000","last_fullsized":"2024-03-12T03:43:54.413772+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:43:22.975800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:31.931946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":471,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.413858+0000","last_change":"2024-03-12T03:43:18.348827+0000","last_active":"2024-03-12T03:43:54.413858+0000","last_peered":"2024-03-12T03:43:54.413858+0000","last_clean":"2024-03-12T03:43:13.354323+0000","last_became_active":"2024-03-12T03:43:18.348827+0000","last_became_peered":"2024-03-12T03:43:18.348827+0000","last_unstale":"2024-03-12T03:43:54.413858+0000","last_undegraded":"2024-03-12T03:43:54.413858+0000","last_fullsized":"2024-03-12T03:43:18.333911+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:15.648255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:15.648255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"269'27","reported_seq":537,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.360642+0000","last_change":"2024-03-12T03:43:18.340176+0000","last_active":"2024-03-12T03:43:54.360642+0000","last_peered":"2024-03-12T03:43:54.360642+0000","last_clean":"2024-03-12T03:43:16.563196+0000","last_became_active":"2024-03-12T03:43:18.340176+0000","last_became_peered":"2024-03-12T03:43:18.340176+0000","last_unstale":"2024-03-12T03:43:54.360642+0000","last_undegraded":"2024-03-12T03:43:18.332765+0000","last_fullsized":"2024-03-12T03:43:18.330467+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:43:16.563042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:16.563042+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:31.501216+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":471,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.416255+0000","last_change":"2024-03-12T03:40:25.773470+0000","last_active":"2024-03-12T03:43:54.416255+0000","last_peered":"2024-03-12T03:43:54.416255+0000","last_clean":"2024-03-12T03:43:54.416255+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:43:54.416255+0000","last_undegraded":"2024-03-12T03:43:54.416255+0000","last_fullsized":"2024-03-12T03:43:54.416255+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:25.773348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:40:25.773348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":463,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.413716+0000","last_change":"2024-03-12T03:40:29.743363+0000","last_active":"2024-03-12T03:43:54.413716+0000","last_peered":"2024-03-12T03:43:54.413716+0000","last_clean":"2024-03-12T03:43:54.413716+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:43:54.413716+0000","last_undegraded":"2024-03-12T03:43:54.413716+0000","last_fullsized":"2024-03-12T03:43:54.413716+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:29.743231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:40:29.743231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":460,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.413657+0000","last_change":"2024-03-12T03:40:31.512991+0000","last_active":"2024-03-12T03:43:54.413657+0000","last_peered":"2024-03-12T03:43:54.413657+0000","last_clean":"2024-03-12T03:43:54.413657+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:43:54.413657+0000","last_undegraded":"2024-03-12T03:43:54.413657+0000","last_fullsized":"2024-03-12T03:43:54.413657+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:31.512880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:40:31.512880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"269'62","reported_seq":494,"reported_epoch":284,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:43:54.364224+0000","last_change":"2024-03-12T03:43:18.343462+0000","last_active":"2024-03-12T03:43:54.364224+0000","last_peered":"2024-03-12T03:43:54.364224+0000","last_clean":"2024-03-12T03:39:28.242684+0000","last_became_active":"2024-03-12T03:43:18.343462+0000","last_became_peered":"2024-03-12T03:43:18.343462+0000","last_unstale":"2024-03-12T03:43:54.364224+0000","last_undegraded":"2024-03-12T03:43:18.336907+0000","last_fullsized":"2024-03-12T03:43:18.336787+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:21.856526+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":380,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.414649+0000","last_change":"2024-03-12T03:43:18.348499+0000","last_active":"2024-03-12T03:43:54.414649+0000","last_peered":"2024-03-12T03:43:54.414649+0000","last_clean":"2024-03-12T03:39:28.243893+0000","last_became_active":"2024-03-12T03:43:18.348499+0000","last_became_peered":"2024-03-12T03:43:18.348499+0000","last_unstale":"2024-03-12T03:43:54.414649+0000","last_undegraded":"2024-03-12T03:43:54.414649+0000","last_fullsized":"2024-03-12T03:43:18.337375+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:26.898782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:39:26.898782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":472,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.359811+0000","last_change":"2024-03-12T03:42:09.304630+0000","last_active":"2024-03-12T03:43:54.359811+0000","last_peered":"2024-03-12T03:43:54.359811+0000","last_clean":"2024-03-12T03:43:54.359811+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:43:54.359811+0000","last_undegraded":"2024-03-12T03:43:54.359811+0000","last_fullsized":"2024-03-12T03:43:54.359811+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:09.304528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:42:09.304528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":304,"reported_epoch":284,"state":"active+undersized","last_fresh":"2024-03-12T03:43:54.415034+0000","last_change":"2024-03-12T03:43:18.353114+0000","last_active":"2024-03-12T03:43:54.415034+0000","last_peered":"2024-03-12T03:43:54.415034+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.353114+0000","last_became_peered":"2024-03-12T03:43:18.353114+0000","last_unstale":"2024-03-12T03:43:54.415034+0000","last_undegraded":"2024-03-12T03:43:54.415034+0000","last_fullsized":"2024-03-12T03:43:18.337255+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.236997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.236997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:43:54.970805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"269'26","reported_seq":518,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.359809+0000","last_change":"2024-03-12T03:43:02.651857+0000","last_active":"2024-03-12T03:43:54.359809+0000","last_peered":"2024-03-12T03:43:54.359809+0000","last_clean":"2024-03-12T03:43:54.359809+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:43:54.359809+0000","last_undegraded":"2024-03-12T03:43:54.359809+0000","last_fullsized":"2024-03-12T03:43:54.359809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'20","last_scrub_stamp":"2024-03-12T03:43:02.651729+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:43:02.651729+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:05.143835+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":485,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.414619+0000","last_change":"2024-03-12T03:40:27.572921+0000","last_active":"2024-03-12T03:43:54.414619+0000","last_peered":"2024-03-12T03:43:54.414619+0000","last_clean":"2024-03-12T03:43:54.414619+0000","last_became_active":"2024-03-12T03:39:33.276119+0000","last_became_peered":"2024-03-12T03:39:33.276119+0000","last_unstale":"2024-03-12T03:43:54.414619+0000","last_undegraded":"2024-03-12T03:43:54.414619+0000","last_fullsized":"2024-03-12T03:43:54.414619+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:27.572788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:28.208044+0000","last_clean_scrub_stamp":"2024-03-12T03:40:27.572788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":488,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.998883+0000","last_change":"2024-03-12T03:43:54.998904+0000","last_active":"2024-03-12T03:43:54.998883+0000","last_peered":"2024-03-12T03:43:54.998883+0000","last_clean":"2024-03-12T03:43:54.998883+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:43:54.998883+0000","last_undegraded":"2024-03-12T03:43:54.998883+0000","last_fullsized":"2024-03-12T03:43:54.998883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:54.998817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:26.625326+0000","last_clean_scrub_stamp":"2024-03-12T03:43:54.998817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:19.507118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":476,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-12T03:43:54.359974+0000","last_change":"2024-03-12T03:43:18.631145+0000","last_active":"2024-03-12T03:43:54.359974+0000","last_peered":"2024-03-12T03:43:54.359974+0000","last_clean":"2024-03-12T03:43:54.359974+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:43:54.359974+0000","last_undegraded":"2024-03-12T03:43:54.359974+0000","last_fullsized":"2024-03-12T03:43:54.359974+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:18.631026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.063092+0000","last_clean_scrub_stamp":"2024-03-12T03:43:18.631026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:23.802026+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":256901120,"num_objects":3920,"num_object_clones":0,"num_object_copies":11760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3920,"num_write_kb":250880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":633208832,"data_stored":633208832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3920,"ondisk_log_size":3920,"up":40,"acting":40,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":628,"num_read_kb":623,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":686386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":28,"acting":28,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1409,"num_read_kb":1134,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":2284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3734,"internal_metadata":0},"log_size":944,"ondisk_log_size":944,"up":79,"acting":79,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070596,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":383868,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93361284,"statfs":{"total":95995035648,"available":95601954816,"internally_reserved":0,"allocated":1482752,"data_stored":934753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":391576010},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332233,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":366940,"kb_used_data":10896,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93378212,"statfs":{"total":95995035648,"available":95619289088,"internally_reserved":0,"allocated":11157504,"data_stored":10632118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364575174},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672708,"kb_used_data":236156,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93072444,"statfs":{"total":95995035648,"available":95306182656,"internally_reserved":0,"allocated":241823744,"data_stored":241255042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738535,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":430196,"kb_used_data":20008,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93314956,"statfs":{"total":95995035648,"available":95554514944,"internally_reserved":0,"allocated":20488192,"data_stored":19939447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411484,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":443108,"kb_used_data":10704,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93302044,"statfs":{"total":95995035648,"available":95541293056,"internally_reserved":0,"allocated":10960896,"data_stored":10437067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":222,"seq":953482739743,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":471640,"kb_used_data":158088,"kb_used_omap":7,"kb_used_meta":313528,"kb_avail":93273512,"statfs":{"total":95995035648,"available":95512076288,"internally_reserved":0,"allocated":161882112,"data_stored":161425991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8180,"internal_metadata":321052684},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953620,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":324920,"kb_used_data":23020,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93420232,"statfs":{"total":95995035648,"available":95662317568,"internally_reserved":0,"allocated":23572480,"data_stored":23019925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659278,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":442936,"kb_used_data":28584,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93302216,"statfs":{"total":95995035648,"available":95541469184,"internally_reserved":0,"allocated":29270016,"data_stored":28721379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3734,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":76808192,"data_stored":76808192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":247005184,"data_stored":247005184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50528256,"data_stored":50528256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28246016,"data_stored":28246016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":102563840,"data_stored":102563840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":39321600,"data_stored":39321600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":88735744,"data_stored":88735744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:44:02.157 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-12T03:44:02.157 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-03-12T03:44:02.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:02.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:02.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:02.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-12T03:44:02.360 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:02.377 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 7 16 21993 21977 196.2 164.312 0.0053207 0.00509133 2024-03-12T03:44:02.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:02.447 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:02.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:02.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:02.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:02.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:02.639 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:02 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 8005/50823 objects degraded (15.751%), 31 pgs degraded (PG_DEGRADED) 2024-03-12T03:44:02.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:02 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 8005/50823 objects degraded (15.751%), 31 pgs degraded (PG_DEGRADED) 2024-03-12T03:44:02.683 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:02.724 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:02 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 8005/50823 objects degraded (15.751%), 31 pgs degraded (PG_DEGRADED) 2024-03-12T03:44:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:02.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:02.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:02.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:02.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:02.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:03.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:03.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:03.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:03.169 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:03.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:03.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:03.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:44:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:44:03.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:03.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:03.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:03.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:03.377 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 8 16 24193 24177 188.861 137.5 0.00259824 0.00528683 2024-03-12T03:44:03.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:03.444 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:03.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:03.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:03.490 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:44:03.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:03.583 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:03.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:03.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:03.720 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:03.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:03 smithi173 ceph-mon[33544]: pgmap v506: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.0 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.09k op/s; 8005/50823 objects degraded (15.751%) 2024-03-12T03:44:03.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:03 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:03 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3600194553' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:44:03.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:03 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:44:03.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:03.842 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:03 smithi169 ceph-mon[29344]: pgmap v506: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.0 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.09k op/s; 8005/50823 objects degraded (15.751%) 2024-03-12T03:44:03.842 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:03 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:03.842 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:03 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3600194553' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:44:03.842 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:03 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:44:03.843 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:03 smithi169 ceph-mon[39949]: pgmap v506: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.0 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.09k op/s; 8005/50823 objects degraded (15.751%) 2024-03-12T03:44:03.843 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:03 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:03.843 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:03 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3600194553' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:44:03.843 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:03 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:44:03.860 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:03.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:03.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:03.962 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:03.999 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:04.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:04.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:04.120 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:04.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:04.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:04.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:04.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:04.276 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:04.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:04.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:04.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 9 16 26842 26826 186.271 165.562 0.0082353 0.00536388 2024-03-12T03:44:04.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:04.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:04.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:04.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:04.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:04.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:04.655 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:04.689 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:04.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:04.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:04 smithi173 ceph-mon[33544]: 6.15 scrub starts 2024-03-12T03:44:04.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:04 smithi173 ceph-mon[33544]: 6.15 scrub ok 2024-03-12T03:44:04.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:04 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:44:04.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:04 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:44:04.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:04 smithi173 ceph-mon[33544]: osdmap e286: 8 total, 7 up, 8 in 2024-03-12T03:44:04.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:04 smithi173 ceph-mon[33544]: 5.6 deep-scrub starts 2024-03-12T03:44:04.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:04 smithi173 ceph-mon[33544]: 5.6 deep-scrub ok 2024-03-12T03:44:04.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:04 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:44:04.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:04 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:44:04.761 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[29344]: 6.15 scrub starts 2024-03-12T03:44:04.761 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[29344]: 6.15 scrub ok 2024-03-12T03:44:04.761 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:44:04.761 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:44:04.762 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[29344]: osdmap e286: 8 total, 7 up, 8 in 2024-03-12T03:44:04.762 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[29344]: 5.6 deep-scrub starts 2024-03-12T03:44:04.762 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[29344]: 5.6 deep-scrub ok 2024-03-12T03:44:04.762 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:44:04.762 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:44:04.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[39949]: 6.15 scrub starts 2024-03-12T03:44:04.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[39949]: 6.15 scrub ok 2024-03-12T03:44:04.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:44:04.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:44:04.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[39949]: osdmap e286: 8 total, 7 up, 8 in 2024-03-12T03:44:04.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[39949]: 5.6 deep-scrub starts 2024-03-12T03:44:04.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[39949]: 5.6 deep-scrub ok 2024-03-12T03:44:04.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:44:04.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:04 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:44:04.773 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:04.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:04.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:04.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:04.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:04.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:04.964 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:04.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:05.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:05.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:05.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:05.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:05.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:05.240 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:05.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:05.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:05.376 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:05.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:05.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 10 16 29419 29403 183.749 161.062 0.00348136 0.00543712 2024-03-12T03:44:05.478 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:05.530 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:05.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:05.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:05.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:05.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:05.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:05 smithi169 ceph-mon[29344]: 4.b scrub starts 2024-03-12T03:44:05.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:05 smithi169 ceph-mon[29344]: 4.b scrub ok 2024-03-12T03:44:05.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:05 smithi169 ceph-mon[29344]: pgmap v508: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 147 MiB/s wr, 2.35k op/s; 9796/58962 objects degraded (16.614%) 2024-03-12T03:44:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:05 smithi169 ceph-mon[29344]: osdmap e287: 8 total, 7 up, 8 in 2024-03-12T03:44:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:05 smithi169 ceph-mon[29344]: 4.8 scrub starts 2024-03-12T03:44:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:05 smithi169 ceph-mon[29344]: 4.8 scrub ok 2024-03-12T03:44:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:05 smithi169 ceph-mon[39949]: 4.b scrub starts 2024-03-12T03:44:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:05 smithi169 ceph-mon[39949]: 4.b scrub ok 2024-03-12T03:44:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:05 smithi169 ceph-mon[39949]: pgmap v508: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 147 MiB/s wr, 2.35k op/s; 9796/58962 objects degraded (16.614%) 2024-03-12T03:44:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:05 smithi169 ceph-mon[39949]: osdmap e287: 8 total, 7 up, 8 in 2024-03-12T03:44:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:05 smithi169 ceph-mon[39949]: 4.8 scrub starts 2024-03-12T03:44:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:05 smithi169 ceph-mon[39949]: 4.8 scrub ok 2024-03-12T03:44:05.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:05 smithi173 ceph-mon[33544]: 4.b scrub starts 2024-03-12T03:44:05.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:05 smithi173 ceph-mon[33544]: 4.b scrub ok 2024-03-12T03:44:05.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:05 smithi173 ceph-mon[33544]: pgmap v508: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 147 MiB/s wr, 2.35k op/s; 9796/58962 objects degraded (16.614%) 2024-03-12T03:44:05.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:05 smithi173 ceph-mon[33544]: osdmap e287: 8 total, 7 up, 8 in 2024-03-12T03:44:05.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:05 smithi173 ceph-mon[33544]: 4.8 scrub starts 2024-03-12T03:44:05.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:05 smithi173 ceph-mon[33544]: 4.8 scrub ok 2024-03-12T03:44:05.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:05.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:05.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:05.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:05.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:05.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:06.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:06.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:06.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:06.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:06.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:06.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:06.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:06.357 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:06.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:06.377 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 11 16 31797 31781 180.554 148.625 0.00496668 0.00553341 2024-03-12T03:44:06.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:06.482 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:06.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:06.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:06.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:06.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:06.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:06.723 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:06.759 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:06.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:06.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:06.895 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:06.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[29344]: 3.3 scrub starts 2024-03-12T03:44:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[29344]: 3.3 scrub ok 2024-03-12T03:44:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[29344]: 3.e scrub starts 2024-03-12T03:44:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[29344]: 3.e scrub ok 2024-03-12T03:44:06.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[29344]: 5.15 scrub starts 2024-03-12T03:44:06.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[29344]: 5.15 scrub ok 2024-03-12T03:44:06.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1297214490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:44:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[39949]: 3.3 scrub starts 2024-03-12T03:44:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[39949]: 3.3 scrub ok 2024-03-12T03:44:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[39949]: 3.e scrub starts 2024-03-12T03:44:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[39949]: 3.e scrub ok 2024-03-12T03:44:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[39949]: 5.15 scrub starts 2024-03-12T03:44:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[39949]: 5.15 scrub ok 2024-03-12T03:44:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:06 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1297214490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:44:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:06 smithi173 ceph-mon[33544]: 3.3 scrub starts 2024-03-12T03:44:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:06 smithi173 ceph-mon[33544]: 3.3 scrub ok 2024-03-12T03:44:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:06 smithi173 ceph-mon[33544]: 3.e scrub starts 2024-03-12T03:44:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:06 smithi173 ceph-mon[33544]: 3.e scrub ok 2024-03-12T03:44:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:06 smithi173 ceph-mon[33544]: 5.15 scrub starts 2024-03-12T03:44:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:06 smithi173 ceph-mon[33544]: 5.15 scrub ok 2024-03-12T03:44:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:06 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1297214490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:44:06.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:07.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:07.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:07.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:07.068 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:44:07.069 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:44:07.069 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:44:07.069 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:44:07.069 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:44:07.069 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:44:07.069 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:44:07.069 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:44:07.069 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:44:07.069 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:44:07.070 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:44:07.070 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:44:07.070 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:44:07.070 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:44:07.070 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:44:07.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:07.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:07.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:07.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:07.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:07.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:07.377 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 12 16 34041 34025 177.195 140.25 0.00818216 0.00563869 2024-03-12T03:44:07.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:07.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:07.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:07.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:07.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:07.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:07.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:07.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:07.725 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:07.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:07.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:07.751 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:07 smithi169 ceph-mon[29344]: 6.9 scrub starts 2024-03-12T03:44:07.751 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:07 smithi169 ceph-mon[29344]: 6.9 scrub ok 2024-03-12T03:44:07.751 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:07 smithi169 ceph-mon[29344]: pgmap v510: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 153 MiB/s wr, 2.44k op/s; 11340/68466 objects degraded (16.563%) 2024-03-12T03:44:07.751 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:07 smithi169 ceph-mon[29344]: 3.1c scrub starts 2024-03-12T03:44:07.751 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:07 smithi169 ceph-mon[29344]: 3.1c scrub ok 2024-03-12T03:44:07.752 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:07 smithi169 ceph-mon[39949]: 6.9 scrub starts 2024-03-12T03:44:07.752 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:07 smithi169 ceph-mon[39949]: 6.9 scrub ok 2024-03-12T03:44:07.752 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:07 smithi169 ceph-mon[39949]: pgmap v510: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 153 MiB/s wr, 2.44k op/s; 11340/68466 objects degraded (16.563%) 2024-03-12T03:44:07.752 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:07 smithi169 ceph-mon[39949]: 3.1c scrub starts 2024-03-12T03:44:07.752 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:07 smithi169 ceph-mon[39949]: 3.1c scrub ok 2024-03-12T03:44:07.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:07.866 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:07.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:07.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:07 smithi173 ceph-mon[33544]: 6.9 scrub starts 2024-03-12T03:44:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:07 smithi173 ceph-mon[33544]: 6.9 scrub ok 2024-03-12T03:44:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:07 smithi173 ceph-mon[33544]: pgmap v510: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 153 MiB/s wr, 2.44k op/s; 11340/68466 objects degraded (16.563%) 2024-03-12T03:44:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:07 smithi173 ceph-mon[33544]: 3.1c scrub starts 2024-03-12T03:44:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:07 smithi173 ceph-mon[33544]: 3.1c scrub ok 2024-03-12T03:44:08.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:08.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:08.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:08.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:08.142 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:08.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:08.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:08.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:08.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:08.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:08.377 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 13 16 36426 36410 175.03 149.062 0.00637177 0.00570825 2024-03-12T03:44:08.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:08.419 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:08.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:08.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:08.555 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:08.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:08.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:08.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:08.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:08.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:08.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:08.827 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:08.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:08 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 11340/68466 objects degraded (16.563%), 31 pgs degraded (PG_DEGRADED) 2024-03-12T03:44:08.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:08 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 11340/68466 objects degraded (16.563%), 31 pgs degraded (PG_DEGRADED) 2024-03-12T03:44:08.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:08.967 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:08.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:08.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:08 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 11340/68466 objects degraded (16.563%), 31 pgs degraded (PG_DEGRADED) 2024-03-12T03:44:09.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:09.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:09.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:09.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:09.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:09.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:09.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:09.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:09.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:09.377 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 14 16 38636 38620 172.393 138.125 0.00856785 0.00579542 2024-03-12T03:44:09.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:09.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:09.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:09.519 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:09.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:09.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:09.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:09.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:09.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:09.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:09.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:09.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:09.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:09.922 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:09.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:09.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:09 smithi169 ceph-mon[29344]: pgmap v511: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.9 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 217 MiB/s wr, 3.47k op/s; 15917/95838 objects degraded (16.608%) 2024-03-12T03:44:09.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:09 smithi169 ceph-mon[39949]: pgmap v511: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.9 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 217 MiB/s wr, 3.47k op/s; 15917/95838 objects degraded (16.608%) 2024-03-12T03:44:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:09 smithi173 ceph-mon[33544]: pgmap v511: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.9 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 217 MiB/s wr, 3.47k op/s; 15917/95838 objects degraded (16.608%) 2024-03-12T03:44:10.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:10.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:10.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:10.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:10.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:10.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:10.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:44:10.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:10.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:10.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:10.377 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 15 16 40208 40192 167.45 98.25 0.00589279 0.00596736 2024-03-12T03:44:10.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:10.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:10.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:10.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:10.614 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:10.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:10.716 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:10.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:10 smithi173 ceph-mon[33544]: 6.17 scrub starts 2024-03-12T03:44:10.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:10 smithi173 ceph-mon[33544]: 6.17 scrub ok 2024-03-12T03:44:10.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:10 smithi173 ceph-mon[33544]: 3.6 scrub starts 2024-03-12T03:44:10.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:10 smithi173 ceph-mon[33544]: 3.6 scrub ok 2024-03-12T03:44:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:10.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:10.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:10.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:10.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:10.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:10.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:10 smithi169 ceph-mon[39949]: 6.17 scrub starts 2024-03-12T03:44:10.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:10 smithi169 ceph-mon[39949]: 6.17 scrub ok 2024-03-12T03:44:10.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:10 smithi169 ceph-mon[39949]: 3.6 scrub starts 2024-03-12T03:44:10.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:10 smithi169 ceph-mon[39949]: 3.6 scrub ok 2024-03-12T03:44:10.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:10 smithi169 ceph-mon[29344]: 6.17 scrub starts 2024-03-12T03:44:10.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:10 smithi169 ceph-mon[29344]: 6.17 scrub ok 2024-03-12T03:44:10.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:10 smithi169 ceph-mon[29344]: 3.6 scrub starts 2024-03-12T03:44:10.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:10 smithi169 ceph-mon[29344]: 3.6 scrub ok 2024-03-12T03:44:10.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:11.015 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:11.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:11.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:11.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:11.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:11.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:11.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:11.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:11.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:11.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 16 16 42039 42023 164.136 114.438 0.00249441 0.00607484 2024-03-12T03:44:11.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:11.416 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:11.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:11.518 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:11.542 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:11.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:11.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:11.678 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:11.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:11.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:11 smithi173 ceph-mon[33544]: 6.e scrub starts 2024-03-12T03:44:11.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:11 smithi173 ceph-mon[33544]: 6.e scrub ok 2024-03-12T03:44:11.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:11 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:44:11.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:11 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:44:11.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:11 smithi173 ceph-mon[33544]: pgmap v512: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.9 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.88k op/s; 15917/95838 objects degraded (16.608%) 2024-03-12T03:44:11.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:11 smithi173 ceph-mon[33544]: 2.2 scrub starts 2024-03-12T03:44:11.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:11 smithi173 ceph-mon[33544]: 2.2 scrub ok 2024-03-12T03:44:11.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:11 smithi173 ceph-mon[33544]: osdmap e288: 8 total, 7 up, 8 in 2024-03-12T03:44:11.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:11 smithi173 ceph-mon[33544]: 6.18 scrub starts 2024-03-12T03:44:11.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:11 smithi173 ceph-mon[33544]: 6.18 scrub ok 2024-03-12T03:44:11.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:11.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:11.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[29344]: 6.e scrub starts 2024-03-12T03:44:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[29344]: 6.e scrub ok 2024-03-12T03:44:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:44:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:44:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[29344]: pgmap v512: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.9 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.88k op/s; 15917/95838 objects degraded (16.608%) 2024-03-12T03:44:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[29344]: 2.2 scrub starts 2024-03-12T03:44:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[29344]: 2.2 scrub ok 2024-03-12T03:44:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[29344]: osdmap e288: 8 total, 7 up, 8 in 2024-03-12T03:44:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[29344]: 6.18 scrub starts 2024-03-12T03:44:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[29344]: 6.18 scrub ok 2024-03-12T03:44:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[39949]: 6.e scrub starts 2024-03-12T03:44:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[39949]: 6.e scrub ok 2024-03-12T03:44:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:44:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:44:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[39949]: pgmap v512: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 1.9 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.88k op/s; 15917/95838 objects degraded (16.608%) 2024-03-12T03:44:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[39949]: 2.2 scrub starts 2024-03-12T03:44:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[39949]: 2.2 scrub ok 2024-03-12T03:44:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[39949]: osdmap e288: 8 total, 7 up, 8 in 2024-03-12T03:44:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[39949]: 6.18 scrub starts 2024-03-12T03:44:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-mon[39949]: 6.18 scrub ok 2024-03-12T03:44:11.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:11.953 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:11.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:12.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:12.091 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:12.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:12.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:44:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:44:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:44:12.192 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:12.230 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:12.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:12.331 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:12.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:12.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:12.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 17 16 43868 43852 161.204 114.312 0.00589275 0.00619864 2024-03-12T03:44:12.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:12.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:12.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:12.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:12.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:12.641 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:12.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:12.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:44:12.712 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:44:12.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:12.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:12.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:12.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:12 smithi173 ceph-mon[33544]: 2.0 deep-scrub starts 2024-03-12T03:44:12.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:12 smithi173 ceph-mon[33544]: 2.0 deep-scrub ok 2024-03-12T03:44:12.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:12 smithi173 ceph-mon[33544]: 5.b deep-scrub starts 2024-03-12T03:44:12.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:12 smithi173 ceph-mon[33544]: 5.7 scrub starts 2024-03-12T03:44:12.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:12 smithi173 ceph-mon[33544]: 5.b deep-scrub ok 2024-03-12T03:44:12.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:12 smithi173 ceph-mon[33544]: 5.7 scrub ok 2024-03-12T03:44:12.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:12 smithi169 ceph-mon[29344]: 2.0 deep-scrub starts 2024-03-12T03:44:12.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:12 smithi169 ceph-mon[29344]: 2.0 deep-scrub ok 2024-03-12T03:44:12.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:12 smithi169 ceph-mon[29344]: 5.b deep-scrub starts 2024-03-12T03:44:12.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:12 smithi169 ceph-mon[29344]: 5.7 scrub starts 2024-03-12T03:44:12.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:12 smithi169 ceph-mon[29344]: 5.b deep-scrub ok 2024-03-12T03:44:12.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:12 smithi169 ceph-mon[29344]: 5.7 scrub ok 2024-03-12T03:44:12.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:12 smithi169 ceph-mon[39949]: 2.0 deep-scrub starts 2024-03-12T03:44:12.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:12 smithi169 ceph-mon[39949]: 2.0 deep-scrub ok 2024-03-12T03:44:12.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:12 smithi169 ceph-mon[39949]: 5.b deep-scrub starts 2024-03-12T03:44:12.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:12 smithi169 ceph-mon[39949]: 5.7 scrub starts 2024-03-12T03:44:12.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:12 smithi169 ceph-mon[39949]: 5.b deep-scrub ok 2024-03-12T03:44:12.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:12 smithi169 ceph-mon[39949]: 5.7 scrub ok 2024-03-12T03:44:12.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:12.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:12.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:13.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:13.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:13.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:13.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:13.181 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:13.205 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:13.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:13.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:44:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:44:13.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:13.343 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:13.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:13.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:13.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 18 16 45943 45927 159.453 129.688 0.0242943 0.00626592 2024-03-12T03:44:13.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:13.480 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:13.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:13.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:13.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:13.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:13.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:13.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:13.754 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:13.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:13.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:13.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:13.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:13.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:13.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:13 smithi169 ceph-mon[29344]: pgmap v514: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 2.5 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.60k op/s; 19832/121326 objects degraded (16.346%) 2024-03-12T03:44:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:13 smithi169 ceph-mon[29344]: 6.a scrub starts 2024-03-12T03:44:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:13 smithi169 ceph-mon[29344]: 6.a scrub ok 2024-03-12T03:44:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:13 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:13 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 19832/121326 objects degraded (16.346%), 31 pgs degraded (PG_DEGRADED) 2024-03-12T03:44:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:13 smithi169 ceph-mon[29344]: osdmap e289: 8 total, 7 up, 8 in 2024-03-12T03:44:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:13 smithi169 ceph-mon[39949]: pgmap v514: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 2.5 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.60k op/s; 19832/121326 objects degraded (16.346%) 2024-03-12T03:44:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:13 smithi169 ceph-mon[39949]: 6.a scrub starts 2024-03-12T03:44:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:13 smithi169 ceph-mon[39949]: 6.a scrub ok 2024-03-12T03:44:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:13 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:13 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 19832/121326 objects degraded (16.346%), 31 pgs degraded (PG_DEGRADED) 2024-03-12T03:44:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:13 smithi169 ceph-mon[39949]: osdmap e289: 8 total, 7 up, 8 in 2024-03-12T03:44:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:13 smithi173 ceph-mon[33544]: pgmap v514: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 2.5 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.60k op/s; 19832/121326 objects degraded (16.346%) 2024-03-12T03:44:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:13 smithi173 ceph-mon[33544]: 6.a scrub starts 2024-03-12T03:44:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:13 smithi173 ceph-mon[33544]: 6.a scrub ok 2024-03-12T03:44:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:13 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:13 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 19832/121326 objects degraded (16.346%), 31 pgs degraded (PG_DEGRADED) 2024-03-12T03:44:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:13 smithi173 ceph-mon[33544]: osdmap e289: 8 total, 7 up, 8 in 2024-03-12T03:44:13.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:14.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:14.136 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:14.182 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:14.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:14.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:14.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:14.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:14.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:14.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 19 16 47767 47751 157.06 114 0.00371373 0.00636194 2024-03-12T03:44:14.431 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:14.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:14.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:14.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:14.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:14.632 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[29344]: 3.13 deep-scrub starts 2024-03-12T03:44:14.632 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[29344]: 3.13 deep-scrub ok 2024-03-12T03:44:14.632 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[29344]: 3.1d scrub starts 2024-03-12T03:44:14.632 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[29344]: 3.1d scrub ok 2024-03-12T03:44:14.632 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[29344]: 5.1e scrub starts 2024-03-12T03:44:14.632 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[29344]: 5.1e scrub ok 2024-03-12T03:44:14.632 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:44:14.633 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[39949]: 3.13 deep-scrub starts 2024-03-12T03:44:14.633 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[39949]: 3.13 deep-scrub ok 2024-03-12T03:44:14.633 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[39949]: 3.1d scrub starts 2024-03-12T03:44:14.633 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[39949]: 3.1d scrub ok 2024-03-12T03:44:14.633 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[39949]: 5.1e scrub starts 2024-03-12T03:44:14.633 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[39949]: 5.1e scrub ok 2024-03-12T03:44:14.633 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:14 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:44:14.713 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:44:14.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:14.754 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:14.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:14.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:14.855 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:44:14.855 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:44:14.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.855 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:48490 9.5 9:a1d1ddf3:::benchmark_data_smithi169_135229_object48489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-12T03:44:14.855 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:44:14.844873+0000", 2024-03-12T03:44:14.855 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0041378509999999997, 2024-03-12T03:44:14.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041815410000000004, 2024-03-12T03:44:14.855 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:44:14.855 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:44:14.855 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 48490 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.844873+0000", 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.844873+0000", 2024-03-12T03:44:14.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.844875+0000", 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9549999999999999e-06 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.844904+0000", 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9655e-05 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.844924+0000", 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9431000000000001e-05 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.844934+0000", 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0210999999999999e-05 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.845090+0000", 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015598899999999999 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.845255+0000", 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016470800000000001 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.845415+0000", 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016004300000000001 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:48492 9.b 9:d4c9a597:::benchmark_data_smithi169_135229_object48491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:44:14.844959+0000", 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0040516019999999996, 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041807279999999999, 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 48492 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:44:14.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.844959+0000", 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.844959+0000", 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.844959+0000", 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.84e-07 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:44:14.861 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.844979+0000", 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9395e-05 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.844986+0000", 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9009999999999997e-06 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.844992+0000", 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3829999999999997e-06 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.848291+0000", 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032991470000000001 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.848415+0000", 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000123819 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.848541+0000", 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000126124 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:48496 9.9 9:98948c6d:::benchmark_data_smithi169_135229_object48495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:44:14.846307+0000", 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0027040860000000001, 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002923765, 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 48496 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846307+0000", 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846307+0000", 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846308+0000", 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.248e-06 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846350+0000", 2024-03-12T03:44:14.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2432999999999998e-05 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846364+0000", 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3565e-05 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846373+0000", 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9560000000000003e-06 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846594+0000", 2024-03-12T03:44:14.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000220879 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846763+0000", 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016963300000000001 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846926+0000", 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016294700000000001 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.867 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:48502 9.5 9:a5e7d8aa:::benchmark_data_smithi169_135229_object48501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:44:14.846804+0000", 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0022069519999999999, 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025197879999999998, 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 48502 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846804+0000", 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:14.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846804+0000", 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846804+0000", 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4700000000000001e-07 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846824+0000", 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9799000000000001e-05 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846844+0000", 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9922e-05 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.846851+0000", 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2590000000000003e-06 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.847210+0000", 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000358263 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.870 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.847341+0000", 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000131106 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,7", 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:14.847487+0000", 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000145857 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 4 2024-03-12T03:44:14.871 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:44:14.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:14.924 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:14.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:14 smithi173 ceph-mon[33544]: 3.13 deep-scrub starts 2024-03-12T03:44:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:14 smithi173 ceph-mon[33544]: 3.13 deep-scrub ok 2024-03-12T03:44:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:14 smithi173 ceph-mon[33544]: 3.1d scrub starts 2024-03-12T03:44:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:14 smithi173 ceph-mon[33544]: 3.1d scrub ok 2024-03-12T03:44:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:14 smithi173 ceph-mon[33544]: 5.1e scrub starts 2024-03-12T03:44:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:14 smithi173 ceph-mon[33544]: 5.1e scrub ok 2024-03-12T03:44:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:14 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:44:15.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:15.076 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:15.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:15.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:15.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:15.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:15.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:15.355 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:15.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:15.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:44:15.376975+0000 min lat: 0.00120676 max lat: 0.160571 avg lat: 0.00647993 2024-03-12T03:44:15.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:44:15.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 20 15 49368 49353 154.213 100.125 0.00302832 0.00647993 2024-03-12T03:44:15.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:15.496 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:15.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:15.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:15.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:15.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:15.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: 4.d scrub starts 2024-03-12T03:44:15.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: 4.d scrub ok 2024-03-12T03:44:15.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: pgmap v516: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 152 MiB/s wr, 2.42k op/s; 20991/126675 objects degraded (16.571%) 2024-03-12T03:44:15.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: 6.3 scrub starts 2024-03-12T03:44:15.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: 6.3 scrub ok 2024-03-12T03:44:15.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: 4.12 scrub starts 2024-03-12T03:44:15.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: 4.12 scrub ok 2024-03-12T03:44:15.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: 3.1f deep-scrub starts 2024-03-12T03:44:15.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: 3.1f deep-scrub ok 2024-03-12T03:44:15.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:15.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:15.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:44:15.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:44:15.653 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:15.653 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/256605676' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-12T03:44:15.653 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-12T03:44:15.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: 4.d scrub starts 2024-03-12T03:44:15.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: 4.d scrub ok 2024-03-12T03:44:15.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: pgmap v516: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 152 MiB/s wr, 2.42k op/s; 20991/126675 objects degraded (16.571%) 2024-03-12T03:44:15.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: 6.3 scrub starts 2024-03-12T03:44:15.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: 6.3 scrub ok 2024-03-12T03:44:15.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: 4.12 scrub starts 2024-03-12T03:44:15.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: 4.12 scrub ok 2024-03-12T03:44:15.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: 3.1f deep-scrub starts 2024-03-12T03:44:15.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: 3.1f deep-scrub ok 2024-03-12T03:44:15.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:15.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:44:15.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:44:15.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:15.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/256605676' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-12T03:44:15.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:15 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-12T03:44:15.702 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-12T03:44:15.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:15.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:15.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:15.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:15.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:15.916 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:15.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:15.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: 4.d scrub starts 2024-03-12T03:44:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: 4.d scrub ok 2024-03-12T03:44:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: pgmap v516: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 152 MiB/s wr, 2.42k op/s; 20991/126675 objects degraded (16.571%) 2024-03-12T03:44:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: 6.3 scrub starts 2024-03-12T03:44:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: 6.3 scrub ok 2024-03-12T03:44:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: 4.12 scrub starts 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: 4.12 scrub ok 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: 3.1f deep-scrub starts 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: 3.1f deep-scrub ok 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:44:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:44:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:44:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/256605676' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-12T03:44:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:15 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-12T03:44:16.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:16.026 INFO:teuthology.orchestra.run.smithi169.stderr:set pool 1 pg_num to 21 2024-03-12T03:44:16.063 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:16.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:16.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:16.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:16.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:16.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:16.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:16.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:16.346 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:16.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:16.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 21 15 51159 51144 152.199 111.938 0.00715675 0.00656579 2024-03-12T03:44:16.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:16.487 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:16.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:16.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:16.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:16.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:16.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:16.726 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:16.763 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:16.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:16.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:16.901 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:16.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:16.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:16.910 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:16 smithi169 ceph-mon[29344]: Adjusting osd_memory_target on smithi173 to 3248M 2024-03-12T03:44:16.910 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:16 smithi169 ceph-mon[29344]: Adjusting osd_memory_target on smithi169 to 3248M 2024-03-12T03:44:16.910 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:16 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:16.910 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:16 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-12T03:44:16.911 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:16 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:16.911 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:16 smithi169 ceph-mon[29344]: osdmap e290: 8 total, 7 up, 8 in 2024-03-12T03:44:16.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:16 smithi169 ceph-mon[39949]: Adjusting osd_memory_target on smithi173 to 3248M 2024-03-12T03:44:16.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:16 smithi169 ceph-mon[39949]: Adjusting osd_memory_target on smithi169 to 3248M 2024-03-12T03:44:16.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:16 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:16.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:16 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-12T03:44:16.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:16 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:16.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:16 smithi169 ceph-mon[39949]: osdmap e290: 8 total, 7 up, 8 in 2024-03-12T03:44:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:16 smithi173 ceph-mon[33544]: Adjusting osd_memory_target on smithi173 to 3248M 2024-03-12T03:44:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:16 smithi173 ceph-mon[33544]: Adjusting osd_memory_target on smithi169 to 3248M 2024-03-12T03:44:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:16 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:16 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-12T03:44:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:16 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:16 smithi173 ceph-mon[33544]: osdmap e290: 8 total, 7 up, 8 in 2024-03-12T03:44:17.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:17.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:17.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:17.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:17.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:17.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:17.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:17.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:17.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:17.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:17.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:17.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 22 16 53025 53009 150.579 116.562 0.0039722 0.00663621 2024-03-12T03:44:17.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:17.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:17.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:17.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:17.486 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:17.547 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:17.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:17.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:17.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:17.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:17.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:17.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:17.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:17.811 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:44:17.811 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:44:17.811 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[29344]: pgmap v517: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 22130/133572 objects degraded (16.568%) 2024-03-12T03:44:17.812 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[29344]: 1.2 scrub starts 2024-03-12T03:44:17.812 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[29344]: 1.2 scrub ok 2024-03-12T03:44:17.812 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[29344]: 5.1b deep-scrub starts 2024-03-12T03:44:17.812 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[29344]: 5.1b deep-scrub ok 2024-03-12T03:44:17.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:44:17.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:44:17.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[39949]: pgmap v517: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 22130/133572 objects degraded (16.568%) 2024-03-12T03:44:17.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[39949]: 1.2 scrub starts 2024-03-12T03:44:17.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[39949]: 1.2 scrub ok 2024-03-12T03:44:17.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[39949]: 5.1b deep-scrub starts 2024-03-12T03:44:17.813 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:17 smithi169 ceph-mon[39949]: 5.1b deep-scrub ok 2024-03-12T03:44:17.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:17.872 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:17.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:17.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:17.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:17 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:44:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:17 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:44:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:17 smithi173 ceph-mon[33544]: pgmap v517: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 22130/133572 objects degraded (16.568%) 2024-03-12T03:44:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:17 smithi173 ceph-mon[33544]: 1.2 scrub starts 2024-03-12T03:44:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:17 smithi173 ceph-mon[33544]: 1.2 scrub ok 2024-03-12T03:44:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:17 smithi173 ceph-mon[33544]: 5.1b deep-scrub starts 2024-03-12T03:44:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:17 smithi173 ceph-mon[33544]: 5.1b deep-scrub ok 2024-03-12T03:44:17.997 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:17.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:17.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:18.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:18.135 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:18.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:18.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:18.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:18.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:18.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:18.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:44:18.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:18.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 23 16 54782 54766 148.806 109.812 0.0043349 0.0067138 2024-03-12T03:44:18.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:18.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:18.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:18.546 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:18.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:18.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: 5.e scrub starts 2024-03-12T03:44:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: 5.e scrub ok 2024-03-12T03:44:18.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: 5.8 scrub starts 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: 5.8 scrub ok 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3375278948' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 22130/133572 objects degraded (16.568%), 31 pgs degraded (PG_DEGRADED) 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[29344]: osdmap e291: 8 total, 7 up, 8 in 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: 5.e scrub starts 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: 5.e scrub ok 2024-03-12T03:44:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: 5.8 scrub starts 2024-03-12T03:44:18.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: 5.8 scrub ok 2024-03-12T03:44:18.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3375278948' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:44:18.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:44:18.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-12T03:44:18.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-12T03:44:18.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 22130/133572 objects degraded (16.568%), 31 pgs degraded (PG_DEGRADED) 2024-03-12T03:44:18.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:44:18.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:44:18.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-12T03:44:18.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:18 smithi169 ceph-mon[39949]: osdmap e291: 8 total, 7 up, 8 in 2024-03-12T03:44:18.679 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:18.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:18.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:18.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:18.813 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:18.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:18.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:18.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:18.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:18.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: 5.e scrub starts 2024-03-12T03:44:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: 5.e scrub ok 2024-03-12T03:44:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: 5.8 scrub starts 2024-03-12T03:44:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: 5.8 scrub ok 2024-03-12T03:44:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3375278948' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:44:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:44:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-12T03:44:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-12T03:44:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 22130/133572 objects degraded (16.568%), 31 pgs degraded (PG_DEGRADED) 2024-03-12T03:44:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:44:18.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:44:18.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-12T03:44:18.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:18 smithi173 ceph-mon[33544]: osdmap e291: 8 total, 7 up, 8 in 2024-03-12T03:44:19.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:19.091 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:19.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:19.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:19.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:19.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:19.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:19.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:19.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:19.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:19.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 24 16 56526 56510 147.147 109 0.0196278 0.00679041 2024-03-12T03:44:19.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:19.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:19.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:19.611 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:19.646 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:19.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:19.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:19 smithi169 ceph-mon[29344]: pgmap v519: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 163 MiB/s wr, 2.60k op/s; 25558/154260 objects degraded (16.568%) 2024-03-12T03:44:19.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:19 smithi169 ceph-mon[29344]: osdmap e292: 8 total, 7 up, 8 in 2024-03-12T03:44:19.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:19 smithi169 ceph-mon[39949]: pgmap v519: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 163 MiB/s wr, 2.60k op/s; 25558/154260 objects degraded (16.568%) 2024-03-12T03:44:19.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:19 smithi169 ceph-mon[39949]: osdmap e292: 8 total, 7 up, 8 in 2024-03-12T03:44:19.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:19.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:19.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:19.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:19.921 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:19.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:19.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:19.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:19 smithi173 ceph-mon[33544]: pgmap v519: 158 pgs: 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 163 MiB/s wr, 2.60k op/s; 25558/154260 objects degraded (16.568%) 2024-03-12T03:44:19.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:19 smithi173 ceph-mon[33544]: osdmap e292: 8 total, 7 up, 8 in 2024-03-12T03:44:20.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:20.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:20.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:20.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:20.198 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:20.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:20.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:20.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:20.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:20.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 25 15 58316 58301 145.738 111.938 0.00472805 0.00685683 2024-03-12T03:44:20.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:20.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:20.480 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:20.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:20.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:20.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:20.616 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:20.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:20.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:20.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:20.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:20.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:20.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:20.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:20 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-12T03:44:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:20 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-12T03:44:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:20 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-12T03:44:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:20 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-12T03:44:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:20 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-12T03:44:20.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:20 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-12T03:44:20.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:21.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:21.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:21.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:21.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:21.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:21.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:21.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:21.304 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:21.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:21.378 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 26 16 60068 60052 144.342 109.438 0.00630187 0.00692317 2024-03-12T03:44:21.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:21.443 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:21.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:21.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:21.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:21.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:21.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:21.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:21.722 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:21.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:21.824 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:21.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:21 smithi169 ceph-mon[29344]: pgmap v522: 168 pgs: 10 unknown, 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 96 MiB/s wr, 1.53k op/s; 25558/154260 objects degraded (16.568%) 2024-03-12T03:44:21.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:21 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-12T03:44:21.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:21 smithi169 ceph-mon[29344]: osdmap e293: 8 total, 7 up, 8 in 2024-03-12T03:44:21.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:21 smithi169 ceph-mon[39949]: pgmap v522: 168 pgs: 10 unknown, 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 96 MiB/s wr, 1.53k op/s; 25558/154260 objects degraded (16.568%) 2024-03-12T03:44:21.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:21 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-12T03:44:21.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:21 smithi169 ceph-mon[39949]: osdmap e293: 8 total, 7 up, 8 in 2024-03-12T03:44:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:21.861 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:21.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:21.960 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-12T03:44:21.960 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:44:21.960 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-12T03:44:21.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-12T03:44:21.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:21 smithi173 ceph-mon[33544]: pgmap v522: 168 pgs: 10 unknown, 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 96 MiB/s wr, 1.53k op/s; 25558/154260 objects degraded (16.568%) 2024-03-12T03:44:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:21 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-12T03:44:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:21 smithi173 ceph-mon[33544]: osdmap e293: 8 total, 7 up, 8 in 2024-03-12T03:44:22.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:22.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:22.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:22.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:22.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:22.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:22.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:44:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:44:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:44:22.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:22.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:22.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:22.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:22.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 27 16 61743 61727 142.873 104.688 0.0150716 0.00699381 2024-03-12T03:44:22.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:22.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:22.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:22.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:22.555 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:22.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:22.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:22.657 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:22.699 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:22.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:22.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:22.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:22.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:22.842 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:22 smithi169 ceph-mon[29344]: osdmap e294: 8 total, 7 up, 8 in 2024-03-12T03:44:22.842 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:22 smithi169 ceph-mon[29344]: osdmap e295: 8 total, 7 up, 8 in 2024-03-12T03:44:22.847 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:22 smithi169 ceph-mon[39949]: osdmap e294: 8 total, 7 up, 8 in 2024-03-12T03:44:22.847 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:22 smithi169 ceph-mon[39949]: osdmap e295: 8 total, 7 up, 8 in 2024-03-12T03:44:22.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:22.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:22.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:22.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:22.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:44:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:44:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:22 smithi173 ceph-mon[33544]: osdmap e294: 8 total, 7 up, 8 in 2024-03-12T03:44:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:22 smithi173 ceph-mon[33544]: osdmap e295: 8 total, 7 up, 8 in 2024-03-12T03:44:23.041 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:44:23.041 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:44:23.072 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:44:23.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:23.118 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:44:23.119 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:44:23.119 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:44:23.119 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:44:23.143 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:23.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:23.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:23.147 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:44:23.189 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:44:23.227 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:44:23.228 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:44:23.228 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:44:23.228 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:44:23.228 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:44:23.228 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:44:23.228 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:44:23.228 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:44:23.228 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:44:23.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:23.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:23.284 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:23.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:23.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 28 16 63213 63197 141.051 91.875 0.026381 0.007083 2024-03-12T03:44:23.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:23.423 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:23.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:23.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:23.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:23.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:23.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:23.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:23.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:23.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:23.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:23.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:23.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:23.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:23.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:23 smithi169 ceph-mon[29344]: pgmap v525: 168 pgs: 4 remapped+peering, 6 unknown, 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 10 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 28938/175896 objects degraded (16.452%); 52 KiB/s, 0 objects/s recovering 2024-03-12T03:44:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:23 smithi169 ceph-mon[29344]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-03-12T03:44:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:23 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:23 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/396526235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:44:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:23 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 28938/175896 objects degraded (16.452%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:23.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:23 smithi169 ceph-mon[29344]: osdmap e296: 8 total, 7 up, 8 in 2024-03-12T03:44:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:23 smithi169 ceph-mon[39949]: pgmap v525: 168 pgs: 4 remapped+peering, 6 unknown, 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 10 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 28938/175896 objects degraded (16.452%); 52 KiB/s, 0 objects/s recovering 2024-03-12T03:44:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:23 smithi169 ceph-mon[39949]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-03-12T03:44:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:23 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:23 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/396526235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:44:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:23 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 28938/175896 objects degraded (16.452%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:23 smithi169 ceph-mon[39949]: osdmap e296: 8 total, 7 up, 8 in 2024-03-12T03:44:23.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:23.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:23.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:23.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:23 smithi173 ceph-mon[33544]: pgmap v525: 168 pgs: 4 remapped+peering, 6 unknown, 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 10 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 28938/175896 objects degraded (16.452%); 52 KiB/s, 0 objects/s recovering 2024-03-12T03:44:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:23 smithi173 ceph-mon[33544]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-03-12T03:44:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:23 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:23 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/396526235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:44:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:23 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 28938/175896 objects degraded (16.452%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:23 smithi173 ceph-mon[33544]: osdmap e296: 8 total, 7 up, 8 in 2024-03-12T03:44:23.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-12T03:44:24.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:24.112 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:24.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:24.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:24.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:24.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:24.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:24.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 29 16 64842 64826 139.698 101.812 0.023268 0.00715252 2024-03-12T03:44:24.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:24.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:24.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:24.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:24.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:24.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:24.627 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:24.654 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:24.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:24.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:24.792 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:24.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:24.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:24.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:24.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:24.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:25.031 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:25.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:25.055 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:25.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:25.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:25.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:25.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:25.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:25.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:25.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:25.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:25.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:44:25.337 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:44:25.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 30 16 66266 66250 138.008 89 0.00460671 0.00724057 2024-03-12T03:44:25.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:25.453 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:25.456 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:25.462 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:25.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:25.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:25.569 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:25.602 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:25 smithi169 ceph-mon[39949]: pgmap v528: 168 pgs: 6 remapped+peering, 4 unknown, 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 10 KiB/s rd, 139 MiB/s wr, 2.23k op/s; 30020/180873 objects degraded (16.597%); 52 KiB/s, 0 objects/s recovering 2024-03-12T03:44:25.602 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:25 smithi169 ceph-mon[39949]: osdmap e297: 8 total, 7 up, 8 in 2024-03-12T03:44:25.603 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:25 smithi169 ceph-mon[29344]: pgmap v528: 168 pgs: 6 remapped+peering, 4 unknown, 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 10 KiB/s rd, 139 MiB/s wr, 2.23k op/s; 30020/180873 objects degraded (16.597%); 52 KiB/s, 0 objects/s recovering 2024-03-12T03:44:25.603 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:25 smithi169 ceph-mon[29344]: osdmap e297: 8 total, 7 up, 8 in 2024-03-12T03:44:25.604 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:25.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:25.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:25 smithi173 ceph-mon[33544]: pgmap v528: 168 pgs: 6 remapped+peering, 4 unknown, 31 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 10 KiB/s rd, 139 MiB/s wr, 2.23k op/s; 30020/180873 objects degraded (16.597%); 52 KiB/s, 0 objects/s recovering 2024-03-12T03:44:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:25 smithi173 ceph-mon[33544]: osdmap e297: 8 total, 7 up, 8 in 2024-03-12T03:44:25.740 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:25.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:25.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:25.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:25.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:25.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:26.018 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:26.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:26.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:26.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:26.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:26.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:26.256 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:26.294 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:26.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:26.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:26.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 31 16 67879 67863 136.808 100.812 0.00698759 0.00730541 2024-03-12T03:44:26.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:26.434 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:26.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:26.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:26.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:26.569 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:26.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:26.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:26.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:26.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:26.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:26.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:26.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:26.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:26.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:26.991 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:26.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:27.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:27.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:27.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:27.230 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:27.269 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:27.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:27.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:44:27.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:27.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 32 16 69392 69376 135.487 94.5625 0.00281005 0.00736829 2024-03-12T03:44:27.415 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:27.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:27.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:27.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:27.568 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:27.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:27.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:27.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:27 smithi169 ceph-mon[29344]: pgmap v530: 168 pgs: 5 remapped+peering, 1 active+clean+remapped, 3 unknown, 31 active+undersized+degraded, 38 active+undersized, 90 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 14 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 30957/186717 objects degraded (16.580%); 71 KiB/s, 0 objects/s recovering 2024-03-12T03:44:27.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:27 smithi169 ceph-mon[39949]: pgmap v530: 168 pgs: 5 remapped+peering, 1 active+clean+remapped, 3 unknown, 31 active+undersized+degraded, 38 active+undersized, 90 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 14 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 30957/186717 objects degraded (16.580%); 71 KiB/s, 0 objects/s recovering 2024-03-12T03:44:27.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:27.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:27.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:27 smithi173 ceph-mon[33544]: pgmap v530: 168 pgs: 5 remapped+peering, 1 active+clean+remapped, 3 unknown, 31 active+undersized+degraded, 38 active+undersized, 90 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 14 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 30957/186717 objects degraded (16.580%); 71 KiB/s, 0 objects/s recovering 2024-03-12T03:44:27.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:27.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:27.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:27.947 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:27.987 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:27.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:27.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:28.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:28.124 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:28.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:28.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:28.261 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:28.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:28.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:28.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:28.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 33 16 70848 70832 134.139 91 0.0077277 0.00744968 2024-03-12T03:44:28.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:28.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:28.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:28.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:28.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:28.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:28.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:28.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:28 smithi169 ceph-mon[29344]: Health check update: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-12T03:44:28.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:28 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 30957/186717 objects degraded (16.580%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:28.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:28 smithi169 ceph-mon[39949]: Health check update: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-12T03:44:28.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:28 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 30957/186717 objects degraded (16.580%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:28.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:28.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:28 smithi173 ceph-mon[33544]: Health check update: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-12T03:44:28.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:28 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 30957/186717 objects degraded (16.580%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:28.786 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:29.076 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:29.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:29.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:29.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:29.209 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:29.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:29.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:29.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:29.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:29.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 34 16 72402 72386 133.05 97.125 0.0101448 0.00751121 2024-03-12T03:44:29.408 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:44:29.408 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:44:29.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:29.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:29.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:29.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:29.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:29.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:29.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:29 smithi173 ceph-mon[33544]: pgmap v531: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 3.8 KiB/s rd, 99 MiB/s wr, 1.58k op/s; 33939/204369 objects degraded (16.607%); 1/204369 objects misplaced (0.000%); 17 KiB/s, 0 objects/s recovering 2024-03-12T03:44:29.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:29 smithi173 ceph-mon[33544]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-12T03:44:29.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:29 smithi169 ceph-mon[29344]: pgmap v531: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 3.8 KiB/s rd, 99 MiB/s wr, 1.58k op/s; 33939/204369 objects degraded (16.607%); 1/204369 objects misplaced (0.000%); 17 KiB/s, 0 objects/s recovering 2024-03-12T03:44:29.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:29 smithi169 ceph-mon[29344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-12T03:44:29.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:29 smithi169 ceph-mon[39949]: pgmap v531: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 3.8 KiB/s rd, 99 MiB/s wr, 1.58k op/s; 33939/204369 objects degraded (16.607%); 1/204369 objects misplaced (0.000%); 17 KiB/s, 0 objects/s recovering 2024-03-12T03:44:29.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:29 smithi169 ceph-mon[39949]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-12T03:44:29.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:29.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:29.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:29.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:29.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:29.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:29.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:30.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:30.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:30.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:30.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:30.144 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:30.177 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:30.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:30.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:30.310 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:30.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:30.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:30.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 35 16 73840 73824 131.816 89.875 0.00369097 0.00758147 2024-03-12T03:44:30.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:30.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:30.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:30.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:30.585 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:30.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:30.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:30.687 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:30.711 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:30.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:30.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:30.846 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:30.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:30.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:30.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:30.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:31.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:31.118 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:31.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:31.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:31.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-12T03:44:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:31.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:31.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:31.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:31.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 36 16 75028 75012 130.217 74.25 0.0185721 0.0076738 2024-03-12T03:44:31.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:31.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:31.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:31.485 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:31 smithi169 ceph-mon[29344]: pgmap v532: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 3.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 33939/204369 objects degraded (16.607%); 1/204369 objects misplaced (0.000%); 14 KiB/s, 0 objects/s recovering 2024-03-12T03:44:31.485 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:31 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:31.485 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:31.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:31 smithi169 ceph-mon[39949]: pgmap v532: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 3.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 33939/204369 objects degraded (16.607%); 1/204369 objects misplaced (0.000%); 14 KiB/s, 0 objects/s recovering 2024-03-12T03:44:31.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:31 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:31.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:31.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:31.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:31.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:31.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:31.627 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:31 smithi173 ceph-mon[33544]: pgmap v532: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 3.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 33939/204369 objects degraded (16.607%); 1/204369 objects misplaced (0.000%); 14 KiB/s, 0 objects/s recovering 2024-03-12T03:44:31.627 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:31 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:44:31.627 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:31.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:31.666 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:31.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:31.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:31.800 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:31.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:31.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:31.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:31.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:32.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:32.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:32.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:32.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:32.088 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:44:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:44:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:44:32.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:32.222 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:32.223 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:32.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:32.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:32.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:32.362 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:32.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:32.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:32.380 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 37 16 76467 76451 129.128 89.9375 0.00884348 0.00773942 2024-03-12T03:44:32.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:32.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:32.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:32.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:32.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:32.632 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:44:32.632 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:44:32.632 INFO:teuthology.orchestra.run.smithi169.stdout: "epoch": 297, 2024-03-12T03:44:32.632 INFO:teuthology.orchestra.run.smithi169.stdout: "fsid": "677d624a-e020-11ee-95c8-87774f69a715", 2024-03-12T03:44:32.632 INFO:teuthology.orchestra.run.smithi169.stdout: "created": "2024-03-12T03:27:53.319986+0000", 2024-03-12T03:44:32.632 INFO:teuthology.orchestra.run.smithi169.stdout: "modified": "2024-03-12T03:44:24.349279+0000", 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "last_up_change": "2024-03-12T03:43:18.160417+0000", 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "last_in_change": "2024-03-12T03:39:32.320120+0000", 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_num": 5801984, 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_set": [ 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "noscrub", 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "pglog_hardlimit", 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "purged_snapdirs", 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_deletes", 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "sortbitwise" 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_version": 19, 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "full_ratio": 0.94999998807907104, 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-12T03:44:32.633 INFO:teuthology.orchestra.run.smithi169.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_snapshot": "", 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_max": 9, 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "max_osd": 8, 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "require_min_compat_client": "luminous", 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "min_compat_client": "luminous", 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "require_osd_release": "squid", 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "allow_crimson": false, 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "pools": [ 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 1, 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": ".mgr", 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:31:04.863493+0000", 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:44:32.634 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 21, 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 21, 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 21, 2024-03-12T03:44:32.635 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 21, 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 21, 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "293", 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "291", 2024-03-12T03:44:32.636 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:44:32.637 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:44:32.638 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "options": { 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_max": 32, 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_min": 1 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr": {} 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 2.2699999809265137, 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 2.2699999809265137, 2024-03-12T03:44:32.639 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 0.87999999523162842, 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 2, 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 2, 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 1, 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 1 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 2, 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "datapool", 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:18.983144+0000", 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 8193, 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:44:32.640 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 3, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 3, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 3, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 3, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 3, 2024-03-12T03:44:32.641 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "70", 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:44:32.642 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 2, 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 70, 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:44:32.643 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:44:32.644 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd": {} 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 2.6700000762939453, 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 2.6700000762939453, 2024-03-12T03:44:32.645 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 0.75, 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 2, 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 2, 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 1, 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 1 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 3, 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": ".rgw.root", 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:19.343301+0000", 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:44:32.646 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 32, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 32, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 32, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 32, 2024-03-12T03:44:32.647 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 32, 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "68", 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:44:32.648 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:44:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:44:32.650 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw": {} 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 1.9900000095367432, 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 1.9900000095367432, 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 0.87999999523162842, 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.75, 2024-03-12T03:44:32.651 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.75, 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 1, 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 1 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 4, 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "default.rgw.log", 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:21.864621+0000", 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:44:32.652 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 32, 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 32, 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 32, 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 32, 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 32, 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:44:32.653 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "70", 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:44:32.654 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:44:32.655 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:44:32.656 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw": {} 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 2.2400000095367432, 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 2.2400000095367432, 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 0.87999999523162842, 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-12T03:44:32.657 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 1, 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 1 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 5, 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "default.rgw.control", 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:23.416265+0000", 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:44:32.658 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 32, 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 32, 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 32, 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 32, 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 32, 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:44:32.659 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "72", 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:44:32.660 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:44:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:44:32.662 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw": {} 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 1.4900000095367432, 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 1.4900000095367432, 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 0.87999999523162842, 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.309999942779541, 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.309999942779541, 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 1, 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:44:32.663 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 1 2024-03-12T03:44:32.664 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.664 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 6, 2024-03-12T03:44:32.665 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "default.rgw.meta", 2024-03-12T03:44:32.665 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:25.476511+0000", 2024-03-12T03:44:32.665 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:44:32.665 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:44:32.665 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:44:32.665 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:44:32.665 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:44:32.665 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:44:32.665 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:44:32.665 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:44:32.666 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:44:32.666 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:44:32.666 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:44:32.666 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:44:32.666 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 32, 2024-03-12T03:44:32.666 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 32, 2024-03-12T03:44:32.666 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 32, 2024-03-12T03:44:32.666 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 32, 2024-03-12T03:44:32.666 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 32, 2024-03-12T03:44:32.666 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:44:32.666 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:44:32.666 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "74", 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:44:32.667 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:44:32.668 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:44:32.669 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: "options": { 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_bias": 4 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw": {} 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 1.9900000095367432, 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 1.9900000095367432, 2024-03-12T03:44:32.670 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 0.87999999523162842, 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.75, 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.75, 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 1, 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 1 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 9, 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "unique_pool_2", 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:43:44.570104+0000", 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:44:32.671 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 16, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 16, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 16, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 16, 2024-03-12T03:44:32.672 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 16, 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "283", 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:44:32.673 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:44:32.674 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:44:32.675 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "rados": {} 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 1.9900000095367432, 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 1.9900000095367432, 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 0.87999999523162842, 2024-03-12T03:44:32.676 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.75, 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.75, 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 1, 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 1 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 0, 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "c44466b0-3112-44dc-b5f3-d6494b958761", 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:44:32.677 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 8, 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 296, 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6801", 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 676779916 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.678 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6802", 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 676779916 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.679 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6804", 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 676779916 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6803", 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 676779916 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.169:6801/676779916", 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.169:6802/676779916", 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.169:6804/676779916", 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.169:6803/676779916", 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 1, 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "184c5c97-a992-4be9-bcd3-c08cb58a0e56", 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:44:32.681 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 13, 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 296, 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6805", 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 130042595 2024-03-12T03:44:32.682 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6806", 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 130042595 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6808", 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 130042595 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6807", 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 130042595 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.169:6805/130042595", 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.169:6806/130042595", 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.169:6808/130042595", 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.169:6807/130042595", 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 2, 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "beb15e06-1ca4-46a7-b640-efd1991367e8", 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:44:32.685 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 18, 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 296, 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6809", 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2751816702 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:44:32.686 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6810", 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2751816702 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6812", 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2751816702 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.687 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6811", 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2751816702 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.169:6809/2751816702", 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.169:6810/2751816702", 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.169:6812/2751816702", 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.169:6811/2751816702", 2024-03-12T03:44:32.688 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 3, 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "a00b4c50-e064-4c9e-85dc-b0a75ea5bf37", 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 0, 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 27, 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 202, 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 222, 2024-03-12T03:44:32.689 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 261, 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 265, 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6813", 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 287982149 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.690 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6814", 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 287982149 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6816", 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 287982149 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:44:32.691 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6815", 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 287982149 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.169:6813/287982149", 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.169:6814/287982149", 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.169:6816/287982149", 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.169:6815/287982149", 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: "exists" 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 4, 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "3a6121ef-6309-41b7-a964-6772c010b3a0", 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 32, 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 296, 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:44:32.693 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6800", 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1397212305 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6801", 2024-03-12T03:44:32.694 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1397212305 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6803", 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1397212305 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6802", 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1397212305 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.173:6800/1397212305", 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.173:6801/1397212305", 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.173:6803/1397212305", 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.173:6802/1397212305", 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:44:32.696 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 5, 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "3c2f6dd9-9784-4a07-8710-55ac3de82964", 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 41, 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 296, 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:44:32.697 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6804", 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3525888417 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.698 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6805", 2024-03-12T03:44:32.699 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3525888417 2024-03-12T03:44:32.699 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.699 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.699 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:44:32.699 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.699 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.699 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6807", 2024-03-12T03:44:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:32.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:32.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:32.700 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3525888417 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6806", 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3525888417 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.173:6804/3525888417", 2024-03-12T03:44:32.701 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.173:6805/3525888417", 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.173:6807/3525888417", 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.173:6806/3525888417", 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 6, 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "d7759df7-1978-4d70-94a2-721540804f59", 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:44:32.702 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 51, 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 296, 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6808", 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2147192629 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6809", 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2147192629 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.704 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6811", 2024-03-12T03:44:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2147192629 2024-03-12T03:44:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:44:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6810", 2024-03-12T03:44:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2147192629 2024-03-12T03:44:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.705 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.173:6808/2147192629", 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.173:6809/2147192629", 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.173:6811/2147192629", 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.173:6810/2147192629", 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 7, 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "8843039e-2b17-4230-9cca-5788a2294d8c", 2024-03-12T03:44:32.706 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:44:32.707 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:44:32.707 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:44:32.707 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:44:32.707 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:44:32.707 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:44:32.707 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 59, 2024-03-12T03:44:32.707 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 296, 2024-03-12T03:44:32.707 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:44:32.707 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:44:32.707 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:44:32.707 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6812", 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 4014976876 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6813", 2024-03-12T03:44:32.708 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 4014976876 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6815", 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 4014976876 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:44:32.709 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6814", 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 4014976876 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.173:6812/4014976876", 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.173:6813/4014976876", 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.173:6815/4014976876", 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.173:6814/4014976876", 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:44:32.710 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_xinfo": [ 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 0, 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:43:40.020140+0000", 2024-03-12T03:44:32.711 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 1, 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:43:40.631033+0000", 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 2, 2024-03-12T03:44:32.712 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:43:52.325920+0000", 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 3, 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:44:32.713 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:43:12.195881+0000", 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 264 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 4, 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:44:09.972166+0000", 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 5, 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:44:32.714 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:43:43.979911+0000", 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 6, 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:44:03.934937+0000", 2024-03-12T03:44:32.715 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 7, 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:44:15.955403+0000", 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_upmap": [], 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_upmap_items": [ 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.716 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "4.4", 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: "mappings": [ 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: "from": 1, 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: "to": 5 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "4.12", 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: "mappings": [ 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: "from": 1, 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: "to": 5 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "4.14", 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: "mappings": [ 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: "from": 1, 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: "to": 2 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_upmap_primaries": [], 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_temp": [ 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.718 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "1.c", 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: 0, 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: 6, 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: 5 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "1.e", 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: 4, 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: 7, 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: 0 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.719 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "1.11", 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: 0, 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: 5, 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: 2 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "1.13", 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: 1, 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: 4, 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: 6 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_temp": [], 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: "blocklist": { 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/4041747161": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:6800/2779395456": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/555866474": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/342292643": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/197210445": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/1430250501": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/3918553041": "2024-03-13T03:28:48.518732+0000", 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/1020514981": "2024-03-13T03:28:21.868923+0000", 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/284191239": "2024-03-13T03:28:21.868923+0000", 2024-03-12T03:44:32.721 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/1212459137": "2024-03-13T03:28:48.518732+0000", 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:6800/204892396": "2024-03-13T03:28:21.868923+0000", 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/2751007072": "2024-03-13T03:28:48.518732+0000", 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/2349376510": "2024-03-13T03:28:21.868923+0000", 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:6800/108891420": "2024-03-13T03:28:48.518732+0000" 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: "range_blocklist": {}, 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profiles": { 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: "default": { 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: "crush-failure-domain": "osd", 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: "k": "2", 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: "m": "1", 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: "plugin": "jerasure", 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: "technique": "reed_sol_van" 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps_queue": [], 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout: "new_removed_snaps": [], 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout: "new_purged_snaps": [], 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_node_flags": {}, 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout: "device_class_flags": {}, 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout: "stretch_mode": { 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout: "stretch_mode_enabled": false, 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout: "stretch_bucket_count": 0, 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout: "degraded_stretch_mode": 0, 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout: "recovering_stretch_mode": 0, 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout: "stretch_mode_bucket": 0 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:32.723 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:44:32.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:32.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:32.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:32.940 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:32.980 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:32.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:32.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:33.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:33.118 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:33.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:33.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:33.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:33.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:44:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:44:33.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:33.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:33.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:33.379 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 38 16 77955 77939 128.177 93 0.0260621 0.00779543 2024-03-12T03:44:33.397 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:33.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:33.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:33.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:33.535 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:33.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:33.636 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:33.668 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:33.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:33.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:33 smithi169 ceph-mon[29344]: pgmap v533: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 3.2 KiB/s rd, 98 MiB/s wr, 1.58k op/s; 36627/221748 objects degraded (16.517%); 1/221748 objects misplaced (0.000%); 12 KiB/s, 0 objects/s recovering 2024-03-12T03:44:33.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:33 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2513062743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-12T03:44:33.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:33 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 36627/221748 objects degraded (16.517%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:33.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:33 smithi169 ceph-mon[39949]: pgmap v533: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 3.2 KiB/s rd, 98 MiB/s wr, 1.58k op/s; 36627/221748 objects degraded (16.517%); 1/221748 objects misplaced (0.000%); 12 KiB/s, 0 objects/s recovering 2024-03-12T03:44:33.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:33 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2513062743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-12T03:44:33.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:33 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 36627/221748 objects degraded (16.517%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:33.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:33 smithi173 ceph-mon[33544]: pgmap v533: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 3.2 KiB/s rd, 98 MiB/s wr, 1.58k op/s; 36627/221748 objects degraded (16.517%); 1/221748 objects misplaced (0.000%); 12 KiB/s, 0 objects/s recovering 2024-03-12T03:44:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:33 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2513062743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-12T03:44:33.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:33 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 36627/221748 objects degraded (16.517%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:33.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:33.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:33.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:33.909 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:33.944 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:33.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:34.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:34.080 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:34.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:34.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:34.205 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:34.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:34.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:34.342 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:34.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:34.380 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 39 15 79391 79376 127.193 89.8125 0.0112361 0.00785769 2024-03-12T03:44:34.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:34.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:34.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:34.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:34.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:34.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:34.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:34.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:34.730 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:34.762 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:34.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:34.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:34.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:34.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:34.889 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:34.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:34.990 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:35.019 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:35.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:35.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:35.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:35.154 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:35.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:35.255 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:35.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:35.291 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:35.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:35.380 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:44:35.378684+0000 min lat: 0.00120676 max lat: 0.160571 avg lat: 0.00792562 2024-03-12T03:44:35.380 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:44:35.380 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 40 16 80688 80672 126.038 81 0.0059731 0.00792562 2024-03-12T03:44:35.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:35.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:35 smithi169 ceph-mon[29344]: pgmap v534: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 2.8 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 37555/225888 objects degraded (16.625%); 1/225888 objects misplaced (0.000%); 10 KiB/s, 0 objects/s recovering 2024-03-12T03:44:35.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:35 smithi169 ceph-mon[39949]: pgmap v534: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 2.8 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 37555/225888 objects degraded (16.625%); 1/225888 objects misplaced (0.000%); 10 KiB/s, 0 objects/s recovering 2024-03-12T03:44:35.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:35.729 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:35.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:35 smithi173 ceph-mon[33544]: pgmap v534: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 2.8 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 37555/225888 objects degraded (16.625%); 1/225888 objects misplaced (0.000%); 10 KiB/s, 0 objects/s recovering 2024-03-12T03:44:35.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:35.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:35.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:35.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:36.004 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:36.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:36.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:36.142 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:36.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:36.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:36.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:36.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:36.380 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 41 15 82073 82058 125.077 86.625 0.00466153 0.00799093 2024-03-12T03:44:36.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:36.419 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:36.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:36.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:36.549 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:36.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:36.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:36.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:36.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:36.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:36.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:36.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:36.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:36.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:36.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:36.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:36.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:37.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:37.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:37.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:37.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:37.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:37.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:37.356 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:37.381 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 42 16 83347 83331 123.99 79.5625 0.0137772 0.00806055 2024-03-12T03:44:37.392 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:37.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:37.424 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:44:37.494 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:37.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:37.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:37.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:37.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:37.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:37.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:37.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:37.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:37 smithi169 ceph-mon[29344]: pgmap v535: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 2.4 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 37966/227121 objects degraded (16.716%); 1/227121 objects misplaced (0.000%); 9.0 KiB/s, 0 objects/s recovering 2024-03-12T03:44:37.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:37 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1378476019' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:44:37.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:37 smithi169 ceph-mon[39949]: pgmap v535: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 2.4 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 37966/227121 objects degraded (16.716%); 1/227121 objects misplaced (0.000%); 9.0 KiB/s, 0 objects/s recovering 2024-03-12T03:44:37.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:37 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1378476019' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:44:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:37 smithi173 ceph-mon[33544]: pgmap v535: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 2.4 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 37966/227121 objects degraded (16.716%); 1/227121 objects misplaced (0.000%); 9.0 KiB/s, 0 objects/s recovering 2024-03-12T03:44:37.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:37 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1378476019' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:44:37.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:37.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:37.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:37.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:37.900 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:37.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:37.940 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:37.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:38.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:38.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:38.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:38.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:38.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:38.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:38.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:38.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:38.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:38.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:38.381 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 43 16 84585 84569 122.906 77.375 0.0243559 0.00813137 2024-03-12T03:44:38.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:38.479 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:38.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:38.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:38.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:38.618 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:38.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:38.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:38 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:44:38.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:38 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1378476019' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:44:38.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:38 smithi169 ceph-mon[29344]: osdmap e298: 8 total, 7 up, 8 in 2024-03-12T03:44:38.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:38 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 37966/227121 objects degraded (16.716%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:38.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:38 smithi169 ceph-mon[29344]: osdmap e299: 8 total, 7 up, 8 in 2024-03-12T03:44:38.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:38 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:44:38.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:38 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1378476019' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:44:38.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:38 smithi169 ceph-mon[39949]: osdmap e298: 8 total, 7 up, 8 in 2024-03-12T03:44:38.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:38 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 37966/227121 objects degraded (16.716%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:38.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:38 smithi169 ceph-mon[39949]: osdmap e299: 8 total, 7 up, 8 in 2024-03-12T03:44:38.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:38.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:38 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:44:38.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:38 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1378476019' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:44:38.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:38 smithi173 ceph-mon[33544]: osdmap e298: 8 total, 7 up, 8 in 2024-03-12T03:44:38.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:38 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 37966/227121 objects degraded (16.716%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:38.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:38 smithi173 ceph-mon[33544]: osdmap e299: 8 total, 7 up, 8 in 2024-03-12T03:44:38.753 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:38.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:38.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:38.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:38.890 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:38.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:38.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:39.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:39.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:39.164 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:39.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:39.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:39.291 INFO:tasks.thrashosds.thrasher:j is {'epoch': 297, 'fsid': '677d624a-e020-11ee-95c8-87774f69a715', 'created': '2024-03-12T03:27:53.319986+0000', 'modified': '2024-03-12T03:44:24.349279+0000', 'last_up_change': '2024-03-12T03:43:18.160417+0000', 'last_in_change': '2024-03-12T03:39:32.320120+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T03:31:04.863493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '293', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '291', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-12T03:33:18.983144+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-12T03:33:19.343301+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-12T03:33:21.864621+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-12T03:33:23.416265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-12T03:33:25.476511+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-12T03:43:44.570104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '283', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'c44466b0-3112-44dc-b5f3-d6494b958761', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6801', 'nonce': 676779916}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6802', 'nonce': 676779916}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6804', 'nonce': 676779916}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6803', 'nonce': 676779916}]}, 'public_addr': '172.21.15.169:6801/676779916', 'cluster_addr': '172.21.15.169:6802/676779916', 'heartbeat_back_addr': '172.21.15.169:6804/676779916', 'heartbeat_front_addr': '172.21.15.169:6803/676779916', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '184c5c97-a992-4be9-bcd3-c08cb58a0e56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6805', 'nonce': 130042595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6806', 'nonce': 130042595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6808', 'nonce': 130042595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6807', 'nonce': 130042595}]}, 'public_addr': '172.21.15.169:6805/130042595', 'cluster_addr': '172.21.15.169:6806/130042595', 'heartbeat_back_addr': '172.21.15.169:6808/130042595', 'heartbeat_front_addr': '172.21.15.169:6807/130042595', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'beb15e06-1ca4-46a7-b640-efd1991367e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6809', 'nonce': 2751816702}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6810', 'nonce': 2751816702}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6812', 'nonce': 2751816702}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6811', 'nonce': 2751816702}]}, 'public_addr': '172.21.15.169:6809/2751816702', 'cluster_addr': '172.21.15.169:6810/2751816702', 'heartbeat_back_addr': '172.21.15.169:6812/2751816702', 'heartbeat_front_addr': '172.21.15.169:6811/2751816702', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a00b4c50-e064-4c9e-85dc-b0a75ea5bf37', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 27, 'last_clean_end': 202, 'up_from': 222, 'up_thru': 261, 'down_at': 265, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6813', 'nonce': 287982149}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6814', 'nonce': 287982149}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6816', 'nonce': 287982149}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6815', 'nonce': 287982149}]}, 'public_addr': '172.21.15.169:6813/287982149', 'cluster_addr': '172.21.15.169:6814/287982149', 'heartbeat_back_addr': '172.21.15.169:6816/287982149', 'heartbeat_front_addr': '172.21.15.169:6815/287982149', 'state': ['exists']}, {'osd': 4, 'uuid': '3a6121ef-6309-41b7-a964-6772c010b3a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 1397212305}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6801', 'nonce': 1397212305}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6803', 'nonce': 1397212305}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 1397212305}]}, 'public_addr': '172.21.15.173:6800/1397212305', 'cluster_addr': '172.21.15.173:6801/1397212305', 'heartbeat_back_addr': '172.21.15.173:6803/1397212305', 'heartbeat_front_addr': '172.21.15.173:6802/1397212305', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '3c2f6dd9-9784-4a07-8710-55ac3de82964', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 3525888417}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6805', 'nonce': 3525888417}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6807', 'nonce': 3525888417}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 3525888417}]}, 'public_addr': '172.21.15.173:6804/3525888417', 'cluster_addr': '172.21.15.173:6805/3525888417', 'heartbeat_back_addr': '172.21.15.173:6807/3525888417', 'heartbeat_front_addr': '172.21.15.173:6806/3525888417', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd7759df7-1978-4d70-94a2-721540804f59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 2147192629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6809', 'nonce': 2147192629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6811', 'nonce': 2147192629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 2147192629}]}, 'public_addr': '172.21.15.173:6808/2147192629', 'cluster_addr': '172.21.15.173:6809/2147192629', 'heartbeat_back_addr': '172.21.15.173:6811/2147192629', 'heartbeat_front_addr': '172.21.15.173:6810/2147192629', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '8843039e-2b17-4230-9cca-5788a2294d8c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 4014976876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6813', 'nonce': 4014976876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6815', 'nonce': 4014976876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 4014976876}]}, 'public_addr': '172.21.15.173:6812/4014976876', 'cluster_addr': '172.21.15.173:6813/4014976876', 'heartbeat_back_addr': '172.21.15.173:6815/4014976876', 'heartbeat_front_addr': '172.21.15.173:6814/4014976876', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:43:40.020140+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:43:40.631033+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:43:52.325920+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:43:12.195881+0000', 'dead_epoch': 264}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:44:09.972166+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:43:43.979911+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:44:03.934937+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:44:15.955403+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.c', 'osds': [0, 6, 5]}, {'pgid': '1.e', 'osds': [4, 7, 0]}, {'pgid': '1.11', 'osds': [0, 5, 2]}, {'pgid': '1.13', 'osds': [1, 4, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.169:0/4041747161': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:6800/2779395456': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/555866474': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/342292643': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/197210445': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/1430250501': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/3918553041': '2024-03-13T03:28:48.518732+0000', '172.21.15.169:0/1020514981': '2024-03-13T03:28:21.868923+0000', '172.21.15.169:0/284191239': '2024-03-13T03:28:21.868923+0000', '172.21.15.169:0/1212459137': '2024-03-13T03:28:48.518732+0000', '172.21.15.169:6800/204892396': '2024-03-13T03:28:21.868923+0000', '172.21.15.169:0/2751007072': '2024-03-13T03:28:48.518732+0000', '172.21.15.169:0/2349376510': '2024-03-13T03:28:21.868923+0000', '172.21.15.169:6800/108891420': '2024-03-13T03:28:48.518732+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-12T03:44:39.292 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:44:39.314 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:39.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:39.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:39.381 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 44 16 86040 86024 122.18 90.9375 0.00906005 0.00818069 2024-03-12T03:44:39.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:39.453 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:39.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:39.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:39.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:39.605 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:39.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:39.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:39 smithi169 ceph-mon[29344]: pgmap v537: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s; 40998/246741 objects degraded (16.616%); 1/246741 objects misplaced (0.000%) 2024-03-12T03:44:39.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:39 smithi169 ceph-mon[39949]: pgmap v537: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s; 40998/246741 objects degraded (16.616%); 1/246741 objects misplaced (0.000%) 2024-03-12T03:44:39.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:39 smithi173 ceph-mon[33544]: pgmap v537: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s; 40998/246741 objects degraded (16.616%); 1/246741 objects misplaced (0.000%) 2024-03-12T03:44:39.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:39.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:39.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:39.886 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:39.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:39.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:40.020 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:40.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:40.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:40.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:40.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:40.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:40.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:40.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:40.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:40.381 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 45 16 87443 87427 121.413 87.6875 0.0121405 0.00823139 2024-03-12T03:44:40.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:40.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:40.440 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:40.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:40.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:40.578 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:40.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:40.623 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:40.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:40.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:40.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:40.813 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:40.845 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:40.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:40.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:40.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:40.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:40.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:41.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:41.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:41.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:41.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:41.265 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:41.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:41.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:41.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:41.381 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 46 15 88864 88849 120.705 88.875 0.0189376 0.00828049 2024-03-12T03:44:41.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:41.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:41.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:41.456 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:41 smithi169 ceph-mon[29344]: pgmap v539: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s; 40998/246741 objects degraded (16.616%); 1/246741 objects misplaced (0.000%) 2024-03-12T03:44:41.456 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:41 smithi169 ceph-mon[39949]: pgmap v539: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s; 40998/246741 objects degraded (16.616%); 1/246741 objects misplaced (0.000%) 2024-03-12T03:44:41.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:41.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:41.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:41.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:41.683 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:41.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:41.720 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:41 smithi173 ceph-mon[33544]: pgmap v539: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s; 40998/246741 objects degraded (16.616%); 1/246741 objects misplaced (0.000%) 2024-03-12T03:44:41.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:41.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:41.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:41.886 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:44:41.886 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:44:41.886 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:44:41.886 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:44:41.886 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:44:41.886 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:44:41.887 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:44:41.887 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:44:41.887 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:44:41.887 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:44:41.887 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:44:41.887 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:44:41.887 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:44:41.887 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:44:41.887 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:44:41.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:41.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:41.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:41.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:42.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:42.069 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:44:42.099 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:44:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:44:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:44:42.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:42.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:42.131 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:42.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:42.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:42.240 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:42.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:42.341 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:42.376 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:42.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:42.381 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 47 16 90284 90268 120.024 88.6875 0.0111752 0.00832705 2024-03-12T03:44:42.478 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:42.510 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:42.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:42.611 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:42.646 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:42.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:42.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:42 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/925796492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:44:42.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:42.773 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:42 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/925796492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:44:42.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:42 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/925796492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:44:42.784 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:42.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:42.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:42.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:42.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:42.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:42.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:43.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:43.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:43.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:43.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:43.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:43.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:43.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:44:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:44:43.299 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:43.331 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:43.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:43.381 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 48 16 91775 91759 119.465 93.1875 0.00421749 0.00836582 2024-03-12T03:44:43.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:43.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:43.472 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:43.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:43.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:43.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:43.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:43.631 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:43 smithi169 ceph-mon[29344]: pgmap v540: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 43542/263334 objects degraded (16.535%); 1/263334 objects misplaced (0.000%) 2024-03-12T03:44:43.631 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:43 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 43542/263334 objects degraded (16.535%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:43.632 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:43 smithi169 ceph-mon[39949]: pgmap v540: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 43542/263334 objects degraded (16.535%); 1/263334 objects misplaced (0.000%) 2024-03-12T03:44:43.632 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:43 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 43542/263334 objects degraded (16.535%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:44:43.683 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:44:43.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:43 smithi173 ceph-mon[33544]: pgmap v540: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 43542/263334 objects degraded (16.535%); 1/263334 objects misplaced (0.000%) 2024-03-12T03:44:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:43 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 43542/263334 objects degraded (16.535%), 31 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:43.754 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:43.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:43.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:43.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:43.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:43.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:44.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:44.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:44.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:44.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:44.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:44.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:44.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:44.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:44.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:44.382 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 49 16 93210 93194 118.857 89.6875 0.0094498 0.00840902 2024-03-12T03:44:44.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:44.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:44.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:44.548 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:44.584 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:44.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:44.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:44.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:44.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:44.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:44.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:44.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:44.824 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:44.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:44.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:44.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:45.003 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:45.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:45.054 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:44:45.054 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:44:45.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.054 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:94186 9.0 9:067e2e67:::benchmark_data_smithi169_135229_object94185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-03-12T03:44:45.054 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:44:45.042949+0000", 2024-03-12T03:44:45.054 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0070896409999999998, 2024-03-12T03:44:45.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071238209999999998, 2024-03-12T03:44:45.054 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:44:45.054 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:44:45.054 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:44:45.054 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:44:45.054 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 94186 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.042949+0000", 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.042949+0000", 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:44:45.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.042950+0000", 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.15e-06 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.042975+0000", 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5673999999999998e-05 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.042986+0000", 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0535e-05 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:44:45.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.042993+0000", 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3370000000000002e-06 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.043044+0000", 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0590000000000002e-05 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.043184+0000", 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013984699999999999 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2,6", 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.043324+0000", 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000140347 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.049048+0000", 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0057242530000000003 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.049448+0000", 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00039953100000000001 2024-03-12T03:44:45.058 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:94190 9.0 9:0c6e7f50:::benchmark_data_smithi169_135229_object94189:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:44:45.044275+0000", 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0057629810000000004, 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058601529999999999, 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 94190 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.059 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.044275+0000", 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.044275+0000", 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:44:45.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.044276+0000", 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0550000000000001e-06 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.044307+0000", 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0979999999999998e-05 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.044331+0000", 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3187999999999999e-05 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:44:45.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.044339+0000", 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2660000000000001e-06 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.045905+0000", 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015664909999999999 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.046067+0000", 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016192800000000001 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2,6", 2024-03-12T03:44:45.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.046222+0000", 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015499700000000001 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.049233+0000", 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0030107799999999998 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.049768+0000", 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00053503399999999999 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.15279.0:94191 9.d e299/278 9:be463c1a:::benchmark_data_smithi169_135229_object94190:head v 299'6132, mlcod=299'6130)", 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:44:45.048688+0000", 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.001349952, 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015129589999999999, 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048688+0000", 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048688+0000", 2024-03-12T03:44:45.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048689+0000", 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8000000000000005e-07 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048727+0000", 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7517e-05 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048737+0000", 2024-03-12T03:44:45.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.083e-05 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048744+0000", 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4710000000000004e-06 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048798+0000", 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4154999999999999e-05 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:44:45.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048818+0000", 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9601000000000001e-05 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.15279.0:94194 9.3 e299/278 9:c71552f4:::benchmark_data_smithi169_135229_object94193:head v 299'5890, mlcod=299'5887)", 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:44:45.048764+0000", 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0012743749999999999, 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001474403, 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:44:45.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048764+0000", 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048764+0000", 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048764+0000", 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9299999999999999e-07 2024-03-12T03:44:45.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048798+0000", 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3297000000000003e-05 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048804+0000", 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5690000000000001e-06 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048809+0000", 2024-03-12T03:44:45.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0900000000000004e-06 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048886+0000", 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6868999999999999e-05 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.048906+0000", 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0392000000000001e-05 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.15279.0:94186 9.0 e299/278 ondisk, result = 0)", 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:44:45.050024+0000", 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.4659e-05, 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00024810799999999998, 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.050024+0000", 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:45.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.050024+0000", 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.050024+0000", 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2099999999999997e-07 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:44:45.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.050028+0000", 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.501e-06 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.050036+0000", 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3750000000000003e-06 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.050042+0000", 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3829999999999997e-06 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.050084+0000", 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1925999999999998e-05 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:45.050091+0000", 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7649999999999996e-06 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 5 2024-03-12T03:44:45.074 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:44:45.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:45.139 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:45.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:45.241 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:45.277 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:45.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:45.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:45.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:45.382 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 50 16 94686 94670 118.324 92.25 0.00771466 0.00844685 2024-03-12T03:44:45.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:45.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:45.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:45.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:45.553 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:45.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:45.655 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:45.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:45 smithi169 ceph-mon[29344]: pgmap v541: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 44432/267378 objects degraded (16.618%); 1/267378 objects misplaced (0.000%) 2024-03-12T03:44:45.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:45 smithi169 ceph-mon[39949]: pgmap v541: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 44432/267378 objects degraded (16.618%); 1/267378 objects misplaced (0.000%) 2024-03-12T03:44:45.685 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:44:45.688 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:45.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:45 smithi173 ceph-mon[33544]: pgmap v541: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 44432/267378 objects degraded (16.618%); 1/267378 objects misplaced (0.000%) 2024-03-12T03:44:45.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:45.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:45.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:45.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:45.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:45.963 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:45.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:45.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:46.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:46.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:46.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:46.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:46.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:46.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:46.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:46.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:46.377 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:46.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:46.381 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 51 16 96135 96119 117.78 90.5625 0.0119366 0.00848624 2024-03-12T03:44:46.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:46.514 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:46.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:46.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:46.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:46.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:46.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:46 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:46.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:46 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:46 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:46.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:46.788 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:46.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:46.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:46.819 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:44:46.822 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:44:46.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:46.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:46.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:46.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-12T03:44:47.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:47.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:47.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:47.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:47.125 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:47.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:47.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:47.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:47.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:47.344 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:47.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:47.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:47.382 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 52 16 97581 97565 117.253 90.375 0.0113045 0.00852411 2024-03-12T03:44:47.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:47.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:47.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:47.585 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:47.622 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:47.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:47.724 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:47 smithi173 ceph-mon[33544]: pgmap v542: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 595 B/s rd, 53 MiB/s wr, 854 op/s; 44899/268779 objects degraded (16.705%); 1/268779 objects misplaced (0.000%) 2024-03-12T03:44:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:47 smithi173 ceph-mon[33544]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:44:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:47.758 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:47.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:47.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:47.898 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:47.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:47 smithi169 ceph-mon[29344]: pgmap v542: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 595 B/s rd, 53 MiB/s wr, 854 op/s; 44899/268779 objects degraded (16.705%); 1/268779 objects misplaced (0.000%) 2024-03-12T03:44:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:47 smithi169 ceph-mon[29344]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:44:47.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:47 smithi169 ceph-mon[39949]: pgmap v542: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 595 B/s rd, 53 MiB/s wr, 854 op/s; 44899/268779 objects degraded (16.705%); 1/268779 objects misplaced (0.000%) 2024-03-12T03:44:47.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:47 smithi169 ceph-mon[39949]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:44:47.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:48.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:48.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:48.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:48.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:48.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:48.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:48.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:48.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:48.382 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 53 16 99070 99054 116.796 93.0625 0.00689751 0.00855726 2024-03-12T03:44:48.424 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:48.458 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:48.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:48.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:48.554 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:48.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:48.564 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":542,"stamp":"2024-03-12T03:44:46.014853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5856857754,"num_objects":89593,"num_object_clones":0,"num_object_copies":268779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":44899,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":89593,"num_whiteouts":0,"num_read":2153,"num_read_kb":1901,"num_write":90326,"num_write_kb":5719873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":1013252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78787,"ondisk_log_size":78787,"up":431,"acting":435,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":502,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17711108,"kb_used_data":14315252,"kb_used_omap":18,"kb_used_meta":3395757,"kb_avail":732250108,"statfs":{"total":767960285184,"available":749824110592,"internally_reserved":0,"allocated":14658818048,"data_stored":14653994236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19339,"internal_metadata":3477255285},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":481427456,"num_objects":7346,"num_object_clones":0,"num_object_copies":22038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7346,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7346,"num_write_kb":470144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.601841"},"pg_stats":[{"pgid":"9.1","version":"299'5428","reported_seq":5476,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:41.178379+0000","last_change":"2024-03-12T03:43:55.390953+0000","last_active":"2024-03-12T03:44:41.178379+0000","last_peered":"2024-03-12T03:44:41.178379+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.591676+0000","last_became_peered":"2024-03-12T03:43:45.591676+0000","last_unstale":"2024-03-12T03:44:41.178379+0000","last_undegraded":"2024-03-12T03:43:55.390315+0000","last_fullsized":"2024-03-12T03:43:45.580214+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":5428,"log_dups_size":0,"ondisk_log_size":5428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:48.943427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355729408,"num_objects":5428,"num_object_clones":0,"num_object_copies":16284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5428,"num_write_kb":347392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":5428}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":817,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.374679+0000","last_change":"2024-03-12T03:43:18.345099+0000","last_active":"2024-03-12T03:44:38.374679+0000","last_peered":"2024-03-12T03:44:38.374679+0000","last_clean":"2024-03-12T03:43:13.340455+0000","last_became_active":"2024-03-12T03:43:18.345099+0000","last_became_peered":"2024-03-12T03:43:18.345099+0000","last_unstale":"2024-03-12T03:44:38.374679+0000","last_undegraded":"2024-03-12T03:44:38.374679+0000","last_fullsized":"2024-03-12T03:43:18.338164+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:28.162735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:28.162735+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"269'20","reported_seq":549,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.363688+0000","last_change":"2024-03-12T03:43:18.351745+0000","last_active":"2024-03-12T03:44:38.363688+0000","last_peered":"2024-03-12T03:44:38.363688+0000","last_clean":"2024-03-12T03:43:14.546139+0000","last_became_active":"2024-03-12T03:43:18.351745+0000","last_became_peered":"2024-03-12T03:43:18.351745+0000","last_unstale":"2024-03-12T03:44:38.363688+0000","last_undegraded":"2024-03-12T03:43:18.333421+0000","last_fullsized":"2024-03-12T03:43:18.333308+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-12T03:43:14.545889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:14.545889+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:43.528094+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":320,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.363317+0000","last_change":"2024-03-12T03:43:18.339149+0000","last_active":"2024-03-12T03:44:38.363317+0000","last_peered":"2024-03-12T03:44:38.363317+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:43:18.339149+0000","last_became_peered":"2024-03-12T03:43:18.339149+0000","last_unstale":"2024-03-12T03:44:38.363317+0000","last_undegraded":"2024-03-12T03:44:38.363317+0000","last_fullsized":"2024-03-12T03:43:18.331129+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.981378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.981378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":295,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.385962+0000","last_change":"2024-03-12T03:43:01.350508+0000","last_active":"2024-03-12T03:44:38.385962+0000","last_peered":"2024-03-12T03:44:38.385962+0000","last_clean":"2024-03-12T03:44:38.385962+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:44:38.385962+0000","last_undegraded":"2024-03-12T03:44:38.385962+0000","last_fullsized":"2024-03-12T03:44:38.385962+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:43:01.350377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:01.350377+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:42.486316+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":519,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.365228+0000","last_change":"2024-03-12T03:44:09.835312+0000","last_active":"2024-03-12T03:44:38.365228+0000","last_peered":"2024-03-12T03:44:38.365228+0000","last_clean":"2024-03-12T03:44:38.365228+0000","last_became_active":"2024-03-12T03:39:33.265665+0000","last_became_peered":"2024-03-12T03:39:33.265665+0000","last_unstale":"2024-03-12T03:44:38.365228+0000","last_undegraded":"2024-03-12T03:44:38.365228+0000","last_fullsized":"2024-03-12T03:44:38.365228+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:09.835231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:09.835231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:23.425757+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"299'5684","reported_seq":5736,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:42.069201+0000","last_change":"2024-03-12T03:43:45.632046+0000","last_active":"2024-03-12T03:44:42.069201+0000","last_peered":"2024-03-12T03:44:42.069201+0000","last_clean":"2024-03-12T03:44:42.069201+0000","last_became_active":"2024-03-12T03:43:45.631752+0000","last_became_peered":"2024-03-12T03:43:45.631752+0000","last_unstale":"2024-03-12T03:44:42.069201+0000","last_undegraded":"2024-03-12T03:44:42.069201+0000","last_fullsized":"2024-03-12T03:44:42.069201+0000","mapping_epoch":278,"log_start":"285'1200","ondisk_log_start":"285'1200","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:49.076842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372506624,"num_objects":5684,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5684,"num_write_kb":363776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"269'31","reported_seq":612,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363134+0000","last_change":"2024-03-12T03:44:13.891462+0000","last_active":"2024-03-12T03:44:38.363134+0000","last_peered":"2024-03-12T03:44:38.363134+0000","last_clean":"2024-03-12T03:44:38.363134+0000","last_became_active":"2024-03-12T03:39:33.268652+0000","last_became_peered":"2024-03-12T03:39:33.268652+0000","last_unstale":"2024-03-12T03:44:38.363134+0000","last_undegraded":"2024-03-12T03:44:38.363134+0000","last_fullsized":"2024-03-12T03:44:38.363134+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:44:13.891303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:44:13.891303+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:28.742455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":517,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.371253+0000","last_change":"2024-03-12T03:43:09.441186+0000","last_active":"2024-03-12T03:44:38.371253+0000","last_peered":"2024-03-12T03:44:38.371253+0000","last_clean":"2024-03-12T03:44:38.371253+0000","last_became_active":"2024-03-12T03:33:21.106461+0000","last_became_peered":"2024-03-12T03:33:21.106461+0000","last_unstale":"2024-03-12T03:44:38.371253+0000","last_undegraded":"2024-03-12T03:44:38.371253+0000","last_fullsized":"2024-03-12T03:44:38.371253+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:09.441075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:43:09.441075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.656422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":510,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.369715+0000","last_change":"2024-03-12T03:43:21.789919+0000","last_active":"2024-03-12T03:44:38.369715+0000","last_peered":"2024-03-12T03:44:38.369715+0000","last_clean":"2024-03-12T03:44:38.369715+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:44:38.369715+0000","last_undegraded":"2024-03-12T03:44:38.369715+0000","last_fullsized":"2024-03-12T03:44:38.369715+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:21.789828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:43:21.789828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.198862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":242,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.386157+0000","last_change":"2024-03-12T03:43:18.344122+0000","last_active":"2024-03-12T03:44:38.386157+0000","last_peered":"2024-03-12T03:44:38.386157+0000","last_clean":"2024-03-12T03:43:13.349376+0000","last_became_active":"2024-03-12T03:43:18.344122+0000","last_became_peered":"2024-03-12T03:43:18.344122+0000","last_unstale":"2024-03-12T03:44:38.386157+0000","last_undegraded":"2024-03-12T03:44:38.386157+0000","last_fullsized":"2024-03-12T03:43:18.337641+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:07.272232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:07.272232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:42.486273+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":658,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.359536+0000","last_change":"2024-03-12T03:43:18.379878+0000","last_active":"2024-03-12T03:44:38.359536+0000","last_peered":"2024-03-12T03:44:38.359536+0000","last_clean":"2024-03-12T03:43:13.353305+0000","last_became_active":"2024-03-12T03:43:18.379878+0000","last_became_peered":"2024-03-12T03:43:18.379878+0000","last_unstale":"2024-03-12T03:44:38.359536+0000","last_undegraded":"2024-03-12T03:43:18.336786+0000","last_fullsized":"2024-03-12T03:43:18.336678+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:41:03.399737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:41:03.399737+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":642,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.367607+0000","last_change":"2024-03-12T03:44:24.371625+0000","last_active":"2024-03-12T03:44:38.367607+0000","last_peered":"2024-03-12T03:44:38.367607+0000","last_clean":"2024-03-12T03:44:38.367607+0000","last_became_active":"2024-03-12T03:44:24.371149+0000","last_became_peered":"2024-03-12T03:44:24.371149+0000","last_unstale":"2024-03-12T03:44:38.367607+0000","last_undegraded":"2024-03-12T03:44:38.367607+0000","last_fullsized":"2024-03-12T03:44:38.367607+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:19.738048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:19.738048+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:42.072150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"299'5548","reported_seq":5597,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:40.980090+0000","last_change":"2024-03-12T03:43:55.386278+0000","last_active":"2024-03-12T03:44:40.980090+0000","last_peered":"2024-03-12T03:44:40.980090+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.592856+0000","last_became_peered":"2024-03-12T03:43:45.592856+0000","last_unstale":"2024-03-12T03:44:40.980090+0000","last_undegraded":"2024-03-12T03:43:55.380028+0000","last_fullsized":"2024-03-12T03:43:45.589896+0000","mapping_epoch":278,"log_start":"285'1200","ondisk_log_start":"285'1200","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4348,"log_dups_size":0,"ondisk_log_size":4348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:14.284774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363593728,"num_objects":5548,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5548,"num_write_kb":355072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":5548}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"269'31","reported_seq":564,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.365041+0000","last_change":"2024-03-12T03:43:59.838390+0000","last_active":"2024-03-12T03:44:38.365041+0000","last_peered":"2024-03-12T03:44:38.365041+0000","last_clean":"2024-03-12T03:44:38.365041+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:44:38.365041+0000","last_undegraded":"2024-03-12T03:44:38.365041+0000","last_fullsized":"2024-03-12T03:44:38.365041+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:43:59.838251+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:43:59.838251+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:06.355307+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":523,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.359534+0000","last_change":"2024-03-12T03:43:09.624124+0000","last_active":"2024-03-12T03:44:38.359534+0000","last_peered":"2024-03-12T03:44:38.359534+0000","last_clean":"2024-03-12T03:44:38.359534+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:44:38.359534+0000","last_undegraded":"2024-03-12T03:44:38.359534+0000","last_fullsized":"2024-03-12T03:44:38.359534+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:09.624047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:09.624047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":501,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.365442+0000","last_change":"2024-03-12T03:43:55.569297+0000","last_active":"2024-03-12T03:44:38.365442+0000","last_peered":"2024-03-12T03:44:38.365442+0000","last_clean":"2024-03-12T03:44:38.365442+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:44:38.365442+0000","last_undegraded":"2024-03-12T03:44:38.365442+0000","last_fullsized":"2024-03-12T03:44:38.365442+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:55.569174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:55.569174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:06.654122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":329,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.371806+0000","last_change":"2024-03-12T03:43:18.353217+0000","last_active":"2024-03-12T03:44:38.371806+0000","last_peered":"2024-03-12T03:44:38.371806+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.353217+0000","last_became_peered":"2024-03-12T03:43:18.353217+0000","last_unstale":"2024-03-12T03:44:38.371806+0000","last_undegraded":"2024-03-12T03:44:38.371806+0000","last_fullsized":"2024-03-12T03:43:18.336780+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:13.114417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.114417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.656304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"299'5581","reported_seq":5630,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:42.058164+0000","last_change":"2024-03-12T03:43:55.387593+0000","last_active":"2024-03-12T03:44:42.058164+0000","last_peered":"2024-03-12T03:44:42.058164+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.631713+0000","last_became_peered":"2024-03-12T03:43:45.631713+0000","last_unstale":"2024-03-12T03:44:42.058164+0000","last_undegraded":"2024-03-12T03:43:55.379093+0000","last_fullsized":"2024-03-12T03:43:45.587023+0000","mapping_epoch":278,"log_start":"285'1100","ondisk_log_start":"285'1100","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:57.946789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365756416,"num_objects":5581,"num_object_clones":0,"num_object_copies":16743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":357184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5581}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":638,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.365056+0000","last_change":"2024-03-12T03:41:04.383619+0000","last_active":"2024-03-12T03:44:38.365056+0000","last_peered":"2024-03-12T03:44:38.365056+0000","last_clean":"2024-03-12T03:44:38.365056+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:44:38.365056+0000","last_undegraded":"2024-03-12T03:44:38.365056+0000","last_fullsized":"2024-03-12T03:44:38.365056+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:41:04.383505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:41:04.383505+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":346,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.369004+0000","last_change":"2024-03-12T03:43:18.346647+0000","last_active":"2024-03-12T03:44:38.369004+0000","last_peered":"2024-03-12T03:44:38.369004+0000","last_clean":"2024-03-12T03:43:13.349764+0000","last_became_active":"2024-03-12T03:43:18.346647+0000","last_became_peered":"2024-03-12T03:43:18.346647+0000","last_unstale":"2024-03-12T03:44:38.369004+0000","last_undegraded":"2024-03-12T03:43:18.336192+0000","last_fullsized":"2024-03-12T03:43:18.335538+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:43:00.715225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.715225+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":320,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.369090+0000","last_change":"2024-03-12T03:43:18.344931+0000","last_active":"2024-03-12T03:44:38.369090+0000","last_peered":"2024-03-12T03:44:38.369090+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:43:18.344931+0000","last_became_peered":"2024-03-12T03:43:18.344931+0000","last_unstale":"2024-03-12T03:44:38.369090+0000","last_undegraded":"2024-03-12T03:44:38.369090+0000","last_fullsized":"2024-03-12T03:43:18.334088+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:08.619817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:08.619817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":656,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:41.758410+0000","last_change":"2024-03-12T03:44:16.927568+0000","last_active":"2024-03-12T03:44:41.758410+0000","last_peered":"2024-03-12T03:44:41.758410+0000","last_clean":"2024-03-12T03:44:41.758410+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:44:41.758410+0000","last_undegraded":"2024-03-12T03:44:41.758410+0000","last_fullsized":"2024-03-12T03:44:41.758410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:44:16.927294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:44:16.927294+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:36.372304+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":490,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.362362+0000","last_change":"2024-03-12T03:43:23.613276+0000","last_active":"2024-03-12T03:44:38.362362+0000","last_peered":"2024-03-12T03:44:38.362362+0000","last_clean":"2024-03-12T03:44:38.362362+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:44:38.362362+0000","last_undegraded":"2024-03-12T03:44:38.362362+0000","last_fullsized":"2024-03-12T03:44:38.362362+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:23.613137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:23.613137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:44.717659+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"299'5755","reported_seq":5806,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:41.007122+0000","last_change":"2024-03-12T03:43:45.597209+0000","last_active":"2024-03-12T03:44:41.007122+0000","last_peered":"2024-03-12T03:44:41.007122+0000","last_clean":"2024-03-12T03:44:41.007122+0000","last_became_active":"2024-03-12T03:43:45.596788+0000","last_became_peered":"2024-03-12T03:43:45.596788+0000","last_unstale":"2024-03-12T03:44:41.007122+0000","last_undegraded":"2024-03-12T03:44:41.007122+0000","last_fullsized":"2024-03-12T03:44:41.007122+0000","mapping_epoch":278,"log_start":"285'1400","ondisk_log_start":"285'1400","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4355,"log_dups_size":0,"ondisk_log_size":4355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:50.337416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377159680,"num_objects":5755,"num_object_clones":0,"num_object_copies":17265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5755,"num_write_kb":368320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":507,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.359333+0000","last_change":"2024-03-12T03:43:18.344476+0000","last_active":"2024-03-12T03:44:38.359333+0000","last_peered":"2024-03-12T03:44:38.359333+0000","last_clean":"2024-03-12T03:43:13.337325+0000","last_became_active":"2024-03-12T03:43:18.344476+0000","last_became_peered":"2024-03-12T03:43:18.344476+0000","last_unstale":"2024-03-12T03:44:38.359333+0000","last_undegraded":"2024-03-12T03:44:38.359333+0000","last_fullsized":"2024-03-12T03:43:18.329711+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:01.229635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:01.229635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"269'42","reported_seq":209,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.372338+0000","last_change":"2024-03-12T03:43:18.345917+0000","last_active":"2024-03-12T03:44:38.372338+0000","last_peered":"2024-03-12T03:44:38.372338+0000","last_clean":"2024-03-12T03:33:28.528154+0000","last_became_active":"2024-03-12T03:43:18.345917+0000","last_became_peered":"2024-03-12T03:43:18.345917+0000","last_unstale":"2024-03-12T03:44:38.372338+0000","last_undegraded":"2024-03-12T03:43:18.336762+0000","last_fullsized":"2024-03-12T03:43:18.336652+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-03-12T03:36:06.083052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.083052+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":324,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.376766+0000","last_change":"2024-03-12T03:43:18.345673+0000","last_active":"2024-03-12T03:44:38.376766+0000","last_peered":"2024-03-12T03:44:38.376766+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:43:18.345673+0000","last_became_peered":"2024-03-12T03:43:18.345673+0000","last_unstale":"2024-03-12T03:44:38.376766+0000","last_undegraded":"2024-03-12T03:44:38.376766+0000","last_fullsized":"2024-03-12T03:43:18.337055+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:41.918538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:41.918538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":523,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.359477+0000","last_change":"2024-03-12T03:43:18.344050+0000","last_active":"2024-03-12T03:44:38.359477+0000","last_peered":"2024-03-12T03:44:38.359477+0000","last_clean":"2024-03-12T03:43:13.338394+0000","last_became_active":"2024-03-12T03:43:18.344050+0000","last_became_peered":"2024-03-12T03:43:18.344050+0000","last_unstale":"2024-03-12T03:44:38.359477+0000","last_undegraded":"2024-03-12T03:44:38.359477+0000","last_fullsized":"2024-03-12T03:43:18.333957+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:23.257942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:23.257942+0000","last_clean_scrub_stamp":"2024-03-12T03:41:23.257942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":652,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.362805+0000","last_change":"2024-03-12T03:44:19.374044+0000","last_active":"2024-03-12T03:44:38.362805+0000","last_peered":"2024-03-12T03:44:38.362805+0000","last_clean":"2024-03-12T03:44:38.362805+0000","last_became_active":"2024-03-12T03:44:19.373715+0000","last_became_peered":"2024-03-12T03:44:19.373715+0000","last_unstale":"2024-03-12T03:44:38.362805+0000","last_undegraded":"2024-03-12T03:44:38.362805+0000","last_fullsized":"2024-03-12T03:44:38.362805+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:04.664919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:04.664919+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"299'5876","reported_seq":5925,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:45.961094+0000","last_change":"2024-03-12T03:43:55.383487+0000","last_active":"2024-03-12T03:44:45.961094+0000","last_peered":"2024-03-12T03:44:45.961094+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.594399+0000","last_became_peered":"2024-03-12T03:43:45.594399+0000","last_unstale":"2024-03-12T03:44:45.961094+0000","last_undegraded":"2024-03-12T03:43:55.379614+0000","last_fullsized":"2024-03-12T03:43:45.586741+0000","mapping_epoch":278,"log_start":"285'1500","ondisk_log_start":"285'1500","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4376,"log_dups_size":0,"ondisk_log_size":4376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:07.446575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385089536,"num_objects":5876,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5876,"num_write_kb":376064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":5876}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"266'11","reported_seq":172,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.369392+0000","last_change":"2024-03-12T03:43:18.345511+0000","last_active":"2024-03-12T03:44:38.369392+0000","last_peered":"2024-03-12T03:44:38.369392+0000","last_clean":"2024-03-12T03:33:28.353771+0000","last_became_active":"2024-03-12T03:43:18.345511+0000","last_became_peered":"2024-03-12T03:43:18.345511+0000","last_unstale":"2024-03-12T03:44:38.369392+0000","last_undegraded":"2024-03-12T03:43:18.332743+0000","last_fullsized":"2024-03-12T03:43:18.332472+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:39:37.039925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:37.039925+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:45.095147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"269'42","reported_seq":569,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.359721+0000","last_change":"2024-03-12T03:42:02.272123+0000","last_active":"2024-03-12T03:44:38.359721+0000","last_peered":"2024-03-12T03:44:38.359721+0000","last_clean":"2024-03-12T03:44:38.359721+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:44:38.359721+0000","last_undegraded":"2024-03-12T03:44:38.359721+0000","last_fullsized":"2024-03-12T03:44:38.359721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'28","last_scrub_stamp":"2024-03-12T03:42:02.271980+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:42:02.271980+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":512,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363577+0000","last_change":"2024-03-12T03:44:09.737622+0000","last_active":"2024-03-12T03:44:38.363577+0000","last_peered":"2024-03-12T03:44:38.363577+0000","last_clean":"2024-03-12T03:44:38.363577+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:44:38.363577+0000","last_undegraded":"2024-03-12T03:44:38.363577+0000","last_fullsized":"2024-03-12T03:44:38.363577+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:44:09.737479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:44:09.737479+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:23.826899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":519,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.371730+0000","last_change":"2024-03-12T03:44:14.121638+0000","last_active":"2024-03-12T03:44:38.371730+0000","last_peered":"2024-03-12T03:44:38.371730+0000","last_clean":"2024-03-12T03:44:38.371730+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:44:38.371730+0000","last_undegraded":"2024-03-12T03:44:38.371730+0000","last_fullsized":"2024-03-12T03:44:38.371730+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:14.121534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:14.121534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:29.551049+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":419,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.359564+0000","last_change":"2024-03-12T03:44:19.370278+0000","last_active":"2024-03-12T03:44:38.359564+0000","last_peered":"2024-03-12T03:44:38.359564+0000","last_clean":"2024-03-12T03:34:01.860694+0000","last_became_active":"2024-03-12T03:44:19.370278+0000","last_became_peered":"2024-03-12T03:44:19.370278+0000","last_unstale":"2024-03-12T03:44:38.359564+0000","last_undegraded":"2024-03-12T03:44:38.359564+0000","last_fullsized":"2024-03-12T03:44:19.357809+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:02.442717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:02.442717+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:41.663783+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"299'5537","reported_seq":5586,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:41.151724+0000","last_change":"2024-03-12T03:43:55.390924+0000","last_active":"2024-03-12T03:44:41.151724+0000","last_peered":"2024-03-12T03:44:41.151724+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.588776+0000","last_became_peered":"2024-03-12T03:43:45.588776+0000","last_unstale":"2024-03-12T03:44:41.151724+0000","last_undegraded":"2024-03-12T03:43:55.379272+0000","last_fullsized":"2024-03-12T03:43:45.585743+0000","mapping_epoch":278,"log_start":"285'1100","ondisk_log_start":"285'1100","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:49.133021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362872832,"num_objects":5537,"num_object_clones":0,"num_object_copies":16611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5537,"num_write_kb":354368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":5537}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":647,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363209+0000","last_change":"2024-03-12T03:43:24.493648+0000","last_active":"2024-03-12T03:44:38.363209+0000","last_peered":"2024-03-12T03:44:38.363209+0000","last_clean":"2024-03-12T03:44:38.363209+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:44:38.363209+0000","last_undegraded":"2024-03-12T03:44:38.363209+0000","last_fullsized":"2024-03-12T03:44:38.363209+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:43:24.493559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:24.493559+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:52.783472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"269'26","reported_seq":557,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.370189+0000","last_change":"2024-03-12T03:43:20.765528+0000","last_active":"2024-03-12T03:44:38.370189+0000","last_peered":"2024-03-12T03:44:38.370189+0000","last_clean":"2024-03-12T03:44:38.370189+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:44:38.370189+0000","last_undegraded":"2024-03-12T03:44:38.370189+0000","last_fullsized":"2024-03-12T03:44:38.370189+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:43:20.765428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:20.765428+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":526,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.362417+0000","last_change":"2024-03-12T03:43:18.347416+0000","last_active":"2024-03-12T03:44:38.362417+0000","last_peered":"2024-03-12T03:44:38.362417+0000","last_clean":"2024-03-12T03:43:13.353362+0000","last_became_active":"2024-03-12T03:43:18.347416+0000","last_became_peered":"2024-03-12T03:43:18.347416+0000","last_unstale":"2024-03-12T03:44:38.362417+0000","last_undegraded":"2024-03-12T03:44:38.362417+0000","last_fullsized":"2024-03-12T03:43:18.331325+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:34.674570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:39:34.674570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":515,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.362452+0000","last_change":"2024-03-12T03:43:18.342056+0000","last_active":"2024-03-12T03:44:38.362452+0000","last_peered":"2024-03-12T03:44:38.362452+0000","last_clean":"2024-03-12T03:43:13.330724+0000","last_became_active":"2024-03-12T03:43:18.342056+0000","last_became_peered":"2024-03-12T03:43:18.342056+0000","last_unstale":"2024-03-12T03:44:38.362452+0000","last_undegraded":"2024-03-12T03:44:38.362452+0000","last_fullsized":"2024-03-12T03:43:18.333058+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:40.532991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:40.532991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675501+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":648,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.372418+0000","last_change":"2024-03-12T03:44:19.379573+0000","last_active":"2024-03-12T03:44:38.372418+0000","last_peered":"2024-03-12T03:44:38.372418+0000","last_clean":"2024-03-12T03:43:13.348756+0000","last_became_active":"2024-03-12T03:44:19.379573+0000","last_became_peered":"2024-03-12T03:44:19.379573+0000","last_unstale":"2024-03-12T03:44:38.372418+0000","last_undegraded":"2024-03-12T03:44:38.372418+0000","last_fullsized":"2024-03-12T03:44:19.360949+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:02.551349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:02.551349+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"299'5493","reported_seq":5545,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:41.017964+0000","last_change":"2024-03-12T03:43:45.597285+0000","last_active":"2024-03-12T03:44:41.017964+0000","last_peered":"2024-03-12T03:44:41.017964+0000","last_clean":"2024-03-12T03:44:41.017964+0000","last_became_active":"2024-03-12T03:43:45.596944+0000","last_became_peered":"2024-03-12T03:43:45.596944+0000","last_unstale":"2024-03-12T03:44:41.017964+0000","last_undegraded":"2024-03-12T03:44:41.017964+0000","last_fullsized":"2024-03-12T03:44:41.017964+0000","mapping_epoch":278,"log_start":"285'1200","ondisk_log_start":"285'1200","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4293,"log_dups_size":0,"ondisk_log_size":4293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:57.787068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359989248,"num_objects":5493,"num_object_clones":0,"num_object_copies":16479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5493,"num_write_kb":351552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":517,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.371517+0000","last_change":"2024-03-12T03:43:27.393346+0000","last_active":"2024-03-12T03:44:38.371517+0000","last_peered":"2024-03-12T03:44:38.371517+0000","last_clean":"2024-03-12T03:44:38.371517+0000","last_became_active":"2024-03-12T03:33:25.176514+0000","last_became_peered":"2024-03-12T03:33:25.176514+0000","last_unstale":"2024-03-12T03:44:38.371517+0000","last_undegraded":"2024-03-12T03:44:38.371517+0000","last_fullsized":"2024-03-12T03:44:38.371517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:27.393251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:43:27.393251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:55.551797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"269'35","reported_seq":570,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363654+0000","last_change":"2024-03-12T03:43:58.738099+0000","last_active":"2024-03-12T03:44:38.363654+0000","last_peered":"2024-03-12T03:44:38.363654+0000","last_clean":"2024-03-12T03:44:38.363654+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:44:38.363654+0000","last_undegraded":"2024-03-12T03:44:38.363654+0000","last_fullsized":"2024-03-12T03:44:38.363654+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-03-12T03:43:58.737948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:58.737948+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:25.826845+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":539,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.369627+0000","last_change":"2024-03-12T03:43:58.972063+0000","last_active":"2024-03-12T03:44:38.369627+0000","last_peered":"2024-03-12T03:44:38.369627+0000","last_clean":"2024-03-12T03:44:38.369627+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:44:38.369627+0000","last_undegraded":"2024-03-12T03:44:38.369627+0000","last_fullsized":"2024-03-12T03:44:38.369627+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:58.971867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:43:58.971867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:18.514654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":519,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.367503+0000","last_change":"2024-03-12T03:43:28.804503+0000","last_active":"2024-03-12T03:44:38.367503+0000","last_peered":"2024-03-12T03:44:38.367503+0000","last_clean":"2024-03-12T03:44:38.367503+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:44:38.367503+0000","last_undegraded":"2024-03-12T03:44:38.367503+0000","last_fullsized":"2024-03-12T03:44:38.367503+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:28.804369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:28.804369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:53.918392+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":648,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.371611+0000","last_change":"2024-03-12T03:44:19.370512+0000","last_active":"2024-03-12T03:44:38.371611+0000","last_peered":"2024-03-12T03:44:38.371611+0000","last_clean":"2024-03-12T03:43:13.344628+0000","last_became_active":"2024-03-12T03:44:19.370512+0000","last_became_peered":"2024-03-12T03:44:19.370512+0000","last_unstale":"2024-03-12T03:44:38.371611+0000","last_undegraded":"2024-03-12T03:44:38.371611+0000","last_fullsized":"2024-03-12T03:44:19.362319+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:10.356260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:10.356260+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"299'5577","reported_seq":5629,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:41.617484+0000","last_change":"2024-03-12T03:43:45.589372+0000","last_active":"2024-03-12T03:44:41.617484+0000","last_peered":"2024-03-12T03:44:41.617484+0000","last_clean":"2024-03-12T03:44:41.617484+0000","last_became_active":"2024-03-12T03:43:45.588933+0000","last_became_peered":"2024-03-12T03:43:45.588933+0000","last_unstale":"2024-03-12T03:44:41.617484+0000","last_undegraded":"2024-03-12T03:44:41.617484+0000","last_fullsized":"2024-03-12T03:44:41.617484+0000","mapping_epoch":278,"log_start":"284'300","ondisk_log_start":"284'300","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:50.687640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365494272,"num_objects":5577,"num_object_clones":0,"num_object_copies":16731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5577,"num_write_kb":356928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":502,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.365528+0000","last_change":"2024-03-12T03:44:11.511999+0000","last_active":"2024-03-12T03:44:38.365528+0000","last_peered":"2024-03-12T03:44:38.365528+0000","last_clean":"2024-03-12T03:44:38.365528+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:44:38.365528+0000","last_undegraded":"2024-03-12T03:44:38.365528+0000","last_fullsized":"2024-03-12T03:44:38.365528+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:11.511918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:44:11.511918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:27.172205+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"269'24","reported_seq":583,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363467+0000","last_change":"2024-03-12T03:43:09.665948+0000","last_active":"2024-03-12T03:44:38.363467+0000","last_peered":"2024-03-12T03:44:38.363467+0000","last_clean":"2024-03-12T03:44:38.363467+0000","last_became_active":"2024-03-12T03:39:33.277868+0000","last_became_peered":"2024-03-12T03:39:33.277868+0000","last_unstale":"2024-03-12T03:44:38.363467+0000","last_undegraded":"2024-03-12T03:44:38.363467+0000","last_fullsized":"2024-03-12T03:44:38.363467+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:43:09.665793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:09.665793+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675665+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":1205,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:39.158662+0000","last_change":"2024-03-12T03:44:11.061486+0000","last_active":"2024-03-12T03:44:39.158662+0000","last_peered":"2024-03-12T03:44:39.158662+0000","last_clean":"2024-03-12T03:44:39.158662+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:44:39.158662+0000","last_undegraded":"2024-03-12T03:44:39.158662+0000","last_fullsized":"2024-03-12T03:44:39.158662+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:44:11.061362+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:25.045059+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":673,"num_read_kb":668,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":515,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.368154+0000","last_change":"2024-03-12T03:43:18.338719+0000","last_active":"2024-03-12T03:44:38.368154+0000","last_peered":"2024-03-12T03:44:38.368154+0000","last_clean":"2024-03-12T03:43:13.332262+0000","last_became_active":"2024-03-12T03:43:18.338719+0000","last_became_peered":"2024-03-12T03:43:18.338719+0000","last_unstale":"2024-03-12T03:44:38.368154+0000","last_undegraded":"2024-03-12T03:44:38.368154+0000","last_fullsized":"2024-03-12T03:43:18.331120+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:42:21.445443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675590+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":519,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.369321+0000","last_change":"2024-03-12T03:43:18.350023+0000","last_active":"2024-03-12T03:44:38.369321+0000","last_peered":"2024-03-12T03:44:38.369321+0000","last_clean":"2024-03-12T03:43:13.346844+0000","last_became_active":"2024-03-12T03:43:18.350023+0000","last_became_peered":"2024-03-12T03:43:18.350023+0000","last_unstale":"2024-03-12T03:44:38.369321+0000","last_undegraded":"2024-03-12T03:44:38.369321+0000","last_fullsized":"2024-03-12T03:43:18.331452+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:11.815414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:11.815414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":635,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.367944+0000","last_change":"2024-03-12T03:44:19.367218+0000","last_active":"2024-03-12T03:44:38.367944+0000","last_peered":"2024-03-12T03:44:38.367944+0000","last_clean":"2024-03-12T03:44:38.367944+0000","last_became_active":"2024-03-12T03:44:19.366060+0000","last_became_peered":"2024-03-12T03:44:19.366060+0000","last_unstale":"2024-03-12T03:44:38.367944+0000","last_undegraded":"2024-03-12T03:44:38.367944+0000","last_fullsized":"2024-03-12T03:44:38.367944+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:19.738048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:19.738048+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:42.072117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"299'5583","reported_seq":5632,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:41.615777+0000","last_change":"2024-03-12T03:43:55.393294+0000","last_active":"2024-03-12T03:44:41.615777+0000","last_peered":"2024-03-12T03:44:41.615777+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.588708+0000","last_became_peered":"2024-03-12T03:43:45.588708+0000","last_unstale":"2024-03-12T03:44:41.615777+0000","last_undegraded":"2024-03-12T03:43:55.382438+0000","last_fullsized":"2024-03-12T03:43:45.583017+0000","mapping_epoch":278,"log_start":"284'300","ondisk_log_start":"284'300","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":5283,"log_dups_size":0,"ondisk_log_size":5283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:11.359288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365887488,"num_objects":5583,"num_object_clones":0,"num_object_copies":16749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5583,"num_write_kb":357312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":5583}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":515,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.378310+0000","last_change":"2024-03-12T03:43:26.022980+0000","last_active":"2024-03-12T03:44:38.378310+0000","last_peered":"2024-03-12T03:44:38.378310+0000","last_clean":"2024-03-12T03:44:38.378310+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:44:38.378310+0000","last_undegraded":"2024-03-12T03:44:38.378310+0000","last_fullsized":"2024-03-12T03:44:38.378310+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:26.022875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:26.022875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:50.896439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"269'28","reported_seq":689,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.371388+0000","last_change":"2024-03-12T03:43:08.426091+0000","last_active":"2024-03-12T03:44:38.371388+0000","last_peered":"2024-03-12T03:44:38.371388+0000","last_clean":"2024-03-12T03:44:38.371388+0000","last_became_active":"2024-03-12T03:33:23.552095+0000","last_became_peered":"2024-03-12T03:33:23.552095+0000","last_unstale":"2024-03-12T03:44:38.371388+0000","last_undegraded":"2024-03-12T03:44:38.371388+0000","last_fullsized":"2024-03-12T03:44:38.371388+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-12T03:43:08.425934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:08.425934+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.656396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":319,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.365290+0000","last_change":"2024-03-12T03:43:18.349427+0000","last_active":"2024-03-12T03:44:38.365290+0000","last_peered":"2024-03-12T03:44:38.365290+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:43:18.349427+0000","last_became_peered":"2024-03-12T03:43:18.349427+0000","last_unstale":"2024-03-12T03:44:38.365290+0000","last_undegraded":"2024-03-12T03:44:38.365290+0000","last_fullsized":"2024-03-12T03:43:18.334581+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:07.572642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:07.572642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":784,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.372141+0000","last_change":"2024-03-12T03:44:19.379591+0000","last_active":"2024-03-12T03:44:38.372141+0000","last_peered":"2024-03-12T03:44:38.372141+0000","last_clean":"2024-03-12T03:44:38.372141+0000","last_became_active":"2024-03-12T03:44:19.379262+0000","last_became_peered":"2024-03-12T03:44:19.379262+0000","last_unstale":"2024-03-12T03:44:38.372141+0000","last_undegraded":"2024-03-12T03:44:38.372141+0000","last_fullsized":"2024-03-12T03:44:38.372141+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:05.222876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:05.222876+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:42.056304+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":502,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.364919+0000","last_change":"2024-03-12T03:43:18.345043+0000","last_active":"2024-03-12T03:44:38.364919+0000","last_peered":"2024-03-12T03:44:38.364919+0000","last_clean":"2024-03-12T03:43:13.352684+0000","last_became_active":"2024-03-12T03:43:18.345043+0000","last_became_peered":"2024-03-12T03:43:18.345043+0000","last_unstale":"2024-03-12T03:44:38.364919+0000","last_undegraded":"2024-03-12T03:44:38.364919+0000","last_fullsized":"2024-03-12T03:43:18.334020+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:00.436718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:00.436718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"299'5427","reported_seq":5478,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:41.136071+0000","last_change":"2024-03-12T03:43:45.595179+0000","last_active":"2024-03-12T03:44:41.136071+0000","last_peered":"2024-03-12T03:44:41.136071+0000","last_clean":"2024-03-12T03:44:41.136071+0000","last_became_active":"2024-03-12T03:43:45.594793+0000","last_became_peered":"2024-03-12T03:43:45.594793+0000","last_unstale":"2024-03-12T03:44:41.136071+0000","last_undegraded":"2024-03-12T03:44:41.136071+0000","last_fullsized":"2024-03-12T03:44:41.136071+0000","mapping_epoch":278,"log_start":"285'1000","ondisk_log_start":"285'1000","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4427,"log_dups_size":0,"ondisk_log_size":4427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:54.033626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355663872,"num_objects":5427,"num_object_clones":0,"num_object_copies":16281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5427,"num_write_kb":347328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":238,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.386091+0000","last_change":"2024-03-12T03:44:03.517544+0000","last_active":"2024-03-12T03:44:38.386091+0000","last_peered":"2024-03-12T03:44:38.386091+0000","last_clean":"2024-03-12T03:44:38.386091+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:44:38.386091+0000","last_undegraded":"2024-03-12T03:44:38.386091+0000","last_fullsized":"2024-03-12T03:44:38.386091+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:44:03.517277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:24.199157+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"269'23","reported_seq":560,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.368505+0000","last_change":"2024-03-12T03:43:29.744286+0000","last_active":"2024-03-12T03:44:38.368505+0000","last_peered":"2024-03-12T03:44:38.368505+0000","last_clean":"2024-03-12T03:44:38.368505+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:44:38.368505+0000","last_undegraded":"2024-03-12T03:44:38.368505+0000","last_fullsized":"2024-03-12T03:44:38.368505+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'23","last_scrub_stamp":"2024-03-12T03:43:29.744179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:29.744179+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.298331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":234,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.385621+0000","last_change":"2024-03-12T03:43:18.343047+0000","last_active":"2024-03-12T03:44:38.385621+0000","last_peered":"2024-03-12T03:44:38.385621+0000","last_clean":"2024-03-12T03:43:13.346919+0000","last_became_active":"2024-03-12T03:43:18.343047+0000","last_became_peered":"2024-03-12T03:43:18.343047+0000","last_unstale":"2024-03-12T03:44:38.385621+0000","last_undegraded":"2024-03-12T03:44:38.385621+0000","last_fullsized":"2024-03-12T03:43:18.336453+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:01.189842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:42:01.189842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:42.486265+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":531,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.368491+0000","last_change":"2024-03-12T03:43:31.765331+0000","last_active":"2024-03-12T03:44:38.368491+0000","last_peered":"2024-03-12T03:44:38.368491+0000","last_clean":"2024-03-12T03:44:38.368491+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:44:38.368491+0000","last_undegraded":"2024-03-12T03:44:38.368491+0000","last_fullsized":"2024-03-12T03:44:38.368491+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:43:31.765257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":513,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.374275+0000","last_change":"2024-03-12T03:43:18.348792+0000","last_active":"2024-03-12T03:44:38.374275+0000","last_peered":"2024-03-12T03:44:38.374275+0000","last_clean":"2024-03-12T03:43:13.342858+0000","last_became_active":"2024-03-12T03:43:18.348792+0000","last_became_peered":"2024-03-12T03:43:18.348792+0000","last_unstale":"2024-03-12T03:44:38.374275+0000","last_undegraded":"2024-03-12T03:43:18.339155+0000","last_fullsized":"2024-03-12T03:43:18.339042+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:42:20.203740+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:42:20.203740+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115434+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":800,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.373377+0000","last_change":"2024-03-12T03:44:19.382230+0000","last_active":"2024-03-12T03:44:38.373377+0000","last_peered":"2024-03-12T03:44:38.373377+0000","last_clean":"2024-03-12T03:44:38.373377+0000","last_became_active":"2024-03-12T03:44:19.378554+0000","last_became_peered":"2024-03-12T03:44:19.378554+0000","last_unstale":"2024-03-12T03:44:38.373377+0000","last_undegraded":"2024-03-12T03:44:38.373377+0000","last_fullsized":"2024-03-12T03:44:38.373377+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:16.082316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:16.082316+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:42.056269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"299'5581","reported_seq":5633,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:41.592801+0000","last_change":"2024-03-12T03:43:45.591852+0000","last_active":"2024-03-12T03:44:41.592801+0000","last_peered":"2024-03-12T03:44:41.592801+0000","last_clean":"2024-03-12T03:44:41.592801+0000","last_became_active":"2024-03-12T03:43:45.591646+0000","last_became_peered":"2024-03-12T03:43:45.591646+0000","last_unstale":"2024-03-12T03:44:41.592801+0000","last_undegraded":"2024-03-12T03:44:41.592801+0000","last_fullsized":"2024-03-12T03:44:41.592801+0000","mapping_epoch":278,"log_start":"284'300","ondisk_log_start":"284'300","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":5281,"log_dups_size":0,"ondisk_log_size":5281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:07.133196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365756416,"num_objects":5581,"num_object_clones":0,"num_object_copies":16743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":357184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":503,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363321+0000","last_change":"2024-03-12T03:43:18.612572+0000","last_active":"2024-03-12T03:44:38.363321+0000","last_peered":"2024-03-12T03:44:38.363321+0000","last_clean":"2024-03-12T03:44:38.363321+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:44:38.363321+0000","last_undegraded":"2024-03-12T03:44:38.363321+0000","last_fullsized":"2024-03-12T03:44:38.363321+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:18.612450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:18.612450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:44.393407+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"269'52","reported_seq":725,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.359739+0000","last_change":"2024-03-12T03:43:18.348827+0000","last_active":"2024-03-12T03:44:38.359739+0000","last_peered":"2024-03-12T03:44:38.359739+0000","last_clean":"2024-03-12T03:43:16.743398+0000","last_became_active":"2024-03-12T03:43:18.348827+0000","last_became_peered":"2024-03-12T03:43:18.348827+0000","last_unstale":"2024-03-12T03:44:38.359739+0000","last_undegraded":"2024-03-12T03:43:18.331846+0000","last_fullsized":"2024-03-12T03:43:18.331727+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'39","last_scrub_stamp":"2024-03-12T03:43:06.482430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:06.482430+0000","objects_scrubbed":11,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":11}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":519,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.360393+0000","last_change":"2024-03-12T03:44:04.812250+0000","last_active":"2024-03-12T03:44:38.360393+0000","last_peered":"2024-03-12T03:44:38.360393+0000","last_clean":"2024-03-12T03:44:38.360393+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:44:38.360393+0000","last_undegraded":"2024-03-12T03:44:38.360393+0000","last_fullsized":"2024-03-12T03:44:38.360393+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:04.812138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:44:04.812138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:05.219042+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":521,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.365470+0000","last_change":"2024-03-12T03:44:10.510880+0000","last_active":"2024-03-12T03:44:38.365470+0000","last_peered":"2024-03-12T03:44:38.365470+0000","last_clean":"2024-03-12T03:44:38.365470+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:44:38.365470+0000","last_undegraded":"2024-03-12T03:44:38.365470+0000","last_fullsized":"2024-03-12T03:44:38.365470+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:44:10.510743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:44:10.510743+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:31.123357+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":363,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.364599+0000","last_change":"2024-03-12T03:43:18.344442+0000","last_active":"2024-03-12T03:44:38.364599+0000","last_peered":"2024-03-12T03:44:38.364599+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.344442+0000","last_became_peered":"2024-03-12T03:43:18.344442+0000","last_unstale":"2024-03-12T03:44:38.364599+0000","last_undegraded":"2024-03-12T03:44:38.364599+0000","last_fullsized":"2024-03-12T03:43:18.330930+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:39.031166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:39.031166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":643,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.360690+0000","last_change":"2024-03-12T03:44:19.371476+0000","last_active":"2024-03-12T03:44:38.360690+0000","last_peered":"2024-03-12T03:44:38.360690+0000","last_clean":"2024-03-12T03:44:38.360690+0000","last_became_active":"2024-03-12T03:44:19.371194+0000","last_became_peered":"2024-03-12T03:44:19.371194+0000","last_unstale":"2024-03-12T03:44:38.360690+0000","last_undegraded":"2024-03-12T03:44:38.360690+0000","last_fullsized":"2024-03-12T03:44:38.360690+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:28.720248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:28.720248+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:41.663863+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":314,"reported_epoch":299,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:44:38.364338+0000","last_change":"2024-03-12T03:44:24.375984+0000","last_active":"2024-03-12T03:44:38.364338+0000","last_peered":"2024-03-12T03:44:38.364338+0000","last_clean":"2024-03-12T03:44:38.364338+0000","last_became_active":"2024-03-12T03:44:24.375475+0000","last_became_peered":"2024-03-12T03:44:24.375475+0000","last_unstale":"2024-03-12T03:44:38.364338+0000","last_undegraded":"2024-03-12T03:44:38.364338+0000","last_fullsized":"2024-03-12T03:44:38.364338+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:10.356260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:10.356260+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:41.758104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"299'5588","reported_seq":5640,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:41.115580+0000","last_change":"2024-03-12T03:43:45.631811+0000","last_active":"2024-03-12T03:44:41.115580+0000","last_peered":"2024-03-12T03:44:41.115580+0000","last_clean":"2024-03-12T03:44:41.115580+0000","last_became_active":"2024-03-12T03:43:45.631620+0000","last_became_peered":"2024-03-12T03:43:45.631620+0000","last_unstale":"2024-03-12T03:44:41.115580+0000","last_undegraded":"2024-03-12T03:44:41.115580+0000","last_fullsized":"2024-03-12T03:44:41.115580+0000","mapping_epoch":278,"log_start":"285'1100","ondisk_log_start":"285'1100","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:52.062274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366215168,"num_objects":5588,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5588,"num_write_kb":357632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"269'26","reported_seq":549,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363327+0000","last_change":"2024-03-12T03:44:03.706713+0000","last_active":"2024-03-12T03:44:38.363327+0000","last_peered":"2024-03-12T03:44:38.363327+0000","last_clean":"2024-03-12T03:44:38.363327+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:44:38.363327+0000","last_undegraded":"2024-03-12T03:44:38.363327+0000","last_fullsized":"2024-03-12T03:44:38.363327+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:44:03.706562+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:44:03.706562+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:09.328755+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":514,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.359919+0000","last_change":"2024-03-12T03:43:18.349816+0000","last_active":"2024-03-12T03:44:38.359919+0000","last_peered":"2024-03-12T03:44:38.359919+0000","last_clean":"2024-03-12T03:43:13.352139+0000","last_became_active":"2024-03-12T03:43:18.349816+0000","last_became_peered":"2024-03-12T03:43:18.349816+0000","last_unstale":"2024-03-12T03:44:38.359919+0000","last_undegraded":"2024-03-12T03:44:38.359919+0000","last_fullsized":"2024-03-12T03:43:18.332719+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:59.344058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:40:59.344058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":234,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.385658+0000","last_change":"2024-03-12T03:43:18.346068+0000","last_active":"2024-03-12T03:44:38.385658+0000","last_peered":"2024-03-12T03:44:38.385658+0000","last_clean":"2024-03-12T03:43:13.348345+0000","last_became_active":"2024-03-12T03:43:18.346068+0000","last_became_peered":"2024-03-12T03:43:18.346068+0000","last_unstale":"2024-03-12T03:44:38.385658+0000","last_undegraded":"2024-03-12T03:44:38.385658+0000","last_fullsized":"2024-03-12T03:43:18.339666+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:03.191398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:03.191398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:42.486258+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":511,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.368522+0000","last_change":"2024-03-12T03:44:05.789257+0000","last_active":"2024-03-12T03:44:38.368522+0000","last_peered":"2024-03-12T03:44:38.368522+0000","last_clean":"2024-03-12T03:44:38.368522+0000","last_became_active":"2024-03-12T03:39:33.271016+0000","last_became_peered":"2024-03-12T03:39:33.271016+0000","last_unstale":"2024-03-12T03:44:38.368522+0000","last_undegraded":"2024-03-12T03:44:38.368522+0000","last_fullsized":"2024-03-12T03:44:38.368522+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:05.789123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:05.789123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:21.216617+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":655,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.372324+0000","last_change":"2024-03-12T03:44:24.379911+0000","last_active":"2024-03-12T03:44:38.372324+0000","last_peered":"2024-03-12T03:44:38.372324+0000","last_clean":"2024-03-12T03:44:38.372324+0000","last_became_active":"2024-03-12T03:44:24.379561+0000","last_became_peered":"2024-03-12T03:44:24.379561+0000","last_unstale":"2024-03-12T03:44:38.372324+0000","last_undegraded":"2024-03-12T03:44:38.372324+0000","last_fullsized":"2024-03-12T03:44:38.372324+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:02.551349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:02.551349+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:41.654619+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"299'5619","reported_seq":5668,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:41.140031+0000","last_change":"2024-03-12T03:43:55.403641+0000","last_active":"2024-03-12T03:44:41.140031+0000","last_peered":"2024-03-12T03:44:41.140031+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.592972+0000","last_became_peered":"2024-03-12T03:43:45.592972+0000","last_unstale":"2024-03-12T03:44:41.140031+0000","last_undegraded":"2024-03-12T03:43:55.398613+0000","last_fullsized":"2024-03-12T03:43:45.585435+0000","mapping_epoch":278,"log_start":"285'1200","ondisk_log_start":"285'1200","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4419,"log_dups_size":0,"ondisk_log_size":4419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:04.063684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368246784,"num_objects":5619,"num_object_clones":0,"num_object_copies":16857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5619,"num_write_kb":359616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":5619}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'51","reported_seq":598,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.371420+0000","last_change":"2024-03-12T03:43:58.539198+0000","last_active":"2024-03-12T03:44:38.371420+0000","last_peered":"2024-03-12T03:44:38.371420+0000","last_clean":"2024-03-12T03:44:38.371420+0000","last_became_active":"2024-03-12T03:33:23.551012+0000","last_became_peered":"2024-03-12T03:33:23.551012+0000","last_unstale":"2024-03-12T03:44:38.371420+0000","last_undegraded":"2024-03-12T03:44:38.371420+0000","last_fullsized":"2024-03-12T03:44:38.371420+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'51","last_scrub_stamp":"2024-03-12T03:43:58.539032+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:43:58.539032+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:59.041804+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":511,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.374143+0000","last_change":"2024-03-12T03:42:15.159053+0000","last_active":"2024-03-12T03:44:38.374143+0000","last_peered":"2024-03-12T03:44:38.374143+0000","last_clean":"2024-03-12T03:44:38.374143+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:44:38.374143+0000","last_undegraded":"2024-03-12T03:44:38.374143+0000","last_fullsized":"2024-03-12T03:44:38.374143+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:15.158916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:15.158916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":230,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.385536+0000","last_change":"2024-03-12T03:44:11.497903+0000","last_active":"2024-03-12T03:44:38.385536+0000","last_peered":"2024-03-12T03:44:38.385536+0000","last_clean":"2024-03-12T03:44:38.385536+0000","last_became_active":"2024-03-12T03:39:33.254395+0000","last_became_peered":"2024-03-12T03:39:33.254395+0000","last_unstale":"2024-03-12T03:44:38.385536+0000","last_undegraded":"2024-03-12T03:44:38.385536+0000","last_fullsized":"2024-03-12T03:44:38.385536+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_clean_scrub_stamp":"2024-03-12T03:44:11.497802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:33.269839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":523,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.371730+0000","last_change":"2024-03-12T03:43:18.339713+0000","last_active":"2024-03-12T03:44:38.371730+0000","last_peered":"2024-03-12T03:44:38.371730+0000","last_clean":"2024-03-12T03:43:15.079953+0000","last_became_active":"2024-03-12T03:43:18.339713+0000","last_became_peered":"2024-03-12T03:43:18.339713+0000","last_unstale":"2024-03-12T03:44:38.371730+0000","last_undegraded":"2024-03-12T03:44:38.371730+0000","last_fullsized":"2024-03-12T03:43:18.336066+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:15.079874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:15.079874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115530+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":421,"reported_epoch":299,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:44:38.363125+0000","last_change":"2024-03-12T03:44:24.371043+0000","last_active":"2024-03-12T03:44:38.363125+0000","last_peered":"2024-03-12T03:44:38.363125+0000","last_clean":"2024-03-12T03:44:38.363125+0000","last_became_active":"2024-03-12T03:44:24.370429+0000","last_became_peered":"2024-03-12T03:44:24.370429+0000","last_unstale":"2024-03-12T03:44:38.363125+0000","last_undegraded":"2024-03-12T03:44:38.363125+0000","last_fullsized":"2024-03-12T03:44:38.363125+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:02.442717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:02.442717+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:41.666115+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"299'5612","reported_seq":5661,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:42.075683+0000","last_change":"2024-03-12T03:43:55.386671+0000","last_active":"2024-03-12T03:44:42.075683+0000","last_peered":"2024-03-12T03:44:42.075683+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.590025+0000","last_became_peered":"2024-03-12T03:43:45.590025+0000","last_unstale":"2024-03-12T03:44:42.075683+0000","last_undegraded":"2024-03-12T03:43:55.378927+0000","last_fullsized":"2024-03-12T03:43:45.586649+0000","mapping_epoch":278,"log_start":"285'1200","ondisk_log_start":"285'1200","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:00.255405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367788032,"num_objects":5612,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5612,"num_write_kb":359168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5612}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":238,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.385089+0000","last_change":"2024-03-12T03:44:17.593848+0000","last_active":"2024-03-12T03:44:38.385089+0000","last_peered":"2024-03-12T03:44:38.385089+0000","last_clean":"2024-03-12T03:44:38.385089+0000","last_became_active":"2024-03-12T03:39:33.278326+0000","last_became_peered":"2024-03-12T03:39:33.278326+0000","last_unstale":"2024-03-12T03:44:38.385089+0000","last_undegraded":"2024-03-12T03:44:38.385089+0000","last_fullsized":"2024-03-12T03:44:38.385089+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:17.593753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:44:17.593753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:24.631360+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"269'30","reported_seq":575,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.359859+0000","last_change":"2024-03-12T03:43:18.341247+0000","last_active":"2024-03-12T03:44:38.359859+0000","last_peered":"2024-03-12T03:44:38.359859+0000","last_clean":"2024-03-12T03:43:15.540480+0000","last_became_active":"2024-03-12T03:43:18.341247+0000","last_became_peered":"2024-03-12T03:43:18.341247+0000","last_unstale":"2024-03-12T03:44:38.359859+0000","last_undegraded":"2024-03-12T03:43:18.330226+0000","last_fullsized":"2024-03-12T03:43:18.330117+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-03-12T03:43:15.540339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:15.540339+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.594572+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":519,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.379209+0000","last_change":"2024-03-12T03:44:05.063402+0000","last_active":"2024-03-12T03:44:38.379209+0000","last_peered":"2024-03-12T03:44:38.379209+0000","last_clean":"2024-03-12T03:44:38.379209+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:44:38.379209+0000","last_undegraded":"2024-03-12T03:44:38.379209+0000","last_fullsized":"2024-03-12T03:44:38.379209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:05.063185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:44:05.063185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:16.230730+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":278,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.372334+0000","last_change":"2024-03-12T03:43:18.345830+0000","last_active":"2024-03-12T03:44:38.372334+0000","last_peered":"2024-03-12T03:44:38.372334+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.345830+0000","last_became_peered":"2024-03-12T03:43:18.345830+0000","last_unstale":"2024-03-12T03:44:38.372334+0000","last_undegraded":"2024-03-12T03:44:38.372334+0000","last_fullsized":"2024-03-12T03:43:18.336109+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:18.651418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:18.651418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":431,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.373682+0000","last_change":"2024-03-12T03:44:24.374633+0000","last_active":"2024-03-12T03:44:38.373682+0000","last_peered":"2024-03-12T03:44:38.373682+0000","last_clean":"2024-03-12T03:44:38.373682+0000","last_became_active":"2024-03-12T03:44:24.373190+0000","last_became_peered":"2024-03-12T03:44:24.373190+0000","last_unstale":"2024-03-12T03:44:38.373682+0000","last_undegraded":"2024-03-12T03:44:38.373682+0000","last_fullsized":"2024-03-12T03:44:38.373682+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:04.664919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:04.664919+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:41.078922+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"299'5473","reported_seq":5525,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:41.597064+0000","last_change":"2024-03-12T03:43:45.593274+0000","last_active":"2024-03-12T03:44:41.597064+0000","last_peered":"2024-03-12T03:44:41.597064+0000","last_clean":"2024-03-12T03:44:41.597064+0000","last_became_active":"2024-03-12T03:43:45.593074+0000","last_became_peered":"2024-03-12T03:43:45.593074+0000","last_unstale":"2024-03-12T03:44:41.597064+0000","last_undegraded":"2024-03-12T03:44:41.597064+0000","last_fullsized":"2024-03-12T03:44:41.597064+0000","mapping_epoch":278,"log_start":"284'200","ondisk_log_start":"284'200","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":5273,"log_dups_size":0,"ondisk_log_size":5273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:11.927395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358678528,"num_objects":5473,"num_object_clones":0,"num_object_copies":16419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5473,"num_write_kb":350272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":648,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.375330+0000","last_change":"2024-03-12T03:43:23.985482+0000","last_active":"2024-03-12T03:44:38.375330+0000","last_peered":"2024-03-12T03:44:38.375330+0000","last_clean":"2024-03-12T03:44:38.375330+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:44:38.375330+0000","last_undegraded":"2024-03-12T03:44:38.375330+0000","last_fullsized":"2024-03-12T03:44:38.375330+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:43:23.985362+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115588+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"269'48","reported_seq":582,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.359352+0000","last_change":"2024-03-12T03:44:04.562483+0000","last_active":"2024-03-12T03:44:38.359352+0000","last_peered":"2024-03-12T03:44:38.359352+0000","last_clean":"2024-03-12T03:44:38.359352+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:44:38.359352+0000","last_undegraded":"2024-03-12T03:44:38.359352+0000","last_fullsized":"2024-03-12T03:44:38.359352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'48","last_scrub_stamp":"2024-03-12T03:44:04.562231+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:44:04.562231+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:04.977841+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":523,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.377058+0000","last_change":"2024-03-12T03:43:59.987280+0000","last_active":"2024-03-12T03:44:38.377058+0000","last_peered":"2024-03-12T03:44:38.377058+0000","last_clean":"2024-03-12T03:44:38.377058+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:44:38.377058+0000","last_undegraded":"2024-03-12T03:44:38.377058+0000","last_fullsized":"2024-03-12T03:44:38.377058+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:43:59.987096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:59.987096+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:19.340844+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":498,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.365200+0000","last_change":"2024-03-12T03:44:12.561600+0000","last_active":"2024-03-12T03:44:38.365200+0000","last_peered":"2024-03-12T03:44:38.365200+0000","last_clean":"2024-03-12T03:44:38.365200+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:44:38.365200+0000","last_undegraded":"2024-03-12T03:44:38.365200+0000","last_fullsized":"2024-03-12T03:44:38.365200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:12.561484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:44:12.561484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:26.689309+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":315,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363355+0000","last_change":"2024-03-12T03:44:24.369004+0000","last_active":"2024-03-12T03:44:38.363355+0000","last_peered":"2024-03-12T03:44:38.363355+0000","last_clean":"2024-03-12T03:44:38.363355+0000","last_became_active":"2024-03-12T03:44:24.365642+0000","last_became_peered":"2024-03-12T03:44:24.365642+0000","last_unstale":"2024-03-12T03:44:38.363355+0000","last_undegraded":"2024-03-12T03:44:38.363355+0000","last_fullsized":"2024-03-12T03:44:38.363355+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:16.082316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:16.082316+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.611359+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":9,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"269'18","reported_seq":165,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.378764+0000","last_change":"2024-03-12T03:43:18.350084+0000","last_active":"2024-03-12T03:44:38.378764+0000","last_peered":"2024-03-12T03:44:38.378764+0000","last_clean":"2024-03-12T03:33:28.523929+0000","last_became_active":"2024-03-12T03:43:18.350084+0000","last_became_peered":"2024-03-12T03:43:18.350084+0000","last_unstale":"2024-03-12T03:44:38.378764+0000","last_undegraded":"2024-03-12T03:43:18.336688+0000","last_fullsized":"2024-03-12T03:43:18.336568+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:43:17.648603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:17.648603+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":524,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.371094+0000","last_change":"2024-03-12T03:44:00.577286+0000","last_active":"2024-03-12T03:44:38.371094+0000","last_peered":"2024-03-12T03:44:38.371094+0000","last_clean":"2024-03-12T03:44:38.371094+0000","last_became_active":"2024-03-12T03:33:21.105781+0000","last_became_peered":"2024-03-12T03:33:21.105781+0000","last_unstale":"2024-03-12T03:44:38.371094+0000","last_undegraded":"2024-03-12T03:44:38.371094+0000","last_fullsized":"2024-03-12T03:44:38.371094+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:00.576797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:44:00.576797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:03.989585+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":505,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.362103+0000","last_change":"2024-03-12T03:43:18.348340+0000","last_active":"2024-03-12T03:44:38.362103+0000","last_peered":"2024-03-12T03:44:38.362103+0000","last_clean":"2024-03-12T03:43:13.352260+0000","last_became_active":"2024-03-12T03:43:18.348340+0000","last_became_peered":"2024-03-12T03:43:18.348340+0000","last_unstale":"2024-03-12T03:44:38.362103+0000","last_undegraded":"2024-03-12T03:44:38.362103+0000","last_fullsized":"2024-03-12T03:43:18.333554+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:01.417809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:01.417809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":506,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.373933+0000","last_change":"2024-03-12T03:44:03.085231+0000","last_active":"2024-03-12T03:44:38.373933+0000","last_peered":"2024-03-12T03:44:38.373933+0000","last_clean":"2024-03-12T03:44:38.373933+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:44:38.373933+0000","last_undegraded":"2024-03-12T03:44:38.373933+0000","last_fullsized":"2024-03-12T03:44:38.373933+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:03.083941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:03.083941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:21.686849+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":638,"reported_epoch":299,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:44:38.367134+0000","last_change":"2024-03-12T03:44:22.591356+0000","last_active":"2024-03-12T03:44:38.367134+0000","last_peered":"2024-03-12T03:44:38.367134+0000","last_clean":"2024-03-12T03:44:38.367134+0000","last_became_active":"2024-03-12T03:44:22.590882+0000","last_became_peered":"2024-03-12T03:44:22.590882+0000","last_unstale":"2024-03-12T03:44:38.367134+0000","last_undegraded":"2024-03-12T03:44:38.367134+0000","last_fullsized":"2024-03-12T03:44:38.367134+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:19.738048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:19.738048+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.145553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"269'27","reported_seq":559,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.363673+0000","last_change":"2024-03-12T03:43:18.358476+0000","last_active":"2024-03-12T03:44:38.363673+0000","last_peered":"2024-03-12T03:44:38.363673+0000","last_clean":"2024-03-12T03:43:13.333279+0000","last_became_active":"2024-03-12T03:43:18.358476+0000","last_became_peered":"2024-03-12T03:43:18.358476+0000","last_unstale":"2024-03-12T03:44:38.363673+0000","last_undegraded":"2024-03-12T03:43:18.343811+0000","last_fullsized":"2024-03-12T03:43:18.343694+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:43:03.583989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:03.583989+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675625+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":510,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.375001+0000","last_change":"2024-03-12T03:42:22.126915+0000","last_active":"2024-03-12T03:44:38.375001+0000","last_peered":"2024-03-12T03:44:38.375001+0000","last_clean":"2024-03-12T03:44:38.375001+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:44:38.375001+0000","last_undegraded":"2024-03-12T03:44:38.375001+0000","last_fullsized":"2024-03-12T03:44:38.375001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:22.126792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:22.126792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115503+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":310,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.368591+0000","last_change":"2024-03-12T03:43:18.346246+0000","last_active":"2024-03-12T03:44:38.368591+0000","last_peered":"2024-03-12T03:44:38.368591+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:43:18.346246+0000","last_became_peered":"2024-03-12T03:43:18.346246+0000","last_unstale":"2024-03-12T03:44:38.368591+0000","last_undegraded":"2024-03-12T03:44:38.368591+0000","last_fullsized":"2024-03-12T03:43:18.335155+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:05.530729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:43:05.530729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":230,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.386590+0000","last_change":"2024-03-12T03:43:03.293930+0000","last_active":"2024-03-12T03:44:38.386590+0000","last_peered":"2024-03-12T03:44:38.386590+0000","last_clean":"2024-03-12T03:44:38.386590+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:44:38.386590+0000","last_undegraded":"2024-03-12T03:44:38.386590+0000","last_fullsized":"2024-03-12T03:44:38.386590+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:03.293463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:03.293463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:42.486290+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":315,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.359191+0000","last_change":"2024-03-12T03:44:24.376476+0000","last_active":"2024-03-12T03:44:38.359191+0000","last_peered":"2024-03-12T03:44:38.359191+0000","last_clean":"2024-03-12T03:44:38.359191+0000","last_became_active":"2024-03-12T03:44:24.376141+0000","last_became_peered":"2024-03-12T03:44:24.376141+0000","last_unstale":"2024-03-12T03:44:38.359191+0000","last_undegraded":"2024-03-12T03:44:38.359191+0000","last_fullsized":"2024-03-12T03:44:38.359191+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:05.222876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:05.222876+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.611391+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"269'21","reported_seq":554,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.362858+0000","last_change":"2024-03-12T03:43:18.347815+0000","last_active":"2024-03-12T03:44:38.362858+0000","last_peered":"2024-03-12T03:44:38.362858+0000","last_clean":"2024-03-12T03:43:13.353750+0000","last_became_active":"2024-03-12T03:43:18.347815+0000","last_became_peered":"2024-03-12T03:43:18.347815+0000","last_unstale":"2024-03-12T03:44:38.362858+0000","last_undegraded":"2024-03-12T03:43:18.332425+0000","last_fullsized":"2024-03-12T03:43:18.332310+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:40:58.312602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:58.312602+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":514,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.362613+0000","last_change":"2024-03-12T03:43:18.343018+0000","last_active":"2024-03-12T03:44:38.362613+0000","last_peered":"2024-03-12T03:44:38.362613+0000","last_clean":"2024-03-12T03:43:13.331412+0000","last_became_active":"2024-03-12T03:43:18.343018+0000","last_became_peered":"2024-03-12T03:43:18.343018+0000","last_unstale":"2024-03-12T03:44:38.362613+0000","last_undegraded":"2024-03-12T03:44:38.362613+0000","last_fullsized":"2024-03-12T03:43:18.331686+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:00.266512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:42:00.266512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675573+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"266'13","reported_seq":174,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.378098+0000","last_change":"2024-03-12T03:43:18.350782+0000","last_active":"2024-03-12T03:44:38.378098+0000","last_peered":"2024-03-12T03:44:38.378098+0000","last_clean":"2024-03-12T03:33:28.372811+0000","last_became_active":"2024-03-12T03:43:18.350782+0000","last_became_peered":"2024-03-12T03:43:18.350782+0000","last_unstale":"2024-03-12T03:44:38.378098+0000","last_undegraded":"2024-03-12T03:43:18.340626+0000","last_fullsized":"2024-03-12T03:43:18.340503+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'12","last_scrub_stamp":"2024-03-12T03:43:12.675070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:12.675070+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":518,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.362953+0000","last_change":"2024-03-12T03:44:08.917228+0000","last_active":"2024-03-12T03:44:38.362953+0000","last_peered":"2024-03-12T03:44:38.362953+0000","last_clean":"2024-03-12T03:44:38.362953+0000","last_became_active":"2024-03-12T03:39:33.265499+0000","last_became_peered":"2024-03-12T03:39:33.265499+0000","last_unstale":"2024-03-12T03:44:38.362953+0000","last_undegraded":"2024-03-12T03:44:38.362953+0000","last_fullsized":"2024-03-12T03:44:38.362953+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:08.917119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:08.917119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:13.594453+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":326,"reported_epoch":299,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:44:38.364362+0000","last_change":"2024-03-12T03:44:22.594970+0000","last_active":"2024-03-12T03:44:38.364362+0000","last_peered":"2024-03-12T03:44:38.364362+0000","last_clean":"2024-03-12T03:44:38.364362+0000","last_became_active":"2024-03-12T03:44:22.594212+0000","last_became_peered":"2024-03-12T03:44:22.594212+0000","last_unstale":"2024-03-12T03:44:38.364362+0000","last_undegraded":"2024-03-12T03:44:38.364362+0000","last_fullsized":"2024-03-12T03:44:38.364362+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:28.720248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:28.720248+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.659595+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":9,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,2],"avail_no_missing":["0","2","5"],"object_location_counts":[{"shards":"0,2,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"269'32","reported_seq":474,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.385829+0000","last_change":"2024-03-12T03:43:18.357702+0000","last_active":"2024-03-12T03:44:38.385829+0000","last_peered":"2024-03-12T03:44:38.385829+0000","last_clean":"2024-03-12T03:42:47.830157+0000","last_became_active":"2024-03-12T03:43:18.357702+0000","last_became_peered":"2024-03-12T03:43:18.357702+0000","last_unstale":"2024-03-12T03:44:38.385829+0000","last_undegraded":"2024-03-12T03:43:18.340096+0000","last_fullsized":"2024-03-12T03:43:18.339982+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-03-12T03:35:58.079645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.079645+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:42.486239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":545,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.371916+0000","last_change":"2024-03-12T03:44:13.063349+0000","last_active":"2024-03-12T03:44:38.371916+0000","last_peered":"2024-03-12T03:44:38.371916+0000","last_clean":"2024-03-12T03:44:38.371916+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:44:38.371916+0000","last_undegraded":"2024-03-12T03:44:38.371916+0000","last_fullsized":"2024-03-12T03:44:38.371916+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:44:13.060719+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:40.549571+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":645,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.362728+0000","last_change":"2024-03-12T03:44:05.557046+0000","last_active":"2024-03-12T03:44:38.362728+0000","last_peered":"2024-03-12T03:44:38.362728+0000","last_clean":"2024-03-12T03:44:38.362728+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:44:38.362728+0000","last_undegraded":"2024-03-12T03:44:38.362728+0000","last_fullsized":"2024-03-12T03:44:38.362728+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:44:05.556852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:44:05.556852+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:09.214886+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":502,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.363388+0000","last_change":"2024-03-12T03:43:18.357954+0000","last_active":"2024-03-12T03:44:38.363388+0000","last_peered":"2024-03-12T03:44:38.363388+0000","last_clean":"2024-03-12T03:43:13.332853+0000","last_became_active":"2024-03-12T03:43:18.357954+0000","last_became_peered":"2024-03-12T03:43:18.357954+0000","last_unstale":"2024-03-12T03:44:38.363388+0000","last_undegraded":"2024-03-12T03:44:38.363388+0000","last_fullsized":"2024-03-12T03:43:18.341976+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:10.355198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:10.355198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675605+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"269'19","reported_seq":545,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.360763+0000","last_change":"2024-03-12T03:43:22.673154+0000","last_active":"2024-03-12T03:44:38.360763+0000","last_peered":"2024-03-12T03:44:38.360763+0000","last_clean":"2024-03-12T03:44:38.360763+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:44:38.360763+0000","last_undegraded":"2024-03-12T03:44:38.360763+0000","last_fullsized":"2024-03-12T03:44:38.360763+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-12T03:43:22.673042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:22.673042+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715919+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":518,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363477+0000","last_change":"2024-03-12T03:43:58.853550+0000","last_active":"2024-03-12T03:44:38.363477+0000","last_peered":"2024-03-12T03:44:38.363477+0000","last_clean":"2024-03-12T03:44:38.363477+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:44:38.363477+0000","last_undegraded":"2024-03-12T03:44:38.363477+0000","last_fullsized":"2024-03-12T03:44:38.363477+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:58.853429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:38:34.264717+0000","last_clean_scrub_stamp":"2024-03-12T03:43:58.853429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:08.981265+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":514,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.370143+0000","last_change":"2024-03-12T03:43:18.347925+0000","last_active":"2024-03-12T03:44:38.370143+0000","last_peered":"2024-03-12T03:44:38.370143+0000","last_clean":"2024-03-12T03:43:13.351418+0000","last_became_active":"2024-03-12T03:43:18.347925+0000","last_became_peered":"2024-03-12T03:43:18.347925+0000","last_unstale":"2024-03-12T03:44:38.370143+0000","last_undegraded":"2024-03-12T03:44:38.370143+0000","last_fullsized":"2024-03-12T03:43:18.334927+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:35.850271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:35.850271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":328,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.368356+0000","last_change":"2024-03-12T03:43:18.350340+0000","last_active":"2024-03-12T03:44:38.368356+0000","last_peered":"2024-03-12T03:44:38.368356+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.350340+0000","last_became_peered":"2024-03-12T03:43:18.350340+0000","last_unstale":"2024-03-12T03:44:38.368356+0000","last_undegraded":"2024-03-12T03:44:38.368356+0000","last_fullsized":"2024-03-12T03:43:18.332506+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.075897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.075897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"269'15","reported_seq":110,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.364956+0000","last_change":"2024-03-12T03:44:14.462001+0000","last_active":"2024-03-12T03:44:38.364956+0000","last_peered":"2024-03-12T03:44:38.364956+0000","last_clean":"2024-03-12T03:44:38.364956+0000","last_became_active":"2024-03-12T03:43:01.037348+0000","last_became_peered":"2024-03-12T03:43:01.037348+0000","last_unstale":"2024-03-12T03:44:38.364956+0000","last_undegraded":"2024-03-12T03:44:38.364956+0000","last_fullsized":"2024-03-12T03:44:38.364956+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'15","last_scrub_stamp":"2024-03-12T03:44:14.461706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:44:14.461706+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:42.352348+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":522,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.373505+0000","last_change":"2024-03-12T03:43:18.347900+0000","last_active":"2024-03-12T03:44:38.373505+0000","last_peered":"2024-03-12T03:44:38.373505+0000","last_clean":"2024-03-12T03:43:13.342510+0000","last_became_active":"2024-03-12T03:43:18.347900+0000","last_became_peered":"2024-03-12T03:43:18.347900+0000","last_unstale":"2024-03-12T03:44:38.373505+0000","last_undegraded":"2024-03-12T03:44:38.373505+0000","last_fullsized":"2024-03-12T03:43:18.337740+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:08.131514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:43:08.131514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":340,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.363437+0000","last_change":"2024-03-12T03:43:18.343467+0000","last_active":"2024-03-12T03:44:38.363437+0000","last_peered":"2024-03-12T03:44:38.363437+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:43:18.343467+0000","last_became_peered":"2024-03-12T03:43:18.343467+0000","last_unstale":"2024-03-12T03:44:38.363437+0000","last_undegraded":"2024-03-12T03:44:38.363437+0000","last_fullsized":"2024-03-12T03:43:18.332412+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:15.808887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:15.808887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":498,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.362444+0000","last_change":"2024-03-12T03:43:04.613657+0000","last_active":"2024-03-12T03:44:38.362444+0000","last_peered":"2024-03-12T03:44:38.362444+0000","last_clean":"2024-03-12T03:44:38.362444+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:44:38.362444+0000","last_undegraded":"2024-03-12T03:44:38.362444+0000","last_fullsized":"2024-03-12T03:44:38.362444+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:04.613384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:43:04.613384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675645+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":420,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.385134+0000","last_change":"2024-03-12T03:44:24.387594+0000","last_active":"2024-03-12T03:44:38.385134+0000","last_peered":"2024-03-12T03:44:38.385134+0000","last_clean":"2024-03-12T03:44:38.385134+0000","last_became_active":"2024-03-12T03:44:24.387247+0000","last_became_peered":"2024-03-12T03:44:24.387247+0000","last_unstale":"2024-03-12T03:44:38.385134+0000","last_undegraded":"2024-03-12T03:44:38.385134+0000","last_fullsized":"2024-03-12T03:44:38.385134+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:02.442717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:02.442717+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:41.438871+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"269'26","reported_seq":179,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.375144+0000","last_change":"2024-03-12T03:43:18.353670+0000","last_active":"2024-03-12T03:44:38.375144+0000","last_peered":"2024-03-12T03:44:38.375144+0000","last_clean":"2024-03-12T03:33:28.559608+0000","last_became_active":"2024-03-12T03:43:18.353670+0000","last_became_peered":"2024-03-12T03:43:18.353670+0000","last_unstale":"2024-03-12T03:44:38.375144+0000","last_undegraded":"2024-03-12T03:43:18.338840+0000","last_fullsized":"2024-03-12T03:43:18.338735+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:43:06.549441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:06.549441+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":509,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.362268+0000","last_change":"2024-03-12T03:43:27.499252+0000","last_active":"2024-03-12T03:44:38.362268+0000","last_peered":"2024-03-12T03:44:38.362268+0000","last_clean":"2024-03-12T03:44:38.362268+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:44:38.362268+0000","last_undegraded":"2024-03-12T03:44:38.362268+0000","last_fullsized":"2024-03-12T03:44:38.362268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:27.499165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:43:27.499165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":502,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.372526+0000","last_change":"2024-03-12T03:43:13.075342+0000","last_active":"2024-03-12T03:44:38.372526+0000","last_peered":"2024-03-12T03:44:38.372526+0000","last_clean":"2024-03-12T03:44:38.372526+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:44:38.372526+0000","last_undegraded":"2024-03-12T03:44:38.372526+0000","last_fullsized":"2024-03-12T03:44:38.372526+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:13.075266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:13.075266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":363,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.362983+0000","last_change":"2024-03-12T03:43:18.344649+0000","last_active":"2024-03-12T03:44:38.362983+0000","last_peered":"2024-03-12T03:44:38.362983+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.344649+0000","last_became_peered":"2024-03-12T03:43:18.344649+0000","last_unstale":"2024-03-12T03:44:38.362983+0000","last_undegraded":"2024-03-12T03:44:38.362983+0000","last_fullsized":"2024-03-12T03:43:18.331783+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"269'10","reported_seq":184,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.368586+0000","last_change":"2024-03-12T03:43:18.337555+0000","last_active":"2024-03-12T03:44:38.368586+0000","last_peered":"2024-03-12T03:44:38.368586+0000","last_clean":"2024-03-12T03:34:34.272872+0000","last_became_active":"2024-03-12T03:43:18.337555+0000","last_became_peered":"2024-03-12T03:43:18.337555+0000","last_unstale":"2024-03-12T03:44:38.368586+0000","last_undegraded":"2024-03-12T03:43:18.333971+0000","last_fullsized":"2024-03-12T03:43:18.333854+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:38:38.943094+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:38:38.943094+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":514,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.363671+0000","last_change":"2024-03-12T03:43:18.349342+0000","last_active":"2024-03-12T03:44:38.363671+0000","last_peered":"2024-03-12T03:44:38.363671+0000","last_clean":"2024-03-12T03:43:13.332505+0000","last_became_active":"2024-03-12T03:43:18.349342+0000","last_became_peered":"2024-03-12T03:43:18.349342+0000","last_unstale":"2024-03-12T03:44:38.363671+0000","last_undegraded":"2024-03-12T03:44:38.363671+0000","last_fullsized":"2024-03-12T03:43:18.335423+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:19.511683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:39:19.511683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":519,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.373551+0000","last_change":"2024-03-12T03:43:57.555296+0000","last_active":"2024-03-12T03:44:38.373551+0000","last_peered":"2024-03-12T03:44:38.373551+0000","last_clean":"2024-03-12T03:44:38.373551+0000","last_became_active":"2024-03-12T03:33:25.173118+0000","last_became_peered":"2024-03-12T03:33:25.173118+0000","last_unstale":"2024-03-12T03:44:38.373551+0000","last_undegraded":"2024-03-12T03:44:38.373551+0000","last_fullsized":"2024-03-12T03:44:38.373551+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:57.555182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:57.555182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:59.844916+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":510,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.373382+0000","last_change":"2024-03-12T03:43:28.028114+0000","last_active":"2024-03-12T03:44:38.373382+0000","last_peered":"2024-03-12T03:44:38.373382+0000","last_clean":"2024-03-12T03:44:38.373382+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:44:38.373382+0000","last_undegraded":"2024-03-12T03:44:38.373382+0000","last_fullsized":"2024-03-12T03:44:38.373382+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:28.028020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:28.028020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115672+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"269'31","reported_seq":562,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.372678+0000","last_change":"2024-03-12T03:43:03.536079+0000","last_active":"2024-03-12T03:44:38.372678+0000","last_peered":"2024-03-12T03:44:38.372678+0000","last_clean":"2024-03-12T03:44:38.372678+0000","last_became_active":"2024-03-12T03:33:23.158755+0000","last_became_peered":"2024-03-12T03:33:23.158755+0000","last_unstale":"2024-03-12T03:44:38.372678+0000","last_undegraded":"2024-03-12T03:44:38.372678+0000","last_fullsized":"2024-03-12T03:44:38.372678+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'23","last_scrub_stamp":"2024-03-12T03:43:03.535926+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:43:03.535926+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.656366+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":124,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.362585+0000","last_change":"2024-03-12T03:43:18.342446+0000","last_active":"2024-03-12T03:44:38.362585+0000","last_peered":"2024-03-12T03:44:38.362585+0000","last_clean":"2024-03-12T03:33:21.096389+0000","last_became_active":"2024-03-12T03:43:18.342446+0000","last_became_peered":"2024-03-12T03:43:18.342446+0000","last_unstale":"2024-03-12T03:44:38.362585+0000","last_undegraded":"2024-03-12T03:43:18.333444+0000","last_fullsized":"2024-03-12T03:43:18.333330+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:25.092313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.092313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":515,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363951+0000","last_change":"2024-03-12T03:44:13.704463+0000","last_active":"2024-03-12T03:44:38.363951+0000","last_peered":"2024-03-12T03:44:38.363951+0000","last_clean":"2024-03-12T03:44:38.363951+0000","last_became_active":"2024-03-12T03:39:33.276435+0000","last_became_peered":"2024-03-12T03:39:33.276435+0000","last_unstale":"2024-03-12T03:44:38.363951+0000","last_undegraded":"2024-03-12T03:44:38.363951+0000","last_fullsized":"2024-03-12T03:44:38.363951+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:13.704331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:44:13.704331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:28.071520+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":505,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.368300+0000","last_change":"2024-03-12T03:43:27.868017+0000","last_active":"2024-03-12T03:44:38.368300+0000","last_peered":"2024-03-12T03:44:38.368300+0000","last_clean":"2024-03-12T03:44:38.368300+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:44:38.368300+0000","last_undegraded":"2024-03-12T03:44:38.368300+0000","last_fullsized":"2024-03-12T03:44:38.368300+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:27.867900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:27.867900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"269'27","reported_seq":554,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.363741+0000","last_change":"2024-03-12T03:43:18.358106+0000","last_active":"2024-03-12T03:44:38.363741+0000","last_peered":"2024-03-12T03:44:38.363741+0000","last_clean":"2024-03-12T03:43:13.333148+0000","last_became_active":"2024-03-12T03:43:18.358106+0000","last_became_peered":"2024-03-12T03:43:18.358106+0000","last_unstale":"2024-03-12T03:44:38.363741+0000","last_undegraded":"2024-03-12T03:43:18.343380+0000","last_fullsized":"2024-03-12T03:43:18.343203+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:39:37.572971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:37.572971+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":523,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.367206+0000","last_change":"2024-03-12T03:43:18.350788+0000","last_active":"2024-03-12T03:44:38.367206+0000","last_peered":"2024-03-12T03:44:38.367206+0000","last_clean":"2024-03-12T03:43:13.349345+0000","last_became_active":"2024-03-12T03:43:18.350788+0000","last_became_peered":"2024-03-12T03:43:18.350788+0000","last_unstale":"2024-03-12T03:44:38.367206+0000","last_undegraded":"2024-03-12T03:43:18.332266+0000","last_fullsized":"2024-03-12T03:43:18.332143+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:43:07.772205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:07.772205+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":656,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.372595+0000","last_change":"2024-03-12T03:43:17.506932+0000","last_active":"2024-03-12T03:44:38.372595+0000","last_peered":"2024-03-12T03:44:38.372595+0000","last_clean":"2024-03-12T03:44:38.372595+0000","last_became_active":"2024-03-12T03:33:25.173844+0000","last_became_peered":"2024-03-12T03:33:25.173844+0000","last_unstale":"2024-03-12T03:44:38.372595+0000","last_undegraded":"2024-03-12T03:44:38.372595+0000","last_fullsized":"2024-03-12T03:44:38.372595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:43:17.506791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:17.506791+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:38.384185+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":527,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.378281+0000","last_change":"2024-03-12T03:43:59.015594+0000","last_active":"2024-03-12T03:44:38.378281+0000","last_peered":"2024-03-12T03:44:38.378281+0000","last_clean":"2024-03-12T03:44:38.378281+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:44:38.378281+0000","last_undegraded":"2024-03-12T03:44:38.378281+0000","last_fullsized":"2024-03-12T03:44:38.378281+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:43:59.015445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:59.015445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:28.367042+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"269'24","reported_seq":179,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.368108+0000","last_change":"2024-03-12T03:43:18.346292+0000","last_active":"2024-03-12T03:44:38.368108+0000","last_peered":"2024-03-12T03:44:38.368108+0000","last_clean":"2024-03-12T03:33:28.567700+0000","last_became_active":"2024-03-12T03:43:18.346292+0000","last_became_peered":"2024-03-12T03:43:18.346292+0000","last_unstale":"2024-03-12T03:44:38.368108+0000","last_undegraded":"2024-03-12T03:43:18.333553+0000","last_fullsized":"2024-03-12T03:43:18.333434+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:36:28.133261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.133261+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":530,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.359891+0000","last_change":"2024-03-12T03:42:18.499466+0000","last_active":"2024-03-12T03:44:38.359891+0000","last_peered":"2024-03-12T03:44:38.359891+0000","last_clean":"2024-03-12T03:44:38.359891+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:44:38.359891+0000","last_undegraded":"2024-03-12T03:44:38.359891+0000","last_fullsized":"2024-03-12T03:44:38.359891+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:18.499342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:42:18.499342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715774+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":526,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.363772+0000","last_change":"2024-03-12T03:43:18.351475+0000","last_active":"2024-03-12T03:44:38.363772+0000","last_peered":"2024-03-12T03:44:38.363772+0000","last_clean":"2024-03-12T03:43:13.338616+0000","last_became_active":"2024-03-12T03:43:18.351475+0000","last_became_peered":"2024-03-12T03:43:18.351475+0000","last_unstale":"2024-03-12T03:44:38.363772+0000","last_undegraded":"2024-03-12T03:44:38.363772+0000","last_fullsized":"2024-03-12T03:43:18.332508+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:08.289718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:08.289718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715627+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":356,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.372812+0000","last_change":"2024-03-12T03:43:18.349876+0000","last_active":"2024-03-12T03:44:38.372812+0000","last_peered":"2024-03-12T03:44:38.372812+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.349876+0000","last_became_peered":"2024-03-12T03:43:18.349876+0000","last_unstale":"2024-03-12T03:44:38.372812+0000","last_undegraded":"2024-03-12T03:44:38.372812+0000","last_fullsized":"2024-03-12T03:43:18.334950+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:16.588551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:16.588551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.656320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"269'27","reported_seq":268,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.386544+0000","last_change":"2024-03-12T03:43:18.338861+0000","last_active":"2024-03-12T03:44:38.386544+0000","last_peered":"2024-03-12T03:44:38.386544+0000","last_clean":"2024-03-12T03:43:13.346680+0000","last_became_active":"2024-03-12T03:43:18.338861+0000","last_became_peered":"2024-03-12T03:43:18.338861+0000","last_unstale":"2024-03-12T03:44:38.386544+0000","last_undegraded":"2024-03-12T03:43:18.337001+0000","last_fullsized":"2024-03-12T03:43:18.336879+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:42:04.171114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:42:04.171114+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:42.486250+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":510,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363602+0000","last_change":"2024-03-12T03:43:10.670801+0000","last_active":"2024-03-12T03:44:38.363602+0000","last_peered":"2024-03-12T03:44:38.363602+0000","last_clean":"2024-03-12T03:44:38.363602+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:44:38.363602+0000","last_undegraded":"2024-03-12T03:44:38.363602+0000","last_fullsized":"2024-03-12T03:44:38.363602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:10.670685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:10.670685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.675681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":509,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.368154+0000","last_change":"2024-03-12T03:43:16.842242+0000","last_active":"2024-03-12T03:44:38.368154+0000","last_peered":"2024-03-12T03:44:38.368154+0000","last_clean":"2024-03-12T03:44:38.368154+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:44:38.368154+0000","last_undegraded":"2024-03-12T03:44:38.368154+0000","last_fullsized":"2024-03-12T03:44:38.368154+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:16.842126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:16.842126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.283787+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":499,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.364762+0000","last_change":"2024-03-12T03:42:20.532434+0000","last_active":"2024-03-12T03:44:38.364762+0000","last_peered":"2024-03-12T03:44:38.364762+0000","last_clean":"2024-03-12T03:44:38.364762+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:44:38.364762+0000","last_undegraded":"2024-03-12T03:44:38.364762+0000","last_fullsized":"2024-03-12T03:44:38.364762+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:20.532318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:20.532318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715744+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"269'16","reported_seq":582,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.360517+0000","last_change":"2024-03-12T03:43:18.345649+0000","last_active":"2024-03-12T03:44:38.360517+0000","last_peered":"2024-03-12T03:44:38.360517+0000","last_clean":"2024-03-12T03:43:13.343489+0000","last_became_active":"2024-03-12T03:43:18.345649+0000","last_became_peered":"2024-03-12T03:43:18.345649+0000","last_unstale":"2024-03-12T03:44:38.360517+0000","last_undegraded":"2024-03-12T03:43:18.330821+0000","last_fullsized":"2024-03-12T03:43:18.330708+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:43:10.580243+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:43:10.580243+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715856+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":509,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.360160+0000","last_change":"2024-03-12T03:44:06.537101+0000","last_active":"2024-03-12T03:44:38.360160+0000","last_peered":"2024-03-12T03:44:38.360160+0000","last_clean":"2024-03-12T03:44:38.360160+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:44:38.360160+0000","last_undegraded":"2024-03-12T03:44:38.360160+0000","last_fullsized":"2024-03-12T03:44:38.360160+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:06.536953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:44:06.536953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:34.764219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":502,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.373287+0000","last_change":"2024-03-12T03:43:22.975945+0000","last_active":"2024-03-12T03:44:38.373287+0000","last_peered":"2024-03-12T03:44:38.373287+0000","last_clean":"2024-03-12T03:44:38.373287+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:44:38.373287+0000","last_undegraded":"2024-03-12T03:44:38.373287+0000","last_fullsized":"2024-03-12T03:44:38.373287+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:43:22.975800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":501,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.360857+0000","last_change":"2024-03-12T03:43:18.348827+0000","last_active":"2024-03-12T03:44:38.360857+0000","last_peered":"2024-03-12T03:44:38.360857+0000","last_clean":"2024-03-12T03:43:13.354323+0000","last_became_active":"2024-03-12T03:43:18.348827+0000","last_became_peered":"2024-03-12T03:43:18.348827+0000","last_unstale":"2024-03-12T03:44:38.360857+0000","last_undegraded":"2024-03-12T03:44:38.360857+0000","last_fullsized":"2024-03-12T03:43:18.333911+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:15.648255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:15.648255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"269'27","reported_seq":567,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.359702+0000","last_change":"2024-03-12T03:43:18.340176+0000","last_active":"2024-03-12T03:44:38.359702+0000","last_peered":"2024-03-12T03:44:38.359702+0000","last_clean":"2024-03-12T03:43:16.563196+0000","last_became_active":"2024-03-12T03:43:18.340176+0000","last_became_peered":"2024-03-12T03:43:18.340176+0000","last_unstale":"2024-03-12T03:44:38.359702+0000","last_undegraded":"2024-03-12T03:43:18.332765+0000","last_fullsized":"2024-03-12T03:43:18.330467+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:43:16.563042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:16.563042+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715936+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":509,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.362647+0000","last_change":"2024-03-12T03:44:13.493021+0000","last_active":"2024-03-12T03:44:38.362647+0000","last_peered":"2024-03-12T03:44:38.362647+0000","last_clean":"2024-03-12T03:44:38.362647+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:44:38.362647+0000","last_undegraded":"2024-03-12T03:44:38.362647+0000","last_fullsized":"2024-03-12T03:44:38.362647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:13.492893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:44:13.492893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:14.124373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":501,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363950+0000","last_change":"2024-03-12T03:44:16.541523+0000","last_active":"2024-03-12T03:44:38.363950+0000","last_peered":"2024-03-12T03:44:38.363950+0000","last_clean":"2024-03-12T03:44:38.363950+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:44:38.363950+0000","last_undegraded":"2024-03-12T03:44:38.363950+0000","last_fullsized":"2024-03-12T03:44:38.363950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_clean_scrub_stamp":"2024-03-12T03:44:16.541173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:30.379373+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":498,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.362666+0000","last_change":"2024-03-12T03:44:10.763845+0000","last_active":"2024-03-12T03:44:38.362666+0000","last_peered":"2024-03-12T03:44:38.362666+0000","last_clean":"2024-03-12T03:44:38.362666+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:44:38.362666+0000","last_undegraded":"2024-03-12T03:44:38.362666+0000","last_fullsized":"2024-03-12T03:44:38.362666+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:10.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:10.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:25.096723+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"269'62","reported_seq":524,"reported_epoch":299,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:44:38.385203+0000","last_change":"2024-03-12T03:43:18.343462+0000","last_active":"2024-03-12T03:44:38.385203+0000","last_peered":"2024-03-12T03:44:38.385203+0000","last_clean":"2024-03-12T03:39:28.242684+0000","last_became_active":"2024-03-12T03:43:18.343462+0000","last_became_peered":"2024-03-12T03:43:18.343462+0000","last_unstale":"2024-03-12T03:44:38.385203+0000","last_undegraded":"2024-03-12T03:43:18.336907+0000","last_fullsized":"2024-03-12T03:43:18.336787+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:42.486299+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":410,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.371799+0000","last_change":"2024-03-12T03:43:18.348499+0000","last_active":"2024-03-12T03:44:38.371799+0000","last_peered":"2024-03-12T03:44:38.371799+0000","last_clean":"2024-03-12T03:39:28.243893+0000","last_became_active":"2024-03-12T03:43:18.348499+0000","last_became_peered":"2024-03-12T03:43:18.348499+0000","last_unstale":"2024-03-12T03:44:38.371799+0000","last_undegraded":"2024-03-12T03:44:38.371799+0000","last_fullsized":"2024-03-12T03:43:18.337375+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:26.898782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:39:26.898782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:39.656332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":502,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.361157+0000","last_change":"2024-03-12T03:42:09.304630+0000","last_active":"2024-03-12T03:44:38.361157+0000","last_peered":"2024-03-12T03:44:38.361157+0000","last_clean":"2024-03-12T03:44:38.361157+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:44:38.361157+0000","last_undegraded":"2024-03-12T03:44:38.361157+0000","last_fullsized":"2024-03-12T03:44:38.361157+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:09.304528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:42:09.304528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":334,"reported_epoch":299,"state":"active+undersized","last_fresh":"2024-03-12T03:44:38.372116+0000","last_change":"2024-03-12T03:43:18.353114+0000","last_active":"2024-03-12T03:44:38.372116+0000","last_peered":"2024-03-12T03:44:38.372116+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.353114+0000","last_became_peered":"2024-03-12T03:43:18.353114+0000","last_unstale":"2024-03-12T03:44:38.372116+0000","last_undegraded":"2024-03-12T03:44:38.372116+0000","last_fullsized":"2024-03-12T03:43:18.337255+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.236997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.236997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.115394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"269'26","reported_seq":548,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.361113+0000","last_change":"2024-03-12T03:43:02.651857+0000","last_active":"2024-03-12T03:44:38.361113+0000","last_peered":"2024-03-12T03:44:38.361113+0000","last_clean":"2024-03-12T03:44:38.361113+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:44:38.361113+0000","last_undegraded":"2024-03-12T03:44:38.361113+0000","last_fullsized":"2024-03-12T03:44:38.361113+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'20","last_scrub_stamp":"2024-03-12T03:43:02.651729+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:43:02.651729+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715809+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":523,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.363512+0000","last_change":"2024-03-12T03:44:14.713075+0000","last_active":"2024-03-12T03:44:38.363512+0000","last_peered":"2024-03-12T03:44:38.363512+0000","last_clean":"2024-03-12T03:44:38.363512+0000","last_became_active":"2024-03-12T03:39:33.276119+0000","last_became_peered":"2024-03-12T03:39:33.276119+0000","last_unstale":"2024-03-12T03:44:38.363512+0000","last_undegraded":"2024-03-12T03:44:38.363512+0000","last_fullsized":"2024-03-12T03:44:38.363512+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:44:14.712916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:37.961917+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":518,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.367413+0000","last_change":"2024-03-12T03:43:54.998904+0000","last_active":"2024-03-12T03:44:38.367413+0000","last_peered":"2024-03-12T03:44:38.367413+0000","last_clean":"2024-03-12T03:44:38.367413+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:44:38.367413+0000","last_undegraded":"2024-03-12T03:44:38.367413+0000","last_fullsized":"2024-03-12T03:44:38.367413+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:54.998817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:26.625326+0000","last_clean_scrub_stamp":"2024-03-12T03:43:54.998817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:19.507118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":506,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-03-12T03:44:38.359388+0000","last_change":"2024-03-12T03:43:18.631145+0000","last_active":"2024-03-12T03:44:38.359388+0000","last_peered":"2024-03-12T03:44:38.359388+0000","last_clean":"2024-03-12T03:44:38.359388+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:44:38.359388+0000","last_undegraded":"2024-03-12T03:44:38.359388+0000","last_fullsized":"2024-03-12T03:44:38.359388+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:18.631026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.063092+0000","last_clean_scrub_stamp":"2024-03-12T03:43:18.631026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:44:40.715884+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5856428032,"num_objects":89362,"num_object_clones":0,"num_object_copies":268086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":44784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89362,"num_write_kb":5719168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14660075520,"data_stored":14660075520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75062,"ondisk_log_size":75062,"up":40,"acting":40,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":673,"num_read_kb":668,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":423698,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":82,"num_write":50,"num_write_kb":660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1008406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":54,"acting":58,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1409,"num_read_kb":1134,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":2284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3734,"internal_metadata":0},"log_size":944,"ondisk_log_size":944,"up":79,"acting":79,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070605,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3615656,"kb_used_data":3129116,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":90129496,"statfs":{"total":95995035648,"available":92292603904,"internally_reserved":0,"allocated":3204214784,"data_stored":3203587951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332242,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1438476,"kb_used_data":1050500,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92306676,"statfs":{"total":95995035648,"available":94522036224,"internally_reserved":0,"allocated":1075712000,"data_stored":1075107780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":397277638},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575019,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3706744,"kb_used_data":3198000,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":90038408,"statfs":{"total":95995035648,"available":92199329792,"internally_reserved":0,"allocated":3274752000,"data_stored":3274104464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738544,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1828900,"kb_used_data":1400668,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":91916252,"statfs":{"total":95995035648,"available":94122242048,"internally_reserved":0,"allocated":1434284032,"data_stored":1433656453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411493,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509456,"kb_used_data":1059012,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92235696,"statfs":{"total":95995035648,"available":94449352704,"internally_reserved":0,"allocated":1084428288,"data_stored":1083825625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":222,"seq":953482739743,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":471640,"kb_used_data":158088,"kb_used_omap":7,"kb_used_meta":313528,"kb_avail":93273512,"statfs":{"total":95995035648,"available":95512076288,"internally_reserved":0,"allocated":161882112,"data_stored":161425991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8180,"internal_metadata":321052684},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953630,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1482344,"kb_used_data":1140192,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92262808,"statfs":{"total":95995035648,"available":94477115392,"internally_reserved":0,"allocated":1167556608,"data_stored":1166925219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659287,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3657892,"kb_used_data":3179676,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":90087260,"statfs":{"total":95995035648,"available":92249354240,"internally_reserved":0,"allocated":3255988224,"data_stored":3255360753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3734,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1456013312,"data_stored":1456013312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3326017536,"data_stored":3326017536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1097924608,"data_stored":1097924608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1168310272,"data_stored":1168310272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3275685888,"data_stored":3275685888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1085603840,"data_stored":1085603840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3250520064,"data_stored":3250520064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:44:48.571 INFO:tasks.thrashosds.thrasher:Setting 3.8 to [4, 5, 0] 2024-03-12T03:44:48.571 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.8', '4', '5', '0'] 2024-03-12T03:44:48.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pg-upmap 3.8 4 5 0 2024-03-12T03:44:48.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:48.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:48.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:48.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:48.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:48 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 44899/268779 objects degraded (16.705%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:48.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:48.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:48.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:48.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:48.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:48.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:48.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:48 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 44899/268779 objects degraded (16.705%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:48.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:48 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 44899/268779 objects degraded (16.705%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:48.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:49.021 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:49.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:49.054 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:49.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:49.156 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:49.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:49.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:49.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:49.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:49.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:49.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:49.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 54 16 100513 100497 116.3 90.1875 0.00790172 0.00859397 2024-03-12T03:44:49.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:49.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:49.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:49.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:49.535 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:49.559 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:49 smithi169 ceph-mon[29344]: pgmap v543: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.42k op/s; 48158/289374 objects degraded (16.642%); 1/289374 objects misplaced (0.000%) 2024-03-12T03:44:49.560 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:49 smithi169 ceph-mon[39949]: pgmap v543: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.42k op/s; 48158/289374 objects degraded (16.642%); 1/289374 objects misplaced (0.000%) 2024-03-12T03:44:49.569 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:49.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:49.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:49.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:49.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:49.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:49 smithi173 ceph-mon[33544]: pgmap v543: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.42k op/s; 48158/289374 objects degraded (16.642%); 1/289374 objects misplaced (0.000%) 2024-03-12T03:44:49.736 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:44:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=cleanup t=2024-03-12T03:44:49.721742047Z level=info msg="Completed cleanup jobs" duration=3.691765ms 2024-03-12T03:44:49.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:49.847 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:49.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:49.947 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:49.950 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:49.984 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:49.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:49.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:50.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:50.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:50.124 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:50.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:50.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:50.259 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:50.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:50.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:50.383 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 55 16 101962 101946 115.832 90.5625 0.00671232 0.0086288 2024-03-12T03:44:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:50.397 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:50.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:50.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:50.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:50.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:50.639 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:50.680 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:50.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:50.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:50.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:50.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:50.820 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:50.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:50.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:50.967 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:50.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:51.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:51.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:51.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:51.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:51.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:51.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:51.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:51.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:51.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:51.383 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 56 16 103410 103394 115.38 90.5 0.0197018 0.00866262 2024-03-12T03:44:51.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:51.496 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:44:51.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:51.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:51.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:51.694 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:51.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:51 smithi173 ceph-mon[33544]: pgmap v544: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 878 B/s rd, 76 MiB/s wr, 1.22k op/s; 48158/289374 objects degraded (16.642%); 1/289374 objects misplaced (0.000%) 2024-03-12T03:44:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:51 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3072595435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:44:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:51 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:44:51.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:51 smithi169 ceph-mon[39949]: pgmap v544: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 878 B/s rd, 76 MiB/s wr, 1.22k op/s; 48158/289374 objects degraded (16.642%); 1/289374 objects misplaced (0.000%) 2024-03-12T03:44:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:51 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3072595435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:44:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:51 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:44:51.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:51 smithi169 ceph-mon[29344]: pgmap v544: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 878 B/s rd, 76 MiB/s wr, 1.22k op/s; 48158/289374 objects degraded (16.642%); 1/289374 objects misplaced (0.000%) 2024-03-12T03:44:51.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:51 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3072595435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:44:51.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:51 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:44:51.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:51.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:51.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:51.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:51.967 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:51.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:52.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:52.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:52.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:52.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:52.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:44:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:44:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:44:52.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:52.235 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:52.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:52.337 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:52.367 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:52.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:52.383 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 57 16 104856 104840 114.941 90.375 0.01353 0.00869588 2024-03-12T03:44:52.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:52.501 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:52.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:52.603 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:52.636 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:52.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:52.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:52 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:44:52.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:52 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:44:52.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:52 smithi173 ceph-mon[33544]: osdmap e300: 8 total, 7 up, 8 in 2024-03-12T03:44:52.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:52 smithi173 ceph-mon[33544]: 1.1 scrub starts 2024-03-12T03:44:52.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:52 smithi173 ceph-mon[33544]: 1.1 scrub ok 2024-03-12T03:44:52.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:52 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:44:52.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:52 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:44:52.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:52 smithi173 ceph-mon[33544]: 1.3 scrub starts 2024-03-12T03:44:52.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:52 smithi173 ceph-mon[33544]: 1.3 scrub ok 2024-03-12T03:44:52.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:52.771 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:44:52.771 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:44:52.771 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[29344]: osdmap e300: 8 total, 7 up, 8 in 2024-03-12T03:44:52.771 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[29344]: 1.1 scrub starts 2024-03-12T03:44:52.771 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[29344]: 1.1 scrub ok 2024-03-12T03:44:52.771 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:44:52.772 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:44:52.772 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[29344]: 1.3 scrub starts 2024-03-12T03:44:52.772 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[29344]: 1.3 scrub ok 2024-03-12T03:44:52.772 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:44:52.772 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:44:52.772 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[39949]: osdmap e300: 8 total, 7 up, 8 in 2024-03-12T03:44:52.772 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[39949]: 1.1 scrub starts 2024-03-12T03:44:52.772 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[39949]: 1.1 scrub ok 2024-03-12T03:44:52.772 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:44:52.772 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:44:52.772 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[39949]: 1.3 scrub starts 2024-03-12T03:44:52.772 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:52 smithi169 ceph-mon[39949]: 1.3 scrub ok 2024-03-12T03:44:52.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:52.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:52.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:52.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:52.916 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:52.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:53.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:53.049 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:53.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:53.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:53.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:53.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:53.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:44:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:44:53.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:53.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:53.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:53.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 58 16 106199 106183 114.406 83.9375 0.0144181 0.00873645 2024-03-12T03:44:53.426 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:53.465 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:53.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:53.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:53.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:53.602 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:53.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:53.700 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[29344]: 1.c scrub starts 2024-03-12T03:44:53.700 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[29344]: 1.c scrub ok 2024-03-12T03:44:53.700 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[29344]: pgmap v546: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s; 50865/307131 objects degraded (16.561%); 1/307131 objects misplaced (0.000%) 2024-03-12T03:44:53.700 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[29344]: 9.d scrub starts 2024-03-12T03:44:53.700 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[29344]: osdmap e301: 8 total, 7 up, 8 in 2024-03-12T03:44:53.700 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[29344]: 1.b scrub starts 2024-03-12T03:44:53.700 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[29344]: 1.b scrub ok 2024-03-12T03:44:53.700 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 50865/307131 objects degraded (16.561%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:53.700 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[39949]: 1.c scrub starts 2024-03-12T03:44:53.700 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[39949]: 1.c scrub ok 2024-03-12T03:44:53.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[39949]: pgmap v546: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s; 50865/307131 objects degraded (16.561%); 1/307131 objects misplaced (0.000%) 2024-03-12T03:44:53.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[39949]: 9.d scrub starts 2024-03-12T03:44:53.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[39949]: osdmap e301: 8 total, 7 up, 8 in 2024-03-12T03:44:53.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[39949]: 1.b scrub starts 2024-03-12T03:44:53.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[39949]: 1.b scrub ok 2024-03-12T03:44:53.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:53 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 50865/307131 objects degraded (16.561%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:53.704 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:53.742 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:53.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:53.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:53.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:53.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:53.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:53.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:53 smithi173 ceph-mon[33544]: 1.c scrub starts 2024-03-12T03:44:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:53 smithi173 ceph-mon[33544]: 1.c scrub ok 2024-03-12T03:44:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:53 smithi173 ceph-mon[33544]: pgmap v546: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s; 50865/307131 objects degraded (16.561%); 1/307131 objects misplaced (0.000%) 2024-03-12T03:44:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:53 smithi173 ceph-mon[33544]: 9.d scrub starts 2024-03-12T03:44:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:53 smithi173 ceph-mon[33544]: osdmap e301: 8 total, 7 up, 8 in 2024-03-12T03:44:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:53 smithi173 ceph-mon[33544]: 1.b scrub starts 2024-03-12T03:44:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:53 smithi173 ceph-mon[33544]: 1.b scrub ok 2024-03-12T03:44:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:53 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 50865/307131 objects degraded (16.561%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:54.020 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:54.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:54.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:54.027 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:44:54.059 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:44:54.092 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:44:54.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:54.130 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:44:54.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:54.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:54.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:54.168 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:44:54.201 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:44:54.235 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:44:54.235 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:44:54.235 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:44:54.235 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:44:54.236 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:44:54.236 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:44:54.236 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:44:54.236 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:44:54.236 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:44:54.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:54.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:44:54.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:54.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:54.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 59 16 107546 107530 113.894 84.1875 0.0236524 0.00877587 2024-03-12T03:44:54.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:54.435 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:54.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:54.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:54.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:54.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:54.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:54.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:54.697 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:54.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:54.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:54.799 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:54.828 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:54.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:54.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:54.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: 1.e scrub starts 2024-03-12T03:44:54.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: 1.e scrub ok 2024-03-12T03:44:54.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:54.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: 1.2 scrub starts 2024-03-12T03:44:54.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: 1.2 scrub ok 2024-03-12T03:44:54.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:44:54.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:44:54.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: osdmap e302: 8 total, 7 up, 8 in 2024-03-12T03:44:54.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: 1.14 scrub starts 2024-03-12T03:44:54.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: 1.14 scrub ok 2024-03-12T03:44:54.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: 1.f scrub starts 2024-03-12T03:44:54.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: 1.f scrub ok 2024-03-12T03:44:54.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/488757416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:44:54.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: 6.1 scrub starts 2024-03-12T03:44:54.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[29344]: 6.1 scrub ok 2024-03-12T03:44:54.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: 1.e scrub starts 2024-03-12T03:44:54.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: 1.e scrub ok 2024-03-12T03:44:54.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:54.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: 1.2 scrub starts 2024-03-12T03:44:54.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: 1.2 scrub ok 2024-03-12T03:44:54.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:44:54.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:44:54.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: osdmap e302: 8 total, 7 up, 8 in 2024-03-12T03:44:54.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: 1.14 scrub starts 2024-03-12T03:44:54.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: 1.14 scrub ok 2024-03-12T03:44:54.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: 1.f scrub starts 2024-03-12T03:44:54.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: 1.f scrub ok 2024-03-12T03:44:54.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/488757416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:44:54.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: 6.1 scrub starts 2024-03-12T03:44:54.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:54 smithi169 ceph-mon[39949]: 6.1 scrub ok 2024-03-12T03:44:54.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:54.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: 1.e scrub starts 2024-03-12T03:44:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: 1.e scrub ok 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: 1.2 scrub starts 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: 1.2 scrub ok 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: osdmap e302: 8 total, 7 up, 8 in 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: 1.14 scrub starts 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: 1.14 scrub ok 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: 1.f scrub starts 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: 1.f scrub ok 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/488757416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: 6.1 scrub starts 2024-03-12T03:44:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:54 smithi173 ceph-mon[33544]: 6.1 scrub ok 2024-03-12T03:44:55.071 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:55.104 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:55.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:55.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:55.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:55.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:55.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:55.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:55.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:55.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:55.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:44:55.382800+0000 min lat: 0.00120676 max lat: 0.160571 avg lat: 0.00881822 2024-03-12T03:44:55.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:44:55.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 60 16 108841 108825 113.345 80.9375 0.00615629 0.00881822 2024-03-12T03:44:55.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:55.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:55.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:55.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:55.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:55.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:55.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:55.791 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:55.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:55.798 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:55 smithi169 ceph-mon[29344]: pgmap v549: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.21k op/s; 51783/311211 objects degraded (16.639%); 1/311211 objects misplaced (0.000%) 2024-03-12T03:44:55.798 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:55 smithi169 ceph-mon[29344]: 1.0 scrub starts 2024-03-12T03:44:55.799 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:55 smithi169 ceph-mon[29344]: 1.0 scrub ok 2024-03-12T03:44:55.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:55 smithi169 ceph-mon[39949]: pgmap v549: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.21k op/s; 51783/311211 objects degraded (16.639%); 1/311211 objects misplaced (0.000%) 2024-03-12T03:44:55.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:55 smithi169 ceph-mon[39949]: 1.0 scrub starts 2024-03-12T03:44:55.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:55 smithi169 ceph-mon[39949]: 1.0 scrub ok 2024-03-12T03:44:55.846 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:44:55.846 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:44:55.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:55.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:55.941 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:55.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:55 smithi173 ceph-mon[33544]: pgmap v549: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.21k op/s; 51783/311211 objects degraded (16.639%); 1/311211 objects misplaced (0.000%) 2024-03-12T03:44:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:55 smithi173 ceph-mon[33544]: 1.0 scrub starts 2024-03-12T03:44:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:55 smithi173 ceph-mon[33544]: 1.0 scrub ok 2024-03-12T03:44:56.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:56.080 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:56.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:56.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:56.220 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:44:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:56.233 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:56.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:56.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:56.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:56.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:56.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:56.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 61 16 110180 110164 112.858 83.6875 0.00847171 0.00885661 2024-03-12T03:44:56.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:56.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:56.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:56.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:56.579 INFO:teuthology.orchestra.run.smithi169.stderr:set 3.8 pg_upmap mapping to [4,5,0] 2024-03-12T03:44:56.588 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:44:56.588 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:44:56.609 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:56.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[39949]: 9.b scrub starts 2024-03-12T03:44:56.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[39949]: 9.d scrub ok 2024-03-12T03:44:56.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[39949]: 1.d scrub starts 2024-03-12T03:44:56.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[39949]: 1.d scrub ok 2024-03-12T03:44:56.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3033866447' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.8", "id": [4, 5, 0]}]: dispatch 2024-03-12T03:44:56.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.8", "id": [4, 5, 0]}]: dispatch 2024-03-12T03:44:56.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[39949]: 1.12 scrub starts 2024-03-12T03:44:56.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[39949]: 1.12 scrub ok 2024-03-12T03:44:56.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[39949]: 1.13 scrub starts 2024-03-12T03:44:56.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[39949]: 1.13 scrub ok 2024-03-12T03:44:56.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[29344]: 9.b scrub starts 2024-03-12T03:44:56.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[29344]: 9.d scrub ok 2024-03-12T03:44:56.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[29344]: 1.d scrub starts 2024-03-12T03:44:56.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[29344]: 1.d scrub ok 2024-03-12T03:44:56.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3033866447' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.8", "id": [4, 5, 0]}]: dispatch 2024-03-12T03:44:56.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.8", "id": [4, 5, 0]}]: dispatch 2024-03-12T03:44:56.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[29344]: 1.12 scrub starts 2024-03-12T03:44:56.655 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[29344]: 1.12 scrub ok 2024-03-12T03:44:56.655 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[29344]: 1.13 scrub starts 2024-03-12T03:44:56.655 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:56 smithi169 ceph-mon[29344]: 1.13 scrub ok 2024-03-12T03:44:56.658 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:56.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:56.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:56.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:56.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:56.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:56.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:56.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:56 smithi173 ceph-mon[33544]: 9.b scrub starts 2024-03-12T03:44:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:56 smithi173 ceph-mon[33544]: 9.d scrub ok 2024-03-12T03:44:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:56 smithi173 ceph-mon[33544]: 1.d scrub starts 2024-03-12T03:44:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:56 smithi173 ceph-mon[33544]: 1.d scrub ok 2024-03-12T03:44:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:56 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3033866447' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.8", "id": [4, 5, 0]}]: dispatch 2024-03-12T03:44:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:56 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.8", "id": [4, 5, 0]}]: dispatch 2024-03-12T03:44:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:56 smithi173 ceph-mon[33544]: 1.12 scrub starts 2024-03-12T03:44:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:56 smithi173 ceph-mon[33544]: 1.12 scrub ok 2024-03-12T03:44:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:56 smithi173 ceph-mon[33544]: 1.13 scrub starts 2024-03-12T03:44:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:56 smithi173 ceph-mon[33544]: 1.13 scrub ok 2024-03-12T03:44:57.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:57.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:57.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:57.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:57.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:57.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:57.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:57.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:57.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:57.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:57.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 62 16 111447 111431 112.315 79.1875 0.0111779 0.00889957 2024-03-12T03:44:57.451 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:44:57.485 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:57.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:44:57.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:57.622 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:57.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:57.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:57.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:57.740 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[29344]: 1.5 scrub starts 2024-03-12T03:44:57.740 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[29344]: 1.5 scrub ok 2024-03-12T03:44:57.740 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[29344]: 1.11 scrub starts 2024-03-12T03:44:57.740 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[29344]: 1.11 scrub ok 2024-03-12T03:44:57.740 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[29344]: pgmap v550: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 52183/312411 objects degraded (16.703%); 1/312411 objects misplaced (0.000%) 2024-03-12T03:44:57.740 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[29344]: 3.d scrub starts 2024-03-12T03:44:57.740 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[29344]: 3.d scrub ok 2024-03-12T03:44:57.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.8", "id": [4, 5, 0]}]': finished 2024-03-12T03:44:57.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[29344]: osdmap e303: 8 total, 7 up, 8 in 2024-03-12T03:44:57.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[29344]: 1.10 scrub starts 2024-03-12T03:44:57.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[29344]: 1.10 scrub ok 2024-03-12T03:44:57.741 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[39949]: 1.5 scrub starts 2024-03-12T03:44:57.741 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[39949]: 1.5 scrub ok 2024-03-12T03:44:57.741 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[39949]: 1.11 scrub starts 2024-03-12T03:44:57.741 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[39949]: 1.11 scrub ok 2024-03-12T03:44:57.742 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[39949]: pgmap v550: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 52183/312411 objects degraded (16.703%); 1/312411 objects misplaced (0.000%) 2024-03-12T03:44:57.742 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[39949]: 3.d scrub starts 2024-03-12T03:44:57.742 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[39949]: 3.d scrub ok 2024-03-12T03:44:57.742 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.8", "id": [4, 5, 0]}]': finished 2024-03-12T03:44:57.742 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[39949]: osdmap e303: 8 total, 7 up, 8 in 2024-03-12T03:44:57.742 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[39949]: 1.10 scrub starts 2024-03-12T03:44:57.742 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:57 smithi169 ceph-mon[39949]: 1.10 scrub ok 2024-03-12T03:44:57.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:57.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:57.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:44:57.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:57.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:57.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:57 smithi173 ceph-mon[33544]: 1.5 scrub starts 2024-03-12T03:44:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:57 smithi173 ceph-mon[33544]: 1.5 scrub ok 2024-03-12T03:44:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:57 smithi173 ceph-mon[33544]: 1.11 scrub starts 2024-03-12T03:44:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:57 smithi173 ceph-mon[33544]: 1.11 scrub ok 2024-03-12T03:44:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:57 smithi173 ceph-mon[33544]: pgmap v550: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 95 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 52183/312411 objects degraded (16.703%); 1/312411 objects misplaced (0.000%) 2024-03-12T03:44:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:57 smithi173 ceph-mon[33544]: 3.d scrub starts 2024-03-12T03:44:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:57 smithi173 ceph-mon[33544]: 3.d scrub ok 2024-03-12T03:44:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:57 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.8", "id": [4, 5, 0]}]': finished 2024-03-12T03:44:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:57 smithi173 ceph-mon[33544]: osdmap e303: 8 total, 7 up, 8 in 2024-03-12T03:44:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:57 smithi173 ceph-mon[33544]: 1.10 scrub starts 2024-03-12T03:44:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:57 smithi173 ceph-mon[33544]: 1.10 scrub ok 2024-03-12T03:44:58.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:58.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:58.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:58.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:58.178 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:58.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:58.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:58.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:58.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:58.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:58.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 63 16 112902 112886 111.976 90.9375 0.00728098 0.00892599 2024-03-12T03:44:58.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:58.455 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:58.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:58.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:58.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:58.597 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:58.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:58.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:58.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:58.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:58.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:58.866 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[29344]: 9.b scrub ok 2024-03-12T03:44:58.866 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[29344]: osdmap e304: 8 total, 7 up, 8 in 2024-03-12T03:44:58.866 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[29344]: 4.1f scrub starts 2024-03-12T03:44:58.866 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[29344]: 4.1f scrub ok 2024-03-12T03:44:58.866 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[29344]: 4.2 scrub starts 2024-03-12T03:44:58.866 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[29344]: 4.2 scrub ok 2024-03-12T03:44:58.866 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 52183/312411 objects degraded (16.703%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:58.866 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[29344]: osdmap e305: 8 total, 7 up, 8 in 2024-03-12T03:44:58.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[39949]: 9.b scrub ok 2024-03-12T03:44:58.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[39949]: osdmap e304: 8 total, 7 up, 8 in 2024-03-12T03:44:58.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[39949]: 4.1f scrub starts 2024-03-12T03:44:58.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[39949]: 4.1f scrub ok 2024-03-12T03:44:58.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[39949]: 4.2 scrub starts 2024-03-12T03:44:58.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[39949]: 4.2 scrub ok 2024-03-12T03:44:58.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 52183/312411 objects degraded (16.703%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:58.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:58 smithi169 ceph-mon[39949]: osdmap e305: 8 total, 7 up, 8 in 2024-03-12T03:44:58.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:58.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:58.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:58 smithi173 ceph-mon[33544]: 9.b scrub ok 2024-03-12T03:44:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:58 smithi173 ceph-mon[33544]: osdmap e304: 8 total, 7 up, 8 in 2024-03-12T03:44:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:58 smithi173 ceph-mon[33544]: 4.1f scrub starts 2024-03-12T03:44:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:58 smithi173 ceph-mon[33544]: 4.1f scrub ok 2024-03-12T03:44:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:58 smithi173 ceph-mon[33544]: 4.2 scrub starts 2024-03-12T03:44:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:58 smithi173 ceph-mon[33544]: 4.2 scrub ok 2024-03-12T03:44:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:58 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 52183/312411 objects degraded (16.703%), 31 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-12T03:44:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:58 smithi173 ceph-mon[33544]: osdmap e305: 8 total, 7 up, 8 in 2024-03-12T03:44:59.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:59.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:59.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:59.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:44:59.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:59.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:59.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:44:59.250 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:44:59.280 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:59.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:44:59.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:44:59.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 64 16 114385 114369 111.674 92.6875 0.00783741 0.00895051 2024-03-12T03:44:59.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:59.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:44:59.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:44:59.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:59.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:44:59.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:44:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:59.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:44:59.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:44:59.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:59.854 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:59.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:44:59.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:59 smithi173 ceph-mon[33544]: pgmap v553: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 94 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 77 MiB/s wr, 1.22k op/s; 55203/331479 objects degraded (16.654%); 1/331479 objects misplaced (0.000%) 2024-03-12T03:44:59.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:59 smithi173 ceph-mon[33544]: 3.11 scrub starts 2024-03-12T03:44:59.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:59 smithi173 ceph-mon[33544]: 3.11 scrub ok 2024-03-12T03:44:59.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:59 smithi173 ceph-mon[33544]: 4.5 scrub starts 2024-03-12T03:44:59.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:44:59 smithi173 ceph-mon[33544]: 4.5 scrub ok 2024-03-12T03:44:59.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:59 smithi169 ceph-mon[29344]: pgmap v553: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 94 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 77 MiB/s wr, 1.22k op/s; 55203/331479 objects degraded (16.654%); 1/331479 objects misplaced (0.000%) 2024-03-12T03:44:59.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:59 smithi169 ceph-mon[29344]: 3.11 scrub starts 2024-03-12T03:44:59.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:59 smithi169 ceph-mon[29344]: 3.11 scrub ok 2024-03-12T03:44:59.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:59 smithi169 ceph-mon[29344]: 4.5 scrub starts 2024-03-12T03:44:59.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:44:59 smithi169 ceph-mon[29344]: 4.5 scrub ok 2024-03-12T03:44:59.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:59 smithi169 ceph-mon[39949]: pgmap v553: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 94 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 77 MiB/s wr, 1.22k op/s; 55203/331479 objects degraded (16.654%); 1/331479 objects misplaced (0.000%) 2024-03-12T03:44:59.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:59 smithi169 ceph-mon[39949]: 3.11 scrub starts 2024-03-12T03:44:59.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:59 smithi169 ceph-mon[39949]: 3.11 scrub ok 2024-03-12T03:44:59.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:59 smithi169 ceph-mon[39949]: 4.5 scrub starts 2024-03-12T03:44:59.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:44:59 smithi169 ceph-mon[39949]: 4.5 scrub ok 2024-03-12T03:44:59.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:44:59.990 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:44:59.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:44:59.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:00.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-12T03:45:00.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:00.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:00.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:00.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:00.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:00.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:00.380 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:00.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 65 16 115748 115732 111.267 85.1875 0.00768271 0.00898349 2024-03-12T03:45:00.414 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:00.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:00.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:00.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:00.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:00.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:00.680 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:00.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:00.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:00.814 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:00.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:00.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:00.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:00 smithi169 ceph-mon[29344]: 5.18 scrub starts 2024-03-12T03:45:00.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:00 smithi169 ceph-mon[29344]: 5.18 scrub ok 2024-03-12T03:45:00.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:00 smithi169 ceph-mon[29344]: 5.10 scrub starts 2024-03-12T03:45:00.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:00 smithi169 ceph-mon[29344]: 5.10 scrub ok 2024-03-12T03:45:00.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:00 smithi169 ceph-mon[29344]: 3.0 scrub starts 2024-03-12T03:45:00.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:00 smithi169 ceph-mon[29344]: 3.0 scrub ok 2024-03-12T03:45:00.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:00 smithi169 ceph-mon[39949]: 5.18 scrub starts 2024-03-12T03:45:00.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:00 smithi169 ceph-mon[39949]: 5.18 scrub ok 2024-03-12T03:45:00.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:00 smithi169 ceph-mon[39949]: 5.10 scrub starts 2024-03-12T03:45:00.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:00 smithi169 ceph-mon[39949]: 5.10 scrub ok 2024-03-12T03:45:00.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:00 smithi169 ceph-mon[39949]: 3.0 scrub starts 2024-03-12T03:45:00.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:00 smithi169 ceph-mon[39949]: 3.0 scrub ok 2024-03-12T03:45:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:00.957 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:00.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:00.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:00 smithi173 ceph-mon[33544]: 5.18 scrub starts 2024-03-12T03:45:00.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:00 smithi173 ceph-mon[33544]: 5.18 scrub ok 2024-03-12T03:45:00.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:00 smithi173 ceph-mon[33544]: 5.10 scrub starts 2024-03-12T03:45:00.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:00 smithi173 ceph-mon[33544]: 5.10 scrub ok 2024-03-12T03:45:00.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:00 smithi173 ceph-mon[33544]: 3.0 scrub starts 2024-03-12T03:45:00.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:00 smithi173 ceph-mon[33544]: 3.0 scrub ok 2024-03-12T03:45:01.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:01.088 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:01.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:01.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:01.218 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:01.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:01.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:01.349 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:01.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:01.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 66 16 117011 116995 110.777 78.9375 0.0162083 0.00902278 2024-03-12T03:45:01.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:01.491 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:01.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:01.593 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:01.625 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:01.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:01.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:01.728 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:01.760 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:01.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:01.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:01.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:01.896 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:01.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[29344]: pgmap v555: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 94 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 55203/331479 objects degraded (16.654%); 1/331479 objects misplaced (0.000%) 2024-03-12T03:45:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[29344]: 6.10 scrub starts 2024-03-12T03:45:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:45:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:45:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[29344]: 6.10 scrub ok 2024-03-12T03:45:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[29344]: 6.1d scrub starts 2024-03-12T03:45:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[29344]: 6.1d scrub ok 2024-03-12T03:45:01.924 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:45:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:45:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[39949]: pgmap v555: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 94 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 55203/331479 objects degraded (16.654%); 1/331479 objects misplaced (0.000%) 2024-03-12T03:45:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[39949]: 6.10 scrub starts 2024-03-12T03:45:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:45:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:45:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[39949]: 6.10 scrub ok 2024-03-12T03:45:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[39949]: 6.1d scrub starts 2024-03-12T03:45:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:01 smithi169 ceph-mon[39949]: 6.1d scrub ok 2024-03-12T03:45:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:01 smithi173 ceph-mon[33544]: pgmap v555: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 38 active+undersized, 94 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 55203/331479 objects degraded (16.654%); 1/331479 objects misplaced (0.000%) 2024-03-12T03:45:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:01 smithi173 ceph-mon[33544]: 6.10 scrub starts 2024-03-12T03:45:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:01 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:45:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:01 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:45:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:01 smithi173 ceph-mon[33544]: 6.10 scrub ok 2024-03-12T03:45:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:01 smithi173 ceph-mon[33544]: 6.1d scrub starts 2024-03-12T03:45:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:01 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:01 smithi173 ceph-mon[33544]: 6.1d scrub ok 2024-03-12T03:45:01.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:02.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:02.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:02.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:02.164 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:02.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:02.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:02.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:02.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:02.385 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 67 16 118182 118166 110.215 73.1875 0.0112203 0.0090688 2024-03-12T03:45:02.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:02.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:02.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:02.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:02.442 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:02.452 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:02.521 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-12T03:45:02.521 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:45:02.522 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-12T03:45:02.522 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-03-12T03:45:02.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 6 0 2024-03-12T03:45:02.537 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:02.626 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:02.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:02.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:02.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:02.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:02.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:02.813 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:02 smithi169 ceph-mon[29344]: osdmap e306: 8 total, 7 up, 8 in 2024-03-12T03:45:02.814 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:02 smithi169 ceph-mon[29344]: 1.a scrub starts 2024-03-12T03:45:02.814 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:02 smithi169 ceph-mon[29344]: 1.a scrub ok 2024-03-12T03:45:02.814 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:02 smithi169 ceph-mon[39949]: osdmap e306: 8 total, 7 up, 8 in 2024-03-12T03:45:02.814 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:02 smithi169 ceph-mon[39949]: 1.a scrub starts 2024-03-12T03:45:02.814 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:02 smithi169 ceph-mon[39949]: 1.a scrub ok 2024-03-12T03:45:02.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:02.904 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:02.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:02.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:02.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:45:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:45:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:02 smithi173 ceph-mon[33544]: osdmap e306: 8 total, 7 up, 8 in 2024-03-12T03:45:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:02 smithi173 ceph-mon[33544]: 1.a scrub starts 2024-03-12T03:45:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:02 smithi173 ceph-mon[33544]: 1.a scrub ok 2024-03-12T03:45:03.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:03.039 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:03.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:03.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:03.174 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:03.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:03.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:03.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:03.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:03.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:03.380 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:45:03.384 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 68 16 119561 119545 109.862 86.1875 0.0148663 0.00909743 2024-03-12T03:45:03.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:03.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:03.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:03.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:03.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:03.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:03.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:03.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:03.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:03.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:03.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:03.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:03.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:03.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:03.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[29344]: pgmap v557: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s; 57825/348219 objects degraded (16.606%); 1/348219 objects misplaced (0.000%) 2024-03-12T03:45:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[29344]: 4.6 scrub starts 2024-03-12T03:45:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[29344]: 4.6 scrub ok 2024-03-12T03:45:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1555041001' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 57825/348219 objects degraded (16.606%), 31 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:45:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:45:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[29344]: osdmap e307: 8 total, 7 up, 8 in 2024-03-12T03:45:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[39949]: pgmap v557: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s; 57825/348219 objects degraded (16.606%); 1/348219 objects misplaced (0.000%) 2024-03-12T03:45:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[39949]: 4.6 scrub starts 2024-03-12T03:45:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[39949]: 4.6 scrub ok 2024-03-12T03:45:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1555041001' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 57825/348219 objects degraded (16.606%), 31 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:45:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:45:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:03 smithi169 ceph-mon[39949]: osdmap e307: 8 total, 7 up, 8 in 2024-03-12T03:45:03.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:03 smithi173 ceph-mon[33544]: pgmap v557: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s; 57825/348219 objects degraded (16.606%); 1/348219 objects misplaced (0.000%) 2024-03-12T03:45:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:03 smithi173 ceph-mon[33544]: 4.6 scrub starts 2024-03-12T03:45:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:03 smithi173 ceph-mon[33544]: 4.6 scrub ok 2024-03-12T03:45:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:03 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1555041001' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:03 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:03 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 57825/348219 objects degraded (16.606%), 31 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:03 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:45:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:03 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:45:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:03 smithi173 ceph-mon[33544]: osdmap e307: 8 total, 7 up, 8 in 2024-03-12T03:45:04.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:04.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:04.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:04.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:04.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:04.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:04.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:04.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:04.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:04.385 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 69 16 120998 120982 109.571 89.8125 0.0044514 0.0091226 2024-03-12T03:45:04.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:04.430 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:04.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:04.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:04.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:04.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:04.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:04.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:04.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:04.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:04.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:04.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:04.830 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:04.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:04.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:04.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[29344]: 3.a scrub starts 2024-03-12T03:45:04.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[29344]: 3.a scrub ok 2024-03-12T03:45:04.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[29344]: 3.1b scrub starts 2024-03-12T03:45:04.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[29344]: 3.1b scrub ok 2024-03-12T03:45:04.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[29344]: 5.c scrub starts 2024-03-12T03:45:04.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[29344]: 5.c scrub ok 2024-03-12T03:45:04.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[29344]: 3.16 scrub starts 2024-03-12T03:45:04.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[29344]: 3.16 scrub ok 2024-03-12T03:45:04.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:45:04.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:45:04.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:04.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[39949]: 3.a scrub starts 2024-03-12T03:45:04.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[39949]: 3.a scrub ok 2024-03-12T03:45:04.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[39949]: 3.1b scrub starts 2024-03-12T03:45:04.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[39949]: 3.1b scrub ok 2024-03-12T03:45:04.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[39949]: 5.c scrub starts 2024-03-12T03:45:04.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[39949]: 5.c scrub ok 2024-03-12T03:45:04.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[39949]: 3.16 scrub starts 2024-03-12T03:45:04.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[39949]: 3.16 scrub ok 2024-03-12T03:45:04.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:45:04.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:04 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:45:04.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:04.958 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:04.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:04 smithi173 ceph-mon[33544]: 3.a scrub starts 2024-03-12T03:45:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:04 smithi173 ceph-mon[33544]: 3.a scrub ok 2024-03-12T03:45:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:04 smithi173 ceph-mon[33544]: 3.1b scrub starts 2024-03-12T03:45:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:04 smithi173 ceph-mon[33544]: 3.1b scrub ok 2024-03-12T03:45:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:04 smithi173 ceph-mon[33544]: 5.c scrub starts 2024-03-12T03:45:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:04 smithi173 ceph-mon[33544]: 5.c scrub ok 2024-03-12T03:45:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:04 smithi173 ceph-mon[33544]: 3.16 scrub starts 2024-03-12T03:45:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:04 smithi173 ceph-mon[33544]: 3.16 scrub ok 2024-03-12T03:45:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:04 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:45:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:04 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:45:05.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:05.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:05.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:05.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:05.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:05.230 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:05.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:05.331 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:05.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:05.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:05.385 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 70 16 122304 122288 109.172 81.625 0.00921822 0.00915572 2024-03-12T03:45:05.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:05.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:05.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:05.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:05.646 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:05.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:05.751 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:05.789 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:05.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:05.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[29344]: 4.1 scrub starts 2024-03-12T03:45:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[29344]: 4.1 scrub ok 2024-03-12T03:45:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[29344]: pgmap v559: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s; 58618/351786 objects degraded (16.663%); 1/351786 objects misplaced (0.000%) 2024-03-12T03:45:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[29344]: 5.1f scrub starts 2024-03-12T03:45:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[29344]: 5.1f scrub ok 2024-03-12T03:45:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[29344]: 6.14 scrub starts 2024-03-12T03:45:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[29344]: 6.14 scrub ok 2024-03-12T03:45:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[29344]: osdmap e308: 8 total, 7 up, 8 in 2024-03-12T03:45:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[39949]: 4.1 scrub starts 2024-03-12T03:45:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[39949]: 4.1 scrub ok 2024-03-12T03:45:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[39949]: pgmap v559: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s; 58618/351786 objects degraded (16.663%); 1/351786 objects misplaced (0.000%) 2024-03-12T03:45:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[39949]: 5.1f scrub starts 2024-03-12T03:45:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[39949]: 5.1f scrub ok 2024-03-12T03:45:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[39949]: 6.14 scrub starts 2024-03-12T03:45:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[39949]: 6.14 scrub ok 2024-03-12T03:45:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:05 smithi169 ceph-mon[39949]: osdmap e308: 8 total, 7 up, 8 in 2024-03-12T03:45:05.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:05.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:05 smithi173 ceph-mon[33544]: 4.1 scrub starts 2024-03-12T03:45:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:05 smithi173 ceph-mon[33544]: 4.1 scrub ok 2024-03-12T03:45:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:05 smithi173 ceph-mon[33544]: pgmap v559: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s; 58618/351786 objects degraded (16.663%); 1/351786 objects misplaced (0.000%) 2024-03-12T03:45:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:05 smithi173 ceph-mon[33544]: 5.1f scrub starts 2024-03-12T03:45:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:05 smithi173 ceph-mon[33544]: 5.1f scrub ok 2024-03-12T03:45:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:05 smithi173 ceph-mon[33544]: 6.14 scrub starts 2024-03-12T03:45:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:05 smithi173 ceph-mon[33544]: 6.14 scrub ok 2024-03-12T03:45:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:05 smithi173 ceph-mon[33544]: osdmap e308: 8 total, 7 up, 8 in 2024-03-12T03:45:06.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:06.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:06.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:06.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:06.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:06.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:06.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:06.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:06.338 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:06.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:06.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:06.385 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 71 16 123750 123734 108.907 90.375 0.0129949 0.00917793 2024-03-12T03:45:06.439 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:06.473 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:06.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:06.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:06.610 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:06.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:06.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:06.744 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:06.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:06.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:06.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:06.886 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:06.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:06.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:06 smithi173 ceph-mon[33544]: 6.1e scrub starts 2024-03-12T03:45:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:06 smithi173 ceph-mon[33544]: 6.1e scrub ok 2024-03-12T03:45:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:06 smithi173 ceph-mon[33544]: 5.2 scrub starts 2024-03-12T03:45:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:06 smithi173 ceph-mon[33544]: 5.2 scrub ok 2024-03-12T03:45:06.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:07.026 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:07.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:07.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:07.164 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:07.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:07.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:06 smithi169 ceph-mon[29344]: 6.1e scrub starts 2024-03-12T03:45:07.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:06 smithi169 ceph-mon[29344]: 6.1e scrub ok 2024-03-12T03:45:07.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:06 smithi169 ceph-mon[29344]: 5.2 scrub starts 2024-03-12T03:45:07.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:06 smithi169 ceph-mon[29344]: 5.2 scrub ok 2024-03-12T03:45:07.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:06 smithi169 ceph-mon[39949]: 6.1e scrub starts 2024-03-12T03:45:07.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:06 smithi169 ceph-mon[39949]: 6.1e scrub ok 2024-03-12T03:45:07.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:06 smithi169 ceph-mon[39949]: 5.2 scrub starts 2024-03-12T03:45:07.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:06 smithi169 ceph-mon[39949]: 5.2 scrub ok 2024-03-12T03:45:07.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:07.296 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:07.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:07.385 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 72 16 125201 125185 108.654 90.6875 0.00517066 0.00919946 2024-03-12T03:45:07.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:07.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:07.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:07.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:07.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:07.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:07.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:07.704 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:07.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:07.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:07.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:07.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:07.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:07.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:07.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:07 smithi173 ceph-mon[33544]: pgmap v561: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s; 58618/351786 objects degraded (16.663%); 1/351786 objects misplaced (0.000%) 2024-03-12T03:45:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:07 smithi173 ceph-mon[33544]: 5.d scrub starts 2024-03-12T03:45:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:07 smithi173 ceph-mon[33544]: 5.d scrub ok 2024-03-12T03:45:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:07 smithi173 ceph-mon[33544]: 4.a scrub starts 2024-03-12T03:45:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:07 smithi173 ceph-mon[33544]: 4.a scrub ok 2024-03-12T03:45:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:07 smithi173 ceph-mon[33544]: 5.5 scrub starts 2024-03-12T03:45:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:07 smithi173 ceph-mon[33544]: 5.5 scrub ok 2024-03-12T03:45:08.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[29344]: pgmap v561: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s; 58618/351786 objects degraded (16.663%); 1/351786 objects misplaced (0.000%) 2024-03-12T03:45:08.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[29344]: 5.d scrub starts 2024-03-12T03:45:08.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[29344]: 5.d scrub ok 2024-03-12T03:45:08.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[29344]: 4.a scrub starts 2024-03-12T03:45:08.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[29344]: 4.a scrub ok 2024-03-12T03:45:08.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[29344]: 5.5 scrub starts 2024-03-12T03:45:08.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[29344]: 5.5 scrub ok 2024-03-12T03:45:08.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[39949]: pgmap v561: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s; 58618/351786 objects degraded (16.663%); 1/351786 objects misplaced (0.000%) 2024-03-12T03:45:08.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[39949]: 5.d scrub starts 2024-03-12T03:45:08.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[39949]: 5.d scrub ok 2024-03-12T03:45:08.036 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[39949]: 4.a scrub starts 2024-03-12T03:45:08.036 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[39949]: 4.a scrub ok 2024-03-12T03:45:08.036 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[39949]: 5.5 scrub starts 2024-03-12T03:45:08.036 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:07 smithi169 ceph-mon[39949]: 5.5 scrub ok 2024-03-12T03:45:08.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:08.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:08.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:08.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:08.255 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:08.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:08.357 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:08.385 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 73 16 126598 126582 108.362 87.3125 0.012171 0.00922419 2024-03-12T03:45:08.398 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:08.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:08.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:08.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:08.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:08.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:08.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:08.676 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:08.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:08.720 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:08 smithi169 ceph-mon[29344]: 6.d scrub starts 2024-03-12T03:45:08.720 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:08 smithi169 ceph-mon[29344]: 6.d scrub ok 2024-03-12T03:45:08.720 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:08 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 58618/351786 objects degraded (16.663%), 31 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:08.720 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:08 smithi169 ceph-mon[29344]: osdmap e309: 8 total, 7 up, 8 in 2024-03-12T03:45:08.720 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:08 smithi169 ceph-mon[39949]: 6.d scrub starts 2024-03-12T03:45:08.721 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:08 smithi169 ceph-mon[39949]: 6.d scrub ok 2024-03-12T03:45:08.721 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:08 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 58618/351786 objects degraded (16.663%), 31 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:08.721 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:08 smithi169 ceph-mon[39949]: osdmap e309: 8 total, 7 up, 8 in 2024-03-12T03:45:08.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:08.814 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:08.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:08.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:08.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:08.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:08.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:08.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:08.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:08 smithi173 ceph-mon[33544]: 6.d scrub starts 2024-03-12T03:45:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:08 smithi173 ceph-mon[33544]: 6.d scrub ok 2024-03-12T03:45:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:08 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 58618/351786 objects degraded (16.663%), 31 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:08 smithi173 ceph-mon[33544]: osdmap e309: 8 total, 7 up, 8 in 2024-03-12T03:45:09.057 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:09.094 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:09.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:09.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:09.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:09.241 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:09.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:09.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:09.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:09.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:09.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:09.386 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 74 16 127951 127935 108.039 84.5625 0.0104398 0.00925099 2024-03-12T03:45:09.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:09.511 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:09.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:09.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:09.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:09.607 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:45:09.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:09.652 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:45:09.665 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:09.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:09.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:09.686 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:45:09.725 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:45:09.765 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:45:09.765 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:09 smithi169 ceph-mon[29344]: pgmap v562: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 799 B/s rd, 95 MiB/s wr, 1.52k op/s; 62066/372339 objects degraded (16.669%); 1/372339 objects misplaced (0.000%) 2024-03-12T03:45:09.765 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:09 smithi169 ceph-mon[29344]: 5.11 scrub starts 2024-03-12T03:45:09.765 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:09 smithi169 ceph-mon[29344]: 5.11 scrub ok 2024-03-12T03:45:09.766 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:09 smithi169 ceph-mon[29344]: 9.6 deep-scrub starts 2024-03-12T03:45:09.766 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:09 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1668627851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:09.766 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:09 smithi169 ceph-mon[39949]: pgmap v562: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 799 B/s rd, 95 MiB/s wr, 1.52k op/s; 62066/372339 objects degraded (16.669%); 1/372339 objects misplaced (0.000%) 2024-03-12T03:45:09.766 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:09 smithi169 ceph-mon[39949]: 5.11 scrub starts 2024-03-12T03:45:09.766 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:09 smithi169 ceph-mon[39949]: 5.11 scrub ok 2024-03-12T03:45:09.766 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:09 smithi169 ceph-mon[39949]: 9.6 deep-scrub starts 2024-03-12T03:45:09.766 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:09 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1668627851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:09.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:09.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:09.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:09.803 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:45:09.840 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:45:09.840 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:45:09.840 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:45:09.840 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:45:09.840 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:45:09.840 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:45:09.840 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:45:09.840 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:45:09.841 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:45:09.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:09.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:09.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:09 smithi173 ceph-mon[33544]: pgmap v562: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 799 B/s rd, 95 MiB/s wr, 1.52k op/s; 62066/372339 objects degraded (16.669%); 1/372339 objects misplaced (0.000%) 2024-03-12T03:45:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:09 smithi173 ceph-mon[33544]: 5.11 scrub starts 2024-03-12T03:45:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:09 smithi173 ceph-mon[33544]: 5.11 scrub ok 2024-03-12T03:45:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:09 smithi173 ceph-mon[33544]: 9.6 deep-scrub starts 2024-03-12T03:45:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:09 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1668627851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:10.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:10.070 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:10.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:10.172 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:10.207 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:10.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:10.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:10.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:10.345 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:10.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:10.386 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 75 16 129211 129195 107.648 78.75 0.00565891 0.00928495 2024-03-12T03:45:10.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:10.484 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:10.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:10.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:10.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:45:10.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:10.593 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:45:10.593 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:45:10.593 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:45:10.593 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155689 7.b 7:d77d3091:::benchmark_data_smithi169_79822_object38272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.729513+0000", 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 437.85803414200001, 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.199297788, 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155689 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729513+0000", 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729513+0000", 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729514+0000", 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4300000000000001e-06 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.595 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729519+0000", 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2549999999999999e-06 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729527+0000", 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1410000000000005e-06 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.729533+0000", 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0410000000000003e-06 2024-03-12T03:45:10.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.790125+0000", 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.060591712999999998 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.790253+0000", 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000128052 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.790394+0000", 2024-03-12T03:45:10.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000141812 2024-03-12T03:45:10.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.598 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.598 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.598 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.798494+0000", 2024-03-12T03:45:10.598 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080992140000000004 2024-03-12T03:45:10.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.598 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.598 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.598 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.798540+0000", 2024-03-12T03:45:10.598 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6567999999999999e-05 2024-03-12T03:45:10.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.598 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928691+0000", 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.130150604 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928746+0000", 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5340000000000002e-05 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928811+0000", 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4617999999999997e-05 2024-03-12T03:45:10.599 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14799.0:155695 7.b 7:d612ca62:::benchmark_data_smithi169_79822_object38278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:37:52.732453+0000", 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 437.85509430100001, 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19647717000000001, 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14799", 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 155695 2024-03-12T03:45:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732453+0000", 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732453+0000", 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732454+0000", 2024-03-12T03:45:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.671e-06 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732461+0000", 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8179999999999998e-06 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732477+0000", 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6169999999999999e-05 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.732486+0000", 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3790000000000006e-06 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.795096+0000", 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062610618000000007 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.795213+0000", 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011686199999999999 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:45:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.795338+0000", 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012426200000000001 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.798582+0000", 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032448910000000002 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.805288+0000", 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067054109999999997 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928826+0000", 2024-03-12T03:45:10.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12353776499999999 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928877+0000", 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.147e-05 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:37:52.928930+0000", 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2852999999999997e-05 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211058 8.7 8:efebe503:::benchmark_data_smithi169_106492_object77575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.681215+0000", 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.906332349, 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0584678489999999, 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211058 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.606 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681215+0000", 2024-03-12T03:45:10.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681215+0000", 2024-03-12T03:45:10.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681228+0000", 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3385e-05 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681233+0000", 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5700000000000003e-06 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681451+0000", 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000218364 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681459+0000", 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5290000000000003e-06 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.683093+0000", 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001634052 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.683194+0000", 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010169400000000001 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.683333+0000", 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013846600000000001 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.688997+0000", 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056637600000000003 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.697167+0000", 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.008170198 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710173+0000", 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013006172999999999 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710182+0000", 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4239999999999999e-06 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.739635+0000", 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0294528089999999 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.739683+0000", 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7425000000000001e-05 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211062 8.7 8:e1bda5df:::benchmark_data_smithi169_106492_object77579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.685338+0000", 2024-03-12T03:45:10.612 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.90220939700001, 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.054851787, 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211062 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685338+0000", 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685338+0000", 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685339+0000", 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3909999999999999e-06 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685345+0000", 2024-03-12T03:45:10.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8769999999999998e-06 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685361+0000", 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5602000000000001e-05 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685369+0000", 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3720000000000005e-06 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686032+0000", 2024-03-12T03:45:10.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00066344899999999998 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686176+0000", 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000143435 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686310+0000", 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013403300000000001 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.689003+0000", 2024-03-12T03:45:10.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026933310000000002 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.700621+0000", 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011617689 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710236+0000", 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0096148210000000008 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710244+0000", 2024-03-12T03:45:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2730000000000002e-06 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740142+0000", 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.029898373 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740189+0000", 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7141000000000001e-05 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211063 8.7 8:e4c4cdd4:::benchmark_data_smithi169_106492_object77580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.685985+0000", 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.90156243999999, 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0542865020000001, 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211063 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685985+0000", 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685985+0000", 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685985+0000", 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4e-07 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.620 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685991+0000", 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0719999999999999e-06 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686001+0000", 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0467e-05 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686009+0000", 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.503e-06 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.689034+0000", 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003025659 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.689186+0000", 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000152027 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.689335+0000", 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014890899999999999 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.691991+0000", 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026562019999999999 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.701569+0000", 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0095778640000000002 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710278+0000", 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087084380000000006 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710285+0000", 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7600000000000002e-06 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740215+0000", 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0299298880000001 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740271+0000", 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5872999999999999e-05 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.624 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211068 8.7 8:e27717b5:::benchmark_data_smithi169_106492_object77585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.693940+0000", 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.893606672, 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0464507999999999, 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211068 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693940+0000", 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693940+0000", 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693942+0000", 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.567e-06 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693948+0000", 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2949999999999999e-06 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693964+0000", 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5559000000000001e-05 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693973+0000", 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0480000000000004e-06 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.694033+0000", 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9882e-05 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.694152+0000", 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011884599999999999 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.694277+0000", 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012499899999999999 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.697558+0000", 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032812539999999999 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.701613+0000", 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040555299999999999 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710314+0000", 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087006870000000004 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710320+0000", 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3690000000000003e-06 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740325+0000", 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0300041929999999 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740391+0000", 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6570999999999997e-05 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.630 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211087 8.7 8:e3d7750f:::benchmark_data_smithi169_106492_object77604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.732661+0000", 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.854885604, 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.015391752, 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211087 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732661+0000", 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732661+0000", 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732663+0000", 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6509999999999999e-06 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732670+0000", 2024-03-12T03:45:10.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.562e-06 2024-03-12T03:45:10.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732685+0000", 2024-03-12T03:45:10.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.543e-05 2024-03-12T03:45:10.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732695+0000", 2024-03-12T03:45:10.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4369999999999996e-06 2024-03-12T03:45:10.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732912+0000", 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000217536 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732932+0000", 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9851999999999999e-05 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740436+0000", 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.007504204 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740552+0000", 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115425 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740681+0000", 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000129512 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.747888+0000", 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072064850000000003 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.747948+0000", 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0829000000000003e-05 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.747962+0000", 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3424999999999999e-05 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748006+0000", 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3905000000000001e-05 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748053+0000", 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7499e-05 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211134 8.7 8:eaa19edb:::benchmark_data_smithi169_106492_object77651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.783398+0000", 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.804148983, 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96989582799999996, 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211134 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783398+0000", 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783398+0000", 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783400+0000", 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6080000000000001e-06 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783405+0000", 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3249999999999998e-06 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783420+0000", 2024-03-12T03:45:10.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5267000000000001e-05 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783429+0000", 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5690000000000005e-06 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783496+0000", 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7441999999999996e-05 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783510+0000", 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4178e-05 2024-03-12T03:45:10.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.743998+0000", 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96048739500000002 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.744100+0000", 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101648 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.744209+0000", 2024-03-12T03:45:10.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010987700000000001 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748122+0000", 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0039126300000000003 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.750470+0000", 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023481650000000001 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753230+0000", 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002759662 2024-03-12T03:45:10.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753263+0000", 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3636000000000001e-05 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753294+0000", 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0426e-05 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211149 8.7 8:e1de3961:::benchmark_data_smithi169_106492_object77666:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:45:10.644 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.803392+0000", 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.784154726, 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.95420632900000002, 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211149 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803392+0000", 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803392+0000", 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803394+0000", 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4899999999999999e-06 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803400+0000", 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5250000000000002e-06 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803413+0000", 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2622e-05 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803421+0000", 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9310000000000008e-06 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803476+0000", 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5510999999999998e-05 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803483+0000", 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6989999999999997e-06 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748175+0000", 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.94469182799999996 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748276+0000", 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100855 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748387+0000", 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011091600000000001 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753338+0000", 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049511879999999996 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753348+0000", 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6150000000000003e-06 2024-03-12T03:45:10.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757502+0000", 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004154952 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757557+0000", 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4391000000000002e-05 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757599+0000", 2024-03-12T03:45:10.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1805999999999999e-05 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211151 8.7 8:e778f11b:::benchmark_data_smithi169_106492_object77668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.804839+0000", 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.78270845599999, 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.95295800399999997, 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.651 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211151 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804839+0000", 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804839+0000", 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804839+0000", 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4799999999999997e-07 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804844+0000", 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.899e-06 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804859+0000", 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4983e-05 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804867+0000", 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5610000000000001e-06 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804920+0000", 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2599000000000002e-05 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804956+0000", 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6235000000000001e-05 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.750501+0000", 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.94554544200000001 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.750600+0000", 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8881000000000004e-05 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.750743+0000", 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014310700000000001 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757648+0000", 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069051119999999997 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757696+0000", 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8081999999999998e-05 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757721+0000", 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4800999999999999e-05 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757762+0000", 2024-03-12T03:45:10.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0550999999999999e-05 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757797+0000", 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4903000000000001e-05 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211165 8.7 8:e0ccb79f:::benchmark_data_smithi169_106492_object77682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.852948+0000", 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.73459877400001, 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.90957021500000002, 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.657 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211165 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852948+0000", 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852948+0000", 2024-03-12T03:45:10.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852949+0000", 2024-03-12T03:45:10.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0899999999999999e-06 2024-03-12T03:45:10.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:10.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:10.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:10.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852955+0000", 2024-03-12T03:45:10.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.519e-06 2024-03-12T03:45:10.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852968+0000", 2024-03-12T03:45:10.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2690999999999999e-05 2024-03-12T03:45:10.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852976+0000", 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9750000000000007e-06 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.853032+0000", 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6852999999999999e-05 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.853043+0000", 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0497e-05 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.662 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753375+0000", 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.90033189899999999 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753473+0000", 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8413999999999997e-05 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753540+0000", 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6861999999999998e-05 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757805+0000", 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042649519999999998 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757850+0000", 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5198999999999999e-05 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762430+0000", 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045799810000000003 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762477+0000", 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7261e-05 2024-03-12T03:45:10.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762519+0000", 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1022000000000001e-05 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211182 8.7 8:e254d1c7:::benchmark_data_smithi169_106492_object77699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.870732+0000", 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.71681484000001, 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89188010699999998, 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.665 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211182 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870732+0000", 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870732+0000", 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870734+0000", 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3179999999999999e-06 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870738+0000", 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9309999999999999e-06 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870750+0000", 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1039999999999999e-05 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870755+0000", 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9070000000000004e-06 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870804+0000", 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8819999999999997e-05 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870811+0000", 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1790000000000002e-06 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.755458+0000", 2024-03-12T03:45:10.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88464690400000001 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.755563+0000", 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000104603 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.755655+0000", 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2262999999999994e-05 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.760061+0000", 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044056720000000002 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762393+0000", 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023321420000000002 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762551+0000", 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000157558 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762585+0000", 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4313999999999997e-05 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762612+0000", 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7455999999999999e-05 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211190 8.7 8:e2ab686f:::benchmark_data_smithi169_106492_object77707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.880970+0000", 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.70657689500001, 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89412791400000002, 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.671 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211190 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880970+0000", 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880970+0000", 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880972+0000", 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.491e-06 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880978+0000", 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4860000000000002e-06 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880996+0000", 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7357999999999999e-05 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.881005+0000", 2024-03-12T03:45:10.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1230000000000006e-06 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.881061+0000", 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6594000000000001e-05 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.881071+0000", 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0067e-05 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757895+0000", 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.87682368799999999 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757991+0000", 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6284000000000002e-05 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.758109+0000", 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117265 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762403+0000", 2024-03-12T03:45:10.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042943770000000003 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762648+0000", 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00024558700000000002 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.775012+0000", 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012363733 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.775060+0000", 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7663000000000001e-05 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.775098+0000", 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8198000000000003e-05 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211197 8.7 8:e6e0739f:::benchmark_data_smithi169_106492_object77714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.890205+0000", 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.69734208400001, 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88828141800000004, 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.677 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211197 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890205+0000", 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890205+0000", 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890206+0000", 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.61e-07 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890209+0000", 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7299999999999999e-06 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890216+0000", 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8040000000000004e-06 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890221+0000", 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9200000000000003e-06 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890270+0000", 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8677999999999997e-05 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890278+0000", 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1249999999999993e-06 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.760085+0000", 2024-03-12T03:45:10.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.869807471 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.760173+0000", 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7177999999999998e-05 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.760276+0000", 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010333299999999999 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762620+0000", 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023441180000000001 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.766983+0000", 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043625510000000001 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778399+0000", 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011416608999999999 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778446+0000", 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6796999999999999e-05 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778486+0000", 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0442999999999999e-05 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211199 8.7 8:e601f7fc:::benchmark_data_smithi169_106492_object77716:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.893694+0000", 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.693853426, 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88494738100000003, 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.683 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211199 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893694+0000", 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893694+0000", 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893695+0000", 2024-03-12T03:45:10.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.173e-06 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893700+0000", 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3380000000000004e-06 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893713+0000", 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2649e-05 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893721+0000", 2024-03-12T03:45:10.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7689999999999996e-06 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893767+0000", 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5949999999999999e-05 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893776+0000", 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0680000000000003e-06 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762678+0000", 2024-03-12T03:45:10.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.86890198600000002 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762763+0000", 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5915999999999995e-05 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762862+0000", 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8690999999999994e-05 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.767024+0000", 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041618280000000002 2024-03-12T03:45:10.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771291+0000", 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042668070000000001 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778547+0000", 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072558479999999996 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778601+0000", 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4703000000000001e-05 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778641+0000", 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9654999999999999e-05 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211211 8.7 8:e437348b:::benchmark_data_smithi169_106492_object77728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.933217+0000", 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.654330104, 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.84557843200000005, 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.689 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211211 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933217+0000", 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933217+0000", 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933218+0000", 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1030000000000001e-06 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933223+0000", 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6909999999999996e-06 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933232+0000", 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8740000000000001e-06 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933237+0000", 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6790000000000002e-06 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933283+0000", 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5688e-05 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933290+0000", 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.007e-06 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.764859+0000", 2024-03-12T03:45:10.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.83156881400000005 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.764956+0000", 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7408e-05 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.765060+0000", 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103928 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771299+0000", 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006238609 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771351+0000", 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.257e-05 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778700+0000", 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0073482479999999999 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778758+0000", 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8440999999999998e-05 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778795+0000", 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7372000000000001e-05 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211219 8.7 8:ec912678:::benchmark_data_smithi169_106492_object77736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.959590+0000", 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.627956657, 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81933082300000004, 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.695 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211219 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959590+0000", 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959590+0000", 2024-03-12T03:45:10.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959592+0000", 2024-03-12T03:45:10.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.156e-06 2024-03-12T03:45:10.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959596+0000", 2024-03-12T03:45:10.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6450000000000004e-06 2024-03-12T03:45:10.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959608+0000", 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1902000000000001e-05 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959616+0000", 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0779999999999996e-06 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959674+0000", 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7806000000000002e-05 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959685+0000", 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0841e-05 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.767051+0000", 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80736609000000004 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.767150+0000", 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9516000000000001e-05 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.767246+0000", 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5678000000000001e-05 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771306+0000", 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040594740000000004 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.775173+0000", 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038675329999999998 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778846+0000", 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036730550000000002 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778887+0000", 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1253e-05 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778921+0000", 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3796e-05 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211222 8.7 8:e61e560a:::benchmark_data_smithi169_106492_object77739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.969003+0000", 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 160.61854381699999, 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81401453999999995, 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211222 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969003+0000", 2024-03-12T03:45:10.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969003+0000", 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969005+0000", 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2640000000000001e-06 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969009+0000", 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3549999999999998e-06 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969019+0000", 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0091e-05 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969026+0000", 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4869999999999996e-06 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969072+0000", 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5597999999999999e-05 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969078+0000", 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.003e-06 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.769181+0000", 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80010249300000003 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.769269+0000", 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8134999999999996e-05 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.769375+0000", 2024-03-12T03:45:10.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000106086 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771421+0000", 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0020459520000000002 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778959+0000", 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075385310000000002 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.706 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.782940+0000", 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003980946 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.782981+0000", 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1287e-05 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.783018+0000", 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6312000000000001e-05 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.707 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.15279.0:42020 9.f e288/278 9:f116cb12:::benchmark_data_smithi169_135229_object42019:head v 288'2652, mlcod=288'2650)", 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:44:11.334197+0000", 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 59.253349952999997, 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.141876372, 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.334197+0000", 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.334197+0000", 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.334197+0000", 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1e-07 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.334251+0000", 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3226999999999997e-05 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.334258+0000", 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0129999999999996e-06 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.334263+0000", 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4559999999999999e-06 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.334309+0000", 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6260000000000001e-05 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.334331+0000", 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1389000000000001e-05 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.476029+0000", 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.141698352 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.476073+0000", 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4365e-05 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:42026 9.0 9:0b657610:::benchmark_data_smithi169_135229_object42025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:44:11.334435+0000", 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 59.253112282000004, 2024-03-12T03:45:10.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13940712899999999, 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 42026 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.334435+0000", 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.334435+0000", 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.334436+0000", 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1600000000000001e-07 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.334456+0000", 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0210000000000001e-05 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.468838+0000", 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.134382011 2024-03-12T03:45:10.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.468852+0000", 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4467000000000001e-05 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.470810+0000", 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001957795 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.470952+0000", 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000142039 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2,6", 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.471081+0000", 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012882599999999999 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.472982+0000", 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001900987 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.473540+0000", 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00055797400000000004 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.473582+0000", 2024-03-12T03:45:10.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2401000000000003e-05 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.473791+0000", 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020916399999999999 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:44:11.473842+0000", 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0538999999999998e-05 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:10.716 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:45:10.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:10.803 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:10.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:10.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:10.905 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:10.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:10 smithi169 ceph-mon[29344]: 3.1a scrub starts 2024-03-12T03:45:10.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:10 smithi169 ceph-mon[29344]: 3.1a scrub ok 2024-03-12T03:45:10.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:10 smithi169 ceph-mon[29344]: 5.3 scrub starts 2024-03-12T03:45:10.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:10 smithi169 ceph-mon[29344]: 5.3 scrub ok 2024-03-12T03:45:10.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:10 smithi169 ceph-mon[39949]: 3.1a scrub starts 2024-03-12T03:45:10.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:10 smithi169 ceph-mon[39949]: 3.1a scrub ok 2024-03-12T03:45:10.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:10 smithi169 ceph-mon[39949]: 5.3 scrub starts 2024-03-12T03:45:10.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:10 smithi169 ceph-mon[39949]: 5.3 scrub ok 2024-03-12T03:45:10.945 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:10.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:10.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:10 smithi173 ceph-mon[33544]: 3.1a scrub starts 2024-03-12T03:45:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:10 smithi173 ceph-mon[33544]: 3.1a scrub ok 2024-03-12T03:45:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:10 smithi173 ceph-mon[33544]: 5.3 scrub starts 2024-03-12T03:45:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:10 smithi173 ceph-mon[33544]: 5.3 scrub ok 2024-03-12T03:45:11.047 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:11.078 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:11.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:11.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:11.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:11.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:11.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:11.349 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:11.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:11.386 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 76 16 130420 130404 107.226 75.5625 0.00943198 0.00932101 2024-03-12T03:45:11.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:11.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:11.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:11.591 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:11.625 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:11.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:11.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:11.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:11.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:11.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:11.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:45:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:45:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[29344]: pgmap v564: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 768 B/s rd, 64 MiB/s wr, 1.03k op/s; 62066/372339 objects degraded (16.669%); 1/372339 objects misplaced (0.000%) 2024-03-12T03:45:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[29344]: 5.9 scrub starts 2024-03-12T03:45:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[29344]: 5.9 scrub ok 2024-03-12T03:45:11.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[29344]: 3.10 scrub starts 2024-03-12T03:45:11.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[29344]: 3.10 scrub ok 2024-03-12T03:45:11.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[29344]: osdmap e310: 8 total, 7 up, 8 in 2024-03-12T03:45:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:45:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:45:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[39949]: pgmap v564: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 768 B/s rd, 64 MiB/s wr, 1.03k op/s; 62066/372339 objects degraded (16.669%); 1/372339 objects misplaced (0.000%) 2024-03-12T03:45:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[39949]: 5.9 scrub starts 2024-03-12T03:45:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[39949]: 5.9 scrub ok 2024-03-12T03:45:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[39949]: 3.10 scrub starts 2024-03-12T03:45:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[39949]: 3.10 scrub ok 2024-03-12T03:45:11.825 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-mon[39949]: osdmap e310: 8 total, 7 up, 8 in 2024-03-12T03:45:11.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:11.904 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:11.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:11 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:45:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:11 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:45:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:11 smithi173 ceph-mon[33544]: pgmap v564: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 768 B/s rd, 64 MiB/s wr, 1.03k op/s; 62066/372339 objects degraded (16.669%); 1/372339 objects misplaced (0.000%) 2024-03-12T03:45:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:11 smithi173 ceph-mon[33544]: 5.9 scrub starts 2024-03-12T03:45:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:11 smithi173 ceph-mon[33544]: 5.9 scrub ok 2024-03-12T03:45:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:11 smithi173 ceph-mon[33544]: 3.10 scrub starts 2024-03-12T03:45:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:11 smithi173 ceph-mon[33544]: 3.10 scrub ok 2024-03-12T03:45:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:11 smithi173 ceph-mon[33544]: osdmap e310: 8 total, 7 up, 8 in 2024-03-12T03:45:12.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:12.040 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:12.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:12.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:12.148 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:45:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:45:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:45:12.186 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:12.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:12.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-12T03:45:12.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:12.331 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:12.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:12.386 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 77 16 131727 131711 106.894 81.6875 0.005906 0.00935042 2024-03-12T03:45:12.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:12.470 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:12.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:12.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:12.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:12.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:12.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:12.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:12.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:12.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:12.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:12.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:12.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:45:12.934 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:45:12.984 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:12 smithi169 ceph-mon[29344]: 4.18 scrub starts 2024-03-12T03:45:12.984 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:12 smithi169 ceph-mon[29344]: 4.18 scrub ok 2024-03-12T03:45:12.984 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:12 smithi169 ceph-mon[29344]: 5.1a scrub starts 2024-03-12T03:45:12.984 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:12 smithi169 ceph-mon[29344]: 5.1a scrub ok 2024-03-12T03:45:12.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:12 smithi169 ceph-mon[39949]: 4.18 scrub starts 2024-03-12T03:45:12.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:12 smithi169 ceph-mon[39949]: 4.18 scrub ok 2024-03-12T03:45:12.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:12 smithi169 ceph-mon[39949]: 5.1a scrub starts 2024-03-12T03:45:12.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:12 smithi169 ceph-mon[39949]: 5.1a scrub ok 2024-03-12T03:45:12.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:45:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:45:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:12 smithi173 ceph-mon[33544]: 4.18 scrub starts 2024-03-12T03:45:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:12 smithi173 ceph-mon[33544]: 4.18 scrub ok 2024-03-12T03:45:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:12 smithi173 ceph-mon[33544]: 5.1a scrub starts 2024-03-12T03:45:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:12 smithi173 ceph-mon[33544]: 5.1a scrub ok 2024-03-12T03:45:12.987 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:13.022 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:13.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:13.098 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:13.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:13.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:13.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:13.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:13.291 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:13.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:13.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:13.386 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 78 16 132916 132900 106.476 74.3125 0.00679031 0.00938696 2024-03-12T03:45:13.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:13.426 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:13.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:13.528 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:13.570 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:13.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:13.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:13.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:13.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:13.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:13.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:13.838 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[29344]: pgmap v566: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 7.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 104 MiB/s wr, 1.66k op/s; 64537/388524 objects degraded (16.611%); 1/388524 objects misplaced (0.000%) 2024-03-12T03:45:13.838 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[29344]: 6.12 scrub starts 2024-03-12T03:45:13.838 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[29344]: 6.12 scrub ok 2024-03-12T03:45:13.838 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[29344]: 9.9 deep-scrub starts 2024-03-12T03:45:13.838 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[29344]: 9.a scrub starts 2024-03-12T03:45:13.838 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 64537/388524 objects degraded (16.611%), 31 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:13.838 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[29344]: osdmap e311: 8 total, 7 up, 8 in 2024-03-12T03:45:13.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[39949]: pgmap v566: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 7.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 104 MiB/s wr, 1.66k op/s; 64537/388524 objects degraded (16.611%); 1/388524 objects misplaced (0.000%) 2024-03-12T03:45:13.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[39949]: 6.12 scrub starts 2024-03-12T03:45:13.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[39949]: 6.12 scrub ok 2024-03-12T03:45:13.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[39949]: 9.9 deep-scrub starts 2024-03-12T03:45:13.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[39949]: 9.a scrub starts 2024-03-12T03:45:13.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 64537/388524 objects degraded (16.611%), 31 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:13.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:13 smithi169 ceph-mon[39949]: osdmap e311: 8 total, 7 up, 8 in 2024-03-12T03:45:13.848 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:13.849 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:13.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:13.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:13.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:13.980 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:13.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:13 smithi173 ceph-mon[33544]: pgmap v566: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 7.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 104 MiB/s wr, 1.66k op/s; 64537/388524 objects degraded (16.611%); 1/388524 objects misplaced (0.000%) 2024-03-12T03:45:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:13 smithi173 ceph-mon[33544]: 6.12 scrub starts 2024-03-12T03:45:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:13 smithi173 ceph-mon[33544]: 6.12 scrub ok 2024-03-12T03:45:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:13 smithi173 ceph-mon[33544]: 9.9 deep-scrub starts 2024-03-12T03:45:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:13 smithi173 ceph-mon[33544]: 9.a scrub starts 2024-03-12T03:45:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:13 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 64537/388524 objects degraded (16.611%), 31 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:13 smithi173 ceph-mon[33544]: osdmap e311: 8 total, 7 up, 8 in 2024-03-12T03:45:14.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:14.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:14.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:14.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:14.256 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:14.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:14.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:14.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:14.386 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 79 16 133839 133823 105.859 57.6875 0.00796254 0.0094421 2024-03-12T03:45:14.431 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:14.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:14.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:14.570 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:14.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:14.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:14.701 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:14.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:14.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:14.803 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:14.837 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:14.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:14.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:14.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:45:14.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:14 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:14 smithi173 ceph-mon[33544]: 3.9 deep-scrub starts 2024-03-12T03:45:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:14 smithi173 ceph-mon[33544]: 3.9 deep-scrub ok 2024-03-12T03:45:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:14 smithi173 ceph-mon[33544]: 4.8 scrub starts 2024-03-12T03:45:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:14 smithi173 ceph-mon[33544]: 4.8 scrub ok 2024-03-12T03:45:15.013 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:14 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:15.013 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:14 smithi169 ceph-mon[29344]: 3.9 deep-scrub starts 2024-03-12T03:45:15.013 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:14 smithi169 ceph-mon[29344]: 3.9 deep-scrub ok 2024-03-12T03:45:15.013 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:14 smithi169 ceph-mon[29344]: 4.8 scrub starts 2024-03-12T03:45:15.013 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:14 smithi169 ceph-mon[29344]: 4.8 scrub ok 2024-03-12T03:45:15.013 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:14 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:15.013 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:14 smithi169 ceph-mon[39949]: 3.9 deep-scrub starts 2024-03-12T03:45:15.013 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:14 smithi169 ceph-mon[39949]: 3.9 deep-scrub ok 2024-03-12T03:45:15.014 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:14 smithi169 ceph-mon[39949]: 4.8 scrub starts 2024-03-12T03:45:15.014 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:14 smithi169 ceph-mon[39949]: 4.8 scrub ok 2024-03-12T03:45:15.015 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:15.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:15.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:15.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:15.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:15.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:15.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:15.256 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:15.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:15.304 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:15.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:15.386 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:45:15.385496+0000 min lat: 0.00120676 max lat: 0.160571 avg lat: 0.00949174 2024-03-12T03:45:15.387 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:45:15.387 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 80 16 134807 134791 105.292 60.5 0.00604333 0.00949174 2024-03-12T03:45:15.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:15.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:15.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:15.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:15.593 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:15.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:15.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:15.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:15.736 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:15.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:15.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:15.797 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[29344]: 3.8 scrub starts 2024-03-12T03:45:15.797 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[29344]: 3.8 scrub ok 2024-03-12T03:45:15.798 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[29344]: pgmap v568: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 8.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s; 65351/392202 objects degraded (16.663%); 1/392202 objects misplaced (0.000%) 2024-03-12T03:45:15.798 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:45:15.798 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:45:15.798 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[29344]: osdmap e312: 8 total, 7 up, 8 in 2024-03-12T03:45:15.798 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:45:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[39949]: 3.8 scrub starts 2024-03-12T03:45:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[39949]: 3.8 scrub ok 2024-03-12T03:45:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[39949]: pgmap v568: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 8.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s; 65351/392202 objects degraded (16.663%); 1/392202 objects misplaced (0.000%) 2024-03-12T03:45:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:45:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:45:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[39949]: osdmap e312: 8 total, 7 up, 8 in 2024-03-12T03:45:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:45:15.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:15.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:15.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:15.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:15.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:15 smithi173 ceph-mon[33544]: 3.8 scrub starts 2024-03-12T03:45:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:15 smithi173 ceph-mon[33544]: 3.8 scrub ok 2024-03-12T03:45:15.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:15 smithi173 ceph-mon[33544]: pgmap v568: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 8.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s; 65351/392202 objects degraded (16.663%); 1/392202 objects misplaced (0.000%) 2024-03-12T03:45:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:15 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:45:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:15 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:45:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:15 smithi173 ceph-mon[33544]: osdmap e312: 8 total, 7 up, 8 in 2024-03-12T03:45:15.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:45:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:16.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:16.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:16.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:16.193 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:16.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:16.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:16.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:16.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:16.387 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 81 16 135745 135729 104.715 58.625 0.00680293 0.00954471 2024-03-12T03:45:16.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:16.480 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:16.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:16.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:16.621 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:16.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:16.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:16.723 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:16.761 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:16.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:16.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:16.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:16.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:16.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:16.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:17.033 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:17.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:17.110 INFO:teuthology.orchestra.run.smithi169.stderr:set osd.6 primary-affinity to 0 (802) 2024-03-12T03:45:17.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[29344]: 6.1a deep-scrub starts 2024-03-12T03:45:17.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[29344]: 6.1a deep-scrub ok 2024-03-12T03:45:17.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[29344]: 9.a scrub ok 2024-03-12T03:45:17.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[29344]: 4.b scrub starts 2024-03-12T03:45:17.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[29344]: 4.b scrub ok 2024-03-12T03:45:17.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:45:17.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:17.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3070014003' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-03-12T03:45:17.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-03-12T03:45:17.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[39949]: 6.1a deep-scrub starts 2024-03-12T03:45:17.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[39949]: 6.1a deep-scrub ok 2024-03-12T03:45:17.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[39949]: 9.a scrub ok 2024-03-12T03:45:17.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[39949]: 4.b scrub starts 2024-03-12T03:45:17.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[39949]: 4.b scrub ok 2024-03-12T03:45:17.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:45:17.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:17.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3070014003' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-03-12T03:45:17.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:16 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-03-12T03:45:17.119 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:45:17.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:17.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:17.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:17.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:17.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:16 smithi173 ceph-mon[33544]: 6.1a deep-scrub starts 2024-03-12T03:45:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:16 smithi173 ceph-mon[33544]: 6.1a deep-scrub ok 2024-03-12T03:45:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:16 smithi173 ceph-mon[33544]: 9.a scrub ok 2024-03-12T03:45:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:16 smithi173 ceph-mon[33544]: 4.b scrub starts 2024-03-12T03:45:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:16 smithi173 ceph-mon[33544]: 4.b scrub ok 2024-03-12T03:45:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:16 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:45:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:16 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:16 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3070014003' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-03-12T03:45:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:16 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-03-12T03:45:17.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:17.300 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:17.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:17.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:17.387 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 82 16 136712 136696 104.175 60.4375 0.00283485 0.00958528 2024-03-12T03:45:17.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:17.435 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:17.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:17.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:17.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:17.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:17.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:17.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:17.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:17.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:17.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:17.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:17.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:17.947 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:17.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:17.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:18.049 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:18.073 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:18.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:18.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:18.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: 4.13 scrub starts 2024-03-12T03:45:18.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: 4.13 scrub ok 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: pgmap v570: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 8.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s; 65351/392202 objects degraded (16.663%); 1/392202 objects misplaced (0.000%) 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: 5.4 scrub starts 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: 5.4 scrub ok 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1228056405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: osdmap e313: 8 total, 7 up, 8 in 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:45:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: 4.13 scrub starts 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: 4.13 scrub ok 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: pgmap v570: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 8.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s; 65351/392202 objects degraded (16.663%); 1/392202 objects misplaced (0.000%) 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: 5.4 scrub starts 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: 5.4 scrub ok 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1228056405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: osdmap e313: 8 total, 7 up, 8 in 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:45:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:17 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:45:18.175 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:18.205 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:18.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:18.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:18.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: 4.13 scrub starts 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: 4.13 scrub ok 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: pgmap v570: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 8.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s; 65351/392202 objects degraded (16.663%); 1/392202 objects misplaced (0.000%) 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: 5.4 scrub starts 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: 5.4 scrub ok 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1228056405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: osdmap e313: 8 total, 7 up, 8 in 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:45:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:17 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:45:18.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:18.344 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:18.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:18.387 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 83 16 137008 136992 103.143 18.5 0.00604768 0.00968788 2024-03-12T03:45:18.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:18.478 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:18.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:18.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:18.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:18.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:18.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:18.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:18.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:18.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:18.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:18.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:18.883 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:18.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:18.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:18.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:19.016 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:19.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:19.118 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:19.150 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:19.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:19.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:19.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:19.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:19.387 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 84 16 138053 138037 102.693 65.3125 0.00524329 0.00973161 2024-03-12T03:45:19.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:19.421 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:19.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:19.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:19 smithi169 ceph-mon[29344]: pgmap v572: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 1 peering, 31 active+undersized+degraded, 37 active+undersized, 93 active+clean; 8.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 682 B/s rd, 68 MiB/s wr, 1.09k op/s; 67987/408195 objects degraded (16.656%); 1/408195 objects misplaced (0.000%) 2024-03-12T03:45:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:19 smithi169 ceph-mon[29344]: osdmap e314: 8 total, 7 up, 8 in 2024-03-12T03:45:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:19 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 65351/392202 objects degraded (16.663%), 31 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:19 smithi169 ceph-mon[29344]: osdmap e315: 8 total, 7 up, 8 in 2024-03-12T03:45:19.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:19 smithi169 ceph-mon[29344]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-12T03:45:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:19 smithi169 ceph-mon[39949]: pgmap v572: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 1 peering, 31 active+undersized+degraded, 37 active+undersized, 93 active+clean; 8.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 682 B/s rd, 68 MiB/s wr, 1.09k op/s; 67987/408195 objects degraded (16.656%); 1/408195 objects misplaced (0.000%) 2024-03-12T03:45:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:19 smithi169 ceph-mon[39949]: osdmap e314: 8 total, 7 up, 8 in 2024-03-12T03:45:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:19 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 65351/392202 objects degraded (16.663%), 31 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:19 smithi169 ceph-mon[39949]: osdmap e315: 8 total, 7 up, 8 in 2024-03-12T03:45:19.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:19 smithi169 ceph-mon[39949]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-12T03:45:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:19 smithi173 ceph-mon[33544]: pgmap v572: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 1 peering, 31 active+undersized+degraded, 37 active+undersized, 93 active+clean; 8.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 682 B/s rd, 68 MiB/s wr, 1.09k op/s; 67987/408195 objects degraded (16.656%); 1/408195 objects misplaced (0.000%) 2024-03-12T03:45:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:19 smithi173 ceph-mon[33544]: osdmap e314: 8 total, 7 up, 8 in 2024-03-12T03:45:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:19 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 65351/392202 objects degraded (16.663%), 31 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:19 smithi173 ceph-mon[33544]: osdmap e315: 8 total, 7 up, 8 in 2024-03-12T03:45:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:19 smithi173 ceph-mon[33544]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-12T03:45:19.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:19.551 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:19.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:19.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:19.689 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:19.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:19.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:19.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:19.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:19.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:19.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:19.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:19.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:20.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:20.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:20.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:20.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:20.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:20.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:20.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:20.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:20.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:20.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:20.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:20.387 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 85 16 139122 139106 102.27 66.8125 0.0173458 0.0097732 2024-03-12T03:45:20.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:20.519 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:20.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:20.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:20.656 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:20.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:20.758 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:20.790 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:20.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:20.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:20.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:20.920 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:20.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:20.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:21.008 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:45:21.008 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:45:21.008 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:21.008 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.15279.0:139757 9.0 e315/278 9:02b0171f:::benchmark_data_smithi169_135229_object139756:head v 315'8772, mlcod=315'8770)", 2024-03-12T03:45:21.008 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:45:21.003981+0000", 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.001919264, 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0019515190000000001, 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:45:21.003981+0000", 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:45:21.003981+0000", 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:45:21.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:21.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:21.010 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:45:21.010 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:45:21.003982+0000", 2024-03-12T03:45:21.010 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.186e-06 2024-03-12T03:45:21.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:21.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:21.010 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:45:21.010 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:45:21.004008+0000", 2024-03-12T03:45:21.010 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5326000000000001e-05 2024-03-12T03:45:21.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:21.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:21.011 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:45:21.011 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:45:21.004020+0000", 2024-03-12T03:45:21.011 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2354999999999999e-05 2024-03-12T03:45:21.011 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:21.011 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:21.011 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:45:21.011 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:45:21.004029+0000", 2024-03-12T03:45:21.011 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9080000000000001e-06 2024-03-12T03:45:21.011 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:21.011 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:21.011 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:45:21.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:45:21.004180+0000", 2024-03-12T03:45:21.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015132399999999999 2024-03-12T03:45:21.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:45:21.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:45:21.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:45:21.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:45:21.004240+0000", 2024-03-12T03:45:21.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9497000000000003e-05 2024-03-12T03:45:21.012 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:21.012 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:45:21.013 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:21.013 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:45:21.013 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:45:21.013 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 1 2024-03-12T03:45:21.013 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:45:21.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:21.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:21.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:21.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:21.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:21.206 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:21.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:21.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:21.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:21.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:21.387 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 86 16 140156 140140 101.833 64.625 0.00403084 0.00981469 2024-03-12T03:45:21.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:21 smithi169 ceph-mon[29344]: pgmap v575: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 1 peering, 31 active+undersized+degraded, 37 active+undersized, 93 active+clean; 8.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 773 B/s rd, 63 MiB/s wr, 1.01k op/s; 67987/408195 objects degraded (16.656%); 1/408195 objects misplaced (0.000%) 2024-03-12T03:45:21.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:21 smithi169 ceph-mon[39949]: pgmap v575: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 1 peering, 31 active+undersized+degraded, 37 active+undersized, 93 active+clean; 8.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 773 B/s rd, 63 MiB/s wr, 1.01k op/s; 67987/408195 objects degraded (16.656%); 1/408195 objects misplaced (0.000%) 2024-03-12T03:45:21.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:21.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:21.482 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:21.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:21.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:21 smithi173 ceph-mon[33544]: pgmap v575: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 1 peering, 31 active+undersized+degraded, 37 active+undersized, 93 active+clean; 8.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 773 B/s rd, 63 MiB/s wr, 1.01k op/s; 67987/408195 objects degraded (16.656%); 1/408195 objects misplaced (0.000%) 2024-03-12T03:45:21.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:21.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:21.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:21.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:21.721 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:21.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:21.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:21.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:21.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:21.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:21.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-12T03:45:21.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:22.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:22.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:22.053 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:45:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:45:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:45:22.132 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:22.168 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:22.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:22.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:22.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:22.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:22.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:45:22.388 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 87 16 141163 141147 101.385 62.9375 0.00464154 0.00985837 2024-03-12T03:45:22.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:22.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:22.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:22.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:22.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:22.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:22.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:22.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:22.706 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:22.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:22.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:22.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:22.840 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:22.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:22.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:22.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:22.955 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:23.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:23.001 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:23.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:23.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:23.137 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:23.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:23.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:23.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:23 smithi169 ceph-mon[39949]: pgmap v576: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 1 peering, 31 active+undersized+degraded, 37 active+undersized, 93 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 5.7 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 69630/418875 objects degraded (16.623%); 1/418875 objects misplaced (0.000%); 26 KiB/s, 0 objects/s recovering 2024-03-12T03:45:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:23 smithi169 ceph-mon[29344]: pgmap v576: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 1 peering, 31 active+undersized+degraded, 37 active+undersized, 93 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 5.7 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 69630/418875 objects degraded (16.623%); 1/418875 objects misplaced (0.000%); 26 KiB/s, 0 objects/s recovering 2024-03-12T03:45:23.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:45:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:45:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:23 smithi173 ceph-mon[33544]: pgmap v576: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 1 peering, 31 active+undersized+degraded, 37 active+undersized, 93 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 5.7 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 69630/418875 objects degraded (16.623%); 1/418875 objects misplaced (0.000%); 26 KiB/s, 0 objects/s recovering 2024-03-12T03:45:23.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:23.270 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:23.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:23.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:23.388 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 88 16 142212 142196 100.978 65.5625 0.00604413 0.00989859 2024-03-12T03:45:23.410 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:23.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:23.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:23.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:23.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:23.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:23.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:23.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:23.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:23.773 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:23.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:23.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:23.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:23.903 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:23.940 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:23.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:24.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:24.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:24.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:24.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:24.185 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:24.222 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:24.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:24.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:24.348 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:24.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:24.388 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 89 16 143178 143162 100.521 60.375 0.00355909 0.00994261 2024-03-12T03:45:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:24 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 69630/418875 objects degraded (16.623%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:24.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:24 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 69630/418875 objects degraded (16.623%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:24.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:24.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:24.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:24 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 69630/418875 objects degraded (16.623%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:24.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:24.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:24.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:24.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:24.623 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:24.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:24.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:24.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:24.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:24.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:24.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:24.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:24.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:25.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:25.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:25.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:25.172 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:25.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:25.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:25.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:25.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:25.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:25.388 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 90 16 144212 144196 100.122 64.625 0.00333223 0.0099803 2024-03-12T03:45:25.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:25.417 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-12T03:45:25.417 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:45:25.418 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-12T03:45:25.418 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-12T03:45:25.418 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-12T03:45:25.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-12T03:45:25.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:25 smithi169 ceph-mon[29344]: pgmap v577: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 4.2 KiB/s rd, 49 MiB/s wr, 784 op/s; 70205/421479 objects degraded (16.657%); 1/421479 objects misplaced (0.000%); 23 KiB/s, 0 objects/s recovering 2024-03-12T03:45:25.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:25 smithi169 ceph-mon[29344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-12T03:45:25.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:25 smithi169 ceph-mon[39949]: pgmap v577: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 4.2 KiB/s rd, 49 MiB/s wr, 784 op/s; 70205/421479 objects degraded (16.657%); 1/421479 objects misplaced (0.000%); 23 KiB/s, 0 objects/s recovering 2024-03-12T03:45:25.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:25 smithi169 ceph-mon[39949]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-12T03:45:25.446 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:25.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:25.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:25.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:25 smithi173 ceph-mon[33544]: pgmap v577: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 4.2 KiB/s rd, 49 MiB/s wr, 784 op/s; 70205/421479 objects degraded (16.657%); 1/421479 objects misplaced (0.000%); 23 KiB/s, 0 objects/s recovering 2024-03-12T03:45:25.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:25 smithi173 ceph-mon[33544]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-12T03:45:25.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:25.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:25.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:25.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:25.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:25.698 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:25.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:25.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:25.828 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:25.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:25.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:25.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:25.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:25.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:25.983 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:26.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:26.084 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:26.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:26.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:26.214 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:26.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:26.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:26.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:26.342 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:26.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:26.388 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Total time run: 90.0615 2024-03-12T03:45:26.388 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Total writes made: 144212 2024-03-12T03:45:26.388 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Write size: 65536 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Object size: 65536 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Bandwidth (MB/sec): 100.079 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev Bandwidth: 37.3584 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max bandwidth (MB/sec): 250.062 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min bandwidth (MB/sec): 18.5 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Average IOPS: 1601 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev IOPS: 597.735 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max IOPS: 4001 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min IOPS: 296 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Average Latency(s): 0.00998495 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev Latency(s): 0.0125967 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max latency(s): 1.02075 2024-03-12T03:45:26.389 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min latency(s): 0.00120676 2024-03-12T03:45:26.393 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-03-12T03:45:26.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:26.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:26.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:26.578 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:26.611 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:26.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:26.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:26.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:26.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:26.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:26.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:26.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:26.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:26.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:26.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:27.008 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:27.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:27.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:27.110 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:27.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:27.147 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:27.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:27.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:27.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:27.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:27.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:27.402 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:27 smithi169 ceph-mon[29344]: pgmap v578: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 3.7 KiB/s rd, 35 MiB/s wr, 557 op/s; 70205/421479 objects degraded (16.657%); 1/421479 objects misplaced (0.000%); 20 KiB/s, 0 objects/s recovering 2024-03-12T03:45:27.402 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:27 smithi169 ceph-mon[39949]: pgmap v578: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 3.7 KiB/s rd, 35 MiB/s wr, 557 op/s; 70205/421479 objects degraded (16.657%); 1/421479 objects misplaced (0.000%); 20 KiB/s, 0 objects/s recovering 2024-03-12T03:45:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:27.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:27.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:27.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:27 smithi173 ceph-mon[33544]: pgmap v578: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 3.7 KiB/s rd, 35 MiB/s wr, 557 op/s; 70205/421479 objects degraded (16.657%); 1/421479 objects misplaced (0.000%); 20 KiB/s, 0 objects/s recovering 2024-03-12T03:45:27.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:27.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:27.546 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:27.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:27.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:27.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:27.691 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:27.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:27.738 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:45:27.768 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:45:27.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:27.805 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:45:27.826 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:27.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:27.842 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:45:27.874 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:45:27.906 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:45:27.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:27.943 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:45:27.943 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:45:27.943 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:45:27.943 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:45:27.943 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:45:27.943 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:45:27.943 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:45:27.944 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:45:27.944 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:45:27.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:27.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:28.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:28.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:28.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:28.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:28.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:28.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:28.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:28.336 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:28.375 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:28.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:28.395 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:28 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1960602482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:28.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:28 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1960602482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:28.403 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:28.476 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:28.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:28 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1960602482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:28.492 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:45:28.492 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:45:28.515 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:28.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:28.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:28.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:28.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:28.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:28.784 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:28.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:28.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:28.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:28.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:28.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:29.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:29.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:29.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:29.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:29.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:29.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:29.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:29.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:45:29.231 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:45:29.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:29.335 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:29.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:29.387 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:29 smithi173 ceph-mon[33544]: pgmap v579: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 3.5 KiB/s rd, 53 MiB/s wr, 875 op/s; 72049/432474 objects degraded (16.660%); 1/432474 objects misplaced (0.000%); 16 KiB/s, 0 objects/s recovering 2024-03-12T03:45:29.388 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:29 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 72049/432474 objects degraded (16.660%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:29.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:29 smithi169 ceph-mon[29344]: pgmap v579: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 3.5 KiB/s rd, 53 MiB/s wr, 875 op/s; 72049/432474 objects degraded (16.660%); 1/432474 objects misplaced (0.000%); 16 KiB/s, 0 objects/s recovering 2024-03-12T03:45:29.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:29 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 72049/432474 objects degraded (16.660%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:29.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:29 smithi169 ceph-mon[39949]: pgmap v579: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 3.5 KiB/s rd, 53 MiB/s wr, 875 op/s; 72049/432474 objects degraded (16.660%); 1/432474 objects misplaced (0.000%); 16 KiB/s, 0 objects/s recovering 2024-03-12T03:45:29.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:29 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 72049/432474 objects degraded (16.660%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:29.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:29.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:29.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:29.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:29.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:29.611 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:29.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:29.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:29.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:29.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:29.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:29.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:29.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:29.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:30.022 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:30.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:30.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:30.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:30.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:30.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:30.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:30.291 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:30.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:30.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:30.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:30.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:30.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:30.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:30.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:30.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:30.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:30.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:30.718 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:30.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:30.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:30.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:30.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:30.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:30.983 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:30.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:30.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:31.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:31.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-12T03:45:31.124 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:31.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:31.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:31.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:31.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:45:31.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:31.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:31.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:31.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:31.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:31.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:31.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:31 smithi169 ceph-mon[39949]: pgmap v580: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 3.0 KiB/s rd, 45 MiB/s wr, 745 op/s; 72049/432474 objects degraded (16.660%); 1/432474 objects misplaced (0.000%); 13 KiB/s, 0 objects/s recovering 2024-03-12T03:45:31.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:31 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:45:31.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:31.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:31 smithi169 ceph-mon[29344]: pgmap v580: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 3.0 KiB/s rd, 45 MiB/s wr, 745 op/s; 72049/432474 objects degraded (16.660%); 1/432474 objects misplaced (0.000%); 13 KiB/s, 0 objects/s recovering 2024-03-12T03:45:31.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:31 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:45:31.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:31.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:31 smithi173 ceph-mon[33544]: pgmap v580: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 3.0 KiB/s rd, 45 MiB/s wr, 745 op/s; 72049/432474 objects degraded (16.660%); 1/432474 objects misplaced (0.000%); 13 KiB/s, 0 objects/s recovering 2024-03-12T03:45:31.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:31 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:45:31.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:31.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:31.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:31.531 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:31.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:31.632 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:31.679 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:31.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:31.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:31.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:31.815 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:31.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:31.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:31.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:31.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:31.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:32.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:32.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:32.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:32.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:45:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:45:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:45:32.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:32.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:32.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:32.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:32.348 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:32.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:32.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:32.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:32.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:32.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:32.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:32.627 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:32.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:32.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:32.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:32.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:32.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:32.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:32.897 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:32.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:32.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:32.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:33.029 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:33.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:33.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:33.166 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:33.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:33.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:45:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:45:33.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:33 smithi173 ceph-mon[33544]: pgmap v581: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 3.4 KiB/s rd, 44 MiB/s wr, 1.07k op/s; 70112/419868 objects degraded (16.699%); 1/419868 objects misplaced (0.000%); 13 KiB/s, 0 objects/s recovering 2024-03-12T03:45:33.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:33.301 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:33.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:33.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:33.421 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:33 smithi169 ceph-mon[29344]: pgmap v581: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 3.4 KiB/s rd, 44 MiB/s wr, 1.07k op/s; 70112/419868 objects degraded (16.699%); 1/419868 objects misplaced (0.000%); 13 KiB/s, 0 objects/s recovering 2024-03-12T03:45:33.421 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:33 smithi169 ceph-mon[39949]: pgmap v581: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 3.4 KiB/s rd, 44 MiB/s wr, 1.07k op/s; 70112/419868 objects degraded (16.699%); 1/419868 objects misplaced (0.000%); 13 KiB/s, 0 objects/s recovering 2024-03-12T03:45:33.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:33.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:33.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:33.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:33.579 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:33.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:33.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:33.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:33.718 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:33.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:33.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:33.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:33.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:33.952 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:33.986 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:33.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:34.088 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:34.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:34.125 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:34.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:34.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:34.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:34.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:34.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:34.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:34.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:34 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 70112/419868 objects degraded (16.699%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:34.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:34.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:34 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 70112/419868 objects degraded (16.699%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:34.359 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:34.398 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:34.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:34.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:34.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:34 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 70112/419868 objects degraded (16.699%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:34.499 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:34.511 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:45:34.511 INFO:teuthology.orchestra.run.smithi169.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-12T03:45:34.511 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:45:34.530 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:34.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:34.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:34.536 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:34.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:34.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:34.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:34.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:34.802 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:34.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:34.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:34.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:34.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:34.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:34.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:35.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:35.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:35.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:35.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:35.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:35.203 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:35.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:35.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:35.335 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:35.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:35.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:35.396 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:45:35.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:35.471 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:35.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:35.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:35.607 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:35.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:35.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:35.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:35 smithi169 ceph-mon[29344]: pgmap v582: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 938 B/s rd, 25 MiB/s wr, 855 op/s; 69473/417006 objects degraded (16.660%); 1/417006 objects misplaced (0.000%) 2024-03-12T03:45:35.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:35 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1166887791' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:35.651 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:35 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:35.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:35 smithi169 ceph-mon[39949]: pgmap v582: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 938 B/s rd, 25 MiB/s wr, 855 op/s; 69473/417006 objects degraded (16.660%); 1/417006 objects misplaced (0.000%) 2024-03-12T03:45:35.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:35 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1166887791' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:35.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:35 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:35.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:35 smithi173 ceph-mon[33544]: pgmap v582: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 938 B/s rd, 25 MiB/s wr, 855 op/s; 69473/417006 objects degraded (16.660%); 1/417006 objects misplaced (0.000%) 2024-03-12T03:45:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:35 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1166887791' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:35.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:35 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:35.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:35.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:35.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:35.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:35.883 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:35.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:35.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:36.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:36.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:36.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:36.162 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:36.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:36.175 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:36.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:36.299 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:36.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:36.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:36.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:36.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:36 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:45:36.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:36 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:45:36.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:36 smithi169 ceph-mon[29344]: osdmap e316: 8 total, 7 up, 8 in 2024-03-12T03:45:36.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:36 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:45:36.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:36 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:45:36.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:36 smithi169 ceph-mon[39949]: osdmap e316: 8 total, 7 up, 8 in 2024-03-12T03:45:36.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:36.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:36.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:36.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:36.575 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:36.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:36.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:36.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:36.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:36.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:36 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:45:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:36 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:45:36.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:36 smithi173 ceph-mon[33544]: osdmap e316: 8 total, 7 up, 8 in 2024-03-12T03:45:36.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:36.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:36.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:36.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:36.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:36.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:36.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:37.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:37.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:37.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:37.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:37.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:37.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:37.357 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:37.388 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:37.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:37.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:37 smithi169 ceph-mon[29344]: pgmap v584: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 939 op/s; 69473/417006 objects degraded (16.660%); 1/417006 objects misplaced (0.000%) 2024-03-12T03:45:37.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:37 smithi169 ceph-mon[39949]: pgmap v584: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 939 op/s; 69473/417006 objects degraded (16.660%); 1/417006 objects misplaced (0.000%) 2024-03-12T03:45:37.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:37.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:37.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:37.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:37.649 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:37.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:37 smithi173 ceph-mon[33544]: pgmap v584: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 939 op/s; 69473/417006 objects degraded (16.660%); 1/417006 objects misplaced (0.000%) 2024-03-12T03:45:37.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:37.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:37.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:37.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:37.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:37.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:38.028 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:38.061 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:38.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:38.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:38.193 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:38.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:38.214 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:45:38.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:38.326 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:38.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:38.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:38.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:38.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:38.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:38.562 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:38.597 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:38.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:38.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:38.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:38 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 69473/417006 objects degraded (16.660%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:38.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:38 smithi169 ceph-mon[39949]: osdmap e317: 8 total, 7 up, 8 in 2024-03-12T03:45:38.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:38 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 69473/417006 objects degraded (16.660%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:38.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:38 smithi169 ceph-mon[29344]: osdmap e317: 8 total, 7 up, 8 in 2024-03-12T03:45:38.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:38.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:38 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 69473/417006 objects degraded (16.660%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:38.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:38 smithi173 ceph-mon[33544]: osdmap e317: 8 total, 7 up, 8 in 2024-03-12T03:45:38.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:38.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:38.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:38.974 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:39.004 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:39.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:39.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:39.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:39.140 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:39.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:39.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:39.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:39.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:39.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:39.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:39.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:39.400 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:39.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:39.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:39.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:39.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:39.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:39.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:39.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:39.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:39.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:39 smithi169 ceph-mon[29344]: pgmap v585: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1.08k op/s; 66648/399966 objects degraded (16.663%); 1/399966 objects misplaced (0.000%) 2024-03-12T03:45:39.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:39 smithi169 ceph-mon[39949]: pgmap v585: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1.08k op/s; 66648/399966 objects degraded (16.663%); 1/399966 objects misplaced (0.000%) 2024-03-12T03:45:39.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:39.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:39 smithi173 ceph-mon[33544]: pgmap v585: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1.08k op/s; 66648/399966 objects degraded (16.663%); 1/399966 objects misplaced (0.000%) 2024-03-12T03:45:39.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:39.827 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:39.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:39.928 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:39.970 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:39.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:39.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:40.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:40.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:40.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:40.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:40.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:40.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:40.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:40.404 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:40.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:40.489 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-12T03:45:40.489 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:45:40.489 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-12T03:45:40.489 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-12T03:45:40.489 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-12T03:45:40.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-12T03:45:40.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:40.542 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:40.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:40.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:40.692 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:40.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:40.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:40.825 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:40.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:40.896 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:45:40.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:40.954 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:40.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:41.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:41.088 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:41.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:41.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:41.169 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:45:41.169 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:45:41.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:41.229 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:41.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:41.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:41.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:41.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:41.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:41.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:41.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:41.615 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:41.651 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:41.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:41.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:41 smithi169 ceph-mon[29344]: pgmap v587: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 829 op/s; 66648/399966 objects degraded (16.663%); 1/399966 objects misplaced (0.000%) 2024-03-12T03:45:41.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:41 smithi169 ceph-mon[39949]: pgmap v587: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 829 op/s; 66648/399966 objects degraded (16.663%); 1/399966 objects misplaced (0.000%) 2024-03-12T03:45:41.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:41 smithi173 ceph-mon[33544]: pgmap v587: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 829 op/s; 66648/399966 objects degraded (16.663%); 1/399966 objects misplaced (0.000%) 2024-03-12T03:45:41.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:41.786 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:41.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:41.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:41.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:41.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:42.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:42.044 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:45:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:45:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:45:42.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:42.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:42.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:42.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:42.195 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:42.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:42.296 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:42.325 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:42.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:42.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:42.427 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:42.458 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:42.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:42.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-12T03:45:42.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:42.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:42.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:42.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:42.733 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:42.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:42.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:42.864 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:42.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:42.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:42.996 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:45:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:45:42.996 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:45:42.996 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-12T03:45:42.996 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:45:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:43.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:43.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:43.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:43.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:43.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:43.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:43.229 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:45:43.253 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:43.284 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:43.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:43.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:43.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:43.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:43.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:43.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:43.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:43.548 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:43.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:43.622 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:43 smithi173 ceph-mon[33544]: pgmap v588: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 63785/381345 objects degraded (16.726%); 1/381345 objects misplaced (0.000%) 2024-03-12T03:45:43.622 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:43 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:45:43.622 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:43 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:45:43.622 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:43 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 63785/381345 objects degraded (16.726%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:43.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:43.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:43 smithi169 ceph-mon[29344]: pgmap v588: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 63785/381345 objects degraded (16.726%); 1/381345 objects misplaced (0.000%) 2024-03-12T03:45:43.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:43 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:45:43.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:43 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:45:43.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:43 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 63785/381345 objects degraded (16.726%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:43 smithi169 ceph-mon[39949]: pgmap v588: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 63785/381345 objects degraded (16.726%); 1/381345 objects misplaced (0.000%) 2024-03-12T03:45:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:43 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:45:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:43 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:45:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:43 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 63785/381345 objects degraded (16.726%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:43.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:43.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:43.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:43.816 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:43.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:43.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:43.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:43.947 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:43.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:44.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:44.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:44.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:44.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:44.230 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:44.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:44.331 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:44.370 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:44.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:44.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:44.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:44.489 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:44.490 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:44 smithi173 ceph-mon[33544]: osdmap e318: 8 total, 7 up, 8 in 2024-03-12T03:45:44.490 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:44 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4110683428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:44.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:44.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:44.536 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:45:44.537 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:45:44.537 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:45:44.537 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:44.537 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:44 smithi169 ceph-mon[29344]: osdmap e318: 8 total, 7 up, 8 in 2024-03-12T03:45:44.537 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:44 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4110683428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:44.537 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:44.538 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:44 smithi169 ceph-mon[39949]: osdmap e318: 8 total, 7 up, 8 in 2024-03-12T03:45:44.538 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:44 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4110683428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:44.541 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:45:44.590 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:45:44.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:44.620 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:45:44.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:44.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:44.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:45:44.656 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:45:44.656 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:45:44.656 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:45:44.656 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:45:44.656 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:45:44.656 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:45:44.656 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:45:44.656 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:45:44.755 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:44.784 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:44.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:44.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:44.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:44.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:45.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:45.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:45.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:45.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:45.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:45.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:45.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:45.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:45.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:45.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:45.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:45.466 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:45.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:45.567 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:45.616 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:45.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:45.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:45.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:45 smithi173 ceph-mon[33544]: pgmap v590: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.66k op/s; 62862/377157 objects degraded (16.667%); 1/377157 objects misplaced (0.000%) 2024-03-12T03:45:45.765 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:45.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:45.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:45.773 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:45 smithi169 ceph-mon[29344]: pgmap v590: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.66k op/s; 62862/377157 objects degraded (16.667%); 1/377157 objects misplaced (0.000%) 2024-03-12T03:45:45.775 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:45 smithi169 ceph-mon[39949]: pgmap v590: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.66k op/s; 62862/377157 objects degraded (16.667%); 1/377157 objects misplaced (0.000%) 2024-03-12T03:45:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:45:45.816 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:45:45.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:45.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:45.906 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:45.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:46.007 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:46.045 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:46.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:46.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:46.180 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:46.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:46.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:46.273 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:45:46.273 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15279.0:169329 9.e e318/278 9:746226d4:::benchmark_data_smithi169_135229_object25114:head v 318'10606, mlcod=318'10603)", 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:46.266591+0000", 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.003756405, 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037885430000000001, 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:46.266591+0000", 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:45:46.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:46.266591+0000", 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:46.266592+0000", 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9800000000000002e-07 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:46.266600+0000", 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3710000000000002e-06 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:45:46.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:46.266614+0000", 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4056000000000001e-05 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:46.266621+0000", 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5610000000000001e-06 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:46.266744+0000", 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122435 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:46.266764+0000", 2024-03-12T03:45:46.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0462000000000002e-05 2024-03-12T03:45:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:45:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:45:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:45:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:45:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-03-12T03:45:46.277 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-03-12T03:45:46.277 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:45:46.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:46.309 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:46.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:46.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:46.440 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3933371921' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:45:46.440 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:46 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:45:46.440 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:46 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:46.452 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:45:46.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:46.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:46.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:46.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:46.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:46.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:46.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:46.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:46.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:46.732 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:46.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:46.795 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3933371921' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:45:46.795 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:46 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:45:46.795 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:46 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:46.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3933371921' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:45:46.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:46 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:45:46.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:46 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:46.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:46.866 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:46.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:46.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:47.008 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:47.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:47.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-12T03:45:47.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:47.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:47.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:47.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:47.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:47.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:47.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:47.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:47.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:47.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:47.502 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:45:47.560 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:47.591 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:47.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:47.691 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:47 smithi173 ceph-mon[33544]: pgmap v591: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 950 op/s; 62862/377157 objects degraded (16.667%); 1/377157 objects misplaced (0.000%) 2024-03-12T03:45:47.691 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:47 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:45:47.691 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:47 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:45:47.691 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:47 smithi173 ceph-mon[33544]: osdmap e319: 8 total, 7 up, 8 in 2024-03-12T03:45:47.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:47.725 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:47.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:47.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:47.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:45:47.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:47.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:47 smithi169 ceph-mon[29344]: pgmap v591: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 950 op/s; 62862/377157 objects degraded (16.667%); 1/377157 objects misplaced (0.000%) 2024-03-12T03:45:47.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:47 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:45:47.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:47 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:45:47.850 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:47 smithi169 ceph-mon[29344]: osdmap e319: 8 total, 7 up, 8 in 2024-03-12T03:45:47.850 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:47 smithi169 ceph-mon[39949]: pgmap v591: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 950 op/s; 62862/377157 objects degraded (16.667%); 1/377157 objects misplaced (0.000%) 2024-03-12T03:45:47.850 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:47 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:45:47.850 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:47 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:45:47.850 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:47 smithi169 ceph-mon[39949]: osdmap e319: 8 total, 7 up, 8 in 2024-03-12T03:45:47.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:47.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:47.970 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:48.007 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:48.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:48.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:48.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:48.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:48.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:48.182 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:48.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:48.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:48.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:48.386 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:48.417 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:48.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:48.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:48.550 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:48.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:48.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:48.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:48 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 62862/377157 objects degraded (16.667%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:48.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:48 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 62862/377157 objects degraded (16.667%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:48.689 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:48.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:48.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:48 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 62862/377157 objects degraded (16.667%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:48.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:48.805 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-12T03:45:48.805 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:45:48.805 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_2 2024-03-12T03:45:48.805 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:45:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:48.826 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:48.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:48.928 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:48.953 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:48.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:49.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:49.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:49.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:49.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:45:49.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:49.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:49.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:49.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:49.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:49.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:49.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:49.476 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:49.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:49.611 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:49.641 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:49.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:49.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:49 smithi169 ceph-mon[29344]: pgmap v593: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 59540/356721 objects degraded (16.691%); 1/356721 objects misplaced (0.000%) 2024-03-12T03:45:49.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:49 smithi169 ceph-mon[29344]: 9.6 scrub starts 2024-03-12T03:45:49.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:49 smithi169 ceph-mon[39949]: pgmap v593: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 59540/356721 objects degraded (16.691%); 1/356721 objects misplaced (0.000%) 2024-03-12T03:45:49.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:49 smithi169 ceph-mon[39949]: 9.6 scrub starts 2024-03-12T03:45:49.735 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:49 smithi173 ceph-mon[33544]: pgmap v593: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 59540/356721 objects degraded (16.691%); 1/356721 objects misplaced (0.000%) 2024-03-12T03:45:49.735 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:49 smithi173 ceph-mon[33544]: 9.6 scrub starts 2024-03-12T03:45:49.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:49.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:49.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:49.883 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:49.884 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:49.940 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:49.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:49.946 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:45:49.947 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:45:49.947 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:45:49.947 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:45:49.947 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:45:50.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:50.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:50.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:50.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:50.175 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:50.212 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:50.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:50.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:50.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:50.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:50.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:50.451 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:50.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:50.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:50.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:50.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:50.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:50.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:50.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:50.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:50.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:50.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:50.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:50.899 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:50.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:50.913 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:51.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:51.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:51.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:51.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:51.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:51.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:51.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:51.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:51.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:51.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:51.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:51.436 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:51.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:51.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:51.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:51.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:51.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:51.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:51 smithi169 ceph-mon[29344]: pgmap v594: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 59540/356721 objects degraded (16.691%); 1/356721 objects misplaced (0.000%) 2024-03-12T03:45:51.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:51 smithi169 ceph-mon[29344]: 9.9 scrub starts 2024-03-12T03:45:51.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:51 smithi169 ceph-mon[29344]: 1.f scrub starts 2024-03-12T03:45:51.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:51 smithi169 ceph-mon[29344]: 1.f scrub ok 2024-03-12T03:45:51.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:51 smithi169 ceph-mon[29344]: 9.6 scrub ok 2024-03-12T03:45:51.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:51 smithi169 ceph-mon[39949]: pgmap v594: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 59540/356721 objects degraded (16.691%); 1/356721 objects misplaced (0.000%) 2024-03-12T03:45:51.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:51 smithi169 ceph-mon[39949]: 9.9 scrub starts 2024-03-12T03:45:51.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:51 smithi169 ceph-mon[39949]: 1.f scrub starts 2024-03-12T03:45:51.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:51 smithi169 ceph-mon[39949]: 1.f scrub ok 2024-03-12T03:45:51.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:51 smithi169 ceph-mon[39949]: 9.6 scrub ok 2024-03-12T03:45:51.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:51.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:51.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:51.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:51 smithi173 ceph-mon[33544]: pgmap v594: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 59540/356721 objects degraded (16.691%); 1/356721 objects misplaced (0.000%) 2024-03-12T03:45:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:51 smithi173 ceph-mon[33544]: 9.9 scrub starts 2024-03-12T03:45:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:51 smithi173 ceph-mon[33544]: 1.f scrub starts 2024-03-12T03:45:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:51 smithi173 ceph-mon[33544]: 1.f scrub ok 2024-03-12T03:45:51.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:51 smithi173 ceph-mon[33544]: 9.6 scrub ok 2024-03-12T03:45:51.773 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:45:51.805 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:45:51.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:51.824 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:45:51.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:51.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:51.853 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:45:51.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-12T03:45:51.887 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:45:51.924 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:45:51.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:51.953 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:45:51.954 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:45:51.954 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:45:51.954 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:45:51.954 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:45:51.954 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:45:51.954 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:45:51.954 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:45:51.955 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:45:51.955 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:45:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:45:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:45:52.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:52.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:52.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:52.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:52.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:52.139 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:52.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:52.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:52.270 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:52.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:52.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:52.331 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:45:52.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:52.403 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:52.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:52.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:52.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:52.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:45:52.598 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:45:52.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:52.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:52.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:52.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:52.727 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:52 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4153789531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:52.727 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:52 smithi169 ceph-mon[29344]: 5.6 scrub starts 2024-03-12T03:45:52.727 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:52 smithi169 ceph-mon[29344]: 5.6 scrub ok 2024-03-12T03:45:52.727 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:52 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4153789531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:52.728 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:52 smithi169 ceph-mon[39949]: 5.6 scrub starts 2024-03-12T03:45:52.728 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:52 smithi169 ceph-mon[39949]: 5.6 scrub ok 2024-03-12T03:45:52.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:52 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4153789531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:52.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:52 smithi173 ceph-mon[33544]: 5.6 scrub starts 2024-03-12T03:45:52.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:52 smithi173 ceph-mon[33544]: 5.6 scrub ok 2024-03-12T03:45:52.786 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:52.836 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:52.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:52.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:52.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:52.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:52.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:52.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:53.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:53.107 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:53.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:53.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:53.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:53.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:45:52 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:45:53.240 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:53.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:53.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:53.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:53.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:53.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:53.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:53.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:53.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:53.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:53.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:53 smithi173 ceph-mon[33544]: pgmap v595: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 56896/339855 objects degraded (16.741%); 1/339855 objects misplaced (0.000%) 2024-03-12T03:45:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:53 smithi173 ceph-mon[33544]: 9.5 scrub starts 2024-03-12T03:45:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:53 smithi173 ceph-mon[33544]: 9.9 scrub ok 2024-03-12T03:45:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:53 smithi173 ceph-mon[33544]: 5.f scrub starts 2024-03-12T03:45:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:53 smithi173 ceph-mon[33544]: 9.e scrub starts 2024-03-12T03:45:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:53 smithi173 ceph-mon[33544]: 5.f scrub ok 2024-03-12T03:45:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:53 smithi173 ceph-mon[33544]: 5.8 scrub starts 2024-03-12T03:45:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:53 smithi173 ceph-mon[33544]: 5.8 scrub ok 2024-03-12T03:45:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:53 smithi173 ceph-mon[33544]: 5.1d scrub starts 2024-03-12T03:45:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:53 smithi173 ceph-mon[33544]: 5.1d scrub ok 2024-03-12T03:45:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:53 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 56896/339855 objects degraded (16.741%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:53.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:53.792 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:53.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:53.891 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:45:53.892 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[29344]: pgmap v595: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 56896/339855 objects degraded (16.741%); 1/339855 objects misplaced (0.000%) 2024-03-12T03:45:53.892 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[29344]: 9.5 scrub starts 2024-03-12T03:45:53.892 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[29344]: 9.9 scrub ok 2024-03-12T03:45:53.892 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[29344]: 5.f scrub starts 2024-03-12T03:45:53.892 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[29344]: 9.e scrub starts 2024-03-12T03:45:53.892 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[29344]: 5.f scrub ok 2024-03-12T03:45:53.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[29344]: 5.8 scrub starts 2024-03-12T03:45:53.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[29344]: 5.8 scrub ok 2024-03-12T03:45:53.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[29344]: 5.1d scrub starts 2024-03-12T03:45:53.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[29344]: 5.1d scrub ok 2024-03-12T03:45:53.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 56896/339855 objects degraded (16.741%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:53.893 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[39949]: pgmap v595: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 56896/339855 objects degraded (16.741%); 1/339855 objects misplaced (0.000%) 2024-03-12T03:45:53.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[39949]: 9.5 scrub starts 2024-03-12T03:45:53.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[39949]: 9.9 scrub ok 2024-03-12T03:45:53.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[39949]: 5.f scrub starts 2024-03-12T03:45:53.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[39949]: 9.e scrub starts 2024-03-12T03:45:53.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[39949]: 5.f scrub ok 2024-03-12T03:45:53.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[39949]: 5.8 scrub starts 2024-03-12T03:45:53.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[39949]: 5.8 scrub ok 2024-03-12T03:45:53.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[39949]: 5.1d scrub starts 2024-03-12T03:45:53.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[39949]: 5.1d scrub ok 2024-03-12T03:45:53.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:53 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 56896/339855 objects degraded (16.741%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:53.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:53.902 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:45:53.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:53.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:53.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:54.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:54.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:54.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:54.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:54.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:54.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:54.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:54.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:54.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:54.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:54.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:54.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:54.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:54.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:54.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:54.586 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:54.586 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[39949]: 5.15 scrub starts 2024-03-12T03:45:54.586 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[39949]: 5.15 scrub ok 2024-03-12T03:45:54.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[39949]: 3.3 scrub starts 2024-03-12T03:45:54.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[39949]: 3.3 scrub ok 2024-03-12T03:45:54.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[39949]: 5.b scrub starts 2024-03-12T03:45:54.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[39949]: 5.b scrub ok 2024-03-12T03:45:54.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:54.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[29344]: 5.15 scrub starts 2024-03-12T03:45:54.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[29344]: 5.15 scrub ok 2024-03-12T03:45:54.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[29344]: 3.3 scrub starts 2024-03-12T03:45:54.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[29344]: 3.3 scrub ok 2024-03-12T03:45:54.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[29344]: 5.b scrub starts 2024-03-12T03:45:54.587 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:54 smithi169 ceph-mon[29344]: 5.b scrub ok 2024-03-12T03:45:54.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:54.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:45:54.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:54.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:54.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:54.724 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:54.753 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:54.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:54.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:54.856 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:54.895 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:54.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:54 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:54 smithi173 ceph-mon[33544]: 5.15 scrub starts 2024-03-12T03:45:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:54 smithi173 ceph-mon[33544]: 5.15 scrub ok 2024-03-12T03:45:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:54 smithi173 ceph-mon[33544]: 3.3 scrub starts 2024-03-12T03:45:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:54 smithi173 ceph-mon[33544]: 3.3 scrub ok 2024-03-12T03:45:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:54 smithi173 ceph-mon[33544]: 5.b scrub starts 2024-03-12T03:45:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:54 smithi173 ceph-mon[33544]: 5.b scrub ok 2024-03-12T03:45:54.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:55.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:55.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:55.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:55.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:55.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:55.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:55.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:55.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:55.402 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":595,"stamp":"2024-03-12T03:45:52.046089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7409774920,"num_objects":113285,"num_object_clones":0,"num_object_copies":339855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":56896,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":113285,"num_whiteouts":0,"num_read":2191,"num_read_kb":1923,"num_write":176306,"num_write_kb":9229907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":744900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77711,"ondisk_log_size":77711,"up":432,"acting":436,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":503,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22455068,"kb_used_data":18438416,"kb_used_omap":18,"kb_used_meta":4016301,"kb_avail":727506148,"statfs":{"total":767960285184,"available":744966295552,"internally_reserved":0,"allocated":18880937984,"data_stored":18875364100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19339,"internal_metadata":4112692341},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,2,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-814874624,"num_objects":-12434,"num_object_clones":0,"num_object_copies":-37302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12434,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12434,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.618091"},"pg_stats":[{"pgid":"9.1","version":"319'10841","reported_seq":10931,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:51.115866+0000","last_change":"2024-03-12T03:45:18.113287+0000","last_active":"2024-03-12T03:45:51.115866+0000","last_peered":"2024-03-12T03:45:51.115866+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:45:18.113287+0000","last_became_peered":"2024-03-12T03:45:18.113287+0000","last_unstale":"2024-03-12T03:45:51.115866+0000","last_undegraded":"2024-03-12T03:45:18.104175+0000","last_fullsized":"2024-03-12T03:45:18.104043+0000","mapping_epoch":313,"log_start":"301'6487","ondisk_log_start":"301'6487","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4354,"log_dups_size":0,"ondisk_log_size":4354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.305320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455540736,"num_objects":6951,"num_object_clones":0,"num_object_copies":20853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10841,"num_write_kb":569344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":6951}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":857,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.448317+0000","last_change":"2024-03-12T03:43:18.345099+0000","last_active":"2024-03-12T03:45:46.448317+0000","last_peered":"2024-03-12T03:45:46.448317+0000","last_clean":"2024-03-12T03:43:13.340455+0000","last_became_active":"2024-03-12T03:43:18.345099+0000","last_became_peered":"2024-03-12T03:43:18.345099+0000","last_unstale":"2024-03-12T03:45:46.448317+0000","last_undegraded":"2024-03-12T03:45:46.448317+0000","last_fullsized":"2024-03-12T03:43:18.338164+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:28.162735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:28.162735+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.304436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"269'20","reported_seq":589,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.461240+0000","last_change":"2024-03-12T03:43:18.351745+0000","last_active":"2024-03-12T03:45:46.461240+0000","last_peered":"2024-03-12T03:45:46.461240+0000","last_clean":"2024-03-12T03:43:14.546139+0000","last_became_active":"2024-03-12T03:43:18.351745+0000","last_became_peered":"2024-03-12T03:43:18.351745+0000","last_unstale":"2024-03-12T03:45:46.461240+0000","last_undegraded":"2024-03-12T03:43:18.333421+0000","last_fullsized":"2024-03-12T03:43:18.333308+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-12T03:43:14.545889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:14.545889+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674115+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":361,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.453987+0000","last_change":"2024-03-12T03:43:18.339149+0000","last_active":"2024-03-12T03:45:46.453987+0000","last_peered":"2024-03-12T03:45:46.453987+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:43:18.339149+0000","last_became_peered":"2024-03-12T03:43:18.339149+0000","last_unstale":"2024-03-12T03:45:46.453987+0000","last_undegraded":"2024-03-12T03:45:46.453987+0000","last_fullsized":"2024-03-12T03:43:18.331129+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:15.981378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:35:15.981378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":357,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.444966+0000","last_change":"2024-03-12T03:45:06.527413+0000","last_active":"2024-03-12T03:45:46.444966+0000","last_peered":"2024-03-12T03:45:46.444966+0000","last_clean":"2024-03-12T03:45:46.444966+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:45:46.444966+0000","last_undegraded":"2024-03-12T03:45:46.444966+0000","last_fullsized":"2024-03-12T03:45:46.444966+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:45:06.527297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.527297+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:23.669122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.463484+0000","last_change":"2024-03-12T03:44:09.835312+0000","last_active":"2024-03-12T03:45:46.463484+0000","last_peered":"2024-03-12T03:45:46.463484+0000","last_clean":"2024-03-12T03:45:46.463484+0000","last_became_active":"2024-03-12T03:39:33.265665+0000","last_became_peered":"2024-03-12T03:39:33.265665+0000","last_unstale":"2024-03-12T03:45:46.463484+0000","last_undegraded":"2024-03-12T03:45:46.463484+0000","last_fullsized":"2024-03-12T03:45:46.463484+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:09.835231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:09.835231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674281+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"319'10727","reported_seq":10819,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:47.176196+0000","last_change":"2024-03-12T03:43:45.632046+0000","last_active":"2024-03-12T03:45:47.176196+0000","last_peered":"2024-03-12T03:45:47.176196+0000","last_clean":"2024-03-12T03:45:47.176196+0000","last_became_active":"2024-03-12T03:43:45.631752+0000","last_became_peered":"2024-03-12T03:43:45.631752+0000","last_unstale":"2024-03-12T03:45:47.176196+0000","last_undegraded":"2024-03-12T03:45:47.176196+0000","last_fullsized":"2024-03-12T03:45:47.176196+0000","mapping_epoch":278,"log_start":"299'6200","ondisk_log_start":"299'6200","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485687296,"num_objects":7411,"num_object_clones":0,"num_object_copies":22233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10727,"num_write_kb":580416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"269'31","reported_seq":652,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.461638+0000","last_change":"2024-03-12T03:44:13.891462+0000","last_active":"2024-03-12T03:45:46.461638+0000","last_peered":"2024-03-12T03:45:46.461638+0000","last_clean":"2024-03-12T03:45:46.461638+0000","last_became_active":"2024-03-12T03:39:33.268652+0000","last_became_peered":"2024-03-12T03:39:33.268652+0000","last_unstale":"2024-03-12T03:45:46.461638+0000","last_undegraded":"2024-03-12T03:45:46.461638+0000","last_fullsized":"2024-03-12T03:45:46.461638+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:44:13.891303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:44:13.891303+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.459612+0000","last_change":"2024-03-12T03:45:18.109064+0000","last_active":"2024-03-12T03:45:46.459612+0000","last_peered":"2024-03-12T03:45:46.459612+0000","last_clean":"2024-03-12T03:45:46.459612+0000","last_became_active":"2024-03-12T03:45:18.108377+0000","last_became_peered":"2024-03-12T03:45:18.108377+0000","last_unstale":"2024-03-12T03:45:46.459612+0000","last_undegraded":"2024-03-12T03:45:46.459612+0000","last_fullsized":"2024-03-12T03:45:46.459612+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.582660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.582660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:24.474403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.449488+0000","last_change":"2024-03-12T03:45:04.055127+0000","last_active":"2024-03-12T03:45:46.449488+0000","last_peered":"2024-03-12T03:45:46.449488+0000","last_clean":"2024-03-12T03:45:46.449488+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:45:46.449488+0000","last_undegraded":"2024-03-12T03:45:46.449488+0000","last_fullsized":"2024-03-12T03:45:46.449488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.054936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.054936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:20.022092+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":282,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.445157+0000","last_change":"2024-03-12T03:43:18.344122+0000","last_active":"2024-03-12T03:45:46.445157+0000","last_peered":"2024-03-12T03:45:46.445157+0000","last_clean":"2024-03-12T03:43:13.349376+0000","last_became_active":"2024-03-12T03:43:18.344122+0000","last_became_peered":"2024-03-12T03:43:18.344122+0000","last_unstale":"2024-03-12T03:45:46.445157+0000","last_undegraded":"2024-03-12T03:45:46.445157+0000","last_fullsized":"2024-03-12T03:43:18.337641+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:07.272232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:07.272232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:47.790414+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":698,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.454028+0000","last_change":"2024-03-12T03:43:18.379878+0000","last_active":"2024-03-12T03:45:46.454028+0000","last_peered":"2024-03-12T03:45:46.454028+0000","last_clean":"2024-03-12T03:43:13.353305+0000","last_became_active":"2024-03-12T03:43:18.379878+0000","last_became_peered":"2024-03-12T03:43:18.379878+0000","last_unstale":"2024-03-12T03:45:46.454028+0000","last_undegraded":"2024-03-12T03:43:18.336786+0000","last_fullsized":"2024-03-12T03:43:18.336678+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:41:03.399737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:41:03.399737+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":690,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.448002+0000","last_change":"2024-03-12T03:44:53.132143+0000","last_active":"2024-03-12T03:45:46.448002+0000","last_peered":"2024-03-12T03:45:46.448002+0000","last_clean":"2024-03-12T03:45:46.448002+0000","last_became_active":"2024-03-12T03:44:24.371149+0000","last_became_peered":"2024-03-12T03:44:24.371149+0000","last_unstale":"2024-03-12T03:45:46.448002+0000","last_undegraded":"2024-03-12T03:45:46.448002+0000","last_fullsized":"2024-03-12T03:45:46.448002+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:53.131506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:53.131506+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:55.822711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"319'11030","reported_seq":11120,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:51.112013+0000","last_change":"2024-03-12T03:43:55.386278+0000","last_active":"2024-03-12T03:45:51.112013+0000","last_peered":"2024-03-12T03:45:51.112013+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.592856+0000","last_became_peered":"2024-03-12T03:43:45.592856+0000","last_unstale":"2024-03-12T03:45:51.112013+0000","last_undegraded":"2024-03-12T03:43:55.380028+0000","last_fullsized":"2024-03-12T03:43:45.589896+0000","mapping_epoch":278,"log_start":"301'6600","ondisk_log_start":"301'6600","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4430,"log_dups_size":0,"ondisk_log_size":4430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.305375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460718080,"num_objects":7030,"num_object_clones":0,"num_object_copies":21090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11030,"num_write_kb":577920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":7030}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"269'31","reported_seq":604,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.463305+0000","last_change":"2024-03-12T03:43:59.838390+0000","last_active":"2024-03-12T03:45:46.463305+0000","last_peered":"2024-03-12T03:45:46.463305+0000","last_clean":"2024-03-12T03:45:46.463305+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:45:46.463305+0000","last_undegraded":"2024-03-12T03:45:46.463305+0000","last_fullsized":"2024-03-12T03:45:46.463305+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:43:59.838251+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:43:59.838251+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674178+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":571,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.458620+0000","last_change":"2024-03-12T03:45:12.928307+0000","last_active":"2024-03-12T03:45:46.458620+0000","last_peered":"2024-03-12T03:45:46.458620+0000","last_clean":"2024-03-12T03:45:46.458620+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:45:46.458620+0000","last_undegraded":"2024-03-12T03:45:46.458620+0000","last_fullsized":"2024-03-12T03:45:46.458620+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_clean_scrub_stamp":"2024-03-12T03:45:12.928199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:13.013623+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":542,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.457823+0000","last_change":"2024-03-12T03:43:55.569297+0000","last_active":"2024-03-12T03:45:46.457823+0000","last_peered":"2024-03-12T03:45:46.457823+0000","last_clean":"2024-03-12T03:45:46.457823+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:45:46.457823+0000","last_undegraded":"2024-03-12T03:45:46.457823+0000","last_fullsized":"2024-03-12T03:45:46.457823+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:55.569174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:55.569174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":277,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.453994+0000","last_change":"2024-03-12T03:45:18.107148+0000","last_active":"2024-03-12T03:45:46.453994+0000","last_peered":"2024-03-12T03:45:46.453994+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:45:18.107148+0000","last_became_peered":"2024-03-12T03:45:18.107148+0000","last_unstale":"2024-03-12T03:45:46.453994+0000","last_undegraded":"2024-03-12T03:45:46.453994+0000","last_fullsized":"2024-03-12T03:45:18.099326+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:13.114417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:13.114417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"319'10624","reported_seq":10713,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:47.171618+0000","last_change":"2024-03-12T03:43:55.387593+0000","last_active":"2024-03-12T03:45:47.171618+0000","last_peered":"2024-03-12T03:45:47.171618+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.631713+0000","last_became_peered":"2024-03-12T03:43:45.631713+0000","last_unstale":"2024-03-12T03:45:47.171618+0000","last_undegraded":"2024-03-12T03:43:55.379093+0000","last_fullsized":"2024-03-12T03:43:45.587023+0000","mapping_epoch":278,"log_start":"299'6100","ondisk_log_start":"299'6100","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4524,"log_dups_size":0,"ondisk_log_size":4524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484704256,"num_objects":7396,"num_object_clones":0,"num_object_copies":22188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10624,"num_write_kb":576640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":7396}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":686,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454386+0000","last_change":"2024-03-12T03:45:02.015686+0000","last_active":"2024-03-12T03:45:46.454386+0000","last_peered":"2024-03-12T03:45:46.454386+0000","last_clean":"2024-03-12T03:45:46.454386+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:45:46.454386+0000","last_undegraded":"2024-03-12T03:45:46.454386+0000","last_fullsized":"2024-03-12T03:45:46.454386+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:45:02.015575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:45:02.015575+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:30.912821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":387,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.449120+0000","last_change":"2024-03-12T03:43:18.346647+0000","last_active":"2024-03-12T03:45:46.449120+0000","last_peered":"2024-03-12T03:45:46.449120+0000","last_clean":"2024-03-12T03:43:13.349764+0000","last_became_active":"2024-03-12T03:43:18.346647+0000","last_became_peered":"2024-03-12T03:43:18.346647+0000","last_unstale":"2024-03-12T03:45:46.449120+0000","last_undegraded":"2024-03-12T03:43:18.336192+0000","last_fullsized":"2024-03-12T03:43:18.335538+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:43:00.715225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.715225+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100323+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":49,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.457869+0000","last_change":"2024-03-12T03:45:13.901275+0000","last_active":"2024-03-12T03:45:46.457869+0000","last_peered":"2024-03-12T03:45:46.457869+0000","last_clean":"2024-03-12T03:45:46.457869+0000","last_became_active":"2024-03-12T03:44:57.602127+0000","last_became_peered":"2024-03-12T03:44:57.602127+0000","last_unstale":"2024-03-12T03:45:46.457869+0000","last_undegraded":"2024-03-12T03:45:46.457869+0000","last_fullsized":"2024-03-12T03:45:46.457869+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:13.901167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:13.901167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:23.528736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":709,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.463436+0000","last_change":"2024-03-12T03:44:16.927568+0000","last_active":"2024-03-12T03:45:46.463436+0000","last_peered":"2024-03-12T03:45:46.463436+0000","last_clean":"2024-03-12T03:45:46.463436+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:45:46.463436+0000","last_undegraded":"2024-03-12T03:45:46.463436+0000","last_fullsized":"2024-03-12T03:45:46.463436+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:44:16.927294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:44:16.927294+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674341+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":538,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454463+0000","last_change":"2024-03-12T03:45:08.175177+0000","last_active":"2024-03-12T03:45:46.454463+0000","last_peered":"2024-03-12T03:45:46.454463+0000","last_clean":"2024-03-12T03:45:46.454463+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:45:46.454463+0000","last_undegraded":"2024-03-12T03:45:46.454463+0000","last_fullsized":"2024-03-12T03:45:46.454463+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:08.174901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:08.174901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:09.715496+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"319'11376","reported_seq":11548,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:51.091931+0000","last_change":"2024-03-12T03:44:54.861121+0000","last_active":"2024-03-12T03:45:51.091931+0000","last_peered":"2024-03-12T03:45:51.091931+0000","last_clean":"2024-03-12T03:45:51.091931+0000","last_became_active":"2024-03-12T03:43:45.596788+0000","last_became_peered":"2024-03-12T03:43:45.596788+0000","last_unstale":"2024-03-12T03:45:51.091931+0000","last_undegraded":"2024-03-12T03:45:51.091931+0000","last_fullsized":"2024-03-12T03:45:51.091931+0000","mapping_epoch":278,"log_start":"302'7000","ondisk_log_start":"302'7000","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'7001","last_scrub_stamp":"2024-03-12T03:44:54.860702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:44:54.860702+0000","objects_scrubbed":6887,"log_size":4376,"log_dups_size":0,"ondisk_log_size":4376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:59.643659+0000","scrub_duration":2766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467337240,"num_objects":7132,"num_object_clones":0,"num_object_copies":21396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11376,"num_write_kb":592193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":547,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.458444+0000","last_change":"2024-03-12T03:43:18.344476+0000","last_active":"2024-03-12T03:45:46.458444+0000","last_peered":"2024-03-12T03:45:46.458444+0000","last_clean":"2024-03-12T03:43:13.337325+0000","last_became_active":"2024-03-12T03:43:18.344476+0000","last_became_peered":"2024-03-12T03:43:18.344476+0000","last_unstale":"2024-03-12T03:45:46.458444+0000","last_undegraded":"2024-03-12T03:45:46.458444+0000","last_fullsized":"2024-03-12T03:43:18.329711+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:01.229635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:01.229635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"269'42","reported_seq":250,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.447002+0000","last_change":"2024-03-12T03:43:18.345917+0000","last_active":"2024-03-12T03:45:46.447002+0000","last_peered":"2024-03-12T03:45:46.447002+0000","last_clean":"2024-03-12T03:33:28.528154+0000","last_became_active":"2024-03-12T03:43:18.345917+0000","last_became_peered":"2024-03-12T03:43:18.345917+0000","last_unstale":"2024-03-12T03:45:46.447002+0000","last_undegraded":"2024-03-12T03:43:18.336762+0000","last_fullsized":"2024-03-12T03:43:18.336652+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-03-12T03:36:06.083052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:06.083052+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.303838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":365,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.447534+0000","last_change":"2024-03-12T03:43:18.345673+0000","last_active":"2024-03-12T03:45:46.447534+0000","last_peered":"2024-03-12T03:45:46.447534+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:43:18.345673+0000","last_became_peered":"2024-03-12T03:43:18.345673+0000","last_unstale":"2024-03-12T03:45:46.447534+0000","last_undegraded":"2024-03-12T03:45:46.447534+0000","last_fullsized":"2024-03-12T03:43:18.337055+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:41.918538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:41.918538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.303878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":563,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.459433+0000","last_change":"2024-03-12T03:43:18.344050+0000","last_active":"2024-03-12T03:45:46.459433+0000","last_peered":"2024-03-12T03:45:46.459433+0000","last_clean":"2024-03-12T03:43:13.338394+0000","last_became_active":"2024-03-12T03:43:18.344050+0000","last_became_peered":"2024-03-12T03:43:18.344050+0000","last_unstale":"2024-03-12T03:45:46.459433+0000","last_undegraded":"2024-03-12T03:45:46.459433+0000","last_fullsized":"2024-03-12T03:43:18.333957+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:23.257942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:23.257942+0000","last_clean_scrub_stamp":"2024-03-12T03:41:23.257942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674059+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":700,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.460801+0000","last_change":"2024-03-12T03:44:55.770605+0000","last_active":"2024-03-12T03:45:46.460801+0000","last_peered":"2024-03-12T03:45:46.460801+0000","last_clean":"2024-03-12T03:45:46.460801+0000","last_became_active":"2024-03-12T03:44:19.373715+0000","last_became_peered":"2024-03-12T03:44:19.373715+0000","last_unstale":"2024-03-12T03:45:46.460801+0000","last_undegraded":"2024-03-12T03:45:46.460801+0000","last_fullsized":"2024-03-12T03:45:46.460801+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:55.770487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:55.770487+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:23.200484+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"319'10940","reported_seq":11029,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:51.026262+0000","last_change":"2024-03-12T03:43:55.383487+0000","last_active":"2024-03-12T03:45:51.026262+0000","last_peered":"2024-03-12T03:45:51.026262+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.594399+0000","last_became_peered":"2024-03-12T03:43:45.594399+0000","last_unstale":"2024-03-12T03:45:51.026262+0000","last_undegraded":"2024-03-12T03:43:55.379614+0000","last_fullsized":"2024-03-12T03:43:45.586741+0000","mapping_epoch":278,"log_start":"302'6600","ondisk_log_start":"302'6600","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456392704,"num_objects":6964,"num_object_clones":0,"num_object_copies":20892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10940,"num_write_kb":572928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":6964}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"266'11","reported_seq":226,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.448073+0000","last_change":"2024-03-12T03:43:18.345511+0000","last_active":"2024-03-12T03:45:46.448073+0000","last_peered":"2024-03-12T03:45:46.448073+0000","last_clean":"2024-03-12T03:33:28.353771+0000","last_became_active":"2024-03-12T03:43:18.345511+0000","last_became_peered":"2024-03-12T03:43:18.345511+0000","last_unstale":"2024-03-12T03:45:46.448073+0000","last_undegraded":"2024-03-12T03:43:18.332743+0000","last_fullsized":"2024-03-12T03:43:18.332472+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:39:37.039925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:37.039925+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"269'42","reported_seq":617,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.458816+0000","last_change":"2024-03-12T03:45:03.863024+0000","last_active":"2024-03-12T03:45:46.458816+0000","last_peered":"2024-03-12T03:45:46.458816+0000","last_clean":"2024-03-12T03:45:46.458816+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:45:46.458816+0000","last_undegraded":"2024-03-12T03:45:46.458816+0000","last_fullsized":"2024-03-12T03:45:46.458816+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:45:03.862874+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.862874+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:10.934654+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":552,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454351+0000","last_change":"2024-03-12T03:44:09.737622+0000","last_active":"2024-03-12T03:45:46.454351+0000","last_peered":"2024-03-12T03:45:46.454351+0000","last_clean":"2024-03-12T03:45:46.454351+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:45:46.454351+0000","last_undegraded":"2024-03-12T03:45:46.454351+0000","last_fullsized":"2024-03-12T03:45:46.454351+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:44:09.737479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:44:09.737479+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.450094+0000","last_change":"2024-03-12T03:44:14.121638+0000","last_active":"2024-03-12T03:45:46.450094+0000","last_peered":"2024-03-12T03:45:46.450094+0000","last_clean":"2024-03-12T03:45:46.450094+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:45:46.450094+0000","last_undegraded":"2024-03-12T03:45:46.450094+0000","last_fullsized":"2024-03-12T03:45:46.450094+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:14.121534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:14.121534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.305555+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":460,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.453494+0000","last_change":"2024-03-12T03:44:19.370278+0000","last_active":"2024-03-12T03:45:46.453494+0000","last_peered":"2024-03-12T03:45:46.453494+0000","last_clean":"2024-03-12T03:34:01.860694+0000","last_became_active":"2024-03-12T03:44:19.370278+0000","last_became_peered":"2024-03-12T03:44:19.370278+0000","last_unstale":"2024-03-12T03:45:46.453494+0000","last_undegraded":"2024-03-12T03:45:46.453494+0000","last_fullsized":"2024-03-12T03:44:19.357809+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:02.442717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:02.442717+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:50.353282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"319'11049","reported_seq":11137,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:51.277704+0000","last_change":"2024-03-12T03:43:55.390924+0000","last_active":"2024-03-12T03:45:51.277704+0000","last_peered":"2024-03-12T03:45:51.277704+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.588776+0000","last_became_peered":"2024-03-12T03:43:45.588776+0000","last_unstale":"2024-03-12T03:45:51.277704+0000","last_undegraded":"2024-03-12T03:43:55.379272+0000","last_fullsized":"2024-03-12T03:43:45.585743+0000","mapping_epoch":278,"log_start":"301'6635","ondisk_log_start":"301'6635","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4414,"log_dups_size":0,"ondisk_log_size":4414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:53.367998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462225408,"num_objects":7053,"num_object_clones":0,"num_object_copies":21159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11049,"num_write_kb":579264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":7053}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":710,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.773075+0000","last_change":"2024-03-12T03:45:09.824960+0000","last_active":"2024-03-12T03:45:46.773075+0000","last_peered":"2024-03-12T03:45:46.773075+0000","last_clean":"2024-03-12T03:45:46.773075+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:45:46.773075+0000","last_undegraded":"2024-03-12T03:45:46.773075+0000","last_fullsized":"2024-03-12T03:45:46.773075+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:09.824819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:09.824819+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:11.296361+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"269'26","reported_seq":605,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.450297+0000","last_change":"2024-03-12T03:44:58.093469+0000","last_active":"2024-03-12T03:45:46.450297+0000","last_peered":"2024-03-12T03:45:46.450297+0000","last_clean":"2024-03-12T03:45:46.450297+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:45:46.450297+0000","last_undegraded":"2024-03-12T03:45:46.450297+0000","last_fullsized":"2024-03-12T03:45:46.450297+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:44:58.093327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:44:58.093327+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:19.165975+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":566,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.454554+0000","last_change":"2024-03-12T03:43:18.347416+0000","last_active":"2024-03-12T03:45:46.454554+0000","last_peered":"2024-03-12T03:45:46.454554+0000","last_clean":"2024-03-12T03:43:13.353362+0000","last_became_active":"2024-03-12T03:43:18.347416+0000","last_became_peered":"2024-03-12T03:43:18.347416+0000","last_unstale":"2024-03-12T03:45:46.454554+0000","last_undegraded":"2024-03-12T03:45:46.454554+0000","last_fullsized":"2024-03-12T03:43:18.331325+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:34.674570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:39:34.674570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":556,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.452900+0000","last_change":"2024-03-12T03:43:18.342056+0000","last_active":"2024-03-12T03:45:46.452900+0000","last_peered":"2024-03-12T03:45:46.452900+0000","last_clean":"2024-03-12T03:43:13.330724+0000","last_became_active":"2024-03-12T03:43:18.342056+0000","last_became_peered":"2024-03-12T03:43:18.342056+0000","last_unstale":"2024-03-12T03:45:46.452900+0000","last_undegraded":"2024-03-12T03:45:46.452900+0000","last_fullsized":"2024-03-12T03:43:18.333058+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:40.532991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:40.532991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":467,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.447363+0000","last_change":"2024-03-12T03:45:18.113289+0000","last_active":"2024-03-12T03:45:46.447363+0000","last_peered":"2024-03-12T03:45:46.447363+0000","last_clean":"2024-03-12T03:38:03.868493+0000","last_became_active":"2024-03-12T03:45:18.113289+0000","last_became_peered":"2024-03-12T03:45:18.113289+0000","last_unstale":"2024-03-12T03:45:46.447363+0000","last_undegraded":"2024-03-12T03:45:46.447363+0000","last_fullsized":"2024-03-12T03:45:18.106124+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:43:02.551349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:43:02.551349+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:50.325129+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"319'11022","reported_seq":11113,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:51.109422+0000","last_change":"2024-03-12T03:43:45.597285+0000","last_active":"2024-03-12T03:45:51.109422+0000","last_peered":"2024-03-12T03:45:51.109422+0000","last_clean":"2024-03-12T03:45:51.109422+0000","last_became_active":"2024-03-12T03:43:45.596944+0000","last_became_peered":"2024-03-12T03:43:45.596944+0000","last_unstale":"2024-03-12T03:45:51.109422+0000","last_undegraded":"2024-03-12T03:45:51.109422+0000","last_fullsized":"2024-03-12T03:45:51.109422+0000","mapping_epoch":278,"log_start":"301'6600","ondisk_log_start":"301'6600","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4422,"log_dups_size":0,"ondisk_log_size":4422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.305361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456654848,"num_objects":6968,"num_object_clones":0,"num_object_copies":20904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":575680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":22,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.453396+0000","last_change":"2024-03-12T03:45:18.106629+0000","last_active":"2024-03-12T03:45:46.453396+0000","last_peered":"2024-03-12T03:45:46.453396+0000","last_clean":"2024-03-12T03:45:46.453396+0000","last_became_active":"2024-03-12T03:45:18.106305+0000","last_became_peered":"2024-03-12T03:45:18.106305+0000","last_unstale":"2024-03-12T03:45:46.453396+0000","last_undegraded":"2024-03-12T03:45:46.453396+0000","last_fullsized":"2024-03-12T03:45:46.453396+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:05.578661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:45:05.578661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:11.308569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"269'35","reported_seq":610,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454670+0000","last_change":"2024-03-12T03:43:58.738099+0000","last_active":"2024-03-12T03:45:46.454670+0000","last_peered":"2024-03-12T03:45:46.454670+0000","last_clean":"2024-03-12T03:45:46.454670+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:45:46.454670+0000","last_undegraded":"2024-03-12T03:45:46.454670+0000","last_fullsized":"2024-03-12T03:45:46.454670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-03-12T03:43:58.737948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:58.737948+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":579,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.449398+0000","last_change":"2024-03-12T03:43:58.972063+0000","last_active":"2024-03-12T03:45:46.449398+0000","last_peered":"2024-03-12T03:45:46.449398+0000","last_clean":"2024-03-12T03:45:46.449398+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:45:46.449398+0000","last_undegraded":"2024-03-12T03:45:46.449398+0000","last_fullsized":"2024-03-12T03:45:46.449398+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:58.971867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:43:58.971867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":567,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.447969+0000","last_change":"2024-03-12T03:44:54.134468+0000","last_active":"2024-03-12T03:45:46.447969+0000","last_peered":"2024-03-12T03:45:46.447969+0000","last_clean":"2024-03-12T03:45:46.447969+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:45:46.447969+0000","last_undegraded":"2024-03-12T03:45:46.447969+0000","last_fullsized":"2024-03-12T03:45:46.447969+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:54.134348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:54.134348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:15.831557+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":315,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.454097+0000","last_change":"2024-03-12T03:45:18.107059+0000","last_active":"2024-03-12T03:45:46.454097+0000","last_peered":"2024-03-12T03:45:46.454097+0000","last_clean":"2024-03-12T03:34:01.860694+0000","last_became_active":"2024-03-12T03:45:18.107059+0000","last_became_peered":"2024-03-12T03:45:18.107059+0000","last_unstale":"2024-03-12T03:45:46.454097+0000","last_undegraded":"2024-03-12T03:45:46.454097+0000","last_fullsized":"2024-03-12T03:45:18.104095+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:42:10.356260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:42:10.356260+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.085066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"319'10945","reported_seq":11116,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:51.768910+0000","last_change":"2024-03-12T03:44:56.809341+0000","last_active":"2024-03-12T03:45:51.768910+0000","last_peered":"2024-03-12T03:45:51.768910+0000","last_clean":"2024-03-12T03:45:51.768910+0000","last_became_active":"2024-03-12T03:43:45.588933+0000","last_became_peered":"2024-03-12T03:43:45.588933+0000","last_unstale":"2024-03-12T03:45:51.768910+0000","last_undegraded":"2024-03-12T03:45:51.768910+0000","last_fullsized":"2024-03-12T03:45:51.768910+0000","mapping_epoch":278,"log_start":"299'5700","ondisk_log_start":"299'5700","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'6912","last_scrub_stamp":"2024-03-12T03:44:56.807997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:44:56.807997+0000","objects_scrubbed":6833,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:59.350475+0000","scrub_duration":1988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453443584,"num_objects":6919,"num_object_clones":0,"num_object_copies":20757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":571648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":542,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.457900+0000","last_change":"2024-03-12T03:44:11.511999+0000","last_active":"2024-03-12T03:45:46.457900+0000","last_peered":"2024-03-12T03:45:46.457900+0000","last_clean":"2024-03-12T03:45:46.457900+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:45:46.457900+0000","last_undegraded":"2024-03-12T03:45:46.457900+0000","last_fullsized":"2024-03-12T03:45:46.457900+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:11.511918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:44:11.511918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"269'24","reported_seq":631,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454306+0000","last_change":"2024-03-12T03:45:02.759428+0000","last_active":"2024-03-12T03:45:46.454306+0000","last_peered":"2024-03-12T03:45:46.454306+0000","last_clean":"2024-03-12T03:45:46.454306+0000","last_became_active":"2024-03-12T03:39:33.277868+0000","last_became_peered":"2024-03-12T03:39:33.277868+0000","last_unstale":"2024-03-12T03:45:46.454306+0000","last_undegraded":"2024-03-12T03:45:46.454306+0000","last_fullsized":"2024-03-12T03:45:46.454306+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:45:02.759243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:02.759243+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:12.385003+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":1316,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:49.300358+0000","last_change":"2024-03-12T03:44:11.061486+0000","last_active":"2024-03-12T03:45:49.300358+0000","last_peered":"2024-03-12T03:45:49.300358+0000","last_clean":"2024-03-12T03:45:49.300358+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:45:49.300358+0000","last_undegraded":"2024-03-12T03:45:49.300358+0000","last_fullsized":"2024-03-12T03:45:49.300358+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:44:11.061362+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.305493+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":738,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":556,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.453628+0000","last_change":"2024-03-12T03:43:18.338719+0000","last_active":"2024-03-12T03:45:46.453628+0000","last_peered":"2024-03-12T03:45:46.453628+0000","last_clean":"2024-03-12T03:43:13.332262+0000","last_became_active":"2024-03-12T03:43:18.338719+0000","last_became_peered":"2024-03-12T03:43:18.338719+0000","last_unstale":"2024-03-12T03:45:46.453628+0000","last_undegraded":"2024-03-12T03:45:46.453628+0000","last_fullsized":"2024-03-12T03:43:18.331120+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:42:21.445443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:55.072343+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.449075+0000","last_change":"2024-03-12T03:43:18.350023+0000","last_active":"2024-03-12T03:45:46.449075+0000","last_peered":"2024-03-12T03:45:46.449075+0000","last_clean":"2024-03-12T03:43:13.346844+0000","last_became_active":"2024-03-12T03:43:18.350023+0000","last_became_peered":"2024-03-12T03:43:18.350023+0000","last_unstale":"2024-03-12T03:45:46.449075+0000","last_undegraded":"2024-03-12T03:45:46.449075+0000","last_fullsized":"2024-03-12T03:43:18.331452+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:11.815414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:11.815414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100308+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":683,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.449717+0000","last_change":"2024-03-12T03:44:52.122439+0000","last_active":"2024-03-12T03:45:46.449717+0000","last_peered":"2024-03-12T03:45:46.449717+0000","last_clean":"2024-03-12T03:45:46.449717+0000","last_became_active":"2024-03-12T03:44:19.366060+0000","last_became_peered":"2024-03-12T03:44:19.366060+0000","last_unstale":"2024-03-12T03:45:46.449717+0000","last_undegraded":"2024-03-12T03:45:46.449717+0000","last_fullsized":"2024-03-12T03:45:46.449717+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:52.122312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:52.122312+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:03.603768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"319'11108","reported_seq":11197,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:51.779949+0000","last_change":"2024-03-12T03:43:55.393294+0000","last_active":"2024-03-12T03:45:51.779949+0000","last_peered":"2024-03-12T03:45:51.779949+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.588708+0000","last_became_peered":"2024-03-12T03:43:45.588708+0000","last_unstale":"2024-03-12T03:45:51.779949+0000","last_undegraded":"2024-03-12T03:43:55.382438+0000","last_fullsized":"2024-03-12T03:43:45.583017+0000","mapping_epoch":278,"log_start":"299'5927","ondisk_log_start":"299'5927","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459145216,"num_objects":7006,"num_object_clones":0,"num_object_copies":21018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11108,"num_write_kb":579648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":7006}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":563,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.448655+0000","last_change":"2024-03-12T03:45:16.222836+0000","last_active":"2024-03-12T03:45:46.448655+0000","last_peered":"2024-03-12T03:45:46.448655+0000","last_clean":"2024-03-12T03:45:46.448655+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:45:46.448655+0000","last_undegraded":"2024-03-12T03:45:46.448655+0000","last_fullsized":"2024-03-12T03:45:46.448655+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:16.222710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:16.222710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:43.800555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"315'29","reported_seq":642,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.461539+0000","last_change":"2024-03-12T03:45:18.109405+0000","last_active":"2024-03-12T03:45:46.461539+0000","last_peered":"2024-03-12T03:45:46.461539+0000","last_clean":"2024-03-12T03:45:46.461539+0000","last_became_active":"2024-03-12T03:45:18.109048+0000","last_became_peered":"2024-03-12T03:45:18.109048+0000","last_unstale":"2024-03-12T03:45:46.461539+0000","last_undegraded":"2024-03-12T03:45:46.461539+0000","last_fullsized":"2024-03-12T03:45:46.461539+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'28","last_scrub_stamp":"2024-03-12T03:44:58.632869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:44:58.632869+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:19.815212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":360,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.454647+0000","last_change":"2024-03-12T03:43:18.349427+0000","last_active":"2024-03-12T03:45:46.454647+0000","last_peered":"2024-03-12T03:45:46.454647+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:43:18.349427+0000","last_became_peered":"2024-03-12T03:43:18.349427+0000","last_unstale":"2024-03-12T03:45:46.454647+0000","last_undegraded":"2024-03-12T03:45:46.454647+0000","last_fullsized":"2024-03-12T03:43:18.334581+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:07.572642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:07.572642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":833,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.446860+0000","last_change":"2024-03-12T03:44:54.092085+0000","last_active":"2024-03-12T03:45:46.446860+0000","last_peered":"2024-03-12T03:45:46.446860+0000","last_clean":"2024-03-12T03:45:46.446860+0000","last_became_active":"2024-03-12T03:44:19.379262+0000","last_became_peered":"2024-03-12T03:44:19.379262+0000","last_unstale":"2024-03-12T03:45:46.446860+0000","last_undegraded":"2024-03-12T03:45:46.446860+0000","last_fullsized":"2024-03-12T03:45:46.446860+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:54.091965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:54.091965+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:18.114887+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":542,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.456707+0000","last_change":"2024-03-12T03:43:18.345043+0000","last_active":"2024-03-12T03:45:46.456707+0000","last_peered":"2024-03-12T03:45:46.456707+0000","last_clean":"2024-03-12T03:43:13.352684+0000","last_became_active":"2024-03-12T03:43:18.345043+0000","last_became_peered":"2024-03-12T03:43:18.345043+0000","last_unstale":"2024-03-12T03:45:46.456707+0000","last_undegraded":"2024-03-12T03:45:46.456707+0000","last_fullsized":"2024-03-12T03:43:18.334020+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:00.436718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:00.436718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"319'10815","reported_seq":10997,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:51.266122+0000","last_change":"2024-03-12T03:45:15.346003+0000","last_active":"2024-03-12T03:45:51.266122+0000","last_peered":"2024-03-12T03:45:51.266122+0000","last_clean":"2024-03-12T03:45:51.266122+0000","last_became_active":"2024-03-12T03:43:45.594793+0000","last_became_peered":"2024-03-12T03:43:45.594793+0000","last_unstale":"2024-03-12T03:45:51.266122+0000","last_undegraded":"2024-03-12T03:45:51.266122+0000","last_fullsized":"2024-03-12T03:45:51.266122+0000","mapping_epoch":278,"log_start":"300'6403","ondisk_log_start":"300'6403","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'8312","last_scrub_stamp":"2024-03-12T03:45:15.345772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.345772+0000","objects_scrubbed":8246,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:36.634416+0000","scrub_duration":2226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452788224,"num_objects":6909,"num_object_clones":0,"num_object_copies":20727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10815,"num_write_kb":567168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":278,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.445289+0000","last_change":"2024-03-12T03:44:03.517544+0000","last_active":"2024-03-12T03:45:46.445289+0000","last_peered":"2024-03-12T03:45:46.445289+0000","last_clean":"2024-03-12T03:45:46.445289+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:45:46.445289+0000","last_undegraded":"2024-03-12T03:45:46.445289+0000","last_fullsized":"2024-03-12T03:45:46.445289+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:44:03.517277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:47.790441+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"269'23","reported_seq":600,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.448798+0000","last_change":"2024-03-12T03:43:29.744286+0000","last_active":"2024-03-12T03:45:46.448798+0000","last_peered":"2024-03-12T03:45:46.448798+0000","last_clean":"2024-03-12T03:45:46.448798+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:45:46.448798+0000","last_undegraded":"2024-03-12T03:45:46.448798+0000","last_fullsized":"2024-03-12T03:45:46.448798+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'23","last_scrub_stamp":"2024-03-12T03:43:29.744179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:29.744179+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":275,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.444695+0000","last_change":"2024-03-12T03:43:18.343047+0000","last_active":"2024-03-12T03:45:46.444695+0000","last_peered":"2024-03-12T03:45:46.444695+0000","last_clean":"2024-03-12T03:43:13.346919+0000","last_became_active":"2024-03-12T03:43:18.343047+0000","last_became_peered":"2024-03-12T03:43:18.343047+0000","last_unstale":"2024-03-12T03:45:46.444695+0000","last_undegraded":"2024-03-12T03:45:46.444695+0000","last_fullsized":"2024-03-12T03:43:18.336453+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:01.189842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:42:01.189842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":580,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.448293+0000","last_change":"2024-03-12T03:45:00.012141+0000","last_active":"2024-03-12T03:45:46.448293+0000","last_peered":"2024-03-12T03:45:46.448293+0000","last_clean":"2024-03-12T03:45:46.448293+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:45:46.448293+0000","last_undegraded":"2024-03-12T03:45:46.448293+0000","last_fullsized":"2024-03-12T03:45:46.448293+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:00.012064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:45:00.012064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:21.470920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":553,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.451828+0000","last_change":"2024-03-12T03:43:18.348792+0000","last_active":"2024-03-12T03:45:46.451828+0000","last_peered":"2024-03-12T03:45:46.451828+0000","last_clean":"2024-03-12T03:43:13.342858+0000","last_became_active":"2024-03-12T03:43:18.348792+0000","last_became_peered":"2024-03-12T03:43:18.348792+0000","last_unstale":"2024-03-12T03:45:46.451828+0000","last_undegraded":"2024-03-12T03:43:18.339155+0000","last_fullsized":"2024-03-12T03:43:18.339042+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:42:20.203740+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:42:20.203740+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.303891+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":848,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.448376+0000","last_change":"2024-03-12T03:44:53.098337+0000","last_active":"2024-03-12T03:45:46.448376+0000","last_peered":"2024-03-12T03:45:46.448376+0000","last_clean":"2024-03-12T03:45:46.448376+0000","last_became_active":"2024-03-12T03:44:19.378554+0000","last_became_peered":"2024-03-12T03:44:19.378554+0000","last_unstale":"2024-03-12T03:45:46.448376+0000","last_undegraded":"2024-03-12T03:45:46.448376+0000","last_fullsized":"2024-03-12T03:45:46.448376+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:53.097930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:53.097930+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:58.415909+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"319'15153","reported_seq":15526,"reported_epoch":319,"state":"active+clean+scrubbing","last_fresh":"2024-03-12T03:45:51.787739+0000","last_change":"2024-03-12T03:45:50.072397+0000","last_active":"2024-03-12T03:45:51.787739+0000","last_peered":"2024-03-12T03:45:51.787739+0000","last_clean":"2024-03-12T03:45:51.787739+0000","last_became_active":"2024-03-12T03:43:45.591646+0000","last_became_peered":"2024-03-12T03:43:45.591646+0000","last_unstale":"2024-03-12T03:45:51.787739+0000","last_undegraded":"2024-03-12T03:45:51.787739+0000","last_fullsized":"2024-03-12T03:45:51.787739+0000","mapping_epoch":278,"log_start":"315'9945","ondisk_log_start":"315'9945","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":5649,"log_size":5208,"log_dups_size":0,"ondisk_log_size":5208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456916992,"num_objects":6972,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11140,"num_write_kb":579584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":551,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454047+0000","last_change":"2024-03-12T03:45:06.865102+0000","last_active":"2024-03-12T03:45:46.454047+0000","last_peered":"2024-03-12T03:45:46.454047+0000","last_clean":"2024-03-12T03:45:46.454047+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:45:46.454047+0000","last_undegraded":"2024-03-12T03:45:46.454047+0000","last_fullsized":"2024-03-12T03:45:46.454047+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:06.864708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.864708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:24.003019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"269'52","reported_seq":779,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.453780+0000","last_change":"2024-03-12T03:43:18.348827+0000","last_active":"2024-03-12T03:45:46.453780+0000","last_peered":"2024-03-12T03:45:46.453780+0000","last_clean":"2024-03-12T03:43:16.743398+0000","last_became_active":"2024-03-12T03:43:18.348827+0000","last_became_peered":"2024-03-12T03:43:18.348827+0000","last_unstale":"2024-03-12T03:45:46.453780+0000","last_undegraded":"2024-03-12T03:43:18.331846+0000","last_fullsized":"2024-03-12T03:43:18.331727+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'39","last_scrub_stamp":"2024-03-12T03:43:06.482430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:06.482430+0000","objects_scrubbed":11,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":11}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":560,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.461430+0000","last_change":"2024-03-12T03:44:04.812250+0000","last_active":"2024-03-12T03:45:46.461430+0000","last_peered":"2024-03-12T03:45:46.461430+0000","last_clean":"2024-03-12T03:45:46.461430+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:45:46.461430+0000","last_undegraded":"2024-03-12T03:45:46.461430+0000","last_fullsized":"2024-03-12T03:45:46.461430+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:04.812138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:44:04.812138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":562,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454897+0000","last_change":"2024-03-12T03:44:10.510880+0000","last_active":"2024-03-12T03:45:46.454897+0000","last_peered":"2024-03-12T03:45:46.454897+0000","last_clean":"2024-03-12T03:45:46.454897+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:45:46.454897+0000","last_undegraded":"2024-03-12T03:45:46.454897+0000","last_fullsized":"2024-03-12T03:45:46.454897+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:44:10.510743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:44:10.510743+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":404,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.461627+0000","last_change":"2024-03-12T03:43:18.344442+0000","last_active":"2024-03-12T03:45:46.461627+0000","last_peered":"2024-03-12T03:45:46.461627+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.344442+0000","last_became_peered":"2024-03-12T03:43:18.344442+0000","last_unstale":"2024-03-12T03:45:46.461627+0000","last_undegraded":"2024-03-12T03:45:46.461627+0000","last_fullsized":"2024-03-12T03:43:18.330930+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:39.031166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:39.031166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":691,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.455438+0000","last_change":"2024-03-12T03:44:51.991449+0000","last_active":"2024-03-12T03:45:46.455438+0000","last_peered":"2024-03-12T03:45:46.455438+0000","last_clean":"2024-03-12T03:45:46.455438+0000","last_became_active":"2024-03-12T03:44:19.371194+0000","last_became_peered":"2024-03-12T03:44:19.371194+0000","last_unstale":"2024-03-12T03:45:46.455438+0000","last_undegraded":"2024-03-12T03:45:46.455438+0000","last_fullsized":"2024-03-12T03:45:46.455438+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:51.991223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:51.991223+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:12.236261+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":362,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:45:46.461282+0000","last_change":"2024-03-12T03:44:52.829443+0000","last_active":"2024-03-12T03:45:46.461282+0000","last_peered":"2024-03-12T03:45:46.461282+0000","last_clean":"2024-03-12T03:45:46.461282+0000","last_became_active":"2024-03-12T03:44:24.375475+0000","last_became_peered":"2024-03-12T03:44:24.375475+0000","last_unstale":"2024-03-12T03:45:46.461282+0000","last_undegraded":"2024-03-12T03:45:46.461282+0000","last_fullsized":"2024-03-12T03:45:46.461282+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:52.828725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:52.828725+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:18.841460+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"319'16276","reported_seq":16737,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:51.319573+0000","last_change":"2024-03-12T03:45:50.733212+0000","last_active":"2024-03-12T03:45:51.319573+0000","last_peered":"2024-03-12T03:45:51.319573+0000","last_clean":"2024-03-12T03:45:51.319573+0000","last_became_active":"2024-03-12T03:43:45.631620+0000","last_became_peered":"2024-03-12T03:43:45.631620+0000","last_unstale":"2024-03-12T03:45:51.319573+0000","last_undegraded":"2024-03-12T03:45:51.319573+0000","last_fullsized":"2024-03-12T03:45:51.319573+0000","mapping_epoch":278,"log_start":"315'11857","ondisk_log_start":"315'11857","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'16240","last_scrub_stamp":"2024-03-12T03:45:50.732977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:45:50.732977+0000","objects_scrubbed":7177,"log_size":4419,"log_dups_size":0,"ondisk_log_size":4419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:47:02.018752+0000","scrub_duration":2360,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461176832,"num_objects":7037,"num_object_clones":0,"num_object_copies":21111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11071,"num_write_kb":579456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"269'26","reported_seq":597,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454179+0000","last_change":"2024-03-12T03:45:15.774203+0000","last_active":"2024-03-12T03:45:46.454179+0000","last_peered":"2024-03-12T03:45:46.454179+0000","last_clean":"2024-03-12T03:45:46.454179+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:45:46.454179+0000","last_undegraded":"2024-03-12T03:45:46.454179+0000","last_fullsized":"2024-03-12T03:45:46.454179+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:15.774076+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.774076+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:24.135652+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":554,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.454087+0000","last_change":"2024-03-12T03:43:18.349816+0000","last_active":"2024-03-12T03:45:46.454087+0000","last_peered":"2024-03-12T03:45:46.454087+0000","last_clean":"2024-03-12T03:43:13.352139+0000","last_became_active":"2024-03-12T03:43:18.349816+0000","last_became_peered":"2024-03-12T03:43:18.349816+0000","last_unstale":"2024-03-12T03:45:46.454087+0000","last_undegraded":"2024-03-12T03:45:46.454087+0000","last_fullsized":"2024-03-12T03:43:18.332719+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:40:59.344058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:40:59.344058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":274,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.444685+0000","last_change":"2024-03-12T03:43:18.346068+0000","last_active":"2024-03-12T03:45:46.444685+0000","last_peered":"2024-03-12T03:45:46.444685+0000","last_clean":"2024-03-12T03:43:13.348345+0000","last_became_active":"2024-03-12T03:43:18.346068+0000","last_became_peered":"2024-03-12T03:43:18.346068+0000","last_unstale":"2024-03-12T03:45:46.444685+0000","last_undegraded":"2024-03-12T03:45:46.444685+0000","last_fullsized":"2024-03-12T03:43:18.339666+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:03.191398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:42:03.191398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:47.790398+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":551,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.453760+0000","last_change":"2024-03-12T03:44:05.789257+0000","last_active":"2024-03-12T03:45:46.453760+0000","last_peered":"2024-03-12T03:45:46.453760+0000","last_clean":"2024-03-12T03:45:46.453760+0000","last_became_active":"2024-03-12T03:39:33.271016+0000","last_became_peered":"2024-03-12T03:39:33.271016+0000","last_unstale":"2024-03-12T03:45:46.453760+0000","last_undegraded":"2024-03-12T03:45:46.453760+0000","last_fullsized":"2024-03-12T03:45:46.453760+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:05.789123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:05.789123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":664,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:50.401886+0000","last_change":"2024-03-12T03:45:50.401930+0000","last_active":"2024-03-12T03:45:50.401886+0000","last_peered":"2024-03-12T03:45:50.401886+0000","last_clean":"2024-03-12T03:45:50.401886+0000","last_became_active":"2024-03-12T03:45:18.108107+0000","last_became_peered":"2024-03-12T03:45:18.108107+0000","last_unstale":"2024-03-12T03:45:50.401886+0000","last_undegraded":"2024-03-12T03:45:50.401886+0000","last_fullsized":"2024-03-12T03:45:50.401886+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:45:50.401804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:45:50.401804+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:47:03.185197+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"319'11091","reported_seq":11180,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:51.318541+0000","last_change":"2024-03-12T03:43:55.403641+0000","last_active":"2024-03-12T03:45:51.318541+0000","last_peered":"2024-03-12T03:45:51.318541+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.592972+0000","last_became_peered":"2024-03-12T03:43:45.592972+0000","last_unstale":"2024-03-12T03:45:51.318541+0000","last_undegraded":"2024-03-12T03:43:55.398613+0000","last_fullsized":"2024-03-12T03:43:45.585435+0000","mapping_epoch":278,"log_start":"301'6718","ondisk_log_start":"301'6718","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459210752,"num_objects":7007,"num_object_clones":0,"num_object_copies":21021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11091,"num_write_kb":579136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":7007}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'51","reported_seq":551,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.450152+0000","last_change":"2024-03-12T03:45:18.106865+0000","last_active":"2024-03-12T03:45:46.450152+0000","last_peered":"2024-03-12T03:45:46.450152+0000","last_clean":"2024-03-12T03:45:46.450152+0000","last_became_active":"2024-03-12T03:45:18.106503+0000","last_became_peered":"2024-03-12T03:45:18.106503+0000","last_unstale":"2024-03-12T03:45:46.450152+0000","last_undegraded":"2024-03-12T03:45:46.450152+0000","last_fullsized":"2024-03-12T03:45:46.450152+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'51","last_scrub_stamp":"2024-03-12T03:45:06.539548+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.539548+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:14.145611+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.451727+0000","last_change":"2024-03-12T03:44:56.159355+0000","last_active":"2024-03-12T03:45:46.451727+0000","last_peered":"2024-03-12T03:45:46.451727+0000","last_clean":"2024-03-12T03:45:46.451727+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:45:46.451727+0000","last_undegraded":"2024-03-12T03:45:46.451727+0000","last_fullsized":"2024-03-12T03:45:46.451727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:56.159248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:44:56.159248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:59.102913+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":270,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.444936+0000","last_change":"2024-03-12T03:44:11.497903+0000","last_active":"2024-03-12T03:45:46.444936+0000","last_peered":"2024-03-12T03:45:46.444936+0000","last_clean":"2024-03-12T03:45:46.444936+0000","last_became_active":"2024-03-12T03:39:33.254395+0000","last_became_peered":"2024-03-12T03:39:33.254395+0000","last_unstale":"2024-03-12T03:45:46.444936+0000","last_undegraded":"2024-03-12T03:45:46.444936+0000","last_fullsized":"2024-03-12T03:45:46.444936+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_clean_scrub_stamp":"2024-03-12T03:44:11.497802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:47.790473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":563,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.446610+0000","last_change":"2024-03-12T03:43:18.339713+0000","last_active":"2024-03-12T03:45:46.446610+0000","last_peered":"2024-03-12T03:45:46.446610+0000","last_clean":"2024-03-12T03:43:15.079953+0000","last_became_active":"2024-03-12T03:43:18.339713+0000","last_became_peered":"2024-03-12T03:43:18.339713+0000","last_unstale":"2024-03-12T03:45:46.446610+0000","last_undegraded":"2024-03-12T03:45:46.446610+0000","last_fullsized":"2024-03-12T03:43:18.336066+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:15.079874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:15.079874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.304473+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":469,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:45:46.453840+0000","last_change":"2024-03-12T03:44:51.818506+0000","last_active":"2024-03-12T03:45:46.453840+0000","last_peered":"2024-03-12T03:45:46.453840+0000","last_clean":"2024-03-12T03:45:46.453840+0000","last_became_active":"2024-03-12T03:44:24.370429+0000","last_became_peered":"2024-03-12T03:44:24.370429+0000","last_unstale":"2024-03-12T03:45:46.453840+0000","last_undegraded":"2024-03-12T03:45:46.453840+0000","last_fullsized":"2024-03-12T03:45:46.453840+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:51.818256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:51.818256+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:57.271837+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"319'10696","reported_seq":10785,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:47.181818+0000","last_change":"2024-03-12T03:43:55.386671+0000","last_active":"2024-03-12T03:45:47.181818+0000","last_peered":"2024-03-12T03:45:47.181818+0000","last_clean":"2024-03-12T03:43:44.570660+0000","last_became_active":"2024-03-12T03:43:45.590025+0000","last_became_peered":"2024-03-12T03:43:45.590025+0000","last_unstale":"2024-03-12T03:45:47.181818+0000","last_undegraded":"2024-03-12T03:43:55.378927+0000","last_fullsized":"2024-03-12T03:43:45.586649+0000","mapping_epoch":278,"log_start":"299'6200","ondisk_log_start":"299'6200","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":4496,"log_dups_size":0,"ondisk_log_size":4496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483262464,"num_objects":7374,"num_object_clones":0,"num_object_copies":22122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10696,"num_write_kb":578240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7374}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":278,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.444423+0000","last_change":"2024-03-12T03:44:17.593848+0000","last_active":"2024-03-12T03:45:46.444423+0000","last_peered":"2024-03-12T03:45:46.444423+0000","last_clean":"2024-03-12T03:45:46.444423+0000","last_became_active":"2024-03-12T03:39:33.278326+0000","last_became_peered":"2024-03-12T03:39:33.278326+0000","last_unstale":"2024-03-12T03:45:46.444423+0000","last_undegraded":"2024-03-12T03:45:46.444423+0000","last_fullsized":"2024-03-12T03:45:46.444423+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:17.593753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:44:17.593753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:47.790457+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"269'30","reported_seq":615,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.458958+0000","last_change":"2024-03-12T03:43:18.341247+0000","last_active":"2024-03-12T03:45:46.458958+0000","last_peered":"2024-03-12T03:45:46.458958+0000","last_clean":"2024-03-12T03:43:15.540480+0000","last_became_active":"2024-03-12T03:43:18.341247+0000","last_became_peered":"2024-03-12T03:43:18.341247+0000","last_unstale":"2024-03-12T03:45:46.458958+0000","last_undegraded":"2024-03-12T03:43:18.330226+0000","last_fullsized":"2024-03-12T03:43:18.330117+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-03-12T03:43:15.540339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:15.540339+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.452102+0000","last_change":"2024-03-12T03:44:05.063402+0000","last_active":"2024-03-12T03:45:46.452102+0000","last_peered":"2024-03-12T03:45:46.452102+0000","last_clean":"2024-03-12T03:45:46.452102+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:45:46.452102+0000","last_undegraded":"2024-03-12T03:45:46.452102+0000","last_fullsized":"2024-03-12T03:45:46.452102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:05.063185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:44:05.063185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.305400+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":319,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.450740+0000","last_change":"2024-03-12T03:43:18.345830+0000","last_active":"2024-03-12T03:45:46.450740+0000","last_peered":"2024-03-12T03:45:46.450740+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.345830+0000","last_became_peered":"2024-03-12T03:43:18.345830+0000","last_unstale":"2024-03-12T03:45:46.450740+0000","last_undegraded":"2024-03-12T03:45:46.450740+0000","last_fullsized":"2024-03-12T03:43:18.336109+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:18.651418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:18.651418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.303866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":479,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.448148+0000","last_change":"2024-03-12T03:44:55.099123+0000","last_active":"2024-03-12T03:45:46.448148+0000","last_peered":"2024-03-12T03:45:46.448148+0000","last_clean":"2024-03-12T03:45:46.448148+0000","last_became_active":"2024-03-12T03:44:24.373190+0000","last_became_peered":"2024-03-12T03:44:24.373190+0000","last_unstale":"2024-03-12T03:45:46.448148+0000","last_undegraded":"2024-03-12T03:45:46.448148+0000","last_fullsized":"2024-03-12T03:45:46.448148+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:55.099010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:55.099010+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:13.562616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"319'10896","reported_seq":10989,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:51.789814+0000","last_change":"2024-03-12T03:43:45.593274+0000","last_active":"2024-03-12T03:45:51.789814+0000","last_peered":"2024-03-12T03:45:51.789814+0000","last_clean":"2024-03-12T03:45:51.789814+0000","last_became_active":"2024-03-12T03:43:45.593074+0000","last_became_peered":"2024-03-12T03:43:45.593074+0000","last_unstale":"2024-03-12T03:45:51.789814+0000","last_undegraded":"2024-03-12T03:45:51.789814+0000","last_fullsized":"2024-03-12T03:45:51.789814+0000","mapping_epoch":278,"log_start":"299'5633","ondisk_log_start":"299'5633","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:44.570660+0000","last_clean_scrub_stamp":"2024-03-12T03:43:44.570660+0000","objects_scrubbed":0,"log_size":5263,"log_dups_size":0,"ondisk_log_size":5263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453902336,"num_objects":6926,"num_object_clones":0,"num_object_copies":20778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10896,"num_write_kb":570304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":710,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.450577+0000","last_change":"2024-03-12T03:45:10.340919+0000","last_active":"2024-03-12T03:45:46.450577+0000","last_peered":"2024-03-12T03:45:46.450577+0000","last_clean":"2024-03-12T03:45:46.450577+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:45:46.450577+0000","last_undegraded":"2024-03-12T03:45:46.450577+0000","last_fullsized":"2024-03-12T03:45:46.450577+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:45:10.340796+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.340796+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:20.915059+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"269'48","reported_seq":630,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.453823+0000","last_change":"2024-03-12T03:45:14.163797+0000","last_active":"2024-03-12T03:45:46.453823+0000","last_peered":"2024-03-12T03:45:46.453823+0000","last_clean":"2024-03-12T03:45:46.453823+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:45:46.453823+0000","last_undegraded":"2024-03-12T03:45:46.453823+0000","last_fullsized":"2024-03-12T03:45:46.453823+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'48","last_scrub_stamp":"2024-03-12T03:45:14.163544+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:45:14.163544+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:31.195127+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":563,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.447725+0000","last_change":"2024-03-12T03:43:59.987280+0000","last_active":"2024-03-12T03:45:46.447725+0000","last_peered":"2024-03-12T03:45:46.447725+0000","last_clean":"2024-03-12T03:45:46.447725+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:45:46.447725+0000","last_undegraded":"2024-03-12T03:45:46.447725+0000","last_fullsized":"2024-03-12T03:45:46.447725+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:43:59.987096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:59.987096+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.305432+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":538,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454547+0000","last_change":"2024-03-12T03:44:12.561600+0000","last_active":"2024-03-12T03:45:46.454547+0000","last_peered":"2024-03-12T03:45:46.454547+0000","last_clean":"2024-03-12T03:45:46.454547+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:45:46.454547+0000","last_undegraded":"2024-03-12T03:45:46.454547+0000","last_fullsized":"2024-03-12T03:45:46.454547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:12.561484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:44:12.561484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":363,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454029+0000","last_change":"2024-03-12T03:44:55.920207+0000","last_active":"2024-03-12T03:45:46.454029+0000","last_peered":"2024-03-12T03:45:46.454029+0000","last_clean":"2024-03-12T03:45:46.454029+0000","last_became_active":"2024-03-12T03:44:24.365642+0000","last_became_peered":"2024-03-12T03:44:24.365642+0000","last_unstale":"2024-03-12T03:45:46.454029+0000","last_undegraded":"2024-03-12T03:45:46.454029+0000","last_fullsized":"2024-03-12T03:45:46.454029+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:55.920045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:55.920045+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:24.990650+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"269'18","reported_seq":206,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.447806+0000","last_change":"2024-03-12T03:43:18.350084+0000","last_active":"2024-03-12T03:45:46.447806+0000","last_peered":"2024-03-12T03:45:46.447806+0000","last_clean":"2024-03-12T03:33:28.523929+0000","last_became_active":"2024-03-12T03:43:18.350084+0000","last_became_peered":"2024-03-12T03:43:18.350084+0000","last_unstale":"2024-03-12T03:45:46.447806+0000","last_undegraded":"2024-03-12T03:43:18.336688+0000","last_fullsized":"2024-03-12T03:43:18.336568+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:43:17.648603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:17.648603+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.304518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.453377+0000","last_change":"2024-03-12T03:45:18.106344+0000","last_active":"2024-03-12T03:45:46.453377+0000","last_peered":"2024-03-12T03:45:46.453377+0000","last_clean":"2024-03-12T03:45:46.453377+0000","last_became_active":"2024-03-12T03:45:18.106080+0000","last_became_peered":"2024-03-12T03:45:18.106080+0000","last_unstale":"2024-03-12T03:45:46.453377+0000","last_undegraded":"2024-03-12T03:45:46.453377+0000","last_fullsized":"2024-03-12T03:45:46.453377+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:10.400135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.400135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:27.248318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":546,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.453519+0000","last_change":"2024-03-12T03:43:18.348340+0000","last_active":"2024-03-12T03:45:46.453519+0000","last_peered":"2024-03-12T03:45:46.453519+0000","last_clean":"2024-03-12T03:43:13.352260+0000","last_became_active":"2024-03-12T03:43:18.348340+0000","last_became_peered":"2024-03-12T03:43:18.348340+0000","last_unstale":"2024-03-12T03:45:46.453519+0000","last_undegraded":"2024-03-12T03:45:46.453519+0000","last_fullsized":"2024-03-12T03:43:18.333554+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:01.417809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:01.417809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":547,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.451353+0000","last_change":"2024-03-12T03:44:03.085231+0000","last_active":"2024-03-12T03:45:46.451353+0000","last_peered":"2024-03-12T03:45:46.451353+0000","last_clean":"2024-03-12T03:45:46.451353+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:45:46.451353+0000","last_undegraded":"2024-03-12T03:45:46.451353+0000","last_fullsized":"2024-03-12T03:45:46.451353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:03.083941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:03.083941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.305460+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":686,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:45:46.447784+0000","last_change":"2024-03-12T03:44:56.067961+0000","last_active":"2024-03-12T03:45:46.447784+0000","last_peered":"2024-03-12T03:45:46.447784+0000","last_clean":"2024-03-12T03:45:46.447784+0000","last_became_active":"2024-03-12T03:44:22.590882+0000","last_became_peered":"2024-03-12T03:44:22.590882+0000","last_unstale":"2024-03-12T03:45:46.447784+0000","last_undegraded":"2024-03-12T03:45:46.447784+0000","last_fullsized":"2024-03-12T03:45:46.447784+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:56.067715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:56.067715+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:01.841825+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"269'27","reported_seq":600,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.454450+0000","last_change":"2024-03-12T03:43:18.358476+0000","last_active":"2024-03-12T03:45:46.454450+0000","last_peered":"2024-03-12T03:45:46.454450+0000","last_clean":"2024-03-12T03:43:13.333279+0000","last_became_active":"2024-03-12T03:43:18.358476+0000","last_became_peered":"2024-03-12T03:43:18.358476+0000","last_unstale":"2024-03-12T03:45:46.454450+0000","last_undegraded":"2024-03-12T03:43:18.343811+0000","last_fullsized":"2024-03-12T03:43:18.343694+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:43:03.583989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:03.583989+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.448659+0000","last_change":"2024-03-12T03:44:58.185913+0000","last_active":"2024-03-12T03:45:46.448659+0000","last_peered":"2024-03-12T03:45:46.448659+0000","last_clean":"2024-03-12T03:45:46.448659+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:45:46.448659+0000","last_undegraded":"2024-03-12T03:45:46.448659+0000","last_fullsized":"2024-03-12T03:45:46.448659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:58.185812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:44:58.185812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:05.942484+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":351,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.449031+0000","last_change":"2024-03-12T03:43:18.346246+0000","last_active":"2024-03-12T03:45:46.449031+0000","last_peered":"2024-03-12T03:45:46.449031+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:43:18.346246+0000","last_became_peered":"2024-03-12T03:43:18.346246+0000","last_unstale":"2024-03-12T03:45:46.449031+0000","last_undegraded":"2024-03-12T03:45:46.449031+0000","last_fullsized":"2024-03-12T03:43:18.335155+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:05.530729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:43:05.530729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":278,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.445563+0000","last_change":"2024-03-12T03:45:04.594041+0000","last_active":"2024-03-12T03:45:46.445563+0000","last_peered":"2024-03-12T03:45:46.445563+0000","last_clean":"2024-03-12T03:45:46.445563+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:45:46.445563+0000","last_undegraded":"2024-03-12T03:45:46.445563+0000","last_fullsized":"2024-03-12T03:45:46.445563+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.593904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.593904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:24.853544+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":363,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.453350+0000","last_change":"2024-03-12T03:44:56.960991+0000","last_active":"2024-03-12T03:45:46.453350+0000","last_peered":"2024-03-12T03:45:46.453350+0000","last_clean":"2024-03-12T03:45:46.453350+0000","last_became_active":"2024-03-12T03:44:24.376141+0000","last_became_peered":"2024-03-12T03:44:24.376141+0000","last_unstale":"2024-03-12T03:45:46.453350+0000","last_undegraded":"2024-03-12T03:45:46.453350+0000","last_fullsized":"2024-03-12T03:45:46.453350+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:56.960733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:56.960733+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:58.874517+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"269'21","reported_seq":595,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.455068+0000","last_change":"2024-03-12T03:43:18.347815+0000","last_active":"2024-03-12T03:45:46.455068+0000","last_peered":"2024-03-12T03:45:46.455068+0000","last_clean":"2024-03-12T03:43:13.353750+0000","last_became_active":"2024-03-12T03:43:18.347815+0000","last_became_peered":"2024-03-12T03:43:18.347815+0000","last_unstale":"2024-03-12T03:45:46.455068+0000","last_undegraded":"2024-03-12T03:43:18.332425+0000","last_fullsized":"2024-03-12T03:43:18.332310+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:40:58.312602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:40:58.312602+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":555,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.453047+0000","last_change":"2024-03-12T03:43:18.343018+0000","last_active":"2024-03-12T03:45:46.453047+0000","last_peered":"2024-03-12T03:45:46.453047+0000","last_clean":"2024-03-12T03:43:13.331412+0000","last_became_active":"2024-03-12T03:43:18.343018+0000","last_became_peered":"2024-03-12T03:43:18.343018+0000","last_unstale":"2024-03-12T03:45:46.453047+0000","last_undegraded":"2024-03-12T03:45:46.453047+0000","last_fullsized":"2024-03-12T03:43:18.331686+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:00.266512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:42:00.266512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"266'13","reported_seq":229,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.450172+0000","last_change":"2024-03-12T03:43:18.350782+0000","last_active":"2024-03-12T03:45:46.450172+0000","last_peered":"2024-03-12T03:45:46.450172+0000","last_clean":"2024-03-12T03:33:28.372811+0000","last_became_active":"2024-03-12T03:43:18.350782+0000","last_became_peered":"2024-03-12T03:43:18.350782+0000","last_unstale":"2024-03-12T03:45:46.450172+0000","last_undegraded":"2024-03-12T03:43:18.340626+0000","last_fullsized":"2024-03-12T03:43:18.340503+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'12","last_scrub_stamp":"2024-03-12T03:43:12.675070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:12.675070+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.304503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.461559+0000","last_change":"2024-03-12T03:44:08.917228+0000","last_active":"2024-03-12T03:45:46.461559+0000","last_peered":"2024-03-12T03:45:46.461559+0000","last_clean":"2024-03-12T03:45:46.461559+0000","last_became_active":"2024-03-12T03:39:33.265499+0000","last_became_peered":"2024-03-12T03:39:33.265499+0000","last_unstale":"2024-03-12T03:45:46.461559+0000","last_undegraded":"2024-03-12T03:45:46.461559+0000","last_fullsized":"2024-03-12T03:45:46.461559+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:08.917119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:08.917119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674255+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":374,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:45:46.452900+0000","last_change":"2024-03-12T03:44:55.797414+0000","last_active":"2024-03-12T03:45:46.452900+0000","last_peered":"2024-03-12T03:45:46.452900+0000","last_clean":"2024-03-12T03:45:46.452900+0000","last_became_active":"2024-03-12T03:44:22.594212+0000","last_became_peered":"2024-03-12T03:44:22.594212+0000","last_unstale":"2024-03-12T03:45:46.452900+0000","last_undegraded":"2024-03-12T03:45:46.452900+0000","last_fullsized":"2024-03-12T03:45:46.452900+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:55.795990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:55.795990+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:08.417723+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,2],"avail_no_missing":["0","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"269'32","reported_seq":515,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.444793+0000","last_change":"2024-03-12T03:43:18.357702+0000","last_active":"2024-03-12T03:45:46.444793+0000","last_peered":"2024-03-12T03:45:46.444793+0000","last_clean":"2024-03-12T03:42:47.830157+0000","last_became_active":"2024-03-12T03:43:18.357702+0000","last_became_peered":"2024-03-12T03:43:18.357702+0000","last_unstale":"2024-03-12T03:45:46.444793+0000","last_undegraded":"2024-03-12T03:43:18.340096+0000","last_fullsized":"2024-03-12T03:43:18.339982+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-03-12T03:35:58.079645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:35:58.079645+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:47.790369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":586,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.450313+0000","last_change":"2024-03-12T03:44:13.063349+0000","last_active":"2024-03-12T03:45:46.450313+0000","last_peered":"2024-03-12T03:45:46.450313+0000","last_clean":"2024-03-12T03:45:46.450313+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:45:46.450313+0000","last_undegraded":"2024-03-12T03:45:46.450313+0000","last_fullsized":"2024-03-12T03:45:46.450313+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:44:13.060719+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.305584+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":700,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454832+0000","last_change":"2024-03-12T03:44:05.557046+0000","last_active":"2024-03-12T03:45:46.454832+0000","last_peered":"2024-03-12T03:45:46.454832+0000","last_clean":"2024-03-12T03:45:46.454832+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:45:46.454832+0000","last_undegraded":"2024-03-12T03:45:46.454832+0000","last_fullsized":"2024-03-12T03:45:46.454832+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:44:05.556852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:44:05.556852+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":543,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.454169+0000","last_change":"2024-03-12T03:43:18.357954+0000","last_active":"2024-03-12T03:45:46.454169+0000","last_peered":"2024-03-12T03:45:46.454169+0000","last_clean":"2024-03-12T03:43:13.332853+0000","last_became_active":"2024-03-12T03:43:18.357954+0000","last_became_peered":"2024-03-12T03:43:18.357954+0000","last_unstale":"2024-03-12T03:45:46.454169+0000","last_undegraded":"2024-03-12T03:45:46.454169+0000","last_fullsized":"2024-03-12T03:43:18.341976+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:42:10.355198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:42:10.355198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"269'19","reported_seq":594,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.461834+0000","last_change":"2024-03-12T03:45:15.958314+0000","last_active":"2024-03-12T03:45:46.461834+0000","last_peered":"2024-03-12T03:45:46.461834+0000","last_clean":"2024-03-12T03:45:46.461834+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:45:46.461834+0000","last_undegraded":"2024-03-12T03:45:46.461834+0000","last_fullsized":"2024-03-12T03:45:46.461834+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'19","last_scrub_stamp":"2024-03-12T03:45:15.957617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.957617+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:41.405966+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.461106+0000","last_change":"2024-03-12T03:43:58.853550+0000","last_active":"2024-03-12T03:45:46.461106+0000","last_peered":"2024-03-12T03:45:46.461106+0000","last_clean":"2024-03-12T03:45:46.461106+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:45:46.461106+0000","last_undegraded":"2024-03-12T03:45:46.461106+0000","last_fullsized":"2024-03-12T03:45:46.461106+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:58.853429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:38:34.264717+0000","last_clean_scrub_stamp":"2024-03-12T03:43:58.853429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674224+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":555,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.450258+0000","last_change":"2024-03-12T03:43:18.347925+0000","last_active":"2024-03-12T03:45:46.450258+0000","last_peered":"2024-03-12T03:45:46.450258+0000","last_clean":"2024-03-12T03:43:13.351418+0000","last_became_active":"2024-03-12T03:43:18.347925+0000","last_became_peered":"2024-03-12T03:43:18.347925+0000","last_unstale":"2024-03-12T03:45:46.450258+0000","last_undegraded":"2024-03-12T03:45:46.450258+0000","last_fullsized":"2024-03-12T03:43:18.334927+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:35.850271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:35.850271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":369,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.453568+0000","last_change":"2024-03-12T03:43:18.350340+0000","last_active":"2024-03-12T03:45:46.453568+0000","last_peered":"2024-03-12T03:45:46.453568+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.350340+0000","last_became_peered":"2024-03-12T03:43:18.350340+0000","last_unstale":"2024-03-12T03:45:46.453568+0000","last_undegraded":"2024-03-12T03:45:46.453568+0000","last_fullsized":"2024-03-12T03:43:18.332506+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:35:59.075897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:35:59.075897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"269'15","reported_seq":151,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454285+0000","last_change":"2024-03-12T03:44:14.462001+0000","last_active":"2024-03-12T03:45:46.454285+0000","last_peered":"2024-03-12T03:45:46.454285+0000","last_clean":"2024-03-12T03:45:46.454285+0000","last_became_active":"2024-03-12T03:43:01.037348+0000","last_became_peered":"2024-03-12T03:43:01.037348+0000","last_unstale":"2024-03-12T03:45:46.454285+0000","last_undegraded":"2024-03-12T03:45:46.454285+0000","last_fullsized":"2024-03-12T03:45:46.454285+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'15","last_scrub_stamp":"2024-03-12T03:44:14.461706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:44:14.461706+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":563,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.451440+0000","last_change":"2024-03-12T03:43:18.347900+0000","last_active":"2024-03-12T03:45:46.451440+0000","last_peered":"2024-03-12T03:45:46.451440+0000","last_clean":"2024-03-12T03:43:13.342510+0000","last_became_active":"2024-03-12T03:43:18.347900+0000","last_became_peered":"2024-03-12T03:43:18.347900+0000","last_unstale":"2024-03-12T03:45:46.451440+0000","last_undegraded":"2024-03-12T03:45:46.451440+0000","last_fullsized":"2024-03-12T03:43:18.337740+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:08.131514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:43:08.131514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.304487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":381,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.454293+0000","last_change":"2024-03-12T03:43:18.343467+0000","last_active":"2024-03-12T03:45:46.454293+0000","last_peered":"2024-03-12T03:45:46.454293+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:43:18.343467+0000","last_became_peered":"2024-03-12T03:43:18.343467+0000","last_unstale":"2024-03-12T03:45:46.454293+0000","last_undegraded":"2024-03-12T03:45:46.454293+0000","last_fullsized":"2024-03-12T03:43:18.332412+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:15.808887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:15.808887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":547,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.452959+0000","last_change":"2024-03-12T03:45:00.842486+0000","last_active":"2024-03-12T03:45:46.452959+0000","last_peered":"2024-03-12T03:45:46.452959+0000","last_clean":"2024-03-12T03:45:46.452959+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:45:46.452959+0000","last_undegraded":"2024-03-12T03:45:46.452959+0000","last_fullsized":"2024-03-12T03:45:46.452959+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:00.842351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:45:00.842351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:09.007022+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":468,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.444799+0000","last_change":"2024-03-12T03:44:53.545462+0000","last_active":"2024-03-12T03:45:46.444799+0000","last_peered":"2024-03-12T03:45:46.444799+0000","last_clean":"2024-03-12T03:45:46.444799+0000","last_became_active":"2024-03-12T03:44:24.387247+0000","last_became_peered":"2024-03-12T03:44:24.387247+0000","last_unstale":"2024-03-12T03:45:46.444799+0000","last_undegraded":"2024-03-12T03:45:46.444799+0000","last_fullsized":"2024-03-12T03:45:46.444799+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:53.545232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:53.545232+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:58.468702+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"269'26","reported_seq":220,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.450335+0000","last_change":"2024-03-12T03:43:18.353670+0000","last_active":"2024-03-12T03:45:46.450335+0000","last_peered":"2024-03-12T03:45:46.450335+0000","last_clean":"2024-03-12T03:33:28.559608+0000","last_became_active":"2024-03-12T03:43:18.353670+0000","last_became_peered":"2024-03-12T03:43:18.353670+0000","last_unstale":"2024-03-12T03:45:46.450335+0000","last_undegraded":"2024-03-12T03:43:18.338840+0000","last_fullsized":"2024-03-12T03:43:18.338735+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:43:06.549441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:06.549441+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.304457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":558,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.455084+0000","last_change":"2024-03-12T03:45:04.074618+0000","last_active":"2024-03-12T03:45:46.455084+0000","last_peered":"2024-03-12T03:45:46.455084+0000","last_clean":"2024-03-12T03:45:46.455084+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:45:46.455084+0000","last_undegraded":"2024-03-12T03:45:46.455084+0000","last_fullsized":"2024-03-12T03:45:46.455084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.074505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.074505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:05.353393+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":551,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.447174+0000","last_change":"2024-03-12T03:44:59.151846+0000","last_active":"2024-03-12T03:45:46.447174+0000","last_peered":"2024-03-12T03:45:46.447174+0000","last_clean":"2024-03-12T03:45:46.447174+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:45:46.447174+0000","last_undegraded":"2024-03-12T03:45:46.447174+0000","last_fullsized":"2024-03-12T03:45:46.447174+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:59.151736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:44:59.151736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:12.516074+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":404,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.453684+0000","last_change":"2024-03-12T03:43:18.344649+0000","last_active":"2024-03-12T03:45:46.453684+0000","last_peered":"2024-03-12T03:45:46.453684+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.344649+0000","last_became_peered":"2024-03-12T03:43:18.344649+0000","last_unstale":"2024-03-12T03:45:46.453684+0000","last_undegraded":"2024-03-12T03:45:46.453684+0000","last_fullsized":"2024-03-12T03:43:18.331783+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"269'10","reported_seq":225,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.448302+0000","last_change":"2024-03-12T03:43:18.337555+0000","last_active":"2024-03-12T03:45:46.448302+0000","last_peered":"2024-03-12T03:45:46.448302+0000","last_clean":"2024-03-12T03:34:34.272872+0000","last_became_active":"2024-03-12T03:43:18.337555+0000","last_became_peered":"2024-03-12T03:43:18.337555+0000","last_unstale":"2024-03-12T03:45:46.448302+0000","last_undegraded":"2024-03-12T03:43:18.333971+0000","last_fullsized":"2024-03-12T03:43:18.333854+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:38:38.943094+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:38:38.943094+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":555,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.454307+0000","last_change":"2024-03-12T03:43:18.349342+0000","last_active":"2024-03-12T03:45:46.454307+0000","last_peered":"2024-03-12T03:45:46.454307+0000","last_clean":"2024-03-12T03:43:13.332505+0000","last_became_active":"2024-03-12T03:43:18.349342+0000","last_became_peered":"2024-03-12T03:43:18.349342+0000","last_unstale":"2024-03-12T03:45:46.454307+0000","last_undegraded":"2024-03-12T03:45:46.454307+0000","last_fullsized":"2024-03-12T03:43:18.335423+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:19.511683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:39:19.511683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":22,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.458352+0000","last_change":"2024-03-12T03:45:18.110285+0000","last_active":"2024-03-12T03:45:46.458352+0000","last_peered":"2024-03-12T03:45:46.458352+0000","last_clean":"2024-03-12T03:45:46.458352+0000","last_became_active":"2024-03-12T03:45:18.109983+0000","last_became_peered":"2024-03-12T03:45:18.109983+0000","last_unstale":"2024-03-12T03:45:46.458352+0000","last_undegraded":"2024-03-12T03:45:46.458352+0000","last_fullsized":"2024-03-12T03:45:46.458352+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:08.466225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:08.466225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:30.385098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.448464+0000","last_change":"2024-03-12T03:45:12.308511+0000","last_active":"2024-03-12T03:45:46.448464+0000","last_peered":"2024-03-12T03:45:46.448464+0000","last_clean":"2024-03-12T03:45:46.448464+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:45:46.448464+0000","last_undegraded":"2024-03-12T03:45:46.448464+0000","last_fullsized":"2024-03-12T03:45:46.448464+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:12.308394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:12.308394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:30.500597+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"269'31","reported_seq":523,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.457736+0000","last_change":"2024-03-12T03:45:18.107459+0000","last_active":"2024-03-12T03:45:46.457736+0000","last_peered":"2024-03-12T03:45:46.457736+0000","last_clean":"2024-03-12T03:45:46.457736+0000","last_became_active":"2024-03-12T03:45:18.107047+0000","last_became_peered":"2024-03-12T03:45:18.107047+0000","last_unstale":"2024-03-12T03:45:46.457736+0000","last_undegraded":"2024-03-12T03:45:46.457736+0000","last_fullsized":"2024-03-12T03:45:46.457736+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:44:57.635788+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:44:57.635788+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:05.239565+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":165,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.453160+0000","last_change":"2024-03-12T03:43:18.342446+0000","last_active":"2024-03-12T03:45:46.453160+0000","last_peered":"2024-03-12T03:45:46.453160+0000","last_clean":"2024-03-12T03:33:21.096389+0000","last_became_active":"2024-03-12T03:43:18.342446+0000","last_became_peered":"2024-03-12T03:43:18.342446+0000","last_unstale":"2024-03-12T03:45:46.453160+0000","last_undegraded":"2024-03-12T03:43:18.333444+0000","last_fullsized":"2024-03-12T03:43:18.333330+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:36:25.092313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:36:25.092313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":556,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454675+0000","last_change":"2024-03-12T03:44:13.704463+0000","last_active":"2024-03-12T03:45:46.454675+0000","last_peered":"2024-03-12T03:45:46.454675+0000","last_clean":"2024-03-12T03:45:46.454675+0000","last_became_active":"2024-03-12T03:39:33.276435+0000","last_became_peered":"2024-03-12T03:39:33.276435+0000","last_unstale":"2024-03-12T03:45:46.454675+0000","last_undegraded":"2024-03-12T03:45:46.454675+0000","last_fullsized":"2024-03-12T03:45:46.454675+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:13.704331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:44:13.704331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":553,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.448968+0000","last_change":"2024-03-12T03:45:01.176270+0000","last_active":"2024-03-12T03:45:46.448968+0000","last_peered":"2024-03-12T03:45:46.448968+0000","last_clean":"2024-03-12T03:45:46.448968+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:45:46.448968+0000","last_undegraded":"2024-03-12T03:45:46.448968+0000","last_fullsized":"2024-03-12T03:45:46.448968+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:01.176150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:01.176150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:26.754979+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"269'27","reported_seq":595,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.454407+0000","last_change":"2024-03-12T03:43:18.358106+0000","last_active":"2024-03-12T03:45:46.454407+0000","last_peered":"2024-03-12T03:45:46.454407+0000","last_clean":"2024-03-12T03:43:13.333148+0000","last_became_active":"2024-03-12T03:43:18.358106+0000","last_became_peered":"2024-03-12T03:43:18.358106+0000","last_unstale":"2024-03-12T03:45:46.454407+0000","last_undegraded":"2024-03-12T03:43:18.343380+0000","last_fullsized":"2024-03-12T03:43:18.343203+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-12T03:39:37.572971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:39:37.572971+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":564,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.447781+0000","last_change":"2024-03-12T03:43:18.350788+0000","last_active":"2024-03-12T03:45:46.447781+0000","last_peered":"2024-03-12T03:45:46.447781+0000","last_clean":"2024-03-12T03:43:13.349345+0000","last_became_active":"2024-03-12T03:43:18.350788+0000","last_became_peered":"2024-03-12T03:43:18.350788+0000","last_unstale":"2024-03-12T03:45:46.447781+0000","last_undegraded":"2024-03-12T03:43:18.332266+0000","last_fullsized":"2024-03-12T03:43:18.332143+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:43:07.772205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:07.772205+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'9","reported_seq":56,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.461749+0000","last_change":"2024-03-12T03:45:18.109442+0000","last_active":"2024-03-12T03:45:46.461749+0000","last_peered":"2024-03-12T03:45:46.461749+0000","last_clean":"2024-03-12T03:45:46.461749+0000","last_became_active":"2024-03-12T03:45:18.108935+0000","last_became_peered":"2024-03-12T03:45:18.108935+0000","last_unstale":"2024-03-12T03:45:46.461749+0000","last_undegraded":"2024-03-12T03:45:46.461749+0000","last_fullsized":"2024-03-12T03:45:46.461749+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:04.572791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.572791+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:18.297679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":567,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.448494+0000","last_change":"2024-03-12T03:43:59.015594+0000","last_active":"2024-03-12T03:45:46.448494+0000","last_peered":"2024-03-12T03:45:46.448494+0000","last_clean":"2024-03-12T03:45:46.448494+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:45:46.448494+0000","last_undegraded":"2024-03-12T03:45:46.448494+0000","last_fullsized":"2024-03-12T03:45:46.448494+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:43:59.015445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:43:59.015445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.305527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"269'24","reported_seq":219,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.448793+0000","last_change":"2024-03-12T03:43:18.346292+0000","last_active":"2024-03-12T03:45:46.448793+0000","last_peered":"2024-03-12T03:45:46.448793+0000","last_clean":"2024-03-12T03:33:28.567700+0000","last_became_active":"2024-03-12T03:43:18.346292+0000","last_became_peered":"2024-03-12T03:43:18.346292+0000","last_unstale":"2024-03-12T03:45:46.448793+0000","last_undegraded":"2024-03-12T03:43:18.333553+0000","last_fullsized":"2024-03-12T03:43:18.333434+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:36:28.133261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:36:28.133261+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":579,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.459943+0000","last_change":"2024-03-12T03:45:08.907717+0000","last_active":"2024-03-12T03:45:46.459943+0000","last_peered":"2024-03-12T03:45:46.459943+0000","last_clean":"2024-03-12T03:45:46.459943+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:45:46.459943+0000","last_undegraded":"2024-03-12T03:45:46.459943+0000","last_fullsized":"2024-03-12T03:45:46.459943+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:08.907581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:08.907581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:15.274744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":566,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.461349+0000","last_change":"2024-03-12T03:43:18.351475+0000","last_active":"2024-03-12T03:45:46.461349+0000","last_peered":"2024-03-12T03:45:46.461349+0000","last_clean":"2024-03-12T03:43:13.338616+0000","last_became_active":"2024-03-12T03:43:18.351475+0000","last_became_peered":"2024-03-12T03:43:18.351475+0000","last_unstale":"2024-03-12T03:45:46.461349+0000","last_undegraded":"2024-03-12T03:45:46.461349+0000","last_fullsized":"2024-03-12T03:43:18.332508+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:08.289718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:41:08.289718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674047+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":304,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.455620+0000","last_change":"2024-03-12T03:45:18.106650+0000","last_active":"2024-03-12T03:45:46.455620+0000","last_peered":"2024-03-12T03:45:46.455620+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:45:18.106650+0000","last_became_peered":"2024-03-12T03:45:18.106650+0000","last_unstale":"2024-03-12T03:45:46.455620+0000","last_undegraded":"2024-03-12T03:45:46.455620+0000","last_fullsized":"2024-03-12T03:45:18.100013+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:16.588551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:16.588551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"269'27","reported_seq":308,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.445350+0000","last_change":"2024-03-12T03:43:18.338861+0000","last_active":"2024-03-12T03:45:46.445350+0000","last_peered":"2024-03-12T03:45:46.445350+0000","last_clean":"2024-03-12T03:43:13.346680+0000","last_became_active":"2024-03-12T03:43:18.338861+0000","last_became_peered":"2024-03-12T03:43:18.338861+0000","last_unstale":"2024-03-12T03:45:46.445350+0000","last_undegraded":"2024-03-12T03:43:18.337001+0000","last_fullsized":"2024-03-12T03:43:18.336879+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-12T03:42:04.171114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:42:04.171114+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:47.790390+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":558,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454589+0000","last_change":"2024-03-12T03:45:03.797399+0000","last_active":"2024-03-12T03:45:46.454589+0000","last_peered":"2024-03-12T03:45:46.454589+0000","last_clean":"2024-03-12T03:45:46.454589+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:45:46.454589+0000","last_undegraded":"2024-03-12T03:45:46.454589+0000","last_fullsized":"2024-03-12T03:45:46.454589+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.797296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.797296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:27.389460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":549,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.448816+0000","last_change":"2024-03-12T03:43:16.842242+0000","last_active":"2024-03-12T03:45:46.448816+0000","last_peered":"2024-03-12T03:45:46.448816+0000","last_clean":"2024-03-12T03:45:46.448816+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:45:46.448816+0000","last_undegraded":"2024-03-12T03:45:46.448816+0000","last_fullsized":"2024-03-12T03:45:46.448816+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:16.842126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:16.842126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100378+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":548,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.461904+0000","last_change":"2024-03-12T03:45:04.926086+0000","last_active":"2024-03-12T03:45:46.461904+0000","last_peered":"2024-03-12T03:45:46.461904+0000","last_clean":"2024-03-12T03:45:46.461904+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:45:46.461904+0000","last_undegraded":"2024-03-12T03:45:46.461904+0000","last_fullsized":"2024-03-12T03:45:46.461904+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.924404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.924404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:13.045572+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"269'16","reported_seq":622,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.461553+0000","last_change":"2024-03-12T03:43:18.345649+0000","last_active":"2024-03-12T03:45:46.461553+0000","last_peered":"2024-03-12T03:45:46.461553+0000","last_clean":"2024-03-12T03:43:13.343489+0000","last_became_active":"2024-03-12T03:43:18.345649+0000","last_became_peered":"2024-03-12T03:43:18.345649+0000","last_unstale":"2024-03-12T03:45:46.461553+0000","last_undegraded":"2024-03-12T03:43:18.330821+0000","last_fullsized":"2024-03-12T03:43:18.330708+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'14","last_scrub_stamp":"2024-03-12T03:43:10.580243+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:43:10.580243+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674073+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":550,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454441+0000","last_change":"2024-03-12T03:44:06.537101+0000","last_active":"2024-03-12T03:45:46.454441+0000","last_peered":"2024-03-12T03:45:46.454441+0000","last_clean":"2024-03-12T03:45:46.454441+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:45:46.454441+0000","last_undegraded":"2024-03-12T03:45:46.454441+0000","last_fullsized":"2024-03-12T03:45:46.454441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:06.536953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:44:06.536953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":550,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.448122+0000","last_change":"2024-03-12T03:45:11.324897+0000","last_active":"2024-03-12T03:45:46.448122+0000","last_peered":"2024-03-12T03:45:46.448122+0000","last_clean":"2024-03-12T03:45:46.448122+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:45:46.448122+0000","last_undegraded":"2024-03-12T03:45:46.448122+0000","last_fullsized":"2024-03-12T03:45:46.448122+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:11.324666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:45:11.324666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:23.575690+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":542,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.455589+0000","last_change":"2024-03-12T03:43:18.348827+0000","last_active":"2024-03-12T03:45:46.455589+0000","last_peered":"2024-03-12T03:45:46.455589+0000","last_clean":"2024-03-12T03:43:13.354323+0000","last_became_active":"2024-03-12T03:43:18.348827+0000","last_became_peered":"2024-03-12T03:43:18.348827+0000","last_unstale":"2024-03-12T03:45:46.455589+0000","last_undegraded":"2024-03-12T03:45:46.455589+0000","last_fullsized":"2024-03-12T03:43:18.333911+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:41:15.648255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:41:15.648255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"269'27","reported_seq":607,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.459704+0000","last_change":"2024-03-12T03:43:18.340176+0000","last_active":"2024-03-12T03:45:46.459704+0000","last_peered":"2024-03-12T03:45:46.459704+0000","last_clean":"2024-03-12T03:43:16.563196+0000","last_became_active":"2024-03-12T03:43:18.340176+0000","last_became_peered":"2024-03-12T03:43:18.340176+0000","last_unstale":"2024-03-12T03:45:46.459704+0000","last_undegraded":"2024-03-12T03:43:18.332765+0000","last_fullsized":"2024-03-12T03:43:18.330467+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:43:16.563042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:16.563042+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:49.674087+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":550,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454727+0000","last_change":"2024-03-12T03:44:13.493021+0000","last_active":"2024-03-12T03:45:46.454727+0000","last_peered":"2024-03-12T03:45:46.454727+0000","last_clean":"2024-03-12T03:45:46.454727+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:45:46.454727+0000","last_undegraded":"2024-03-12T03:45:46.454727+0000","last_fullsized":"2024-03-12T03:45:46.454727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:13.492893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:44:13.492893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":541,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.458199+0000","last_change":"2024-03-12T03:44:16.541523+0000","last_active":"2024-03-12T03:45:46.458199+0000","last_peered":"2024-03-12T03:45:46.458199+0000","last_clean":"2024-03-12T03:45:46.458199+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:45:46.458199+0000","last_undegraded":"2024-03-12T03:45:46.458199+0000","last_fullsized":"2024-03-12T03:45:46.458199+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_clean_scrub_stamp":"2024-03-12T03:44:16.541173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":539,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.453113+0000","last_change":"2024-03-12T03:44:10.763845+0000","last_active":"2024-03-12T03:45:46.453113+0000","last_peered":"2024-03-12T03:45:46.453113+0000","last_clean":"2024-03-12T03:45:46.453113+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:45:46.453113+0000","last_undegraded":"2024-03-12T03:45:46.453113+0000","last_fullsized":"2024-03-12T03:45:46.453113+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:10.763492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:10.763492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"269'62","reported_seq":565,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:45:46.444360+0000","last_change":"2024-03-12T03:43:18.343462+0000","last_active":"2024-03-12T03:45:46.444360+0000","last_peered":"2024-03-12T03:45:46.444360+0000","last_clean":"2024-03-12T03:39:28.242684+0000","last_became_active":"2024-03-12T03:43:18.343462+0000","last_became_peered":"2024-03-12T03:43:18.343462+0000","last_unstale":"2024-03-12T03:45:46.444360+0000","last_undegraded":"2024-03-12T03:43:18.336907+0000","last_fullsized":"2024-03-12T03:43:18.336787+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:47.790422+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":358,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.445058+0000","last_change":"2024-03-12T03:45:18.105152+0000","last_active":"2024-03-12T03:45:46.445058+0000","last_peered":"2024-03-12T03:45:46.445058+0000","last_clean":"2024-03-12T03:39:28.243893+0000","last_became_active":"2024-03-12T03:45:18.105152+0000","last_became_peered":"2024-03-12T03:45:18.105152+0000","last_unstale":"2024-03-12T03:45:46.445058+0000","last_undegraded":"2024-03-12T03:45:46.445058+0000","last_fullsized":"2024-03-12T03:45:18.100708+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:26.898782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:39:26.898782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:47.790380+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":551,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.458186+0000","last_change":"2024-03-12T03:44:58.765175+0000","last_active":"2024-03-12T03:45:46.458186+0000","last_peered":"2024-03-12T03:45:46.458186+0000","last_clean":"2024-03-12T03:45:46.458186+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:45:46.458186+0000","last_undegraded":"2024-03-12T03:45:46.458186+0000","last_fullsized":"2024-03-12T03:45:46.458186+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:58.765061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:44:58.765061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:08.411798+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":375,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-12T03:45:46.450612+0000","last_change":"2024-03-12T03:43:18.353114+0000","last_active":"2024-03-12T03:45:46.450612+0000","last_peered":"2024-03-12T03:45:46.450612+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:43:18.353114+0000","last_became_peered":"2024-03-12T03:43:18.353114+0000","last_unstale":"2024-03-12T03:45:46.450612+0000","last_undegraded":"2024-03-12T03:45:46.450612+0000","last_fullsized":"2024-03-12T03:43:18.337255+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:36:19.236997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:36:19.236997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.303854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"269'26","reported_seq":597,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.458093+0000","last_change":"2024-03-12T03:45:10.855287+0000","last_active":"2024-03-12T03:45:46.458093+0000","last_peered":"2024-03-12T03:45:46.458093+0000","last_clean":"2024-03-12T03:45:46.458093+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:45:46.458093+0000","last_undegraded":"2024-03-12T03:45:46.458093+0000","last_fullsized":"2024-03-12T03:45:46.458093+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:10.855156+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.855156+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:36.005696+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":564,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.454197+0000","last_change":"2024-03-12T03:44:14.713075+0000","last_active":"2024-03-12T03:45:46.454197+0000","last_peered":"2024-03-12T03:45:46.454197+0000","last_clean":"2024-03-12T03:45:46.454197+0000","last_became_active":"2024-03-12T03:39:33.276119+0000","last_became_peered":"2024-03-12T03:39:33.276119+0000","last_unstale":"2024-03-12T03:45:46.454197+0000","last_undegraded":"2024-03-12T03:45:46.454197+0000","last_fullsized":"2024-03-12T03:45:46.454197+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:44:14.712916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.447879+0000","last_change":"2024-03-12T03:43:54.998904+0000","last_active":"2024-03-12T03:45:46.447879+0000","last_peered":"2024-03-12T03:45:46.447879+0000","last_clean":"2024-03-12T03:45:46.447879+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:45:46.447879+0000","last_undegraded":"2024-03-12T03:45:46.447879+0000","last_fullsized":"2024-03-12T03:45:46.447879+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:54.998817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:26.625326+0000","last_clean_scrub_stamp":"2024-03-12T03:43:54.998817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:45:51.100593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":554,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-12T03:45:46.459348+0000","last_change":"2024-03-12T03:45:14.952828+0000","last_active":"2024-03-12T03:45:46.459348+0000","last_peered":"2024-03-12T03:45:46.459348+0000","last_clean":"2024-03-12T03:45:46.459348+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:45:46.459348+0000","last_undegraded":"2024-03-12T03:45:46.459348+0000","last_fullsized":"2024-03-12T03:45:46.459348+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_clean_scrub_stamp":"2024-03-12T03:45:14.952550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:46:41.194144+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7409106968,"num_objects":113055,"num_object_clones":0,"num_object_copies":339165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":56781,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113055,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175371,"num_write_kb":9229569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18567213056,"data_stored":18567200840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73984,"ondisk_log_size":73984,"up":40,"acting":40,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":743,"num_read_kb":738,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":661928,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":36,"num_write":21,"num_write_kb":293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":54,"acting":58,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":84,"acting":84,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1406,"num_read_kb":1131,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":2284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3734,"internal_metadata":0},"log_size":945,"ondisk_log_size":945,"up":79,"acting":79,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070619,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4706968,"kb_used_data":4071824,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":89038184,"statfs":{"total":95995035648,"available":91175100416,"internally_reserved":0,"allocated":4169547776,"data_stored":4168811463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":650312138},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332256,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1799932,"kb_used_data":1359220,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":91945220,"statfs":{"total":95995035648,"available":94151905280,"internally_reserved":0,"allocated":1391841280,"data_stored":1391131652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":451279302},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4906416,"kb_used_data":4257316,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":88838736,"statfs":{"total":95995035648,"available":90970865664,"internally_reserved":0,"allocated":4359491584,"data_stored":4358734568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738558,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2283924,"kb_used_data":1797388,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91461228,"statfs":{"total":95995035648,"available":93656297472,"internally_reserved":0,"allocated":1840525312,"data_stored":1839792325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411507,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1850172,"kb_used_data":1359476,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":91894980,"statfs":{"total":95995035648,"available":94100459520,"internally_reserved":0,"allocated":1392103424,"data_stored":1391395353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":222,"seq":953482739743,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":471640,"kb_used_data":158088,"kb_used_omap":7,"kb_used_meta":313528,"kb_avail":93273512,"statfs":{"total":95995035648,"available":95512076288,"internally_reserved":0,"allocated":161882112,"data_stored":161425991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8180,"internal_metadata":321052684},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953643,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1720920,"kb_used_data":1350992,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92024232,"statfs":{"total":95995035648,"available":94232813568,"internally_reserved":0,"allocated":1383415808,"data_stored":1382679011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659301,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4715096,"kb_used_data":4084112,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":89030056,"statfs":{"total":95995035648,"available":91166777344,"internally_reserved":0,"allocated":4182130688,"data_stored":4181393737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3734,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1823408128,"data_stored":1823408128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4350611456,"data_stored":4350607384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1381695488,"data_stored":1381695488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1375010816,"data_stored":1375010816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4135718912,"data_stored":4135714840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1374420992,"data_stored":1374420992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4126347264,"data_stored":4126343192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:45:55.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool set unique_pool_2 pgp_num 16 2024-03-12T03:45:55.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:55.449 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:55.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:55.517 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:45:55.539 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:55.551 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:55.595 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:55.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:55.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:55 smithi169 ceph-mon[39949]: from='client.25286 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:45:55.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:55 smithi169 ceph-mon[39949]: pgmap v596: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s; 56031/335934 objects degraded (16.679%); 1/335934 objects misplaced (0.000%) 2024-03-12T03:45:55.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:55 smithi169 ceph-mon[39949]: 9.5 scrub ok 2024-03-12T03:45:55.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:55 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4285620721' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:55.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:55 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:55.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:55 smithi169 ceph-mon[29344]: from='client.25286 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:45:55.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:55 smithi169 ceph-mon[29344]: pgmap v596: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s; 56031/335934 objects degraded (16.679%); 1/335934 objects misplaced (0.000%) 2024-03-12T03:45:55.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:55 smithi169 ceph-mon[29344]: 9.5 scrub ok 2024-03-12T03:45:55.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:55 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4285620721' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:55.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:55 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:55.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:55.721 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:55.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:55.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:55.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:55.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:55.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:55 smithi173 ceph-mon[33544]: from='client.25286 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:45:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:55 smithi173 ceph-mon[33544]: pgmap v596: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s; 56031/335934 objects degraded (16.679%); 1/335934 objects misplaced (0.000%) 2024-03-12T03:45:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:55 smithi173 ceph-mon[33544]: 9.5 scrub ok 2024-03-12T03:45:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:55 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4285620721' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:55 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:45:55.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:55.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:56.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:56.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:56.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:56.232 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:56.275 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:56.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:56.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:56.410 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:56.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:56.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:56.544 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:56.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:56.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:56.670 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:56.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:56.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:56.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:56 smithi169 ceph-mon[29344]: 9.e scrub ok 2024-03-12T03:45:56.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:56 smithi169 ceph-mon[29344]: 6.9 scrub starts 2024-03-12T03:45:56.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:56 smithi169 ceph-mon[29344]: 6.9 scrub ok 2024-03-12T03:45:56.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:56 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:45:56.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:56 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:45:56.795 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:56 smithi169 ceph-mon[29344]: osdmap e320: 8 total, 7 up, 8 in 2024-03-12T03:45:56.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:56 smithi169 ceph-mon[39949]: 9.e scrub ok 2024-03-12T03:45:56.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:56 smithi169 ceph-mon[39949]: 6.9 scrub starts 2024-03-12T03:45:56.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:56 smithi169 ceph-mon[39949]: 6.9 scrub ok 2024-03-12T03:45:56.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:56 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:45:56.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:56 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:45:56.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:56 smithi169 ceph-mon[39949]: osdmap e320: 8 total, 7 up, 8 in 2024-03-12T03:45:56.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:56.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:56.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:56.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:56.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:56.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:56 smithi173 ceph-mon[33544]: 9.e scrub ok 2024-03-12T03:45:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:56 smithi173 ceph-mon[33544]: 6.9 scrub starts 2024-03-12T03:45:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:56 smithi173 ceph-mon[33544]: 6.9 scrub ok 2024-03-12T03:45:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:56 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:45:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:56 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:45:56.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:56 smithi173 ceph-mon[33544]: osdmap e320: 8 total, 7 up, 8 in 2024-03-12T03:45:57.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:57.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:57.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:57.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:45:57.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:57.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:45:57.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:57.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:57.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:57.439 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:57.465 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:57.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:57.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:57.499 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:45:57.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:57.605 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:57.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:57.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:57.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:57.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:57.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:57.811 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:57 smithi169 ceph-mon[29344]: pgmap v598: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 533 B/s rd, 0 B/s wr, 1.03k op/s; 56031/335934 objects degraded (16.679%); 1/335934 objects misplaced (0.000%) 2024-03-12T03:45:57.812 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:57 smithi169 ceph-mon[29344]: 3.6 deep-scrub starts 2024-03-12T03:45:57.812 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:57 smithi169 ceph-mon[29344]: 3.6 deep-scrub ok 2024-03-12T03:45:57.812 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:57 smithi169 ceph-mon[29344]: 3.1d scrub starts 2024-03-12T03:45:57.812 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:57 smithi169 ceph-mon[29344]: 3.1d scrub ok 2024-03-12T03:45:57.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:57 smithi169 ceph-mon[39949]: pgmap v598: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 533 B/s rd, 0 B/s wr, 1.03k op/s; 56031/335934 objects degraded (16.679%); 1/335934 objects misplaced (0.000%) 2024-03-12T03:45:57.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:57 smithi169 ceph-mon[39949]: 3.6 deep-scrub starts 2024-03-12T03:45:57.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:57 smithi169 ceph-mon[39949]: 3.6 deep-scrub ok 2024-03-12T03:45:57.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:57 smithi169 ceph-mon[39949]: 3.1d scrub starts 2024-03-12T03:45:57.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:57 smithi169 ceph-mon[39949]: 3.1d scrub ok 2024-03-12T03:45:57.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:57.877 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:57.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:57.943 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:57 smithi173 ceph-mon[33544]: pgmap v598: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 533 B/s rd, 0 B/s wr, 1.03k op/s; 56031/335934 objects degraded (16.679%); 1/335934 objects misplaced (0.000%) 2024-03-12T03:45:57.943 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:57 smithi173 ceph-mon[33544]: 3.6 deep-scrub starts 2024-03-12T03:45:57.943 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:57 smithi173 ceph-mon[33544]: 3.6 deep-scrub ok 2024-03-12T03:45:57.944 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:57 smithi173 ceph-mon[33544]: 3.1d scrub starts 2024-03-12T03:45:57.944 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:57 smithi173 ceph-mon[33544]: 3.1d scrub ok 2024-03-12T03:45:57.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:58.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:58.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:58.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:58.158 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:58.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:58.248 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:45:58.260 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:58.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:58.334 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:58.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:58.414 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:45:58.414 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:45:58.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:45:58.473 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:58.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:58.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:45:58.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:58.611 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:58.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:58.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:58.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:58 smithi169 ceph-mon[39949]: 3.e scrub starts 2024-03-12T03:45:58.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:58 smithi169 ceph-mon[39949]: 3.e scrub ok 2024-03-12T03:45:58.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:58 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 56031/335934 objects degraded (16.679%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:58.628 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:58 smithi169 ceph-mon[39949]: osdmap e321: 8 total, 7 up, 8 in 2024-03-12T03:45:58.628 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:58 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2511519703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:58.628 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:58 smithi169 ceph-mon[29344]: 3.e scrub starts 2024-03-12T03:45:58.628 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:58 smithi169 ceph-mon[29344]: 3.e scrub ok 2024-03-12T03:45:58.628 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:58 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 56031/335934 objects degraded (16.679%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:58.628 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:58 smithi169 ceph-mon[29344]: osdmap e321: 8 total, 7 up, 8 in 2024-03-12T03:45:58.628 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:58 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2511519703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:58.713 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:58.728 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:45:58.728 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:45:58.728 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:45:58.728 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:45:58.728 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:45:58.728 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:45:58.728 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:45:58.729 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:45:58.729 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:45:58.729 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:45:58.729 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:45:58.729 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:45:58.729 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:45:58.729 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:45:58.729 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:45:58.753 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:58.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:58.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:58.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:58.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:58.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:58.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:58.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:58 smithi173 ceph-mon[33544]: 3.e scrub starts 2024-03-12T03:45:58.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:58 smithi173 ceph-mon[33544]: 3.e scrub ok 2024-03-12T03:45:58.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:58 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 56031/335934 objects degraded (16.679%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:45:58.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:58 smithi173 ceph-mon[33544]: osdmap e321: 8 total, 7 up, 8 in 2024-03-12T03:45:58.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:58 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2511519703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:45:58.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:45:59.027 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:59.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:45:59.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-12T03:45:59.128 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:59.158 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:59.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:59.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:59.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:45:59.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:59.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:59.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:45:59.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:45:59.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:59.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:59.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:45:59.513 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:45:59.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:59.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:59.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:45:59.740 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:45:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:59.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:45:59.797 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:59 smithi173 ceph-mon[33544]: pgmap v599: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 53056/317853 objects degraded (16.692%); 1/317853 objects misplaced (0.000%) 2024-03-12T03:45:59.797 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:59 smithi173 ceph-mon[33544]: 6.18 deep-scrub starts 2024-03-12T03:45:59.797 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:59 smithi173 ceph-mon[33544]: 6.18 deep-scrub ok 2024-03-12T03:45:59.798 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:59 smithi173 ceph-mon[33544]: 1.14 scrub starts 2024-03-12T03:45:59.798 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:45:59 smithi173 ceph-mon[33544]: 1.14 scrub ok 2024-03-12T03:45:59.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:59 smithi169 ceph-mon[29344]: pgmap v599: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 53056/317853 objects degraded (16.692%); 1/317853 objects misplaced (0.000%) 2024-03-12T03:45:59.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:59 smithi169 ceph-mon[29344]: 6.18 deep-scrub starts 2024-03-12T03:45:59.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:59 smithi169 ceph-mon[29344]: 6.18 deep-scrub ok 2024-03-12T03:45:59.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:59 smithi169 ceph-mon[29344]: 1.14 scrub starts 2024-03-12T03:45:59.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:45:59 smithi169 ceph-mon[29344]: 1.14 scrub ok 2024-03-12T03:45:59.823 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:59 smithi169 ceph-mon[39949]: pgmap v599: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 53056/317853 objects degraded (16.692%); 1/317853 objects misplaced (0.000%) 2024-03-12T03:45:59.823 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:59 smithi169 ceph-mon[39949]: 6.18 deep-scrub starts 2024-03-12T03:45:59.823 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:59 smithi169 ceph-mon[39949]: 6.18 deep-scrub ok 2024-03-12T03:45:59.823 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:59 smithi169 ceph-mon[39949]: 1.14 scrub starts 2024-03-12T03:45:59.823 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:45:59 smithi169 ceph-mon[39949]: 1.14 scrub ok 2024-03-12T03:45:59.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:45:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:45:59.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:45:59.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:45:59.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:00.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:00.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:00.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:00.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:00.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:00.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:00.245 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:00.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:00.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:00.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:00.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:00.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:00.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:00.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:00.570 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:00.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:00.661 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:00 smithi173 ceph-mon[33544]: 4.3 scrub starts 2024-03-12T03:46:00.661 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:00 smithi173 ceph-mon[33544]: 4.3 scrub ok 2024-03-12T03:46:00.661 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:00 smithi173 ceph-mon[33544]: 6.a scrub starts 2024-03-12T03:46:00.661 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:00 smithi173 ceph-mon[33544]: 6.a scrub ok 2024-03-12T03:46:00.661 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:00 smithi173 ceph-mon[33544]: 4.e scrub starts 2024-03-12T03:46:00.661 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:00 smithi173 ceph-mon[33544]: 4.e scrub ok 2024-03-12T03:46:00.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:00 smithi173 ceph-mon[33544]: 4.7 scrub starts 2024-03-12T03:46:00.662 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:00 smithi173 ceph-mon[33544]: 4.7 scrub ok 2024-03-12T03:46:00.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:00.689 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:00.710 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:46:00.710 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:46:00.712 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:00.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:00.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[29344]: 4.3 scrub starts 2024-03-12T03:46:00.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[29344]: 4.3 scrub ok 2024-03-12T03:46:00.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[29344]: 6.a scrub starts 2024-03-12T03:46:00.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[29344]: 6.a scrub ok 2024-03-12T03:46:00.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[29344]: 4.e scrub starts 2024-03-12T03:46:00.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[29344]: 4.e scrub ok 2024-03-12T03:46:00.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[29344]: 4.7 scrub starts 2024-03-12T03:46:00.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[29344]: 4.7 scrub ok 2024-03-12T03:46:00.810 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[39949]: 4.3 scrub starts 2024-03-12T03:46:00.810 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[39949]: 4.3 scrub ok 2024-03-12T03:46:00.810 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[39949]: 6.a scrub starts 2024-03-12T03:46:00.810 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[39949]: 6.a scrub ok 2024-03-12T03:46:00.810 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[39949]: 4.e scrub starts 2024-03-12T03:46:00.810 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[39949]: 4.e scrub ok 2024-03-12T03:46:00.810 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[39949]: 4.7 scrub starts 2024-03-12T03:46:00.810 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:00 smithi169 ceph-mon[39949]: 4.7 scrub ok 2024-03-12T03:46:00.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:00.867 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:00.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:00.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:00.968 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:01.008 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:01.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:01.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:01.110 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:01.143 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:01.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:01.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:01.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:01.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:01.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:01.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:01.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:01.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:01.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:01.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:01.570 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:46:01.570 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:46:01.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.571 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:189378 9.7 9:e5cd62f4:::benchmark_data_smithi169_135229_object45163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-03-12T03:46:01.571 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:46:01.546836+0000", 2024-03-12T03:46:01.571 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.019336261, 2024-03-12T03:46:01.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.019372059000000001, 2024-03-12T03:46:01.571 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:01.571 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:46:01.572 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:01.572 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:46:01.572 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:46:01.572 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 189378 2024-03-12T03:46:01.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.572 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:01.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:01.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.546836+0000", 2024-03-12T03:46:01.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:01.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:01.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.546836+0000", 2024-03-12T03:46:01.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:01.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:01.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.546837+0000", 2024-03-12T03:46:01.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6199999999999998e-07 2024-03-12T03:46:01.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:01.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.546840+0000", 2024-03-12T03:46:01.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.038e-06 2024-03-12T03:46:01.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.546846+0000", 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8510000000000004e-06 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.546851+0000", 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0000000000000004e-06 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.552464+0000", 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0056130349999999997 2024-03-12T03:46:01.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.552590+0000", 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000126749 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7", 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.552712+0000", 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121196 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.557166+0000", 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.004454698 2024-03-12T03:46:01.576 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15279.0:189383 9.b e321/278 9:de0d08fe:::benchmark_data_smithi169_135229_object45168:head v 321'11743, mlcod=321'11741)", 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:46:01.559586+0000", 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0065865519999999999, 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0067164410000000001, 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:01.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.559586+0000", 2024-03-12T03:46:01.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:01.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:01.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.559586+0000", 2024-03-12T03:46:01.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:01.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:01.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.559587+0000", 2024-03-12T03:46:01.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1800000000000004e-07 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.559599+0000", 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.208e-05 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.559610+0000", 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1044e-05 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.559619+0000", 2024-03-12T03:46:01.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7139999999999999e-06 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.559696+0000", 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7142000000000001e-05 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.559714+0000", 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8380999999999999e-05 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.580 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15279.0:189385 9.b e321/278 9:d3eaeffa:::benchmark_data_smithi169_135229_object45170:head v 321'11744, mlcod=321'11741)", 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:46:01.560581+0000", 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.005591258, 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0057869369999999998, 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.560581+0000", 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.560581+0000", 2024-03-12T03:46:01.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.560582+0000", 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7899999999999998e-07 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.560586+0000", 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6700000000000002e-06 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:01.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.560594+0000", 2024-03-12T03:46:01.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7970000000000001e-06 2024-03-12T03:46:01.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:01.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.560600+0000", 2024-03-12T03:46:01.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0399999999999998e-06 2024-03-12T03:46:01.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:01.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.562463+0000", 2024-03-12T03:46:01.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001862554 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.562480+0000", 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7181e-05 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:189388 9.f 9:faf24612:::benchmark_data_smithi169_135229_object45173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:46:01.562162+0000", 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0040105490000000004, 2024-03-12T03:46:01.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042669029999999998, 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 189388 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.562162+0000", 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:01.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.562162+0000", 2024-03-12T03:46:01.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:01.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:01.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.562163+0000", 2024-03-12T03:46:01.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.468e-06 2024-03-12T03:46:01.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:01.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.562169+0000", 2024-03-12T03:46:01.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0050000000000001e-06 2024-03-12T03:46:01.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:01.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.562184+0000", 2024-03-12T03:46:01.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4951e-05 2024-03-12T03:46:01.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:01.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.562192+0000", 2024-03-12T03:46:01.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9759999999999995e-06 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.562722+0000", 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00052930700000000002 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.562842+0000", 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120643 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1", 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.562981+0000", 2024-03-12T03:46:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013903200000000001 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.566095+0000", 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00311341 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15279.0:189387 9.b e321/278)", 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:46:01.563917+0000", 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0022557990000000002, 2024-03-12T03:46:01.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025883989999999999, 2024-03-12T03:46:01.590 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:01.590 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-03-12T03:46:01.590 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:01.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:01.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.563917+0000", 2024-03-12T03:46:01.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:01.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:01.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.563917+0000", 2024-03-12T03:46:01.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:01.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:01.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.563917+0000", 2024-03-12T03:46:01.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6899999999999997e-07 2024-03-12T03:46:01.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:01.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.563922+0000", 2024-03-12T03:46:01.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0429999999999998e-06 2024-03-12T03:46:01.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:01.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.563934+0000", 2024-03-12T03:46:01.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1311999999999999e-05 2024-03-12T03:46:01.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:01.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:01.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:01.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:01.563940+0000", 2024-03-12T03:46:01.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4960000000000001e-06 2024-03-12T03:46:01.592 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:01.592 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:01.593 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:01.593 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:01.593 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-03-12T03:46:01.593 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 5 2024-03-12T03:46:01.593 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:46:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:01.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:01.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:01.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-mon[29344]: pgmap v601: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 916 op/s; 53056/317853 objects degraded (16.692%); 1/317853 objects misplaced (0.000%) 2024-03-12T03:46:01.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-mon[29344]: 5.7 scrub starts 2024-03-12T03:46:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-mon[29344]: 5.7 scrub ok 2024-03-12T03:46:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-mon[29344]: 9.0 deep-scrub starts 2024-03-12T03:46:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:01.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-mon[39949]: pgmap v601: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 916 op/s; 53056/317853 objects degraded (16.692%); 1/317853 objects misplaced (0.000%) 2024-03-12T03:46:01.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-mon[39949]: 5.7 scrub starts 2024-03-12T03:46:01.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-mon[39949]: 5.7 scrub ok 2024-03-12T03:46:01.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-mon[39949]: 9.0 deep-scrub starts 2024-03-12T03:46:01.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:01.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:01.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:01.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:01.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:01.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:01.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:01.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:01.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:01.917 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:01 smithi173 ceph-mon[33544]: pgmap v601: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 916 op/s; 53056/317853 objects degraded (16.692%); 1/317853 objects misplaced (0.000%) 2024-03-12T03:46:01.917 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:01 smithi173 ceph-mon[33544]: 5.7 scrub starts 2024-03-12T03:46:01.917 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:01 smithi173 ceph-mon[33544]: 5.7 scrub ok 2024-03-12T03:46:01.917 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:01 smithi173 ceph-mon[33544]: 9.0 deep-scrub starts 2024-03-12T03:46:01.917 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:01 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:01.918 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:01 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:01.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:02.008 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:02.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:02.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:02.140 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:02.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:02.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:46:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:46:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:46:02.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:02.275 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:02.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:02.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:02.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-12T03:46:02.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:02.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:02.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:02.561 INFO:teuthology.orchestra.run.smithi169.stderr:set pool 9 pgp_num to 16 2024-03-12T03:46:02.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:02.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:02.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:02.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:02.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:46:02.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:02.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:02.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:02 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3591476458' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-12T03:46:02.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:02.848 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:46:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:02.858 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:02.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:02.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:02 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3591476458' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-12T03:46:02.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:02 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3591476458' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-12T03:46:02.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:03.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:03.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:03.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:03.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:03.143 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:03.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:03.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:03.221 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:46:02 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:46:03.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:03.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:03.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:03.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:03.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:03.414 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:03.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:03.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:03.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:03.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:03.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:03.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:03.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:03.689 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:03.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:03.778 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[29344]: pgmap v602: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 50531/301323 objects degraded (16.770%); 1/301323 objects misplaced (0.000%) 2024-03-12T03:46:03.778 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3591476458' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-03-12T03:46:03.778 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[29344]: osdmap e322: 8 total, 7 up, 8 in 2024-03-12T03:46:03.778 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[29344]: 5.1b scrub starts 2024-03-12T03:46:03.779 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[29344]: 5.1b scrub ok 2024-03-12T03:46:03.779 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 50531/301323 objects degraded (16.770%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:03.779 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[29344]: osdmap e323: 8 total, 7 up, 8 in 2024-03-12T03:46:03.779 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[39949]: pgmap v602: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 50531/301323 objects degraded (16.770%); 1/301323 objects misplaced (0.000%) 2024-03-12T03:46:03.779 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3591476458' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-03-12T03:46:03.779 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[39949]: osdmap e322: 8 total, 7 up, 8 in 2024-03-12T03:46:03.779 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[39949]: 5.1b scrub starts 2024-03-12T03:46:03.779 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[39949]: 5.1b scrub ok 2024-03-12T03:46:03.780 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 50531/301323 objects degraded (16.770%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:03.780 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:03 smithi169 ceph-mon[39949]: osdmap e323: 8 total, 7 up, 8 in 2024-03-12T03:46:03.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:03.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:03.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:03.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:03.876 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:03.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:03.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:03.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:03.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:03 smithi173 ceph-mon[33544]: pgmap v602: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 50531/301323 objects degraded (16.770%); 1/301323 objects misplaced (0.000%) 2024-03-12T03:46:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:03 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3591476458' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-03-12T03:46:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:03 smithi173 ceph-mon[33544]: osdmap e322: 8 total, 7 up, 8 in 2024-03-12T03:46:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:03 smithi173 ceph-mon[33544]: 5.1b scrub starts 2024-03-12T03:46:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:03 smithi173 ceph-mon[33544]: 5.1b scrub ok 2024-03-12T03:46:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:03 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 50531/301323 objects degraded (16.770%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:03 smithi173 ceph-mon[33544]: osdmap e323: 8 total, 7 up, 8 in 2024-03-12T03:46:04.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:04.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:04.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:04.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:04.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:04.224 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:04.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:04.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:04.370 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:04.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:04.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:04.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:04.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:04.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:04.641 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:46:04.642 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:04.642 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:04 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:46:04.642 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:04 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:46:04.642 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:04 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4091362737' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:46:04.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:04.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:04 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:46:04.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:04 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:46:04.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:04 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4091362737' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:46:04.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:04.646 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:04.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:04.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:04.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:04.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:04.882 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:04.912 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:04.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:04 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:46:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:04 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:46:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:04 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4091362737' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:46:05.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:05.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:05.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:05.182 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:05.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:05.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:05.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:05.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:05.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:05.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:05.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:05.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:05.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:05.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:05.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:05.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:05.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:05 smithi169 ceph-mon[29344]: pgmap v605: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.12k op/s; 49723/297735 objects degraded (16.700%); 1/297735 objects misplaced (0.000%) 2024-03-12T03:46:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:05 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:46:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:05 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4091362737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:46:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:05 smithi169 ceph-mon[29344]: 2.2 scrub starts 2024-03-12T03:46:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:05 smithi169 ceph-mon[29344]: osdmap e324: 8 total, 7 up, 8 in 2024-03-12T03:46:05.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:05 smithi169 ceph-mon[29344]: 2.2 scrub ok 2024-03-12T03:46:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:05 smithi169 ceph-mon[39949]: pgmap v605: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.12k op/s; 49723/297735 objects degraded (16.700%); 1/297735 objects misplaced (0.000%) 2024-03-12T03:46:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:05 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:46:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:05 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4091362737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:46:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:05 smithi169 ceph-mon[39949]: 2.2 scrub starts 2024-03-12T03:46:05.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:05 smithi169 ceph-mon[39949]: osdmap e324: 8 total, 7 up, 8 in 2024-03-12T03:46:05.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:05 smithi169 ceph-mon[39949]: 2.2 scrub ok 2024-03-12T03:46:05.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:05.756 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:05.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:05.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:05.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:05.875 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:05 smithi173 ceph-mon[33544]: pgmap v605: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.12k op/s; 49723/297735 objects degraded (16.700%); 1/297735 objects misplaced (0.000%) 2024-03-12T03:46:05.875 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:05 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:46:05.875 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:05 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4091362737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:46:05.875 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:05 smithi173 ceph-mon[33544]: 2.2 scrub starts 2024-03-12T03:46:05.876 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:05 smithi173 ceph-mon[33544]: osdmap e324: 8 total, 7 up, 8 in 2024-03-12T03:46:05.876 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:05 smithi173 ceph-mon[33544]: 2.2 scrub ok 2024-03-12T03:46:05.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:05.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:05.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:05.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:06.028 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:06.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:06.069 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:46:06.070 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:46:06.070 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:46:06.070 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:46:06.070 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:46:06.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:06.166 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:06.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:06.268 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:06.300 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:06.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:06.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:06.434 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:06.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:06.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:06.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:06.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:06.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:06.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:06.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:06.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:06.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:06.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:06.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:06.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:06.980 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:06.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:06.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:07.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:07.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:07.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:07.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:07.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:07.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-12T03:46:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:07.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:07.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:07.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:07.383 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:07.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:07.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:07.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:07.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:07.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:07.641 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:46:07.664 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:07.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:07.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:07.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:07.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:07.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:07.823 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:07 smithi173 ceph-mon[33544]: pgmap v607: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.12k op/s; 49723/297735 objects degraded (16.700%); 1/297735 objects misplaced (0.000%) 2024-03-12T03:46:07.867 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:46:07.900 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:07 smithi169 ceph-mon[39949]: pgmap v607: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.12k op/s; 49723/297735 objects degraded (16.700%); 1/297735 objects misplaced (0.000%) 2024-03-12T03:46:07.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:07 smithi169 ceph-mon[29344]: pgmap v607: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.12k op/s; 49723/297735 objects degraded (16.700%); 1/297735 objects misplaced (0.000%) 2024-03-12T03:46:07.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:07.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:07.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:08.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:08.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:08.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:08.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:08.213 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:08.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:08.272 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-12T03:46:08.272 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-03-12T03:46:08.272 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-03-12T03:46:08.272 DEBUG:teuthology.orchestra.run.smithi169:osd.3> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3.service 2024-03-12T03:46:08.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl start ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3 2024-03-12T03:46:08.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:08.353 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:08.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:08.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:08.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:08.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:08.568 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:08 smithi169 systemd[1]: Starting Ceph osd.3 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:46:08.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:08.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:08.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:08.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:08.764 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:08.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:08.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:08 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 49723/297735 objects degraded (16.700%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:08.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:08 smithi173 ceph-mon[33544]: osdmap e325: 8 total, 7 up, 8 in 2024-03-12T03:46:08.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:08.899 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:08.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:08 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 49723/297735 objects degraded (16.700%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:08 smithi169 ceph-mon[29344]: osdmap e325: 8 total, 7 up, 8 in 2024-03-12T03:46:08.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:08 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 49723/297735 objects degraded (16.700%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:08.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:08 smithi169 ceph-mon[39949]: osdmap e325: 8 total, 7 up, 8 in 2024-03-12T03:46:09.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:09.047 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:09.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:09.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:09.181 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:09.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:09.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:09.313 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:09.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:09.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:09.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:09.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:09.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:09.574 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:46:09.574 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:46:09.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:46:09.575 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:46:09.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.575 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:107734 7.10 7:09752c19:::benchmark_data_smithi169_79822_object107733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-12T03:46:09.575 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:36:48.406825+0000", 2024-03-12T03:46:09.575 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 561.15946974099995, 2024-03-12T03:46:09.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90505219699999995, 2024-03-12T03:46:09.575 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.575 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.576 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.576 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:09.576 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:09.576 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 107734 2024-03-12T03:46:09.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.576 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406825+0000", 2024-03-12T03:46:09.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406825+0000", 2024-03-12T03:46:09.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406827+0000", 2024-03-12T03:46:09.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6139999999999999e-06 2024-03-12T03:46:09.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406856+0000", 2024-03-12T03:46:09.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9417999999999999e-05 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406870+0000", 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3414999999999999e-05 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406881+0000", 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1232e-05 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406940+0000", 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8978999999999999e-05 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:48.406951+0000", 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0831e-05 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.301975+0000", 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89502379700000001 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.302104+0000", 2024-03-12T03:46:09.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012923399999999999 2024-03-12T03:46:09.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:46:09.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.302398+0000", 2024-03-12T03:46:09.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029388400000000001 2024-03-12T03:46:09.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.303969+0000", 2024-03-12T03:46:09.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001571476 2024-03-12T03:46:09.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.309180+0000", 2024-03-12T03:46:09.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0052102030000000001 2024-03-12T03:46:09.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.311767+0000", 2024-03-12T03:46:09.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025875160000000002 2024-03-12T03:46:09.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.311830+0000", 2024-03-12T03:46:09.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2638000000000006e-05 2024-03-12T03:46:09.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:36:49.311878+0000", 2024-03-12T03:46:09.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7960000000000002e-05 2024-03-12T03:46:09.585 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.585 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.585 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.586 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:189349 7.e 7:78c2a40a:::benchmark_data_smithi169_79822_object71932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:46:09.586 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:30.631242+0000", 2024-03-12T03:46:09.586 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 458.93505307300001, 2024-03-12T03:46:09.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19704829900000001, 2024-03-12T03:46:09.587 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.587 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.587 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.587 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:09.587 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:09.587 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 189349 2024-03-12T03:46:09.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.588 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631242+0000", 2024-03-12T03:46:09.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631242+0000", 2024-03-12T03:46:09.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631243+0000", 2024-03-12T03:46:09.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.15e-06 2024-03-12T03:46:09.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631249+0000", 2024-03-12T03:46:09.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5809999999999996e-06 2024-03-12T03:46:09.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631260+0000", 2024-03-12T03:46:09.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0712e-05 2024-03-12T03:46:09.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631267+0000", 2024-03-12T03:46:09.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.277e-06 2024-03-12T03:46:09.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631324+0000", 2024-03-12T03:46:09.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7567e-05 2024-03-12T03:46:09.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631472+0000", 2024-03-12T03:46:09.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014747800000000001 2024-03-12T03:46:09.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2,3", 2024-03-12T03:46:09.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.631643+0000", 2024-03-12T03:46:09.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000171435 2024-03-12T03:46:09.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.635257+0000", 2024-03-12T03:46:09.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036141390000000002 2024-03-12T03:46:09.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.636921+0000", 2024-03-12T03:46:09.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001663072 2024-03-12T03:46:09.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.637788+0000", 2024-03-12T03:46:09.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086715000000000002 2024-03-12T03:46:09.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.828001+0000", 2024-03-12T03:46:09.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19021288 2024-03-12T03:46:09.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.828134+0000", 2024-03-12T03:46:09.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000133619 2024-03-12T03:46:09.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.828290+0000", 2024-03-12T03:46:09.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015623899999999999 2024-03-12T03:46:09.600 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.600 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.600 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.600 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:189350 7.e 7:7b91d00a:::benchmark_data_smithi169_79822_object71933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-12T03:46:09.600 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:30.633610+0000", 2024-03-12T03:46:09.600 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 458.93268516500001, 2024-03-12T03:46:09.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19495178699999999, 2024-03-12T03:46:09.601 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.601 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.601 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.601 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:09.601 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:09.601 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 189350 2024-03-12T03:46:09.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.601 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633610+0000", 2024-03-12T03:46:09.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633610+0000", 2024-03-12T03:46:09.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633611+0000", 2024-03-12T03:46:09.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6400000000000001e-07 2024-03-12T03:46:09.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633617+0000", 2024-03-12T03:46:09.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2450000000000003e-06 2024-03-12T03:46:09.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633628+0000", 2024-03-12T03:46:09.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0404999999999999e-05 2024-03-12T03:46:09.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633636+0000", 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3969999999999994e-06 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633748+0000", 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112451 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.633883+0000", 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000134535 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2,3", 2024-03-12T03:46:09.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.634034+0000", 2024-03-12T03:46:09.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000150956 2024-03-12T03:46:09.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.635379+0000", 2024-03-12T03:46:09.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013455209999999999 2024-03-12T03:46:09.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.636983+0000", 2024-03-12T03:46:09.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016039610000000001 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.637883+0000", 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00089951699999999996 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.828366+0000", 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19048279300000001 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.828439+0000", 2024-03-12T03:46:09.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3374000000000006e-05 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:30.828562+0000", 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012276799999999999 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209774 8.8 8:1fd2d222:::benchmark_data_smithi169_106492_object76291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.684941+0000", 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.88135381199999, 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0621361359999999, 2024-03-12T03:46:09.608 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209774 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.684941+0000", 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.684941+0000", 2024-03-12T03:46:09.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.684942+0000", 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0049999999999999e-06 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.684947+0000", 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5700000000000003e-06 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.684957+0000", 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0108999999999999e-05 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.684972+0000", 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5432000000000001e-05 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.685017+0000", 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4901000000000003e-05 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.685025+0000", 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4789999999999999e-06 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.728071+0000", 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0430456610000001 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.728207+0000", 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013618899999999999 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.728350+0000", 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014358700000000001 2024-03-12T03:46:09.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.734003+0000", 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.005652893 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.734065+0000", 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1401000000000004e-05 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.746970+0000", 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.01290579 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.747026+0000", 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5318000000000001e-05 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.747077+0000", 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1801000000000001e-05 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.615 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209781 8.8 8:1e1821c8:::benchmark_data_smithi169_106492_object76298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.615 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.690301+0000", 2024-03-12T03:46:09.615 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.875993943, 2024-03-12T03:46:09.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0570561009999999, 2024-03-12T03:46:09.615 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.615 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.615 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209781 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.690301+0000", 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.690301+0000", 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.690302+0000", 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5999999999999999e-07 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.690304+0000", 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5059999999999998e-06 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.690310+0000", 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6500000000000001e-06 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.690315+0000", 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9919999999999998e-06 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.690766+0000", 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000451509 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.690774+0000", 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9279999999999993e-06 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.730816+0000", 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.040041258 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.730914+0000", 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8286000000000001e-05 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.731025+0000", 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111078 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.734121+0000", 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0030959590000000001 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.734153+0000", 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2478000000000003e-05 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.747265+0000", 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013111566999999999 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.747316+0000", 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1065e-05 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.747357+0000", 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1365000000000001e-05 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209797 8.8 8:197f2acc:::benchmark_data_smithi169_106492_object76314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.707724+0000", 2024-03-12T03:46:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.85857147799999, 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0456861420000001, 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209797 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.707724+0000", 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.707724+0000", 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.707725+0000", 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.111e-06 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.707729+0000", 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.549e-06 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.707738+0000", 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9660000000000002e-06 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.707744+0000", 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5600000000000001e-06 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.707798+0000", 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3672999999999999e-05 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.707807+0000", 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9889999999999998e-06 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.732535+0000", 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0247288880000001 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.732620+0000", 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5017000000000003e-05 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.732740+0000", 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119139 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.737706+0000", 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049665869999999997 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.737744+0000", 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7404999999999998e-05 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753300+0000", 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.01555602 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753363+0000", 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3663000000000004e-05 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753410+0000", 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6575e-05 2024-03-12T03:46:09.627 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209801 8.8 8:1ce0a161:::benchmark_data_smithi169_106492_object76318:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.712449+0000", 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.85384616900001, 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0410975309999999, 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209801 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.712449+0000", 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.712449+0000", 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.712450+0000", 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.404e-06 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.712457+0000", 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4509999999999997e-06 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.712470+0000", 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2762000000000001e-05 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.712477+0000", 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2989999999999999e-06 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.713725+0000", 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001247827 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.713733+0000", 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6289999999999999e-06 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.734176+0000", 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.020442947 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.734258+0000", 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1798000000000005e-05 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.734343+0000", 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4822999999999997e-05 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.737792+0000", 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034491740000000002 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.737801+0000", 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8089999999999999e-06 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753447+0000", 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.015646501 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753510+0000", 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2538000000000003e-05 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753547+0000", 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6569000000000002e-05 2024-03-12T03:46:09.634 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209864 8.8 8:14514634:::benchmark_data_smithi169_106492_object76381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.782513+0000", 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.783782156, 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97123173699999998, 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209864 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.635 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.782513+0000", 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.782513+0000", 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.782515+0000", 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.626e-06 2024-03-12T03:46:09.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.782520+0000", 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8209999999999998e-06 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.782534+0000", 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3509000000000001e-05 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.782542+0000", 2024-03-12T03:46:09.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.943e-06 2024-03-12T03:46:09.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.783768+0000", 2024-03-12T03:46:09.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012265760000000001 2024-03-12T03:46:09.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.783777+0000", 2024-03-12T03:46:09.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5080000000000005e-06 2024-03-12T03:46:09.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.736290+0000", 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95251258100000002 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.736394+0000", 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104545 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.736502+0000", 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108372 2024-03-12T03:46:09.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.740950+0000", 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044475449999999998 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.740966+0000", 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6413999999999999e-05 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753633+0000", 2024-03-12T03:46:09.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012666336 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753708+0000", 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5480000000000002e-05 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753745+0000", 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6480999999999999e-05 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:09.643 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:09.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:09.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.644 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209894 8.8 8:1d3a11d4:::benchmark_data_smithi169_106492_object76411:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.644 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.812895+0000", 2024-03-12T03:46:09.644 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.753400081, 2024-03-12T03:46:09.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94099949299999996, 2024-03-12T03:46:09.644 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.644 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209894 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.812895+0000", 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.812895+0000", 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.812897+0000", 2024-03-12T03:46:09.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4309999999999999e-06 2024-03-12T03:46:09.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.812902+0000", 2024-03-12T03:46:09.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6450000000000002e-06 2024-03-12T03:46:09.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.812914+0000", 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1884e-05 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.812922+0000", 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9850000000000006e-06 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.812976+0000", 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4218999999999997e-05 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.812985+0000", 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2830000000000001e-06 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.737825+0000", 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.92484089300000005 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.737889+0000", 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3524999999999999e-05 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.737986+0000", 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6898999999999995e-05 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.740971+0000", 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029854880000000001 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.741008+0000", 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6248000000000003e-05 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753802+0000", 2024-03-12T03:46:09.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012794128 2024-03-12T03:46:09.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753855+0000", 2024-03-12T03:46:09.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3335999999999999e-05 2024-03-12T03:46:09.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753895+0000", 2024-03-12T03:46:09.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9529e-05 2024-03-12T03:46:09.650 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.651 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.651 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.651 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209899 8.8 8:111e9127:::benchmark_data_smithi169_106492_object76416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.651 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.814158+0000", 2024-03-12T03:46:09.651 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.75213763100001, 2024-03-12T03:46:09.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94005227000000002, 2024-03-12T03:46:09.651 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.651 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.651 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.651 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209899 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.814158+0000", 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.814158+0000", 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.814158+0000", 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3200000000000003e-07 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.814163+0000", 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4460000000000003e-06 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.814176+0000", 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3021e-05 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.814184+0000", 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6079999999999998e-06 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.814238+0000", 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4620000000000002e-05 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.814247+0000", 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2490000000000007e-06 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.739133+0000", 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.92488531500000004 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.739208+0000", 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5160000000000005e-05 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.739276+0000", 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7781999999999996e-05 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.741056+0000", 2024-03-12T03:46:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017802359999999999 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.742728+0000", 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016718379999999999 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753955+0000", 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011226834999999999 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.754169+0000", 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021430099999999999 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.754210+0000", 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0927000000000003e-05 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209907 8.8 8:118c1c9b:::benchmark_data_smithi169_106492_object76424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.825848+0000", 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.74044723, 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.92850627399999996, 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.657 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209907 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.825848+0000", 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.825848+0000", 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.825850+0000", 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5969999999999999e-06 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.825855+0000", 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6239999999999999e-06 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.825879+0000", 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4144000000000001e-05 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.825888+0000", 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6459999999999999e-06 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.825931+0000", 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2966000000000002e-05 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.825937+0000", 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0020000000000003e-06 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.741092+0000", 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91515523799999998 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.741211+0000", 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119251 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.741340+0000", 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012836299999999999 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.746922+0000", 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0055817769999999996 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.746930+0000", 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4689999999999997e-06 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.754265+0000", 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.007334719 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.754317+0000", 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2240999999999997e-05 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.754354+0000", 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7237000000000003e-05 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209942 8.8 8:1ecd438a:::benchmark_data_smithi169_106492_object76459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.866101+0000", 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.70019432999999, 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90595345599999999, 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209942 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.866101+0000", 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.866101+0000", 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.866102+0000", 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.358e-06 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.866108+0000", 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.93e-06 2024-03-12T03:46:09.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.866120+0000", 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2177999999999999e-05 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.866128+0000", 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0520000000000002e-06 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.866180+0000", 2024-03-12T03:46:09.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1350999999999997e-05 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.866186+0000", 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0850000000000002e-06 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.742766+0000", 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87657988099999995 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.742862+0000", 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6634999999999999e-05 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.742939+0000", 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6897999999999995e-05 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.747136+0000", 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041967230000000003 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.747154+0000", 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8425000000000001e-05 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.771944+0000", 2024-03-12T03:46:09.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024790133999999998 2024-03-12T03:46:09.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.772007+0000", 2024-03-12T03:46:09.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2199000000000005e-05 2024-03-12T03:46:09.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.772054+0000", 2024-03-12T03:46:09.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7607e-05 2024-03-12T03:46:09.670 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.670 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.670 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.670 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209952 8.8 8:1596f561:::benchmark_data_smithi169_106492_object76469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.670 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.878173+0000", 2024-03-12T03:46:09.670 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.688122588, 2024-03-12T03:46:09.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89951130999999995, 2024-03-12T03:46:09.671 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.671 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.671 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.671 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.671 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.671 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209952 2024-03-12T03:46:09.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.672 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.878173+0000", 2024-03-12T03:46:09.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.878173+0000", 2024-03-12T03:46:09.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.878174+0000", 2024-03-12T03:46:09.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3909999999999999e-06 2024-03-12T03:46:09.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.878180+0000", 2024-03-12T03:46:09.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2260000000000002e-06 2024-03-12T03:46:09.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.878193+0000", 2024-03-12T03:46:09.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2349e-05 2024-03-12T03:46:09.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.878201+0000", 2024-03-12T03:46:09.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3140000000000004e-06 2024-03-12T03:46:09.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.878258+0000", 2024-03-12T03:46:09.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6975999999999998e-05 2024-03-12T03:46:09.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.878271+0000", 2024-03-12T03:46:09.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3577000000000001e-05 2024-03-12T03:46:09.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.744460+0000", 2024-03-12T03:46:09.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.86618844299999997 2024-03-12T03:46:09.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.744570+0000", 2024-03-12T03:46:09.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0001103 2024-03-12T03:46:09.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.744687+0000", 2024-03-12T03:46:09.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117087 2024-03-12T03:46:09.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.747409+0000", 2024-03-12T03:46:09.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027217500000000002 2024-03-12T03:46:09.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753905+0000", 2024-03-12T03:46:09.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0064963180000000001 2024-03-12T03:46:09.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.777582+0000", 2024-03-12T03:46:09.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.023677032000000001 2024-03-12T03:46:09.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.777636+0000", 2024-03-12T03:46:09.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3202000000000003e-05 2024-03-12T03:46:09.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.777684+0000", 2024-03-12T03:46:09.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8344999999999999e-05 2024-03-12T03:46:09.681 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.681 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.681 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.681 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209960 8.8 8:1970c532:::benchmark_data_smithi169_106492_object76477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.681 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.888621+0000", 2024-03-12T03:46:09.681 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.677673922, 2024-03-12T03:46:09.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89327928700000003, 2024-03-12T03:46:09.682 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.682 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.682 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209960 2024-03-12T03:46:09.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.682 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.888621+0000", 2024-03-12T03:46:09.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.888621+0000", 2024-03-12T03:46:09.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.888623+0000", 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9259999999999999e-06 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.888629+0000", 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0530000000000003e-06 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.888644+0000", 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5017999999999999e-05 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.888653+0000", 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5830000000000007e-06 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.888717+0000", 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3671999999999996e-05 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.888725+0000", 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6300000000000004e-06 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.747446+0000", 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85872043899999995 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.747554+0000", 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107988 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.747663+0000", 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000109318 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.753596+0000", 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0059328280000000002 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.754366+0000", 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077014499999999997 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.781755+0000", 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027389040999999999 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.781847+0000", 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1712999999999994e-05 2024-03-12T03:46:09.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.781901+0000", 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3933e-05 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209976 8.8 8:1b2ddf9d:::benchmark_data_smithi169_106492_object76493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.929917+0000", 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.63637839899999, 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85215487300000003, 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.688 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209976 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.929917+0000", 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.929917+0000", 2024-03-12T03:46:09.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.929919+0000", 2024-03-12T03:46:09.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7630000000000001e-06 2024-03-12T03:46:09.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.929926+0000", 2024-03-12T03:46:09.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6979999999999998e-06 2024-03-12T03:46:09.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.929944+0000", 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7441000000000001e-05 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.929954+0000", 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0579e-05 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.930034+0000", 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0041000000000002e-05 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.930047+0000", 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3200999999999999e-05 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.750513+0000", 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82046599099999995 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.750634+0000", 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120176 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.750742+0000", 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108631 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.754371+0000", 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036283090000000001 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.754410+0000", 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9360999999999998e-05 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.781945+0000", 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027535385999999999 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.782031+0000", 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6074000000000003e-05 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.782072+0000", 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0222000000000002e-05 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.694 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:209981 8.8 8:1cfbd55a:::benchmark_data_smithi169_106492_object76498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:27.948742+0000", 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.61755351799999, 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83349706099999998, 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209981 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.948742+0000", 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.948742+0000", 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.948744+0000", 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1670000000000002e-06 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.948750+0000", 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2659999999999998e-06 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.948764+0000", 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3937999999999999e-05 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.948772+0000", 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4889999999999998e-06 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.948835+0000", 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3712000000000003e-05 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:27.948848+0000", 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3222e-05 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.754439+0000", 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.80559058500000003 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.754536+0000", 2024-03-12T03:46:09.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6879000000000005e-05 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.754642+0000", 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000105831 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.768844+0000", 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014202674 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.768859+0000", 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4632e-05 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.782142+0000", 2024-03-12T03:46:09.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013282608 2024-03-12T03:46:09.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.700 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:09 smithi173 ceph-mon[33544]: pgmap v608: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.24k op/s; 46608/278949 objects degraded (16.708%); 1/278949 objects misplaced (0.000%) 2024-03-12T03:46:09.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.782196+0000", 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4506999999999998e-05 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.782239+0000", 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2551e-05 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:210105 8.8 8:15ec353b:::benchmark_data_smithi169_106492_object76622:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.701 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:28.277300+0000", 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.28899507899999, 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50509822699999996, 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 210105 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.277300+0000", 2024-03-12T03:46:09.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.277300+0000", 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.277301+0000", 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2419999999999999e-06 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.277307+0000", 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7150000000000003e-06 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.277323+0000", 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5775000000000001e-05 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.277332+0000", 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3980000000000005e-06 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.277396+0000", 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3555999999999999e-05 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.277405+0000", 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7499999999999992e-06 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.759630+0000", 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48222505799999998 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.759748+0000", 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011891 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.759868+0000", 2024-03-12T03:46:09.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119249 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.768909+0000", 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0090412180000000002 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.771665+0000", 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027556989999999999 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.782294+0000", 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010629339999999999 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.782365+0000", 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0868e-05 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.782398+0000", 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3448999999999998e-05 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:210118 8.8 8:1de46dfc:::benchmark_data_smithi169_106492_object76635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:28.312883+0000", 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.253412462, 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46960537899999999, 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 210118 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.312883+0000", 2024-03-12T03:46:09.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.312883+0000", 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.312885+0000", 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.812e-06 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.312894+0000", 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2439999999999999e-06 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.312911+0000", 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6762e-05 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.312920+0000", 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6989999999999998e-06 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.313000+0000", 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9812000000000001e-05 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.313010+0000", 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0427e-05 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.762506+0000", 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44949577400000001 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.762630+0000", 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012417600000000001 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.762748+0000", 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117088 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.768968+0000", 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0062208189999999998 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.771671+0000", 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027028989999999999 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.782426+0000", 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.01075475 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.782454+0000", 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7936000000000002e-05 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:28.782488+0000", 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4180999999999997e-05 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:145281 9.6 9:6322c19f:::benchmark_data_smithi169_135229_object1066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:27.349699+0000", 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 42.216596463000002, 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.176032988, 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 145281 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.349699+0000", 2024-03-12T03:46:09.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.349699+0000", 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.349700+0000", 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.249e-06 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.349705+0000", 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.259e-06 2024-03-12T03:46:09.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.349718+0000", 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.313e-05 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.349726+0000", 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0420000000000003e-06 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.487817+0000", 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13809052499999999 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.487924+0000", 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010692999999999999 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2", 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.488057+0000", 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013286299999999999 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.525555+0000", 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.037498160000000003 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.525565+0000", 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.012e-05 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.525619+0000", 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4404000000000003e-05 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.525669+0000", 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9793000000000003e-05 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.525732+0000", 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2513000000000003e-05 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:145284 9.6 9:6c0aa463:::benchmark_data_smithi169_135229_object1069:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:27.352519+0000", 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 42.213776531000001, 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17343059199999999, 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 145284 2024-03-12T03:46:09.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.352519+0000", 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.352519+0000", 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.352520+0000", 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6199999999999996e-07 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.352524+0000", 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.639e-06 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.352536+0000", 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1392999999999999e-05 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.352543+0000", 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2830000000000004e-06 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.490362+0000", 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13781945900000001 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.490478+0000", 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000116152 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2", 2024-03-12T03:46:09.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.490580+0000", 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101383 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.525571+0000", 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.034990753999999999 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.525781+0000", 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021072199999999999 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.525842+0000", 2024-03-12T03:46:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0469e-05 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.525895+0000", 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3216e-05 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:27.525949+0000", 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4259999999999999e-05 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:09.724 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:46:09.744 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:09.783 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:09.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:09.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:09.915 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:09.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:09.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:09.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:09 smithi169 ceph-mon[29344]: pgmap v608: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.24k op/s; 46608/278949 objects degraded (16.708%); 1/278949 objects misplaced (0.000%) 2024-03-12T03:46:09.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:09 smithi169 ceph-mon[39949]: pgmap v608: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.24k op/s; 46608/278949 objects degraded (16.708%); 1/278949 objects misplaced (0.000%) 2024-03-12T03:46:10.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:10.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:10.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:10.164 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:10.196 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:10.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:10.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:10.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:10.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:10.396 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:10.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:10.475 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:10.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:10.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:10.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:10.611 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:10.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:10.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:10.713 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:10.747 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:10.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:10.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:10.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:10.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:10.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:10.923 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:10 smithi169 podman[148454]: 2024-03-12 03:46:10.511233695 +0000 UTC m=+0.083577297 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:46:10.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:11.019 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:11.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:11.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:11.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:11.157 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:11.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:11.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:11.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:11.291 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:11.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:11.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:11.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-12T03:46:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:11.443 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:11.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:11.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:11.580 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:11.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:11.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:11.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:11.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:11.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:11.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:11 smithi173 ceph-mon[33544]: pgmap v610: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 938 op/s; 46608/278949 objects degraded (16.708%); 1/278949 objects misplaced (0.000%) 2024-03-12T03:46:11.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:11.841 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:46:11.862 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:11.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:11.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:11 smithi169 ceph-mon[29344]: pgmap v610: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 938 op/s; 46608/278949 objects degraded (16.708%); 1/278949 objects misplaced (0.000%) 2024-03-12T03:46:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:11 smithi169 ceph-mon[39949]: pgmap v610: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 938 op/s; 46608/278949 objects degraded (16.708%); 1/278949 objects misplaced (0.000%) 2024-03-12T03:46:11.924 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:46:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:46:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:46:11.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:12.009 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:12.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:12.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:12.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:12.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:12.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:12.278 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:12.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:12.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:12.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:12.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:12.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:12.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:12.551 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:12.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:12.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:12.685 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:12.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:12.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:12.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:12.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:12.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:12.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:12.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:12.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:12.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:13.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:13.094 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:13.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:13.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:13.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:13.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:46:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:46:13.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:13.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:13.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:13.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:13.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:13.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:13.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:13.516 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:13.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:13.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:13.605 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:13 smithi169 podman[148454]: 2024-03-12 03:46:13.209730135 +0000 UTC m=+2.782073712 container create 3435d2172dd83c23c9e98d0964dce6ca21c8dbab0a3e4d4e91ac8a82a614c2ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-03-12T03:46:13.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:13.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:13.651 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:13.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:13.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:13.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:13.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:13.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:13.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:13 smithi169 ceph-mon[29344]: pgmap v611: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 44374/264519 objects degraded (16.775%); 1/264519 objects misplaced (0.000%) 2024-03-12T03:46:13.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:13 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:46:13.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:13 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:46:13.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:13 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 44374/264519 objects degraded (16.775%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:13.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:13 smithi169 ceph-mon[39949]: pgmap v611: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 44374/264519 objects degraded (16.775%); 1/264519 objects misplaced (0.000%) 2024-03-12T03:46:13.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:13 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:46:13.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:13 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:46:13.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:13 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 44374/264519 objects degraded (16.775%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:13.904 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15279.0:204356 9.0 e326/278 9:09225fb2:::benchmark_data_smithi169_135229_object60141:head v 326'15452, mlcod=326'15435)", 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:46:13.856109+0000", 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.044650495999999998, 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.044681092999999998, 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.856109+0000", 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.856109+0000", 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.856109+0000", 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3900000000000004e-07 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.856115+0000", 2024-03-12T03:46:13.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7749999999999998e-06 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.856128+0000", 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2612999999999999e-05 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.856135+0000", 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2300000000000002e-06 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:13.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.888731+0000", 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.032596199999999999 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.888751+0000", 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.969e-05 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.900708+0000", 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.01195713 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15279.0:204381 9.0 e326/278 9:04b8f03c:::benchmark_data_smithi169_135229_object60166:head v 326'15453, mlcod=326'15435)", 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:46:13.862589+0000", 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.038170366999999997, 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.038282214000000002, 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.862589+0000", 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.862589+0000", 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.862590+0000", 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2699999999999998e-07 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.862598+0000", 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0830000000000004e-06 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.862612+0000", 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4698000000000001e-05 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.862620+0000", 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6410000000000002e-06 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.890727+0000", 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.028107271 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.890747+0000", 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9752999999999999e-05 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15279.0:204394 9.0 e326/278 9:0eeca874:::benchmark_data_smithi169_135229_object60179:head v 326'15454, mlcod=326'15437)", 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:46:13.874814+0000", 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.025945057000000001, 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026129146999999998, 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:13.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.874814+0000", 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.874814+0000", 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.874815+0000", 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2900000000000003e-07 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.874821+0000", 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4699999999999999e-06 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.874831+0000", 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0081999999999999e-05 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.874838+0000", 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.917e-06 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.892755+0000", 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.017916499999999998 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.892773+0000", 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8555000000000001e-05 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15279.0:204401 9.0 e326/278 9:0c388a45:::benchmark_data_smithi169_135229_object60186:head v 326'15455, mlcod=326'15443)", 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:46:13.880967+0000", 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.019791757, 2024-03-12T03:46:13.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020051439000000001, 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.880967+0000", 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.880967+0000", 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:13.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.880968+0000", 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0800000000000004e-07 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.880976+0000", 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5329999999999997e-06 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.880988+0000", 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2768999999999999e-05 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:13.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.880996+0000", 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5870000000000004e-06 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.895228+0000", 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014231636000000001 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.895260+0000", 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2738999999999999e-05 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.918 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15279.0:204410 9.0 e326/278 9:00935ba1:::benchmark_data_smithi169_135229_object60195:head v 326'15456, mlcod=326'15444)", 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:46:13.885885+0000", 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.014874513000000001, 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.015196256999999999, 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.885885+0000", 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.885885+0000", 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.885885+0000", 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3399999999999998e-07 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.885892+0000", 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.793e-06 2024-03-12T03:46:13.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.885904+0000", 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2383e-05 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.885911+0000", 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9269999999999999e-06 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.897843+0000", 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011931748000000001 2024-03-12T03:46:13.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.897861+0000", 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8190999999999999e-05 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15279.0:204415 9.0 e326/278)", 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:46:13.890597+0000", 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.010162098, 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010541976, 2024-03-12T03:46:13.922 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.890597+0000", 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.890597+0000", 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:13.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.890598+0000", 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3600000000000003e-07 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.890604+0000", 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.737e-06 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.890618+0000", 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3653e-05 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.890625+0000", 2024-03-12T03:46:13.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3080000000000002e-06 2024-03-12T03:46:13.925 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.925 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:13.925 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.925 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15279.0:204425 9.1 e326/313)", 2024-03-12T03:46:13.925 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:46:13.901054+0000", 2024-03-12T03:46:13.925 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4294967295.9997053, 2024-03-12T03:46:13.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013374599999999999, 2024-03-12T03:46:13.925 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.901054+0000", 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.901054+0000", 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.901055+0000", 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5900000000000005e-07 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.901063+0000", 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.14e-06 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.901093+0000", 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0153000000000001e-05 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:13.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:13.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:46:13.901140+0000", 2024-03-12T03:46:13.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7274999999999997e-05 2024-03-12T03:46:13.928 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.928 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:13.928 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.928 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:13.928 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-03-12T03:46:13.928 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 7 2024-03-12T03:46:13.928 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:46:13.929 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:13 smithi173 ceph-mon[33544]: pgmap v611: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 44374/264519 objects degraded (16.775%); 1/264519 objects misplaced (0.000%) 2024-03-12T03:46:13.929 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:13 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:46:13.929 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:13 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:46:13.929 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:13 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 44374/264519 objects degraded (16.775%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:13.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:13.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:13.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:14.031 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:14.064 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:14.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:14.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:14.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:14.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:14.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:14.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:14.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:14.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:14.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:14.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:14.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:14.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:14.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:14.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:14.624 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:14.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:14.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-12T03:46:14.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:14.759 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:14.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:14.851 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:14 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:14.851 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:14 smithi173 ceph-mon[33544]: osdmap e326: 8 total, 7 up, 8 in 2024-03-12T03:46:14.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:14.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:14.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:14.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:14 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:14 smithi169 ceph-mon[29344]: osdmap e326: 8 total, 7 up, 8 in 2024-03-12T03:46:14.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:14 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:14 smithi169 ceph-mon[39949]: osdmap e326: 8 total, 7 up, 8 in 2024-03-12T03:46:15.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:15.040 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:15.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:15.102 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:46:15.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:15.172 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:15.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:15.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:15.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:15.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:15.427 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:15.462 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:15.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:15.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:15.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:15.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:15.641 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:15 smithi169 ceph-mon[29344]: pgmap v613: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 43661/261138 objects degraded (16.720%); 1/261138 objects misplaced (0.000%) 2024-03-12T03:46:15.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:15 smithi169 ceph-mon[39949]: pgmap v613: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 43661/261138 objects degraded (16.720%); 1/261138 objects misplaced (0.000%) 2024-03-12T03:46:15.642 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:15 smithi169 podman[148454]: 2024-03-12 03:46:15.367983904 +0000 UTC m=+4.940327477 container init 3435d2172dd83c23c9e98d0964dce6ca21c8dbab0a3e4d4e91ac8a82a614c2ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-03-12T03:46:15.642 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:15 smithi169 podman[148454]: 2024-03-12 03:46:15.375015729 +0000 UTC m=+4.947359304 container start 3435d2172dd83c23c9e98d0964dce6ca21c8dbab0a3e4d4e91ac8a82a614c2ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-12T03:46:15.642 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:15 smithi169 podman[148454]: 2024-03-12 03:46:15.476002888 +0000 UTC m=+5.048346464 container attach 3435d2172dd83c23c9e98d0964dce6ca21c8dbab0a3e4d4e91ac8a82a614c2ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-12T03:46:15.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:15.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:15.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:15.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:15.884 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:15.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:15.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:15 smithi173 ceph-mon[33544]: pgmap v613: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 43661/261138 objects degraded (16.720%); 1/261138 objects misplaced (0.000%) 2024-03-12T03:46:16.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:16.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:16.126 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:16.166 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:16.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:16.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:16.268 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:16.305 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:16.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:16.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:16.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:16.424 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[148648]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-12T03:46:16.424 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[148648]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-12T03:46:16.424 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[148648]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-12T03:46:16.424 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[148648]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-12T03:46:16.424 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[148648]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-12T03:46:16.424 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[148648]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-12T03:46:16.424 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate[148648]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-12T03:46:16.424 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 bash[148454]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-12T03:46:16.424 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 bash[148454]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-12T03:46:16.424 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 bash[148454]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-12T03:46:16.425 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 bash[148454]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-12T03:46:16.425 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 bash[148454]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-12T03:46:16.425 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 bash[148454]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-12T03:46:16.425 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 bash[148454]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-12T03:46:16.425 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:16 smithi169 podman[148454]: 2024-03-12 03:46:16.130770547 +0000 UTC m=+5.703114130 container died 3435d2172dd83c23c9e98d0964dce6ca21c8dbab0a3e4d4e91ac8a82a614c2ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-12T03:46:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:16.443 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:16.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:16.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:16.579 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:16.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:16.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:16.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:16.719 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:16.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:16.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:16.860 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:16.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:16.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:16.994 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:16.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:17.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:17.130 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:17.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:17.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:17.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:17.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:17.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:17.348 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:17 smithi173 ceph-mon[33544]: pgmap v614: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 742 op/s; 43661/261138 objects degraded (16.720%); 1/261138 objects misplaced (0.000%) 2024-03-12T03:46:17.348 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:17 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:17.349 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:17 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:17.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:17.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:17.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:17.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:17 smithi169 ceph-mon[39949]: pgmap v614: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 742 op/s; 43661/261138 objects degraded (16.720%); 1/261138 objects misplaced (0.000%) 2024-03-12T03:46:17.522 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:17 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:17.522 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:17 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:17.522 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:17 smithi169 ceph-mon[29344]: pgmap v614: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 742 op/s; 43661/261138 objects degraded (16.720%); 1/261138 objects misplaced (0.000%) 2024-03-12T03:46:17.522 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:17 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:17.522 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:17 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:17.522 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:17 smithi169 podman[148885]: 2024-03-12 03:46:17.301504687 +0000 UTC m=+1.162944532 container remove 3435d2172dd83c23c9e98d0964dce6ca21c8dbab0a3e4d4e91ac8a82a614c2ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-12T03:46:17.531 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:17.554 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:46:17.554 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:46:17.554 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:46:17.554 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:46:17.554 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:46:17.570 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:17.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:17.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:17.713 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:17.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:17.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:17.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:17.883 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:17.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:17.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:17.950 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:46:17.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:17.991 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:46:18.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:18.031 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:18.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:18.034 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:46:18.091 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:46:18.091 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:17 smithi169 podman[149181]: 2024-03-12 03:46:17.882092927 +0000 UTC m=+0.039310087 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:46:18.092 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:18 smithi169 podman[149181]: 2024-03-12 03:46:18.044466898 +0000 UTC m=+0.201684044 container create aa2a33b518fa524484a9721065801e882b8211ec8605dad6e715ef09ebbd9cbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-12T03:46:18.125 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:46:18.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:18.160 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:46:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:18.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:18.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:18.204 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:46:18.204 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:46:18.204 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:46:18.205 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:46:18.205 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-12T03:46:18.205 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:46:18.205 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:46:18.205 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:46:18.206 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:46:18.235 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:46:18.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-12T03:46:18.271 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:18.315 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:18.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:18.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:18.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:18 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:46:18.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:18 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1300900760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:46:18.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:18 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4167004464' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:46:18.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:18 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:46:18.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:18 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:46:18.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:18 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1300900760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:46:18.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:18 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4167004464' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:46:18.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:18 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:46:18.449 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:18.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:18.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:18.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:18 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:46:18.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:18 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1300900760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:46:18.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:18 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4167004464' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:46:18.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:18 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:46:18.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:18.585 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:18.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:18.664 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:46:18.687 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:18.722 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:18.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:18.764 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-12T03:46:18.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:18.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:18.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:18.925 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:18 smithi169 podman[149181]: 2024-03-12 03:46:18.604512307 +0000 UTC m=+0.761729461 container init aa2a33b518fa524484a9721065801e882b8211ec8605dad6e715ef09ebbd9cbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:46:18.925 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:18 smithi169 podman[149181]: 2024-03-12 03:46:18.610241817 +0000 UTC m=+0.767458967 container start aa2a33b518fa524484a9721065801e882b8211ec8605dad6e715ef09ebbd9cbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-12T03:46:18.925 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:18 smithi169 ceph-osd[149393]: -- 172.21.15.169:0/2391589073 <== mon.2 v2:172.21.15.169:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558f13b61680 con 0x558f13b2b400 2024-03-12T03:46:18.925 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:18 smithi169 bash[149181]: aa2a33b518fa524484a9721065801e882b8211ec8605dad6e715ef09ebbd9cbd 2024-03-12T03:46:18.925 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:18 smithi169 systemd[1]: Started Ceph osd.3 for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:46:18.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:19.002 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:19.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:19.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:19.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:19.134 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:19.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:19.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:19.262 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:19.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:19.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:19.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:19.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:19.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:19.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:19 smithi173 ceph-mon[33544]: pgmap v615: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.23k op/s; 40610/243138 objects degraded (16.702%); 1/243138 objects misplaced (0.000%) 2024-03-12T03:46:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:19 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:46:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:19 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:46:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:19 smithi173 ceph-mon[33544]: osdmap e327: 8 total, 7 up, 8 in 2024-03-12T03:46:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:19 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 40610/243138 objects degraded (16.702%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:19 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:46:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:19 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:46:19.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:19 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:19.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:19.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:19.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:19.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:19.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:19.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:19.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:19.673 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[149269]: 2024-03-12T03:46:19.170+0000 7efe91d1e7c0 -1 Falling back to public interface 2024-03-12T03:46:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[29344]: pgmap v615: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.23k op/s; 40610/243138 objects degraded (16.702%); 1/243138 objects misplaced (0.000%) 2024-03-12T03:46:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:46:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:46:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[29344]: osdmap e327: 8 total, 7 up, 8 in 2024-03-12T03:46:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 40610/243138 objects degraded (16.702%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:46:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:46:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[39949]: pgmap v615: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.23k op/s; 40610/243138 objects degraded (16.702%); 1/243138 objects misplaced (0.000%) 2024-03-12T03:46:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:46:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:46:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[39949]: osdmap e327: 8 total, 7 up, 8 in 2024-03-12T03:46:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 40610/243138 objects degraded (16.702%), 31 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:46:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:46:19.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:19 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:19.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:19.795 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:19.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:19.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:19.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:19.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:19.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:20.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:20.066 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:20.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:20.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:20.211 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:20.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:20.226 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:46:20.227 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:46:20.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:20.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:20.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:20.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:20.491 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:20.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:20.593 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:20.624 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:20.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:20.644 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:46:20.644 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:46:20.725 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:20.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:20.765 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:20.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:20.784 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:20.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:20.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:20.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:21.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:21.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:21.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:21.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:21.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:21.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:21.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:21.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:21.314 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:21.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:21.317 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:21 smithi173 ceph-mon[33544]: pgmap v617: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 890 op/s; 40610/243138 objects degraded (16.702%); 1/243138 objects misplaced (0.000%) 2024-03-12T03:46:21.366 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-12T03:46:21.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:21.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:21.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:21.602 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:21.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:21.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:21.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:21 smithi169 ceph-mon[29344]: pgmap v617: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 890 op/s; 40610/243138 objects degraded (16.702%); 1/243138 objects misplaced (0.000%) 2024-03-12T03:46:21.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:21 smithi169 ceph-mon[39949]: pgmap v617: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 890 op/s; 40610/243138 objects degraded (16.702%); 1/243138 objects misplaced (0.000%) 2024-03-12T03:46:21.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:21.776 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:46:21.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:21.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:21.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:21.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:21.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:21.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:22.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:22.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:22.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:22.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:22.173 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:22.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:46:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:46:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:46:22.206 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:22.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:22.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:22.227 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:46:22.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:22.342 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:22.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:22.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:22.474 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:22.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:22.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:22 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:46:22.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:22 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:46:22.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:22 smithi173 ceph-mon[33544]: osdmap e328: 8 total, 7 up, 8 in 2024-03-12T03:46:22.567 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:22 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:46:22.567 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:22 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:46:22.567 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:22 smithi169 ceph-mon[29344]: osdmap e328: 8 total, 7 up, 8 in 2024-03-12T03:46:22.568 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:22 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:46:22.568 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:22 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:46:22.568 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:22 smithi169 ceph-mon[39949]: osdmap e328: 8 total, 7 up, 8 in 2024-03-12T03:46:22.576 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:22.636 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:22.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:22.708 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:22.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:46:22.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:22.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:22.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:22.855 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:22 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[149269]: 2024-03-12T03:46:22.588+0000 7efe91d1e7c0 -1 osd.3 264 log_to_monitors true 2024-03-12T03:46:22.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:22.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:22.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:23.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:23.063 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:23.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:23.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:23.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:23.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:23.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:23.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:46:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:46:23.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:23.357 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:23.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:23.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:23.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:23 smithi169 ceph-mon[29344]: pgmap v619: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 38147/227139 objects degraded (16.795%); 1/227139 objects misplaced (0.000%) 2024-03-12T03:46:23.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:23 smithi169 ceph-mon[29344]: from='osd.3 v2:172.21.15.169:6813/3056769071' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:46:23.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:23 smithi169 ceph-mon[29344]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:46:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:23 smithi169 ceph-mon[39949]: pgmap v619: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 38147/227139 objects degraded (16.795%); 1/227139 objects misplaced (0.000%) 2024-03-12T03:46:23.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:23 smithi169 ceph-mon[39949]: from='osd.3 v2:172.21.15.169:6813/3056769071' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:46:23.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:23 smithi169 ceph-mon[39949]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:46:23.448 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:46:23.449 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:46:23.449 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-03-12T03:46:23.449 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:46:23.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:23.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:23.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:23.585 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:23 smithi173 ceph-mon[33544]: pgmap v619: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 95 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 38147/227139 objects degraded (16.795%); 1/227139 objects misplaced (0.000%) 2024-03-12T03:46:23.585 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:23 smithi173 ceph-mon[33544]: from='osd.3 v2:172.21.15.169:6813/3056769071' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:46:23.585 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:23 smithi173 ceph-mon[33544]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:46:23.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:23.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:23.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:23.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:23.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:23.779 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:23.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:23.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:23.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:23.913 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:23.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:23.923 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:46:23 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[149269]: 2024-03-12T03:46:23.451+0000 7efe83176700 -1 osd.3 329 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:46:23.931 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:46:23.931 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:46:24.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:24.047 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:24.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:24.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:24.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:24.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:24.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:24.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:24.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:24.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:24.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:24.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:24 smithi169 ceph-mon[29344]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:46:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:24 smithi169 ceph-mon[29344]: osdmap e329: 8 total, 7 up, 8 in 2024-03-12T03:46:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:24 smithi169 ceph-mon[29344]: from='osd.3 v2:172.21.15.169:6813/3056769071' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:46:24.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:24 smithi169 ceph-mon[29344]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:46:24.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:24 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 38147/227139 objects degraded (16.795%), 31 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:24 smithi169 ceph-mon[39949]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:46:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:24 smithi169 ceph-mon[39949]: osdmap e329: 8 total, 7 up, 8 in 2024-03-12T03:46:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:24 smithi169 ceph-mon[39949]: from='osd.3 v2:172.21.15.169:6813/3056769071' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:46:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:24 smithi169 ceph-mon[39949]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:46:24.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:24 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 38147/227139 objects degraded (16.795%), 31 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:24.440 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:24.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:24.487 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:24.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:24.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:24.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:24.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:24.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:24.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:24.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:24 smithi173 ceph-mon[33544]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:46:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:24 smithi173 ceph-mon[33544]: osdmap e329: 8 total, 7 up, 8 in 2024-03-12T03:46:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:24 smithi173 ceph-mon[33544]: from='osd.3 v2:172.21.15.169:6813/3056769071' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:46:24.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:24 smithi173 ceph-mon[33544]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:46:24.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:24 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 38147/227139 objects degraded (16.795%), 31 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:24.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:24.819 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:24.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:24.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:24.944 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:24.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:25.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:25.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:25.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:25.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:25.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:25.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:25.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:25.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:25.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:25.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:25.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:25.433 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:25.458 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:25.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:25.476 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-12T03:46:25.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:46:25.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:46:25.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[39949]: pgmap v621: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.11k op/s; 37272/223179 objects degraded (16.700%); 1/223179 objects misplaced (0.000%) 2024-03-12T03:46:25.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[39949]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T03:46:25.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[39949]: osd.3 v2:172.21.15.169:6813/3056769071 boot 2024-03-12T03:46:25.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[39949]: osdmap e330: 8 total, 8 up, 8 in 2024-03-12T03:46:25.541 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:46:25.541 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:46:25.541 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:46:25.541 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[29344]: pgmap v621: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.11k op/s; 37272/223179 objects degraded (16.700%); 1/223179 objects misplaced (0.000%) 2024-03-12T03:46:25.541 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[29344]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T03:46:25.541 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[29344]: osd.3 v2:172.21.15.169:6813/3056769071 boot 2024-03-12T03:46:25.541 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[29344]: osdmap e330: 8 total, 8 up, 8 in 2024-03-12T03:46:25.541 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:25 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:46:25.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:25.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:25.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:25.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:25.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:25.715 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:25.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:25.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:25 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:46:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:25 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:46:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:25 smithi173 ceph-mon[33544]: pgmap v621: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.11k op/s; 37272/223179 objects degraded (16.700%); 1/223179 objects misplaced (0.000%) 2024-03-12T03:46:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:25 smithi173 ceph-mon[33544]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T03:46:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:25 smithi173 ceph-mon[33544]: osd.3 v2:172.21.15.169:6813/3056769071 boot 2024-03-12T03:46:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:25 smithi173 ceph-mon[33544]: osdmap e330: 8 total, 8 up, 8 in 2024-03-12T03:46:25.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:25 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:46:25.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:25.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:25.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:25.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:25.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:25.972 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:25.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:26.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:26.098 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:26.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:26.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:26.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-12T03:46:26.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:26.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:26.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:26.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:26.387 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:26.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:26.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:26 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:46:26.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:26 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:46:26.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:26 smithi169 ceph-mon[29344]: osdmap e331: 8 total, 8 up, 8 in 2024-03-12T03:46:26.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:26 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:46:26.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:26 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:46:26.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:26 smithi169 ceph-mon[39949]: osdmap e331: 8 total, 8 up, 8 in 2024-03-12T03:46:26.457 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:46:26.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:26 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:46:26.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:26 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:46:26.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:26 smithi173 ceph-mon[33544]: osdmap e331: 8 total, 8 up, 8 in 2024-03-12T03:46:26.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:26.527 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:26.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:26.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:26.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:26.664 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:26.670 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:26.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:26.693 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:46:26.735 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:46:26.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:26.773 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:46:26.808 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:26.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:26.814 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:46:26.853 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:46:26.890 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:46:26.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:26.930 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:46:26.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:26.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:26.972 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:46:26.976 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:46:26.976 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:46:26.977 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:46:26.977 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:46:26.977 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:46:26.977 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:46:26.977 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:46:26.977 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:46:27.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:27.081 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:27.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:27.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:27.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:27.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:27.216 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:27.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:27.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:27.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:27.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:27.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:27.503 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:27.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:27.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:27.643 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:27.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:27.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[29344]: pgmap v624: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.30k op/s; 37272/223179 objects degraded (16.700%); 1/223179 objects misplaced (0.000%) 2024-03-12T03:46:27.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[29344]: 9.1 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13417 2024-03-12T03:46:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[29344]: 9.2 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13574 2024-03-12T03:46:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[29344]: 9.4 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13626 2024-03-12T03:46:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[29344]: 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 329'13473 2024-03-12T03:46:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[29344]: 9.8 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13681 2024-03-12T03:46:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[29344]: 9.f starting backfill to osd.3 from (0'0,0'0] MAX to 329'13648 2024-03-12T03:46:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[29344]: osdmap e332: 8 total, 8 up, 8 in 2024-03-12T03:46:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[29344]: 9.3 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13654 2024-03-12T03:46:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[29344]: 9.7 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13764 2024-03-12T03:46:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/965380323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:46:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[39949]: pgmap v624: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.30k op/s; 37272/223179 objects degraded (16.700%); 1/223179 objects misplaced (0.000%) 2024-03-12T03:46:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[39949]: 9.1 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13417 2024-03-12T03:46:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[39949]: 9.2 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13574 2024-03-12T03:46:27.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[39949]: 9.4 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13626 2024-03-12T03:46:27.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[39949]: 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 329'13473 2024-03-12T03:46:27.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[39949]: 9.8 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13681 2024-03-12T03:46:27.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[39949]: 9.f starting backfill to osd.3 from (0'0,0'0] MAX to 329'13648 2024-03-12T03:46:27.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[39949]: osdmap e332: 8 total, 8 up, 8 in 2024-03-12T03:46:27.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[39949]: 9.3 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13654 2024-03-12T03:46:27.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[39949]: 9.7 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13764 2024-03-12T03:46:27.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:27 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/965380323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:46:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:27 smithi173 ceph-mon[33544]: pgmap v624: 168 pgs: 4 active+clean+remapped, 31 active+undersized+degraded, 37 active+undersized, 96 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.30k op/s; 37272/223179 objects degraded (16.700%); 1/223179 objects misplaced (0.000%) 2024-03-12T03:46:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:27 smithi173 ceph-mon[33544]: 9.1 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13417 2024-03-12T03:46:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:27 smithi173 ceph-mon[33544]: 9.2 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13574 2024-03-12T03:46:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:27 smithi173 ceph-mon[33544]: 9.4 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13626 2024-03-12T03:46:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:27 smithi173 ceph-mon[33544]: 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 329'13473 2024-03-12T03:46:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:27 smithi173 ceph-mon[33544]: 9.8 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13681 2024-03-12T03:46:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:27 smithi173 ceph-mon[33544]: 9.f starting backfill to osd.3 from (0'0,0'0] MAX to 329'13648 2024-03-12T03:46:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:27 smithi173 ceph-mon[33544]: osdmap e332: 8 total, 8 up, 8 in 2024-03-12T03:46:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:27 smithi173 ceph-mon[33544]: 9.3 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13654 2024-03-12T03:46:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:27 smithi173 ceph-mon[33544]: 9.7 starting backfill to osd.3 from (0'0,0'0] MAX to 329'13764 2024-03-12T03:46:27.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:27 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/965380323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:46:27.745 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:27.782 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:27.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:27.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:27.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:27.920 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:27.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:27.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:28.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:28.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:28.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:28.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:28.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:28.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:28.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:28.301 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:28.341 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:28.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:28.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:28.347 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:28 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:46:28.347 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:28 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:46:28.347 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:28 smithi169 ceph-mon[29344]: osdmap e333: 8 total, 8 up, 8 in 2024-03-12T03:46:28.347 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:28 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:46:28.347 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:28 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:46:28.348 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:28 smithi169 ceph-mon[39949]: osdmap e333: 8 total, 8 up, 8 in 2024-03-12T03:46:28.443 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:28.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:28 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:46:28.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:28 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:46:28.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:28 smithi173 ceph-mon[33544]: osdmap e333: 8 total, 8 up, 8 in 2024-03-12T03:46:28.483 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:28.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:28.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:28.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:28.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:28.742 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:28.782 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:28.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:28.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:28.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:28.920 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:28.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:28.978 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:46:28.978 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:46:28.978 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:46:28.978 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:46:28.978 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:46:29.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:29.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:29.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:29.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:29.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:29.194 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:29.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:29.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:29.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:29.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:29.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:29.434 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:29.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:29.481 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:29.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:29.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:29.621 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:29.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:29.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:29 smithi173 ceph-mon[33544]: pgmap v627: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 activating+undersized+degraded+remapped, 131 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 15 active+undersized, 1 active+undersized+remapped, 10 active+undersized+degraded, 2 peering, 1 active+clean+remapped, 3 remapped+peering; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 995 op/s; 21872/211239 objects degraded (10.354%); 1/211239 objects misplaced (0.000%); 1.9 MiB/s, 40 objects/s recovering 2024-03-12T03:46:29.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:29 smithi173 ceph-mon[33544]: osdmap e334: 8 total, 8 up, 8 in 2024-03-12T03:46:29.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:29 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 21872/211239 objects degraded (10.354%), 15 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:29.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:29 smithi173 ceph-mon[33544]: osdmap e335: 8 total, 8 up, 8 in 2024-03-12T03:46:29.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:29 smithi169 ceph-mon[29344]: pgmap v627: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 activating+undersized+degraded+remapped, 131 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 15 active+undersized, 1 active+undersized+remapped, 10 active+undersized+degraded, 2 peering, 1 active+clean+remapped, 3 remapped+peering; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 995 op/s; 21872/211239 objects degraded (10.354%); 1/211239 objects misplaced (0.000%); 1.9 MiB/s, 40 objects/s recovering 2024-03-12T03:46:29.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:29 smithi169 ceph-mon[29344]: osdmap e334: 8 total, 8 up, 8 in 2024-03-12T03:46:29.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:29 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 21872/211239 objects degraded (10.354%), 15 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:29.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:29 smithi169 ceph-mon[29344]: osdmap e335: 8 total, 8 up, 8 in 2024-03-12T03:46:29.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:29 smithi169 ceph-mon[39949]: pgmap v627: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 activating+undersized+degraded+remapped, 131 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 15 active+undersized, 1 active+undersized+remapped, 10 active+undersized+degraded, 2 peering, 1 active+clean+remapped, 3 remapped+peering; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 995 op/s; 21872/211239 objects degraded (10.354%); 1/211239 objects misplaced (0.000%); 1.9 MiB/s, 40 objects/s recovering 2024-03-12T03:46:29.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:29 smithi169 ceph-mon[39949]: osdmap e334: 8 total, 8 up, 8 in 2024-03-12T03:46:29.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:29 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 21872/211239 objects degraded (10.354%), 15 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2024-03-12T03:46:29.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:29 smithi169 ceph-mon[39949]: osdmap e335: 8 total, 8 up, 8 in 2024-03-12T03:46:29.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-12T03:46:29.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:29.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:29.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:29.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:29.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:46:29.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:29.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:29.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:29.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:30.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:30.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:30.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:30.102 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:46:30.156 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:30.203 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:30.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:30.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:30.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:30.311 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:46:30.357 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:30.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:30.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:30.459 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:30.502 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:30.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:30.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:30.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:30.626 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:30.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:30.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:30 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1834837145' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:46:30.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:30 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:46:30.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:30 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1834837145' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:46:30.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:30 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:46:30.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:30 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1834837145' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:46:30.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:30 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:46:30.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:30.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:30.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:30.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:30.904 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:30.914 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:30.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:31.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:31.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:31.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:31.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:31.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:31.199 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:31.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:31.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:31.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:31.343 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.0 to 1 (10000) 2024-03-12T03:46:31.354 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:31.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:31.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:31.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:31.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:31.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:31.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:31 smithi173 ceph-mon[33544]: pgmap v630: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 activating+undersized+degraded+remapped, 157 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped, 2 peering, 3 remapped+peering; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 21823/211239 objects degraded (10.331%); 1.8 MiB/s, 47 objects/s recovering 2024-03-12T03:46:31.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:31 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:46:31.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:31 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:46:31.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:31 smithi173 ceph-mon[33544]: osdmap e336: 8 total, 8 up, 8 in 2024-03-12T03:46:31.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:31 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3471987649' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:46:31.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:31 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:46:31.610 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:31 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:31.610 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:31.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:31.650 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:31.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:31.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[29344]: pgmap v630: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 activating+undersized+degraded+remapped, 157 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped, 2 peering, 3 remapped+peering; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 21823/211239 objects degraded (10.331%); 1.8 MiB/s, 47 objects/s recovering 2024-03-12T03:46:31.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[29344]: osdmap e336: 8 total, 8 up, 8 in 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3471987649' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[39949]: pgmap v630: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 activating+undersized+degraded+remapped, 157 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped, 2 peering, 3 remapped+peering; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 21823/211239 objects degraded (10.331%); 1.8 MiB/s, 47 objects/s recovering 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[39949]: osdmap e336: 8 total, 8 up, 8 in 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3471987649' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:46:31.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:31.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:31.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:31.792 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:31.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:31.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:31.929 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:31.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:32.000 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:46:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:46:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:46:32.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:32.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-12T03:46:32.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:32.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:32.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:32.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:32.217 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:32.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:32.286 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:46:32.286 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:46:32.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:46:32.286 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:46:32.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.286 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:129154 7.15 7:abc81e19:::benchmark_data_smithi169_79822_object11737:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:46:32.286 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:15.204199+0000", 2024-03-12T03:46:32.287 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 557.07643524599996, 2024-03-12T03:46:32.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0467048430000001, 2024-03-12T03:46:32.287 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.287 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:46:32.287 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.287 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.287 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.287 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 129154 2024-03-12T03:46:32.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.287 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204199+0000", 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204199+0000", 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204201+0000", 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.13e-06 2024-03-12T03:46:32.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204205+0000", 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9250000000000003e-06 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204215+0000", 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9799999999999993e-06 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204221+0000", 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8950000000000003e-06 2024-03-12T03:46:32.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204265+0000", 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3579999999999999e-05 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204404+0000", 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013930600000000001 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1,2", 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.204657+0000", 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025272100000000002 2024-03-12T03:46:32.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.205872+0000", 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012149540000000001 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.206748+0000", 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00087584100000000003 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.208928+0000", 2024-03-12T03:46:32.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021800690000000002 2024-03-12T03:46:32.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.231311+0000", 2024-03-12T03:46:32.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.022383713999999999 2024-03-12T03:46:32.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.231324+0000", 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2982e-05 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.250844+0000", 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.019519925 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.250904+0000", 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9821000000000002e-05 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:129167 7.15 7:a82f6f0d:::benchmark_data_smithi169_79822_object11750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:15.216355+0000", 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 557.06427949900001, 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0349462650000001, 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:46:32.294 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 129167 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216355+0000", 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216355+0000", 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216357+0000", 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1009999999999999e-06 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216363+0000", 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4940000000000003e-06 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216375+0000", 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2055999999999999e-05 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216384+0000", 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1279999999999997e-06 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216440+0000", 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6094999999999997e-05 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216568+0000", 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012828199999999999 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1,2", 2024-03-12T03:46:32.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.216853+0000", 2024-03-12T03:46:32.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028503600000000001 2024-03-12T03:46:32.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.218510+0000", 2024-03-12T03:46:32.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016566549999999999 2024-03-12T03:46:32.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.220906+0000", 2024-03-12T03:46:32.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023956540000000001 2024-03-12T03:46:32.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.231461+0000", 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010555787000000001 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.231472+0000", 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1031e-05 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.251263+0000", 2024-03-12T03:46:32.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0197909190000001 2024-03-12T03:46:32.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.251301+0000", 2024-03-12T03:46:32.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8027e-05 2024-03-12T03:46:32.300 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.300 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.300 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.301 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:129180 7.15 7:a934eea3:::benchmark_data_smithi169_79822_object11763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-12T03:46:32.301 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:15.237990+0000", 2024-03-12T03:46:32.301 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 557.04264443700004, 2024-03-12T03:46:32.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0297124339999999, 2024-03-12T03:46:32.301 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.301 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.301 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.301 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 129180 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.237990+0000", 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.237990+0000", 2024-03-12T03:46:32.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.237992+0000", 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.497e-06 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.237997+0000", 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8030000000000002e-06 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.238013+0000", 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5271e-05 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.238021+0000", 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9889999999999992e-06 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.238076+0000", 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5757999999999997e-05 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.238084+0000", 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8599999999999993e-06 2024-03-12T03:46:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.251326+0000", 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0132414750000001 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.251411+0000", 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5657999999999999e-05 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2", 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.251593+0000", 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018110700000000001 2024-03-12T03:46:32.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.255444+0000", 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038511219999999998 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.263712+0000", 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0082679350000000006 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267631+0000", 2024-03-12T03:46:32.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003919769 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267663+0000", 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1665999999999998e-05 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267703+0000", 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9524000000000003e-05 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:129218 7.15 7:ac84363a:::benchmark_data_smithi169_79822_object11801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:15.278835+0000", 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 557.00179913800002, 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98898992100000005, 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 129218 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278835+0000", 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278835+0000", 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278837+0000", 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.443e-06 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278842+0000", 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1410000000000004e-06 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278856+0000", 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4423000000000001e-05 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278865+0000", 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4409999999999993e-06 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278919+0000", 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4038e-05 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.278927+0000", 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2900000000000002e-06 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.252599+0000", 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97367206100000003 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.252718+0000", 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118445 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2", 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.252881+0000", 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016331700000000001 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.257985+0000", 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0051034950000000004 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.265507+0000", 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0075223210000000002 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267745+0000", 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022385999999999999 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267790+0000", 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4743000000000002e-05 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267825+0000", 2024-03-12T03:46:32.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5163000000000002e-05 2024-03-12T03:46:32.314 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.314 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.314 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.314 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:129221 7.15 7:abe42fdc:::benchmark_data_smithi169_79822_object11804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-12T03:46:32.314 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:15.285585+0000", 2024-03-12T03:46:32.314 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 556.99505010899998, 2024-03-12T03:46:32.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98232308099999999, 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 129221 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285585+0000", 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285585+0000", 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285586+0000", 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.44e-06 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285592+0000", 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6300000000000003e-06 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285605+0000", 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2934e-05 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285620+0000", 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5019e-05 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285680+0000", 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0352000000000001e-05 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.285689+0000", 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.611e-06 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.253492+0000", 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96780261499999998 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.253563+0000", 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0690000000000003e-05 2024-03-12T03:46:32.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2", 2024-03-12T03:46:32.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.253679+0000", 2024-03-12T03:46:32.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000116539 2024-03-12T03:46:32.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.257991+0000", 2024-03-12T03:46:32.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043116459999999997 2024-03-12T03:46:32.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.265538+0000", 2024-03-12T03:46:32.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0075468699999999998 2024-03-12T03:46:32.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267856+0000", 2024-03-12T03:46:32.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023183660000000001 2024-03-12T03:46:32.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267882+0000", 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6027e-05 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267908+0000", 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5341999999999999e-05 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.321 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:129271 7.15 7:ab2867bc:::benchmark_data_smithi169_79822_object11854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:15.337635+0000", 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 556.94299935100003, 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93035491299999995, 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 129271 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337635+0000", 2024-03-12T03:46:32.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337635+0000", 2024-03-12T03:46:32.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337637+0000", 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8390000000000001e-06 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337642+0000", 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3839999999999996e-06 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337653+0000", 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0139999999999999e-05 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337659+0000", 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8190000000000003e-06 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337720+0000", 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0788000000000001e-05 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:15.337729+0000", 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9500000000000007e-06 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.254290+0000", 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91656037300000004 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.254411+0000", 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012097200000000001 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2", 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.254623+0000", 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021296100000000001 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.261126+0000", 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0065024999999999996 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.265577+0000", 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044507549999999998 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267935+0000", 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023586200000000001 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267968+0000", 2024-03-12T03:46:32.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2815999999999999e-05 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:16.267990+0000", 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1996000000000001e-05 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:157476 7.11 7:8b1b26a8:::benchmark_data_smithi169_79822_object40059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:54.829219+0000", 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 517.45141546800005, 2024-03-12T03:46:32.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.939989089, 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 157476 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829219+0000", 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829219+0000", 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829220+0000", 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.004e-06 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829224+0000", 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8990000000000003e-06 2024-03-12T03:46:32.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829233+0000", 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2720000000000003e-06 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829239+0000", 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7180000000000001e-06 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829300+0000", 2024-03-12T03:46:32.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0857999999999999e-05 2024-03-12T03:46:32.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.829306+0000", 2024-03-12T03:46:32.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3289999999999999e-06 2024-03-12T03:46:32.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.765307+0000", 2024-03-12T03:46:32.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93600046199999998 2024-03-12T03:46:32.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.765388+0000", 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1761999999999994e-05 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.765626+0000", 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000237128 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769059+0000", 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003432928 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769100+0000", 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1183000000000002e-05 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769145+0000", 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5318999999999998e-05 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769176+0000", 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0868000000000003e-05 2024-03-12T03:46:32.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769208+0000", 2024-03-12T03:46:32.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2359e-05 2024-03-12T03:46:32.335 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.335 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.335 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.335 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:157480 7.11 7:8cdca059:::benchmark_data_smithi169_79822_object40063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-12T03:46:32.335 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:54.833037+0000", 2024-03-12T03:46:32.335 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 517.44759771199995, 2024-03-12T03:46:32.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94306616099999996, 2024-03-12T03:46:32.336 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.336 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.336 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.336 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.336 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.336 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 157480 2024-03-12T03:46:32.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.336 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833037+0000", 2024-03-12T03:46:32.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833037+0000", 2024-03-12T03:46:32.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833039+0000", 2024-03-12T03:46:32.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7179999999999999e-06 2024-03-12T03:46:32.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833044+0000", 2024-03-12T03:46:32.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5779999999999998e-06 2024-03-12T03:46:32.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833059+0000", 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4848000000000001e-05 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833068+0000", 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7220000000000005e-06 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833132+0000", 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4257999999999994e-05 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.833157+0000", 2024-03-12T03:46:32.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4570999999999999e-05 2024-03-12T03:46:32.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.767762+0000", 2024-03-12T03:46:32.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93460503100000003 2024-03-12T03:46:32.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.767831+0000", 2024-03-12T03:46:32.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9350999999999994e-05 2024-03-12T03:46:32.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:46:32.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.768013+0000", 2024-03-12T03:46:32.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018161499999999999 2024-03-12T03:46:32.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.772936+0000", 2024-03-12T03:46:32.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049235320000000004 2024-03-12T03:46:32.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.773001+0000", 2024-03-12T03:46:32.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4875000000000005e-05 2024-03-12T03:46:32.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776000+0000", 2024-03-12T03:46:32.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002998775 2024-03-12T03:46:32.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776056+0000", 2024-03-12T03:46:32.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6474000000000002e-05 2024-03-12T03:46:32.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:32.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776103+0000", 2024-03-12T03:46:32.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6813000000000001e-05 2024-03-12T03:46:32.345 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.345 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.345 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.346 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:157528 7.11 7:8ad61dec:::benchmark_data_smithi169_79822_object40111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-12T03:46:32.346 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:37:54.893117+0000", 2024-03-12T03:46:32.346 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 517.38751764699998, 2024-03-12T03:46:32.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88316590900000003, 2024-03-12T03:46:32.346 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.346 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.346 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.346 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.346 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.346 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 157528 2024-03-12T03:46:32.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.346 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893117+0000", 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893117+0000", 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893118+0000", 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2279999999999999e-06 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893123+0000", 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9520000000000002e-06 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893133+0000", 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0115000000000001e-05 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893141+0000", 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3100000000000003e-06 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893181+0000", 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0275000000000003e-05 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:54.893187+0000", 2024-03-12T03:46:32.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8839999999999999e-06 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769313+0000", 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87612656099999997 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769427+0000", 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000113409 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:46:32.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.769620+0000", 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000193556 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.773065+0000", 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034451809999999999 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.773080+0000", 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4368e-05 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776195+0000", 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0031153470000000001 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776241+0000", 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5568000000000001e-05 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:37:55.776283+0000", 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2154999999999999e-05 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.352 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:193831 7.8 7:17000021:::benchmark_data_smithi169_79822_object76414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:36.449932+0000", 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 475.830702309, 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97274706300000002, 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193831 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.449932+0000", 2024-03-12T03:46:32.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.449932+0000", 2024-03-12T03:46:32.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.449934+0000", 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.074e-06 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.449944+0000", 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1200000000000008e-06 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.449956+0000", 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.27e-05 2024-03-12T03:46:32.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.449974+0000", 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8247999999999999e-05 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.450110+0000", 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013577700000000001 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.450119+0000", 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9120000000000004e-06 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.417091+0000", 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96697190899999996 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.417182+0000", 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0586000000000003e-05 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.417473+0000", 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029137100000000001 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422452+0000", 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049790820000000001 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422512+0000", 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9846000000000003e-05 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422574+0000", 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2218999999999995e-05 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422636+0000", 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1651999999999999e-05 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422679+0000", 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3566999999999998e-05 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.359 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:193840 7.8 7:12e91ce1:::benchmark_data_smithi169_79822_object76423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:36.459371+0000", 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 475.82126397500002, 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96367646900000004, 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193840 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.459371+0000", 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.459371+0000", 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.459371+0000", 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1900000000000001e-07 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.459375+0000", 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7620000000000001e-06 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.459383+0000", 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9400000000000002e-06 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.459389+0000", 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2670000000000003e-06 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.460921+0000", 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015317810000000001 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.461000+0000", 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9171999999999993e-05 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.419254+0000", 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95825351700000005 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.419390+0000", 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013585600000000001 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.419603+0000", 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000213017 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422764+0000", 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003160726 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422819+0000", 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5849e-05 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422938+0000", 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119085 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.423002+0000", 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4008000000000001e-05 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.423047+0000", 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4669999999999998e-05 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:193879 7.8 7:16eafa45:::benchmark_data_smithi169_79822_object76462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:36.491804+0000", 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 475.788830679, 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93352429599999998, 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.366 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193879 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491804+0000", 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491804+0000", 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491805+0000", 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1319999999999999e-06 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491810+0000", 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8760000000000004e-06 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491821+0000", 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1094000000000001e-05 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491830+0000", 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5299999999999996e-06 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491868+0000", 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8293000000000001e-05 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.491878+0000", 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0455000000000001e-05 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.420888+0000", 2024-03-12T03:46:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.92901003500000001 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.420990+0000", 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101359 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.421191+0000", 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020108600000000001 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422883+0000", 2024-03-12T03:46:32.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016925880000000001 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.422897+0000", 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3202e-05 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.425124+0000", 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022270739999999999 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.425279+0000", 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015494599999999999 2024-03-12T03:46:32.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.425328+0000", 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9626000000000002e-05 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:193909 7.8 7:113cb578:::benchmark_data_smithi169_79822_object76492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:36.511598+0000", 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 475.76903700100002, 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91856142100000004, 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.372 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193909 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511598+0000", 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511598+0000", 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511600+0000", 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.905e-06 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511605+0000", 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0189999999999997e-06 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511616+0000", 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1584e-05 2024-03-12T03:46:32.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511624+0000", 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5410000000000003e-06 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511690+0000", 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6323999999999997e-05 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:36.511699+0000", 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2820000000000002e-06 2024-03-12T03:46:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.423085+0000", 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.911386166 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.423157+0000", 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1669999999999997e-05 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.423363+0000", 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000205427 2024-03-12T03:46:32.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.429984+0000", 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0066215149999999997 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.430007+0000", 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3006999999999998e-05 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.430052+0000", 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5311e-05 2024-03-12T03:46:32.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.430111+0000", 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9051000000000001e-05 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:37.430159+0000", 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7618999999999999e-05 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.378 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14799.0:214218 7.5 7:a5b8afe0:::benchmark_data_smithi169_79822_object96801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:38:55.695612+0000", 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 456.58502253299997, 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90713400899999996, 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14799", 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1727386922", 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 214218 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695612+0000", 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695612+0000", 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695613+0000", 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3239999999999999e-06 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695619+0000", 2024-03-12T03:46:32.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7259999999999999e-06 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695631+0000", 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2241e-05 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695638+0000", 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7399999999999998e-06 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695697+0000", 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9312999999999999e-05 2024-03-12T03:46:32.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:55.695708+0000", 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0505e-05 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.597283+0000", 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90157510900000004 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.597378+0000", 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5352000000000005e-05 2024-03-12T03:46:32.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.597649+0000", 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000271026 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.601493+0000", 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038440229999999998 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602663+0000", 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00116942 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602688+0000", 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5034000000000001e-05 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602720+0000", 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1809e-05 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:38:56.602746+0000", 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6387e-05 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.384 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:136482 9.1 9:8c6774bb:::benchmark_data_smithi169_135229_object136481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:17.113369+0000", 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 75.167265830000005, 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.020501748, 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136482 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.113369+0000", 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.113369+0000", 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.113370+0000", 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2419999999999999e-06 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.113412+0000", 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1980000000000001e-05 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.113422+0000", 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7079999999999992e-06 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.113430+0000", 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8679999999999999e-06 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.123492+0000", 2024-03-12T03:46:32.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010062786000000001 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.123504+0000", 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2065999999999999e-05 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.117188+0000", 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99368320499999996 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.118596+0000", 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00140876 2024-03-12T03:46:32.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.118803+0000", 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020672999999999999 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.131479+0000", 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012675906000000001 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.133719+0000", 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022399270000000001 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.133781+0000", 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1969000000000002e-05 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.133871+0000", 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9600999999999998e-05 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:136514 9.1 9:8a90bfc5:::benchmark_data_smithi169_135229_object136513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-12T03:46:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:17.162294+0000", 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 75.118340814999996, 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97167573699999998, 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136514 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162294+0000", 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162294+0000", 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162294+0000", 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3499999999999996e-07 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162329+0000", 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4301999999999998e-05 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162337+0000", 2024-03-12T03:46:32.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9920000000000007e-06 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162343+0000", 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.37e-06 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162883+0000", 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054012899999999998 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162898+0000", 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4354e-05 2024-03-12T03:46:32.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.126948+0000", 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96405053799999996 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.127103+0000", 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000155227 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.127270+0000", 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016620299999999999 2024-03-12T03:46:32.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.131536+0000", 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042664679999999998 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.133877+0000", 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023412929999999999 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.133911+0000", 2024-03-12T03:46:32.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3680999999999998e-05 2024-03-12T03:46:32.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.396 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.396 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.396 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.133970+0000", 2024-03-12T03:46:32.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8545000000000003e-05 2024-03-12T03:46:32.396 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.396 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:136534 9.1 9:8d1b151d:::benchmark_data_smithi169_135229_object136533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:17.178457+0000", 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 75.102177878999996, 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95566303200000002, 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136534 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178457+0000", 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178457+0000", 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178458+0000", 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1310000000000001e-06 2024-03-12T03:46:32.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178501+0000", 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3600000000000003e-05 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178512+0000", 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0899e-05 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178519+0000", 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0639999999999996e-06 2024-03-12T03:46:32.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178572+0000", 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2095999999999997e-05 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178579+0000", 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.943e-06 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.129362+0000", 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95078218400000003 2024-03-12T03:46:32.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.129533+0000", 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017161699999999999 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.129723+0000", 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018984199999999999 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.134020+0000", 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042966050000000002 2024-03-12T03:46:32.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.134037+0000", 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7572999999999999e-05 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.134071+0000", 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3800000000000002e-05 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.134120+0000", 2024-03-12T03:46:32.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8677999999999997e-05 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:136578 9.1 9:8d8890e7:::benchmark_data_smithi169_135229_object136577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:17.219682+0000", 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 75.060952416999996, 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91693437899999997, 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:46:32.403 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136578 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.219682+0000", 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.219682+0000", 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.219683+0000", 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.218e-06 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.219742+0000", 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8369999999999998e-05 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.219753+0000", 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1178999999999999e-05 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.219760+0000", 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9399999999999996e-06 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.220223+0000", 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00046343000000000003 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.220232+0000", 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0440000000000001e-06 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.131601+0000", 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91136816899999995 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.131751+0000", 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015015199999999999 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.131919+0000", 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016819200000000001 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.134165+0000", 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002246315 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.136416+0000", 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022507629999999998 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.136516+0000", 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9989999999999996e-05 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.136617+0000", 2024-03-12T03:46:32.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100617 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:136605 9.1 9:8a5baff4:::benchmark_data_smithi169_135229_object136604:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:17.244697+0000", 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 75.035937349999998, 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.903490817, 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:46:32.409 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136605 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.244697+0000", 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.244697+0000", 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.244698+0000", 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4600000000000003e-07 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.244742+0000", 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.401e-05 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.244757+0000", 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4918e-05 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.244765+0000", 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9149999999999996e-06 2024-03-12T03:46:32.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.269257+0000", 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024491947999999999 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.269267+0000", 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0022e-05 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.140509+0000", 2024-03-12T03:46:32.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87124190700000004 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.140640+0000", 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013155 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.140792+0000", 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000152103 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.145679+0000", 2024-03-12T03:46:32.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048862430000000002 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.148023+0000", 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023446180000000001 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.148086+0000", 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3010999999999997e-05 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.148188+0000", 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010172599999999999 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:136613 9.1 9:8d6f6f47:::benchmark_data_smithi169_135229_object136612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:17.254853+0000", 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 75.025781600000002, 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89725414599999997, 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:46:32.415 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136613 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.254853+0000", 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.254853+0000", 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.254855+0000", 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7400000000000001e-06 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.254882+0000", 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7569999999999999e-05 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.254894+0000", 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1285e-05 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.254901+0000", 2024-03-12T03:46:32.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5039999999999996e-06 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.272240+0000", 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.017338536000000002 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.272251+0000", 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1406000000000001e-05 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.143149+0000", 2024-03-12T03:46:32.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87089840200000002 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.143280+0000", 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013056799999999999 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.143433+0000", 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000153187 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.147989+0000", 2024-03-12T03:46:32.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045556110000000002 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.151966+0000", 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039771989999999998 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.152018+0000", 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1496000000000003e-05 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.152107+0000", 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9642000000000007e-05 2024-03-12T03:46:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.421 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.421 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.421 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:46:32.421 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:46:32.421 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:46:32.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:32.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:32.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:32.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:32.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:32.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:32.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:32.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:32 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-12T03:46:32.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:32 smithi169 ceph-mon[29344]: osdmap e337: 8 total, 8 up, 8 in 2024-03-12T03:46:32.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:32 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-12T03:46:32.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:32 smithi169 ceph-mon[39949]: osdmap e337: 8 total, 8 up, 8 in 2024-03-12T03:46:32.707 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:32.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:32.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:32 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-12T03:46:32.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:32 smithi173 ceph-mon[33544]: osdmap e337: 8 total, 8 up, 8 in 2024-03-12T03:46:32.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:32.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:32.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:32.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:32.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:32.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:33.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:33.124 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:33.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:33.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:33.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:33.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:33.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:33.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:46:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:46:33.353 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:33 smithi173 ceph-mon[33544]: pgmap v633: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 159 active+clean, 4 active+undersized+degraded+remapped+backfill_wait, 1 peering; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 31669/197166 objects degraded (16.062%); 20 MiB/s, 321 objects/s recovering 2024-03-12T03:46:33.353 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:33 smithi173 ceph-mon[33544]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:46:33.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:33.379 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:46:33.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:33.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:33.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:33.424 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:33.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:33.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:33.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:33.579 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:33 smithi169 ceph-mon[29344]: pgmap v633: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 159 active+clean, 4 active+undersized+degraded+remapped+backfill_wait, 1 peering; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 31669/197166 objects degraded (16.062%); 20 MiB/s, 321 objects/s recovering 2024-03-12T03:46:33.579 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:33 smithi169 ceph-mon[29344]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:46:33.580 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:33 smithi169 ceph-mon[39949]: pgmap v633: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 159 active+clean, 4 active+undersized+degraded+remapped+backfill_wait, 1 peering; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 31669/197166 objects degraded (16.062%); 20 MiB/s, 321 objects/s recovering 2024-03-12T03:46:33.580 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:33 smithi169 ceph-mon[39949]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:46:33.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:33.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:33.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:33.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:33.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:33.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:33.923 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:33.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-12T03:46:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:33.963 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:33.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:34.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:34.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:34.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:34.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:34.253 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:34.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:34.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:34.355 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:34.355 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:34 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:46:34.356 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:34 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:46:34.356 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:34 smithi169 ceph-mon[39949]: osdmap e338: 8 total, 8 up, 8 in 2024-03-12T03:46:34.356 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:34 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 31669/197166 objects degraded (16.062%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:34.356 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:34 smithi169 ceph-mon[39949]: osdmap e339: 8 total, 8 up, 8 in 2024-03-12T03:46:34.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:34.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:34.433 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:46:34.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:34.496 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:46:34.536 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:46:34.540 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:34.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:34.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:34.579 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:46:34.617 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:46:34.617 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:34.617 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:34 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:46:34.617 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:34 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:46:34.617 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:34 smithi169 ceph-mon[29344]: osdmap e338: 8 total, 8 up, 8 in 2024-03-12T03:46:34.617 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:34 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 31669/197166 objects degraded (16.062%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:34.617 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:34 smithi169 ceph-mon[29344]: osdmap e339: 8 total, 8 up, 8 in 2024-03-12T03:46:34.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:34.643 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:46:34.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:34.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:34.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:34.678 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:46:34.722 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:46:34.727 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:46:34.727 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:46:34.727 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:46:34.727 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:46:34.727 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:46:34.727 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:46:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:46:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:46:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:34 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:46:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:34 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:46:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:34 smithi173 ceph-mon[33544]: osdmap e338: 8 total, 8 up, 8 in 2024-03-12T03:46:34.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:34 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 31669/197166 objects degraded (16.062%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:34.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:34 smithi173 ceph-mon[33544]: osdmap e339: 8 total, 8 up, 8 in 2024-03-12T03:46:34.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:34.812 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:34.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:34.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:34.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:34.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:34.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:35.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:35.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:35.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:35.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:35.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:35.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:35.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:35.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:35.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:35.366 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:35.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:35.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:35.508 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:35.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:35.609 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:35.644 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:35.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:35.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:35.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:35 smithi169 ceph-mon[29344]: 5.a scrub starts 2024-03-12T03:46:35.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:35 smithi169 ceph-mon[29344]: 5.a scrub ok 2024-03-12T03:46:35.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:35 smithi169 ceph-mon[29344]: pgmap v636: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 30058/193860 objects degraded (15.505%); 34 MiB/s, 539 objects/s recovering 2024-03-12T03:46:35.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:35 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1967060125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:46:35.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:35 smithi169 ceph-mon[39949]: 5.a scrub starts 2024-03-12T03:46:35.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:35 smithi169 ceph-mon[39949]: 5.a scrub ok 2024-03-12T03:46:35.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:35 smithi169 ceph-mon[39949]: pgmap v636: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 30058/193860 objects degraded (15.505%); 34 MiB/s, 539 objects/s recovering 2024-03-12T03:46:35.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:35 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1967060125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:46:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:35 smithi173 ceph-mon[33544]: 5.a scrub starts 2024-03-12T03:46:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:35 smithi173 ceph-mon[33544]: 5.a scrub ok 2024-03-12T03:46:35.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:35 smithi173 ceph-mon[33544]: pgmap v636: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 30058/193860 objects degraded (15.505%); 34 MiB/s, 539 objects/s recovering 2024-03-12T03:46:35.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:35 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1967060125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:46:35.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:35.784 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:35.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:35.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:35.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:35.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:35.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:36.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:36.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:36.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:36.159 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:36.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:36.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:36.287 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:36.326 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:36.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:36.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:36.466 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:36.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:36.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:36.602 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:36.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:36.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:36.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:36.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:36.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:36.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:36.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:36.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:36.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:37.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:37.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:37.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:37.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:37.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:37.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:37.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:37.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:37.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:37.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:37.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:37.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:37.443 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:37.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:37.544 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:37.582 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:37.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:37.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:37 smithi169 ceph-mon[29344]: pgmap v637: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 886 B/s rd, 0 B/s wr, 1.00k op/s; 30058/193860 objects degraded (15.505%); 23 MiB/s, 374 objects/s recovering 2024-03-12T03:46:37.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:37 smithi169 ceph-mon[39949]: pgmap v637: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 886 B/s rd, 0 B/s wr, 1.00k op/s; 30058/193860 objects degraded (15.505%); 23 MiB/s, 374 objects/s recovering 2024-03-12T03:46:37.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:37.718 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:37.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:37 smithi173 ceph-mon[33544]: pgmap v637: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 886 B/s rd, 0 B/s wr, 1.00k op/s; 30058/193860 objects degraded (15.505%); 23 MiB/s, 374 objects/s recovering 2024-03-12T03:46:37.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:37.849 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:37.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:37.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:37.987 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:37.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:38.088 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:38.121 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:38.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:38.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:38.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:38.247 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:38.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:38.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:38.387 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:38.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:38.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:38.525 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:38.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:38.627 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:38.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:38.665 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:38.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:38.766 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:38.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:38.808 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:38.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:38.909 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:38.951 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:38.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:38.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:39.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:39.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:39.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:39.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:39.223 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:39.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:39.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:39.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:39.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:39.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:39.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:39.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:39.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:39.592 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:39.632 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:39.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:39.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:39 smithi169 ceph-mon[29344]: 6.f scrub starts 2024-03-12T03:46:39.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:39 smithi169 ceph-mon[29344]: 6.f scrub ok 2024-03-12T03:46:39.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:39 smithi169 ceph-mon[29344]: pgmap v638: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 758 B/s rd, 0 B/s wr, 1.49k op/s; 25367/175860 objects degraded (14.425%); 27 MiB/s, 434 objects/s recovering 2024-03-12T03:46:39.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:39 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 25367/175860 objects degraded (14.425%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:39.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:39 smithi169 ceph-mon[39949]: 6.f scrub starts 2024-03-12T03:46:39.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:39 smithi169 ceph-mon[39949]: 6.f scrub ok 2024-03-12T03:46:39.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:39 smithi169 ceph-mon[39949]: pgmap v638: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 758 B/s rd, 0 B/s wr, 1.49k op/s; 25367/175860 objects degraded (14.425%); 27 MiB/s, 434 objects/s recovering 2024-03-12T03:46:39.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:39 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 25367/175860 objects degraded (14.425%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:39.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:39 smithi173 ceph-mon[33544]: 6.f scrub starts 2024-03-12T03:46:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:39 smithi173 ceph-mon[33544]: 6.f scrub ok 2024-03-12T03:46:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:39 smithi173 ceph-mon[33544]: pgmap v638: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 758 B/s rd, 0 B/s wr, 1.49k op/s; 25367/175860 objects degraded (14.425%); 27 MiB/s, 434 objects/s recovering 2024-03-12T03:46:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:39 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 25367/175860 objects degraded (14.425%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:39.765 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:39.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:39.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:39.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:39.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:39.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:40.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:40.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:40.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:40.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:40.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:40.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:40.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:40.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:40.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:40.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:40.455 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:40.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:40.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:40.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:40.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:40.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:40.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:40.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:40.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:40.734 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:40.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:40.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:40.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:40.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:40.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:40.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:41.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:41.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:41.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:41.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:41.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:41.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:41.248 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:41.286 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:41.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:41.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:41.434 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:41.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:41 smithi173 ceph-mon[33544]: pgmap v639: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 888 op/s; 25367/175860 objects degraded (14.425%); 23 MiB/s, 366 objects/s recovering 2024-03-12T03:46:41.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:41.572 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:41.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:41.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:41 smithi169 ceph-mon[29344]: pgmap v639: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 888 op/s; 25367/175860 objects degraded (14.425%); 23 MiB/s, 366 objects/s recovering 2024-03-12T03:46:41.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:41 smithi169 ceph-mon[39949]: pgmap v639: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 888 op/s; 25367/175860 objects degraded (14.425%); 23 MiB/s, 366 objects/s recovering 2024-03-12T03:46:41.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:41.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:41.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:41.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:41.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:41.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:41.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:41.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:41.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:41.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:41.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:42.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:42.117 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:42.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:42.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:46:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:46:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:46:42.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:42.256 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:42.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:42.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:42.393 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:42.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:42.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:42.494 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:42.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:42.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:42.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:42.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:42.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:42.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:42.803 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:42.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:42.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:42.928 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:42.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:43.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:43.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:43.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:43.167 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:43.202 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:43.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:43.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:43.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:43.339 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:43.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:43.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:43.376 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:46:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:46:43.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:43.476 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:43.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:43.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:43.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:43.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:43.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:43.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:43 smithi169 ceph-mon[29344]: pgmap v640: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s; 21959/161277 objects degraded (13.616%); 25 MiB/s, 398 objects/s recovering 2024-03-12T03:46:43.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:43 smithi169 ceph-mon[39949]: pgmap v640: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s; 21959/161277 objects degraded (13.616%); 25 MiB/s, 398 objects/s recovering 2024-03-12T03:46:43.710 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:43 smithi173 ceph-mon[33544]: pgmap v640: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s; 21959/161277 objects degraded (13.616%); 25 MiB/s, 398 objects/s recovering 2024-03-12T03:46:43.748 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:43.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:43.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:43.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:43.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:43.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:43.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:44.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:44.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:44.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:44.128 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:44.158 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:44.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:44.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:44.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:44.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:44.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:44.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:44.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:44.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:44.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:44.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:44.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:44.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:44.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:44.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:44 smithi169 ceph-mon[29344]: 2.1 scrub starts 2024-03-12T03:46:44.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:44 smithi169 ceph-mon[29344]: 2.1 scrub ok 2024-03-12T03:46:44.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:44.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:44 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 21959/161277 objects degraded (13.616%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:44.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:44 smithi169 ceph-mon[39949]: 2.1 scrub starts 2024-03-12T03:46:44.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:44 smithi169 ceph-mon[39949]: 2.1 scrub ok 2024-03-12T03:46:44.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:44.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:44 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 21959/161277 objects degraded (13.616%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:44.702 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:44.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:44 smithi173 ceph-mon[33544]: 2.1 scrub starts 2024-03-12T03:46:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:44 smithi173 ceph-mon[33544]: 2.1 scrub ok 2024-03-12T03:46:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:44 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 21959/161277 objects degraded (13.616%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:44.804 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:44.866 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:44.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:44.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:45.011 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:45.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:45.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:45.112 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:45.151 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:45.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:45.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:45.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:45.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:45.350 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:45.355 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:45.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:45.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:45.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:45.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:45.497 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:45 smithi169 ceph-mon[29344]: 4.1c scrub starts 2024-03-12T03:46:45.498 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:45 smithi169 ceph-mon[29344]: 4.1c scrub ok 2024-03-12T03:46:45.498 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:45 smithi169 ceph-mon[29344]: pgmap v641: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 959 B/s rd, 0 B/s wr, 1.12k op/s; 20806/157965 objects degraded (13.171%); 24 MiB/s, 385 objects/s recovering 2024-03-12T03:46:45.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:45 smithi169 ceph-mon[39949]: 4.1c scrub starts 2024-03-12T03:46:45.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:45 smithi169 ceph-mon[39949]: 4.1c scrub ok 2024-03-12T03:46:45.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:45 smithi169 ceph-mon[39949]: pgmap v641: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 959 B/s rd, 0 B/s wr, 1.12k op/s; 20806/157965 objects degraded (13.171%); 24 MiB/s, 385 objects/s recovering 2024-03-12T03:46:45.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:45.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:45.576 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:45.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:45.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:45.727 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:45.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:45 smithi173 ceph-mon[33544]: 4.1c scrub starts 2024-03-12T03:46:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:45 smithi173 ceph-mon[33544]: 4.1c scrub ok 2024-03-12T03:46:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:45 smithi173 ceph-mon[33544]: pgmap v641: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 959 B/s rd, 0 B/s wr, 1.12k op/s; 20806/157965 objects degraded (13.171%); 24 MiB/s, 385 objects/s recovering 2024-03-12T03:46:45.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:45.869 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:45.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:45.970 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:46.014 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:46.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:46.116 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:46.155 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:46.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:46.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:46.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:46.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:46.394 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:46.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:46.436 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:46.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:46.437 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.1 to 1 (10000) 2024-03-12T03:46:46.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:46.575 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:46.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:46.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:46.719 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:46.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:46.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:46:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:46 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:46:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:46 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:46:46.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:46 smithi173 ceph-mon[33544]: osdmap e340: 8 total, 8 up, 8 in 2024-03-12T03:46:46.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3541158362' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:46:46.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:46 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:46:46.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:46 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:46.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:46 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:46.782 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:46:46.782 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:46:46.783 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[29344]: osdmap e340: 8 total, 8 up, 8 in 2024-03-12T03:46:46.783 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3541158362' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:46:46.783 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:46:46.783 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:46.783 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:46.783 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:46:46.783 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:46:46.783 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[39949]: osdmap e340: 8 total, 8 up, 8 in 2024-03-12T03:46:46.783 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3541158362' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:46:46.783 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:46:46.784 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:46:46.784 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:46 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:46.821 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:46.863 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:46.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:46.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:46.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:47.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:47.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:47.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:47.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:47.146 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:47.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:47.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:47.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:47.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:47.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:47.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:47.417 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:47.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:47.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:47.555 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:47.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:47.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:47.657 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:47.690 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:47.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:47 smithi173 ceph-mon[33544]: pgmap v643: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s; 20806/157965 objects degraded (13.171%); 26 MiB/s, 410 objects/s recovering 2024-03-12T03:46:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:47 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-12T03:46:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:47 smithi173 ceph-mon[33544]: osdmap e341: 8 total, 8 up, 8 in 2024-03-12T03:46:47.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:47.831 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:47.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:47 smithi169 ceph-mon[29344]: pgmap v643: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s; 20806/157965 objects degraded (13.171%); 26 MiB/s, 410 objects/s recovering 2024-03-12T03:46:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:47 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-12T03:46:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:47 smithi169 ceph-mon[29344]: osdmap e341: 8 total, 8 up, 8 in 2024-03-12T03:46:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:47 smithi169 ceph-mon[39949]: pgmap v643: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s; 20806/157965 objects degraded (13.171%); 26 MiB/s, 410 objects/s recovering 2024-03-12T03:46:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:47 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-12T03:46:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:47 smithi169 ceph-mon[39949]: osdmap e341: 8 total, 8 up, 8 in 2024-03-12T03:46:47.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:47.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:47.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:48.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:48.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:48.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:48.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:48.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:48.249 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:48.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:48.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:48.390 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:48.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:48.459 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-12T03:46:48.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:48.521 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:48.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:48.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:48.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:48.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:48.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:48.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:48.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:48 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 20806/157965 objects degraded (13.171%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:48.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:48 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 20806/157965 objects degraded (13.171%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:48.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:48 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 20806/157965 objects degraded (13.171%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:48.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:48.802 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:48.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:48.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:48.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:48.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:49.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:49.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:49.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:49.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:49.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:49.219 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:49.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:49.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:49.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:49.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:49.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:49.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:49.512 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:49.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:49.524 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:49.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:49.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:49.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:49.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:49.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:49 smithi173 ceph-mon[33544]: 5.d scrub starts 2024-03-12T03:46:49.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:49 smithi173 ceph-mon[33544]: 5.d scrub ok 2024-03-12T03:46:49.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:49 smithi173 ceph-mon[33544]: pgmap v645: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 16593/140376 objects degraded (11.820%); 32 MiB/s, 509 objects/s recovering 2024-03-12T03:46:49.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:49.768 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:49.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:49.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:49.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:49 smithi169 ceph-mon[29344]: 5.d scrub starts 2024-03-12T03:46:49.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:49 smithi169 ceph-mon[29344]: 5.d scrub ok 2024-03-12T03:46:49.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:49 smithi169 ceph-mon[29344]: pgmap v645: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 16593/140376 objects degraded (11.820%); 32 MiB/s, 509 objects/s recovering 2024-03-12T03:46:49.790 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:49 smithi169 ceph-mon[39949]: 5.d scrub starts 2024-03-12T03:46:49.790 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:49 smithi169 ceph-mon[39949]: 5.d scrub ok 2024-03-12T03:46:49.790 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:49 smithi169 ceph-mon[39949]: pgmap v645: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 16593/140376 objects degraded (11.820%); 32 MiB/s, 509 objects/s recovering 2024-03-12T03:46:49.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:49.936 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:49.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:50.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:50.077 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:50.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:50.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:50.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:50.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:50.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:50.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:50.344 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:50.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:50.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:50.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:50.484 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:50.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:50.586 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:50.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:50.628 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:50.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:50.713 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:46:50.713 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:46:50.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:50.730 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:50 smithi169 ceph-mon[29344]: 6.14 scrub starts 2024-03-12T03:46:50.730 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:50 smithi169 ceph-mon[29344]: 6.14 scrub ok 2024-03-12T03:46:50.730 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:50 smithi169 ceph-mon[39949]: 6.14 scrub starts 2024-03-12T03:46:50.731 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:50 smithi169 ceph-mon[39949]: 6.14 scrub ok 2024-03-12T03:46:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:50 smithi173 ceph-mon[33544]: 6.14 scrub starts 2024-03-12T03:46:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:50 smithi173 ceph-mon[33544]: 6.14 scrub ok 2024-03-12T03:46:50.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:50.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:50.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:50.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:50.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:50.905 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:50.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:51.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:51.044 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:51.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:51.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:51.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:51.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:51.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:51.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:51.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:51.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:51.455 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:51.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:51.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:51.587 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:51.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:51.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:51.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-12T03:46:51.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:51.712 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:51.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:51.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:51.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:51 smithi169 ceph-mon[29344]: pgmap v646: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 871 op/s; 16593/140376 objects degraded (11.820%); 21 MiB/s, 328 objects/s recovering 2024-03-12T03:46:51.742 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:51 smithi169 ceph-mon[39949]: pgmap v646: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 871 op/s; 16593/140376 objects degraded (11.820%); 21 MiB/s, 328 objects/s recovering 2024-03-12T03:46:51.813 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:51.843 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:51.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:51.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:51.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:51.974 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:51.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:51 smithi173 ceph-mon[33544]: pgmap v646: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 4 active+undersized+degraded+remapped+backfill_wait; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 871 op/s; 16593/140376 objects degraded (11.820%); 21 MiB/s, 328 objects/s recovering 2024-03-12T03:46:52.018 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:46:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:46:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:46:52.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:52.106 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:52.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:52.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:52.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:52.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:52.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:52.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:52.363 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:52.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:52.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:52.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:52.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:52.591 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:52.634 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:52.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:52.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:52.776 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:52.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:52.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:52.915 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:52.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:52.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:52 smithi169 ceph-mon[29344]: osdmap e342: 8 total, 8 up, 8 in 2024-03-12T03:46:52.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:52 smithi169 ceph-mon[29344]: 6.5 scrub starts 2024-03-12T03:46:52.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:52 smithi169 ceph-mon[29344]: 6.5 scrub ok 2024-03-12T03:46:52.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:52 smithi169 ceph-mon[39949]: osdmap e342: 8 total, 8 up, 8 in 2024-03-12T03:46:52.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:52 smithi169 ceph-mon[39949]: 6.5 scrub starts 2024-03-12T03:46:52.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:52 smithi169 ceph-mon[39949]: 6.5 scrub ok 2024-03-12T03:46:52.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:52 smithi173 ceph-mon[33544]: osdmap e342: 8 total, 8 up, 8 in 2024-03-12T03:46:52.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:52 smithi173 ceph-mon[33544]: 6.5 scrub starts 2024-03-12T03:46:52.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:52 smithi173 ceph-mon[33544]: 6.5 scrub ok 2024-03-12T03:46:53.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:53.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:53.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:53.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:53.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:53.202 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:53.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:53.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:53.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:53.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:53.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:53.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:53.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:53.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:53.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:53.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:46:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:46:53.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:53.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:53.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:53.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:53.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:53.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:53.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:53.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:53.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[29344]: pgmap v648: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 2 active+undersized+degraded+remapped+backfill_wait; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.66k op/s; 13209/124896 objects degraded (10.576%); 35 MiB/s, 554 objects/s recovering 2024-03-12T03:46:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[29344]: 5.1 scrub starts 2024-03-12T03:46:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[29344]: 5.1 scrub ok 2024-03-12T03:46:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[29344]: osdmap e343: 8 total, 8 up, 8 in 2024-03-12T03:46:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[29344]: 1.9 scrub starts 2024-03-12T03:46:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[29344]: 1.9 scrub ok 2024-03-12T03:46:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 13209/124896 objects degraded (10.576%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[39949]: pgmap v648: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 2 active+undersized+degraded+remapped+backfill_wait; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.66k op/s; 13209/124896 objects degraded (10.576%); 35 MiB/s, 554 objects/s recovering 2024-03-12T03:46:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[39949]: 5.1 scrub starts 2024-03-12T03:46:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[39949]: 5.1 scrub ok 2024-03-12T03:46:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[39949]: osdmap e343: 8 total, 8 up, 8 in 2024-03-12T03:46:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[39949]: 1.9 scrub starts 2024-03-12T03:46:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[39949]: 1.9 scrub ok 2024-03-12T03:46:53.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:53 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 13209/124896 objects degraded (10.576%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:53 smithi173 ceph-mon[33544]: pgmap v648: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfilling, 160 active+clean, 2 active+undersized+degraded+remapped+backfill_wait; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.66k op/s; 13209/124896 objects degraded (10.576%); 35 MiB/s, 554 objects/s recovering 2024-03-12T03:46:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:53 smithi173 ceph-mon[33544]: 5.1 scrub starts 2024-03-12T03:46:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:53 smithi173 ceph-mon[33544]: 5.1 scrub ok 2024-03-12T03:46:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:53 smithi173 ceph-mon[33544]: osdmap e343: 8 total, 8 up, 8 in 2024-03-12T03:46:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:53 smithi173 ceph-mon[33544]: 1.9 scrub starts 2024-03-12T03:46:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:53 smithi173 ceph-mon[33544]: 1.9 scrub ok 2024-03-12T03:46:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:53 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 13209/124896 objects degraded (10.576%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:53.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:54.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:54.040 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:54.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:54.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:54.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:54.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:54.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:54.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:54.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:54.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:54.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:54.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:54.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:54.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:54.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:54.596 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:54.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:54.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:54 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:54.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:54 smithi169 ceph-mon[29344]: 5.1c scrub starts 2024-03-12T03:46:54.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:54 smithi169 ceph-mon[29344]: 5.1c scrub ok 2024-03-12T03:46:54.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:54 smithi169 ceph-mon[29344]: 3.15 scrub starts 2024-03-12T03:46:54.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:54 smithi169 ceph-mon[29344]: 3.15 scrub ok 2024-03-12T03:46:54.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:54 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:54.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:54 smithi169 ceph-mon[39949]: 5.1c scrub starts 2024-03-12T03:46:54.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:54 smithi169 ceph-mon[39949]: 5.1c scrub ok 2024-03-12T03:46:54.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:54 smithi169 ceph-mon[39949]: 3.15 scrub starts 2024-03-12T03:46:54.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:54 smithi169 ceph-mon[39949]: 3.15 scrub ok 2024-03-12T03:46:54.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:54.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:54.739 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:54.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:54.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:54.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:54.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:54.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:54 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:54 smithi173 ceph-mon[33544]: 5.1c scrub starts 2024-03-12T03:46:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:54 smithi173 ceph-mon[33544]: 5.1c scrub ok 2024-03-12T03:46:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:54 smithi173 ceph-mon[33544]: 3.15 scrub starts 2024-03-12T03:46:54.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:54 smithi173 ceph-mon[33544]: 3.15 scrub ok 2024-03-12T03:46:55.003 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:55.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:55.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:55.153 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:55.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:55.255 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:55.297 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:55.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:55.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:55.442 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:55.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:55.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:55.556 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:46:55.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:55.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:55.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:55.608 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:46:55.618 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:46:55.659 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:46:55.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:55.707 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:46:55.713 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:55.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:55.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:55.750 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:46:55.750 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:55 smithi169 ceph-mon[29344]: pgmap v650: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean, 1 active+undersized+degraded+remapped+backfill_wait; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 668 B/s rd, 0 B/s wr, 1.17k op/s; 11891/121626 objects degraded (9.777%); 27 MiB/s, 429 objects/s recovering 2024-03-12T03:46:55.750 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:55 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/494933936' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:46:55.750 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:55 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:46:55.750 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:55 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2187792571' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:46:55.750 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:55 smithi169 ceph-mon[39949]: pgmap v650: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean, 1 active+undersized+degraded+remapped+backfill_wait; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 668 B/s rd, 0 B/s wr, 1.17k op/s; 11891/121626 objects degraded (9.777%); 27 MiB/s, 429 objects/s recovering 2024-03-12T03:46:55.751 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:55 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/494933936' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:46:55.751 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:55 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:46:55.751 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:55 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2187792571' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:46:55.792 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:46:55.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:55.835 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:55.845 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:46:55.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:55.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:55.887 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:46:55.889 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:46:55.889 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:46:55.889 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:46:55.890 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:46:55.890 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:46:55.890 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:46:55.890 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:46:55.890 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:46:55.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:55 smithi173 ceph-mon[33544]: pgmap v650: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean, 1 active+undersized+degraded+remapped+backfill_wait; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 668 B/s rd, 0 B/s wr, 1.17k op/s; 11891/121626 objects degraded (9.777%); 27 MiB/s, 429 objects/s recovering 2024-03-12T03:46:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:55 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/494933936' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:46:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:55 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:46:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:55 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2187792571' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:46:56.005 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:56.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:56.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:56.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:56.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:56.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:56.240 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:56.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:56.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:56.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:56.376 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:56.423 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:56.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:56.524 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:56.565 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:56.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:56.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:56.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:56.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:56.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:56.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:56.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:56.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:56.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:56.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:56 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:46:56.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:56 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:46:56.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:56 smithi169 ceph-mon[29344]: osdmap e344: 8 total, 8 up, 8 in 2024-03-12T03:46:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:56 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:46:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:56 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:46:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:56 smithi169 ceph-mon[39949]: osdmap e344: 8 total, 8 up, 8 in 2024-03-12T03:46:56.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:56.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:56.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:56 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:46:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:56 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:46:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:56 smithi173 ceph-mon[33544]: osdmap e344: 8 total, 8 up, 8 in 2024-03-12T03:46:57.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:57.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:57.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:57.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:57.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:57.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:57.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:57.352 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:57.398 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:57.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:57.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:46:57.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:57.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:57.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:46:57.603 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.3 to 1 (10000) 2024-03-12T03:46:57.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:57.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:57.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:57.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:57.827 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:57.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:57.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:57 smithi169 ceph-mon[29344]: pgmap v652: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean, 1 active+undersized+degraded+remapped+backfill_wait; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s; 11891/121626 objects degraded (9.777%); 28 MiB/s, 448 objects/s recovering 2024-03-12T03:46:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:57 smithi169 ceph-mon[29344]: 6.2 deep-scrub starts 2024-03-12T03:46:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:57 smithi169 ceph-mon[29344]: 6.2 deep-scrub ok 2024-03-12T03:46:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:57 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1258983592' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:46:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:57 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:46:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:57 smithi169 ceph-mon[39949]: pgmap v652: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean, 1 active+undersized+degraded+remapped+backfill_wait; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s; 11891/121626 objects degraded (9.777%); 28 MiB/s, 448 objects/s recovering 2024-03-12T03:46:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:57 smithi169 ceph-mon[39949]: 6.2 deep-scrub starts 2024-03-12T03:46:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:57 smithi169 ceph-mon[39949]: 6.2 deep-scrub ok 2024-03-12T03:46:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:57 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1258983592' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:46:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:57 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:46:57.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:57.972 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:57.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:57.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:57 smithi173 ceph-mon[33544]: pgmap v652: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean, 1 active+undersized+degraded+remapped+backfill_wait; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s; 11891/121626 objects degraded (9.777%); 28 MiB/s, 448 objects/s recovering 2024-03-12T03:46:57.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:57 smithi173 ceph-mon[33544]: 6.2 deep-scrub starts 2024-03-12T03:46:57.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:57 smithi173 ceph-mon[33544]: 6.2 deep-scrub ok 2024-03-12T03:46:57.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:57 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1258983592' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:46:57.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:57 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:46:58.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:46:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:58.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:58.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:46:58.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:58.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:58.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:58.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:58.352 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:46:58.391 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:58.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:46:58.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:58.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:58.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:58.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:58.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:58.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:58.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:58.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[29344]: 4.c deep-scrub starts 2024-03-12T03:46:58.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[29344]: 4.c deep-scrub ok 2024-03-12T03:46:58.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-12T03:46:58.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[29344]: osdmap e345: 8 total, 8 up, 8 in 2024-03-12T03:46:58.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[29344]: 6.8 scrub starts 2024-03-12T03:46:58.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[29344]: 6.8 scrub ok 2024-03-12T03:46:58.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[29344]: 5.8 scrub starts 2024-03-12T03:46:58.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[29344]: 5.8 scrub ok 2024-03-12T03:46:58.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 11891/121626 objects degraded (9.777%), 5 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:58.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[39949]: 4.c deep-scrub starts 2024-03-12T03:46:58.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[39949]: 4.c deep-scrub ok 2024-03-12T03:46:58.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-12T03:46:58.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[39949]: osdmap e345: 8 total, 8 up, 8 in 2024-03-12T03:46:58.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[39949]: 6.8 scrub starts 2024-03-12T03:46:58.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[39949]: 6.8 scrub ok 2024-03-12T03:46:58.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[39949]: 5.8 scrub starts 2024-03-12T03:46:58.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[39949]: 5.8 scrub ok 2024-03-12T03:46:58.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:58 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 11891/121626 objects degraded (9.777%), 5 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:58.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:46:58.765 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:58.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:58.847 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:58.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:46:58.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:58 smithi173 ceph-mon[33544]: 4.c deep-scrub starts 2024-03-12T03:46:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:58 smithi173 ceph-mon[33544]: 4.c deep-scrub ok 2024-03-12T03:46:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:58 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-12T03:46:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:58 smithi173 ceph-mon[33544]: osdmap e345: 8 total, 8 up, 8 in 2024-03-12T03:46:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:58 smithi173 ceph-mon[33544]: 6.8 scrub starts 2024-03-12T03:46:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:58 smithi173 ceph-mon[33544]: 6.8 scrub ok 2024-03-12T03:46:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:58 smithi173 ceph-mon[33544]: 5.8 scrub starts 2024-03-12T03:46:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:58 smithi173 ceph-mon[33544]: 5.8 scrub ok 2024-03-12T03:46:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:58 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 11891/121626 objects degraded (9.777%), 5 pgs degraded (PG_DEGRADED) 2024-03-12T03:46:59.023 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:59.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:59.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:59.125 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:46:59.166 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:59.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:59.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:46:59.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:46:59.267 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:59.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-12T03:46:59.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:59.312 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:59.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:59.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:59.478 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:59.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:59.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:46:59.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:59.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:46:59.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:46:59.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:46:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:59.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:46:59.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:46:59.872 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:46:59 smithi169 ceph-mon[29344]: pgmap v654: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 2.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 963 op/s; 8223/107562 objects degraded (7.645%); 35 MiB/s, 554 objects/s recovering 2024-03-12T03:46:59.873 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:46:59 smithi169 ceph-mon[39949]: pgmap v654: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 2.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 963 op/s; 8223/107562 objects degraded (7.645%); 35 MiB/s, 554 objects/s recovering 2024-03-12T03:46:59.886 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:46:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:46:59.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:46:59.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:46:59 smithi173 ceph-mon[33544]: pgmap v654: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 2.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 963 op/s; 8223/107562 objects degraded (7.645%); 35 MiB/s, 554 objects/s recovering 2024-03-12T03:46:59.980 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:46:59.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:46:59.989 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:46:59.989 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:46:59.989 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-03-12T03:46:59.989 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-03-12T03:46:59.989 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:47:00.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:00.064 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:00.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:00.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:00.207 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:00.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:00.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:00.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:00.354 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:00.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:00.456 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:00.492 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:00.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:00.503 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:00.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:00.527 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:00.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:00.657 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:00.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:00.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:00.792 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:00.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:00.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:00.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:00.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:01.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:01.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:01.069 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:01.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:01.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:01.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:01.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:01.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:01.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:01.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:01.343 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:01.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:01.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:01.484 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:01.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:01.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:01.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:01.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:01.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:01.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:01.756 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:01.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:01.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-mon[29344]: pgmap v655: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 2.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 675 B/s rd, 0 B/s wr, 868 op/s; 8223/105162 objects degraded (7.819%); 27 MiB/s, 439 objects/s recovering 2024-03-12T03:47:01.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-mon[29344]: 4.9 scrub starts 2024-03-12T03:47:01.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-mon[29344]: 4.9 scrub ok 2024-03-12T03:47:01.827 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-mon[29344]: 9.3 scrub starts 2024-03-12T03:47:01.827 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:47:01.827 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:01.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-mon[39949]: pgmap v655: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 2.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 675 B/s rd, 0 B/s wr, 868 op/s; 8223/105162 objects degraded (7.819%); 27 MiB/s, 439 objects/s recovering 2024-03-12T03:47:01.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-mon[39949]: 4.9 scrub starts 2024-03-12T03:47:01.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-mon[39949]: 4.9 scrub ok 2024-03-12T03:47:01.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-mon[39949]: 9.3 scrub starts 2024-03-12T03:47:01.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:47:01.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:01.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:01.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:01 smithi173 ceph-mon[33544]: pgmap v655: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing+deep, 3 active+undersized+degraded+remapped+backfilling, 162 active+clean; 2.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 675 B/s rd, 0 B/s wr, 868 op/s; 8223/105162 objects degraded (7.819%); 27 MiB/s, 439 objects/s recovering 2024-03-12T03:47:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:01 smithi173 ceph-mon[33544]: 4.9 scrub starts 2024-03-12T03:47:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:01 smithi173 ceph-mon[33544]: 4.9 scrub ok 2024-03-12T03:47:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:01 smithi173 ceph-mon[33544]: 9.3 scrub starts 2024-03-12T03:47:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:01 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:47:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:01 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:01.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:02.035 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:02.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:02.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:02.149 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:47:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:47:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:47:02.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:02.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:02.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:02.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:02.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:02.309 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:02.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:02.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:02.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:02.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:02.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:02.632 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-12T03:47:02.647 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:02.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:02.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:02.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:02.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:02.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:02.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:02.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:02.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:02.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:02 smithi169 ceph-mon[29344]: 9.3 scrub ok 2024-03-12T03:47:02.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:02 smithi169 ceph-mon[39949]: 9.3 scrub ok 2024-03-12T03:47:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:02 smithi173 ceph-mon[33544]: 9.3 scrub ok 2024-03-12T03:47:03.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:03.070 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:03.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:03.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:03.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:03.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:03.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:03.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:03.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:03.345 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:03.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:03.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:03.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:03.481 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:03.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:03.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:47:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:47:03.583 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:03.615 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:03.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:03.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:03.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:03.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:03.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:03.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:03.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:03.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:03.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:03 smithi169 ceph-mon[29344]: pgmap v656: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 163 active+clean; 1.8 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 5794/91026 objects degraded (6.365%); 28 MiB/s, 451 objects/s recovering 2024-03-12T03:47:03.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:03 smithi169 ceph-mon[29344]: 4.1a deep-scrub starts 2024-03-12T03:47:03.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:03 smithi169 ceph-mon[29344]: 4.1a deep-scrub ok 2024-03-12T03:47:03.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:03 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 5794/91026 objects degraded (6.365%), 5 pgs degraded (PG_DEGRADED) 2024-03-12T03:47:03.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:03 smithi169 ceph-mon[29344]: osdmap e346: 8 total, 8 up, 8 in 2024-03-12T03:47:03.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:03 smithi169 ceph-mon[39949]: pgmap v656: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 163 active+clean; 1.8 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 5794/91026 objects degraded (6.365%); 28 MiB/s, 451 objects/s recovering 2024-03-12T03:47:03.905 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:03 smithi169 ceph-mon[39949]: 4.1a deep-scrub starts 2024-03-12T03:47:03.905 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:03 smithi169 ceph-mon[39949]: 4.1a deep-scrub ok 2024-03-12T03:47:03.905 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:03 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 5794/91026 objects degraded (6.365%), 5 pgs degraded (PG_DEGRADED) 2024-03-12T03:47:03.905 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:03 smithi169 ceph-mon[39949]: osdmap e346: 8 total, 8 up, 8 in 2024-03-12T03:47:03.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:03 smithi173 ceph-mon[33544]: pgmap v656: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 163 active+clean; 1.8 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 5794/91026 objects degraded (6.365%); 28 MiB/s, 451 objects/s recovering 2024-03-12T03:47:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:03 smithi173 ceph-mon[33544]: 4.1a deep-scrub starts 2024-03-12T03:47:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:03 smithi173 ceph-mon[33544]: 4.1a deep-scrub ok 2024-03-12T03:47:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:03 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 5794/91026 objects degraded (6.365%), 5 pgs degraded (PG_DEGRADED) 2024-03-12T03:47:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:03 smithi173 ceph-mon[33544]: osdmap e346: 8 total, 8 up, 8 in 2024-03-12T03:47:03.997 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:03.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:03.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:04.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:04.129 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:04.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:04.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:04.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:04.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:04.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:04.356 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:04.382 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:04.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:04.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:04.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:04.511 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:04.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:04.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:04.646 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:04.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:04.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:04.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:04.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:04.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:04.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:04.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:04.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:04.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:04.920 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:04.921 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:04 smithi169 ceph-mon[29344]: 5.6 scrub starts 2024-03-12T03:47:04.921 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:04 smithi169 ceph-mon[29344]: 5.6 scrub ok 2024-03-12T03:47:04.921 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:04 smithi169 ceph-mon[29344]: osdmap e347: 8 total, 8 up, 8 in 2024-03-12T03:47:04.921 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:04.921 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:04 smithi169 ceph-mon[39949]: 5.6 scrub starts 2024-03-12T03:47:04.921 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:04 smithi169 ceph-mon[39949]: 5.6 scrub ok 2024-03-12T03:47:04.921 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:04 smithi169 ceph-mon[39949]: osdmap e347: 8 total, 8 up, 8 in 2024-03-12T03:47:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:04 smithi173 ceph-mon[33544]: 5.6 scrub starts 2024-03-12T03:47:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:04 smithi173 ceph-mon[33544]: 5.6 scrub ok 2024-03-12T03:47:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:04 smithi173 ceph-mon[33544]: osdmap e347: 8 total, 8 up, 8 in 2024-03-12T03:47:05.003 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:05.027 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:05.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:05.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:05.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:05.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:05.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:05.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:05.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:05.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:05.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:05.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:05.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:05.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:05.550 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:05.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:05.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:05.689 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:05.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:05.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:05.816 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:05.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:05.873 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:05 smithi169 ceph-mon[29344]: pgmap v658: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 163 active+clean; 1.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 5099/86601 objects degraded (5.888%); 32 MiB/s, 515 objects/s recovering 2024-03-12T03:47:05.873 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:05 smithi169 ceph-mon[29344]: 5.12 scrub starts 2024-03-12T03:47:05.873 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:05 smithi169 ceph-mon[29344]: 5.12 scrub ok 2024-03-12T03:47:05.874 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:05 smithi169 ceph-mon[29344]: osdmap e348: 8 total, 8 up, 8 in 2024-03-12T03:47:05.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:05 smithi169 ceph-mon[39949]: pgmap v658: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 163 active+clean; 1.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 5099/86601 objects degraded (5.888%); 32 MiB/s, 515 objects/s recovering 2024-03-12T03:47:05.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:05 smithi169 ceph-mon[39949]: 5.12 scrub starts 2024-03-12T03:47:05.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:05 smithi169 ceph-mon[39949]: 5.12 scrub ok 2024-03-12T03:47:05.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:05 smithi169 ceph-mon[39949]: osdmap e348: 8 total, 8 up, 8 in 2024-03-12T03:47:05.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:05.944 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:05.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:05 smithi173 ceph-mon[33544]: pgmap v658: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 163 active+clean; 1.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 5099/86601 objects degraded (5.888%); 32 MiB/s, 515 objects/s recovering 2024-03-12T03:47:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:05 smithi173 ceph-mon[33544]: 5.12 scrub starts 2024-03-12T03:47:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:05 smithi173 ceph-mon[33544]: 5.12 scrub ok 2024-03-12T03:47:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:05 smithi173 ceph-mon[33544]: osdmap e348: 8 total, 8 up, 8 in 2024-03-12T03:47:06.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:06.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:06.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:06.080 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:47:06.080 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:47:06.080 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:47:06.080 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:47:06.080 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:47:06.080 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:47:06.080 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:47:06.080 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:47:06.080 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:47:06.080 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:47:06.080 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:47:06.081 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:47:06.081 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:47:06.081 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:47:06.081 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:47:06.081 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:47:06.171 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:06.195 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:06.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:06.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:06.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:06.321 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:06.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:06.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:06.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:06.452 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:06.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:06.553 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:06.588 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:06.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:06.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:06.725 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:06.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:06.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:06.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:06.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:06.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[29344]: 1.6 scrub starts 2024-03-12T03:47:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[29344]: 1.6 scrub ok 2024-03-12T03:47:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[29344]: 4.1b scrub starts 2024-03-12T03:47:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[29344]: 4.1b scrub ok 2024-03-12T03:47:06.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[29344]: 3.f scrub starts 2024-03-12T03:47:06.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[29344]: 3.f scrub ok 2024-03-12T03:47:06.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1180953401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:47:06.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1699680408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:47:06.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[29344]: 4.f scrub starts 2024-03-12T03:47:06.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[29344]: 4.f scrub ok 2024-03-12T03:47:06.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1699680408' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-12T03:47:06.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[29344]: osdmap e349: 8 total, 8 up, 8 in 2024-03-12T03:47:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[39949]: 1.6 scrub starts 2024-03-12T03:47:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[39949]: 1.6 scrub ok 2024-03-12T03:47:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[39949]: 4.1b scrub starts 2024-03-12T03:47:06.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[39949]: 4.1b scrub ok 2024-03-12T03:47:06.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[39949]: 3.f scrub starts 2024-03-12T03:47:06.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[39949]: 3.f scrub ok 2024-03-12T03:47:06.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1180953401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:47:06.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1699680408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:47:06.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[39949]: 4.f scrub starts 2024-03-12T03:47:06.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[39949]: 4.f scrub ok 2024-03-12T03:47:06.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1699680408' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-12T03:47:06.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:06 smithi169 ceph-mon[39949]: osdmap e349: 8 total, 8 up, 8 in 2024-03-12T03:47:06.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:06 smithi173 ceph-mon[33544]: 1.6 scrub starts 2024-03-12T03:47:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:06 smithi173 ceph-mon[33544]: 1.6 scrub ok 2024-03-12T03:47:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:06 smithi173 ceph-mon[33544]: 4.1b scrub starts 2024-03-12T03:47:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:06 smithi173 ceph-mon[33544]: 4.1b scrub ok 2024-03-12T03:47:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:06 smithi173 ceph-mon[33544]: 3.f scrub starts 2024-03-12T03:47:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:06 smithi173 ceph-mon[33544]: 3.f scrub ok 2024-03-12T03:47:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:06 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1180953401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:47:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:06 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1699680408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:47:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:06 smithi173 ceph-mon[33544]: 4.f scrub starts 2024-03-12T03:47:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:06 smithi173 ceph-mon[33544]: 4.f scrub ok 2024-03-12T03:47:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:06 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1699680408' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-12T03:47:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:06 smithi173 ceph-mon[33544]: osdmap e349: 8 total, 8 up, 8 in 2024-03-12T03:47:07.000 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:07.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:07.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:07.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:07.139 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:07.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:07.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:07.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:07.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:07.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:07.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:07.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:07.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:07.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:07.545 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:07.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:07.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:07.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:07.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:07.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:07.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:07.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:07.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:07 smithi169 ceph-mon[29344]: pgmap v661: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 163 active+clean; 1.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 5099/86601 objects degraded (5.888%); 19 MiB/s, 299 objects/s recovering 2024-03-12T03:47:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:07 smithi169 ceph-mon[29344]: osdmap e350: 8 total, 8 up, 8 in 2024-03-12T03:47:07.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:07 smithi169 ceph-mon[39949]: pgmap v661: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 163 active+clean; 1.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 5099/86601 objects degraded (5.888%); 19 MiB/s, 299 objects/s recovering 2024-03-12T03:47:07.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:07 smithi169 ceph-mon[39949]: osdmap e350: 8 total, 8 up, 8 in 2024-03-12T03:47:07.927 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:07.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:07.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:07.956 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.4 to 1 (10000) 2024-03-12T03:47:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:07 smithi173 ceph-mon[33544]: pgmap v661: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 163 active+clean; 1.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 5099/86601 objects degraded (5.888%); 19 MiB/s, 299 objects/s recovering 2024-03-12T03:47:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:07 smithi173 ceph-mon[33544]: osdmap e350: 8 total, 8 up, 8 in 2024-03-12T03:47:08.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:08.054 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:08.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:08.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:08.179 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:08.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:08.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:08.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:08.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:08.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:08.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:08.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:08.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:08.433 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:08.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:08.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:08.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:08.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:08.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:08.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:08.702 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:08.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:08.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:08.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:08.842 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:08.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:08 smithi169 ceph-mon[29344]: 3.b scrub starts 2024-03-12T03:47:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:08 smithi169 ceph-mon[29344]: 3.b scrub ok 2024-03-12T03:47:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:08 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 5099/86601 objects degraded (5.888%), 5 pgs degraded (PG_DEGRADED) 2024-03-12T03:47:08.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:08 smithi169 ceph-mon[29344]: osdmap e351: 8 total, 8 up, 8 in 2024-03-12T03:47:08.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:08 smithi169 ceph-mon[39949]: 3.b scrub starts 2024-03-12T03:47:08.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:08 smithi169 ceph-mon[39949]: 3.b scrub ok 2024-03-12T03:47:08.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:08 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 5099/86601 objects degraded (5.888%), 5 pgs degraded (PG_DEGRADED) 2024-03-12T03:47:08.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:08 smithi169 ceph-mon[39949]: osdmap e351: 8 total, 8 up, 8 in 2024-03-12T03:47:08.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:08.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:08 smithi173 ceph-mon[33544]: 3.b scrub starts 2024-03-12T03:47:08.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:08 smithi173 ceph-mon[33544]: 3.b scrub ok 2024-03-12T03:47:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:08 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 5099/86601 objects degraded (5.888%), 5 pgs degraded (PG_DEGRADED) 2024-03-12T03:47:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:08 smithi173 ceph-mon[33544]: osdmap e351: 8 total, 8 up, 8 in 2024-03-12T03:47:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:08.989 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:08.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:09.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:09.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:09.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:09.168 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:09.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:09.264 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:09.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:09.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:09.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:09.404 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:09.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:09.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:09.544 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:09.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:09.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:09.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:09.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:09.784 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:09.818 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:09.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:09.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:09.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:09 smithi169 ceph-mon[39949]: 3.14 deep-scrub starts 2024-03-12T03:47:09.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:09 smithi169 ceph-mon[39949]: 3.14 deep-scrub ok 2024-03-12T03:47:09.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:09 smithi169 ceph-mon[39949]: pgmap v664: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 164 active+clean; 1.6 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 2507/79173 objects degraded (3.166%); 34 MiB/s, 540 objects/s recovering 2024-03-12T03:47:09.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:09 smithi169 ceph-mon[39949]: 3.5 scrub starts 2024-03-12T03:47:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:09 smithi169 ceph-mon[39949]: 3.5 scrub ok 2024-03-12T03:47:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:09 smithi169 ceph-mon[29344]: 3.14 deep-scrub starts 2024-03-12T03:47:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:09 smithi169 ceph-mon[29344]: 3.14 deep-scrub ok 2024-03-12T03:47:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:09 smithi169 ceph-mon[29344]: pgmap v664: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 164 active+clean; 1.6 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 2507/79173 objects degraded (3.166%); 34 MiB/s, 540 objects/s recovering 2024-03-12T03:47:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:09 smithi169 ceph-mon[29344]: 3.5 scrub starts 2024-03-12T03:47:09.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:09 smithi169 ceph-mon[29344]: 3.5 scrub ok 2024-03-12T03:47:09.956 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:09.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:09 smithi173 ceph-mon[33544]: 3.14 deep-scrub starts 2024-03-12T03:47:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:09 smithi173 ceph-mon[33544]: 3.14 deep-scrub ok 2024-03-12T03:47:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:09 smithi173 ceph-mon[33544]: pgmap v664: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 164 active+clean; 1.6 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 2507/79173 objects degraded (3.166%); 34 MiB/s, 540 objects/s recovering 2024-03-12T03:47:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:09 smithi173 ceph-mon[33544]: 3.5 scrub starts 2024-03-12T03:47:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:09 smithi173 ceph-mon[33544]: 3.5 scrub ok 2024-03-12T03:47:10.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:10.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:10.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:10.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:10.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:10.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:10.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:10.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:10.356 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:10.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:10.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:10.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:10.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:10.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:10.594 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:10.633 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:10.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:10.734 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:10.770 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:10.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:10.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:10.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:10.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:10.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:10 smithi173 ceph-mon[33544]: 4.15 scrub starts 2024-03-12T03:47:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:10 smithi173 ceph-mon[33544]: 4.15 scrub ok 2024-03-12T03:47:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:10 smithi173 ceph-mon[33544]: 1.7 scrub starts 2024-03-12T03:47:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:10 smithi173 ceph-mon[33544]: 1.7 scrub ok 2024-03-12T03:47:11.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:11.047 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:11.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:11.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:10 smithi169 ceph-mon[39949]: 4.15 scrub starts 2024-03-12T03:47:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:10 smithi169 ceph-mon[39949]: 4.15 scrub ok 2024-03-12T03:47:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:10 smithi169 ceph-mon[39949]: 1.7 scrub starts 2024-03-12T03:47:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:10 smithi169 ceph-mon[39949]: 1.7 scrub ok 2024-03-12T03:47:11.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:10 smithi169 ceph-mon[29344]: 4.15 scrub starts 2024-03-12T03:47:11.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:10 smithi169 ceph-mon[29344]: 4.15 scrub ok 2024-03-12T03:47:11.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:10 smithi169 ceph-mon[29344]: 1.7 scrub starts 2024-03-12T03:47:11.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:10 smithi169 ceph-mon[29344]: 1.7 scrub ok 2024-03-12T03:47:11.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:11.188 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:11.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:11.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:11.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:11.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:11.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:11.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:11.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:11.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:11.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:11.677 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:11.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:11.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:11.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:11.818 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:11.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:11.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:11.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:11.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:11 smithi173 ceph-mon[33544]: pgmap v666: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 166 active+clean; 1.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 1656/76455 objects degraded (2.166%); 40 MiB/s, 637 objects/s recovering 2024-03-12T03:47:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:11 smithi173 ceph-mon[33544]: 3.c scrub starts 2024-03-12T03:47:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:11 smithi173 ceph-mon[33544]: 3.c scrub ok 2024-03-12T03:47:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:11 smithi173 ceph-mon[33544]: 1.4 deep-scrub starts 2024-03-12T03:47:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:11 smithi173 ceph-mon[33544]: 1.4 deep-scrub ok 2024-03-12T03:47:12.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:12.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:12.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:12.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:11 smithi169 ceph-mon[29344]: pgmap v666: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 166 active+clean; 1.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 1656/76455 objects degraded (2.166%); 40 MiB/s, 637 objects/s recovering 2024-03-12T03:47:12.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:11 smithi169 ceph-mon[29344]: 3.c scrub starts 2024-03-12T03:47:12.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:11 smithi169 ceph-mon[29344]: 3.c scrub ok 2024-03-12T03:47:12.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:11 smithi169 ceph-mon[29344]: 1.4 deep-scrub starts 2024-03-12T03:47:12.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:11 smithi169 ceph-mon[29344]: 1.4 deep-scrub ok 2024-03-12T03:47:12.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:11 smithi169 ceph-mon[39949]: pgmap v666: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 166 active+clean; 1.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 1656/76455 objects degraded (2.166%); 40 MiB/s, 637 objects/s recovering 2024-03-12T03:47:12.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:11 smithi169 ceph-mon[39949]: 3.c scrub starts 2024-03-12T03:47:12.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:11 smithi169 ceph-mon[39949]: 3.c scrub ok 2024-03-12T03:47:12.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:11 smithi169 ceph-mon[39949]: 1.4 deep-scrub starts 2024-03-12T03:47:12.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:11 smithi169 ceph-mon[39949]: 1.4 deep-scrub ok 2024-03-12T03:47:12.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:47:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:47:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:47:12.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:12.232 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:12.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:12.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:12.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:12.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:12.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:12.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:12.472 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:12.508 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:12.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:12.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:12.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:12.647 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:12.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:12.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:12.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:12.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:12.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:12.923 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:12.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:12 smithi173 ceph-mon[33544]: 6.17 scrub starts 2024-03-12T03:47:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:12 smithi173 ceph-mon[33544]: 6.17 scrub ok 2024-03-12T03:47:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:12 smithi173 ceph-mon[33544]: 6.15 scrub starts 2024-03-12T03:47:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:12 smithi173 ceph-mon[33544]: 6.15 scrub ok 2024-03-12T03:47:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:12 smithi173 ceph-mon[33544]: 3.17 scrub starts 2024-03-12T03:47:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:12 smithi173 ceph-mon[33544]: 3.17 scrub ok 2024-03-12T03:47:13.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:13.057 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:13.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:13.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:12 smithi169 ceph-mon[29344]: 6.17 scrub starts 2024-03-12T03:47:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:12 smithi169 ceph-mon[29344]: 6.17 scrub ok 2024-03-12T03:47:13.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:12 smithi169 ceph-mon[29344]: 6.15 scrub starts 2024-03-12T03:47:13.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:12 smithi169 ceph-mon[29344]: 6.15 scrub ok 2024-03-12T03:47:13.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:12 smithi169 ceph-mon[29344]: 3.17 scrub starts 2024-03-12T03:47:13.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:12 smithi169 ceph-mon[29344]: 3.17 scrub ok 2024-03-12T03:47:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:12 smithi169 ceph-mon[39949]: 6.17 scrub starts 2024-03-12T03:47:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:12 smithi169 ceph-mon[39949]: 6.17 scrub ok 2024-03-12T03:47:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:12 smithi169 ceph-mon[39949]: 6.15 scrub starts 2024-03-12T03:47:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:12 smithi169 ceph-mon[39949]: 6.15 scrub ok 2024-03-12T03:47:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:12 smithi169 ceph-mon[39949]: 3.17 scrub starts 2024-03-12T03:47:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:12 smithi169 ceph-mon[39949]: 3.17 scrub ok 2024-03-12T03:47:13.196 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:13.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:13.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:13.323 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:13.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:13.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:13.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:13.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:13.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:13.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:47:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:47:13.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:13.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:13.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:13.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:13.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:13.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:13.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:13.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:13.862 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:13.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:13.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:13.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:13 smithi173 ceph-mon[33544]: pgmap v667: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 1.3 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.09k op/s; 1320/66864 objects degraded (1.974%); 31 MiB/s, 499 objects/s recovering 2024-03-12T03:47:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:13 smithi173 ceph-mon[33544]: 4.0 scrub starts 2024-03-12T03:47:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:13 smithi173 ceph-mon[33544]: 4.0 scrub ok 2024-03-12T03:47:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:13 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 1320/66864 objects degraded (1.974%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:47:13.999 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:14.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:14.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:14.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:13 smithi169 ceph-mon[29344]: pgmap v667: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 1.3 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.09k op/s; 1320/66864 objects degraded (1.974%); 31 MiB/s, 499 objects/s recovering 2024-03-12T03:47:14.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:13 smithi169 ceph-mon[29344]: 4.0 scrub starts 2024-03-12T03:47:14.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:13 smithi169 ceph-mon[29344]: 4.0 scrub ok 2024-03-12T03:47:14.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:13 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 1320/66864 objects degraded (1.974%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:47:14.037 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:13 smithi169 ceph-mon[39949]: pgmap v667: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 1.3 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.09k op/s; 1320/66864 objects degraded (1.974%); 31 MiB/s, 499 objects/s recovering 2024-03-12T03:47:14.037 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:13 smithi169 ceph-mon[39949]: 4.0 scrub starts 2024-03-12T03:47:14.038 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:13 smithi169 ceph-mon[39949]: 4.0 scrub ok 2024-03-12T03:47:14.038 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:13 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 1320/66864 objects degraded (1.974%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:47:14.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:14.135 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:14.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:14.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:14.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:14.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:14.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:14.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:14.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:14.411 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:14.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:14.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:14.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:14.550 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:14.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:14.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:14.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:14.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:14.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:14.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:14.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[29344]: 6.e scrub starts 2024-03-12T03:47:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[29344]: 6.e scrub ok 2024-03-12T03:47:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[29344]: 2.0 scrub starts 2024-03-12T03:47:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[29344]: 2.0 scrub ok 2024-03-12T03:47:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[29344]: 6.0 scrub starts 2024-03-12T03:47:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[29344]: 6.0 scrub ok 2024-03-12T03:47:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[29344]: 6.c scrub starts 2024-03-12T03:47:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[29344]: 6.c scrub ok 2024-03-12T03:47:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[39949]: 6.e scrub starts 2024-03-12T03:47:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[39949]: 6.e scrub ok 2024-03-12T03:47:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:14.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[39949]: 2.0 scrub starts 2024-03-12T03:47:14.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[39949]: 2.0 scrub ok 2024-03-12T03:47:14.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[39949]: 6.0 scrub starts 2024-03-12T03:47:14.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[39949]: 6.0 scrub ok 2024-03-12T03:47:14.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[39949]: 6.c scrub starts 2024-03-12T03:47:14.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:14 smithi169 ceph-mon[39949]: 6.c scrub ok 2024-03-12T03:47:14.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:14.963 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:14.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:14.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:14.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:14 smithi173 ceph-mon[33544]: 6.e scrub starts 2024-03-12T03:47:14.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:14 smithi173 ceph-mon[33544]: 6.e scrub ok 2024-03-12T03:47:14.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:14 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:14.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:14 smithi173 ceph-mon[33544]: 2.0 scrub starts 2024-03-12T03:47:14.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:14 smithi173 ceph-mon[33544]: 2.0 scrub ok 2024-03-12T03:47:14.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:14 smithi173 ceph-mon[33544]: 6.0 scrub starts 2024-03-12T03:47:14.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:14 smithi173 ceph-mon[33544]: 6.0 scrub ok 2024-03-12T03:47:14.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:14 smithi173 ceph-mon[33544]: 6.c scrub starts 2024-03-12T03:47:14.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:14 smithi173 ceph-mon[33544]: 6.c scrub ok 2024-03-12T03:47:15.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:15.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:15.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:15.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:15.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:15.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:15.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:15.375 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:15.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:15.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:15.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:15.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:15.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:15.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:15.615 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:15.662 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:15.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:15.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:15.764 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:15.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-12T03:47:15.825 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:15.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:15.903 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[39949]: 4.d scrub starts 2024-03-12T03:47:15.903 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[39949]: 4.d scrub ok 2024-03-12T03:47:15.903 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[39949]: pgmap v668: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 1320/59949 objects degraded (2.202%); 24 MiB/s, 391 objects/s recovering 2024-03-12T03:47:15.903 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[39949]: 6.1c scrub starts 2024-03-12T03:47:15.903 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[39949]: 6.1c scrub ok 2024-03-12T03:47:15.903 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[39949]: 1.14 scrub starts 2024-03-12T03:47:15.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[39949]: 1.14 scrub ok 2024-03-12T03:47:15.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[39949]: 3.19 deep-scrub starts 2024-03-12T03:47:15.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[39949]: 3.19 deep-scrub ok 2024-03-12T03:47:15.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[29344]: 4.d scrub starts 2024-03-12T03:47:15.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[29344]: 4.d scrub ok 2024-03-12T03:47:15.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[29344]: pgmap v668: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 1320/59949 objects degraded (2.202%); 24 MiB/s, 391 objects/s recovering 2024-03-12T03:47:15.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[29344]: 6.1c scrub starts 2024-03-12T03:47:15.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[29344]: 6.1c scrub ok 2024-03-12T03:47:15.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[29344]: 1.14 scrub starts 2024-03-12T03:47:15.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[29344]: 1.14 scrub ok 2024-03-12T03:47:15.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[29344]: 3.19 deep-scrub starts 2024-03-12T03:47:15.904 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:15 smithi169 ceph-mon[29344]: 3.19 deep-scrub ok 2024-03-12T03:47:15.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:15.957 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:15.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:15 smithi173 ceph-mon[33544]: 4.d scrub starts 2024-03-12T03:47:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:15 smithi173 ceph-mon[33544]: 4.d scrub ok 2024-03-12T03:47:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:15 smithi173 ceph-mon[33544]: pgmap v668: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 1320/59949 objects degraded (2.202%); 24 MiB/s, 391 objects/s recovering 2024-03-12T03:47:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:15 smithi173 ceph-mon[33544]: 6.1c scrub starts 2024-03-12T03:47:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:15 smithi173 ceph-mon[33544]: 6.1c scrub ok 2024-03-12T03:47:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:15 smithi173 ceph-mon[33544]: 1.14 scrub starts 2024-03-12T03:47:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:15 smithi173 ceph-mon[33544]: 1.14 scrub ok 2024-03-12T03:47:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:15 smithi173 ceph-mon[33544]: 3.19 deep-scrub starts 2024-03-12T03:47:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:15 smithi173 ceph-mon[33544]: 3.19 deep-scrub ok 2024-03-12T03:47:16.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:16.105 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:16.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:16.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:16.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:16.243 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:16.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:16.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:16.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:16.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:16.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:16.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:16.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:16.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:16.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:16.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:16.752 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:16.788 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:16.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:16.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:16.916 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:16 smithi169 ceph-mon[29344]: 1.8 scrub starts 2024-03-12T03:47:16.916 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:16 smithi169 ceph-mon[29344]: 4.1e scrub starts 2024-03-12T03:47:16.916 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:16 smithi169 ceph-mon[29344]: 1.8 scrub ok 2024-03-12T03:47:16.916 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:16 smithi169 ceph-mon[29344]: 4.1e scrub ok 2024-03-12T03:47:16.916 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:16 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:16.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:16 smithi169 ceph-mon[39949]: 1.8 scrub starts 2024-03-12T03:47:16.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:16 smithi169 ceph-mon[39949]: 4.1e scrub starts 2024-03-12T03:47:16.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:16 smithi169 ceph-mon[39949]: 1.8 scrub ok 2024-03-12T03:47:16.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:16 smithi169 ceph-mon[39949]: 4.1e scrub ok 2024-03-12T03:47:16.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:16 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:16.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:16.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:16 smithi173 ceph-mon[33544]: 1.8 scrub starts 2024-03-12T03:47:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:16 smithi173 ceph-mon[33544]: 4.1e scrub starts 2024-03-12T03:47:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:16 smithi173 ceph-mon[33544]: 1.8 scrub ok 2024-03-12T03:47:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:16 smithi173 ceph-mon[33544]: 4.1e scrub ok 2024-03-12T03:47:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:16 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:17.026 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:17.063 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:17.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:17.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:17.200 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:17.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:17.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:17.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:17.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:17.351 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:17.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:17.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:17.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:17.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:17.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:17.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:17.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:17.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:17.772 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:17.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:17.844 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[29344]: pgmap v669: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 783 op/s; 1320/59949 objects degraded (2.202%); 10 MiB/s, 166 objects/s recovering 2024-03-12T03:47:17.845 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[29344]: 3.12 scrub starts 2024-03-12T03:47:17.845 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[29344]: 3.12 scrub ok 2024-03-12T03:47:17.845 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[29344]: 5.b scrub starts 2024-03-12T03:47:17.845 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[29344]: 5.b scrub ok 2024-03-12T03:47:17.845 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[29344]: 6.4 scrub starts 2024-03-12T03:47:17.845 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[29344]: 6.4 scrub ok 2024-03-12T03:47:17.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[39949]: pgmap v669: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 783 op/s; 1320/59949 objects degraded (2.202%); 10 MiB/s, 166 objects/s recovering 2024-03-12T03:47:17.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[39949]: 3.12 scrub starts 2024-03-12T03:47:17.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[39949]: 3.12 scrub ok 2024-03-12T03:47:17.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[39949]: 5.b scrub starts 2024-03-12T03:47:17.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[39949]: 5.b scrub ok 2024-03-12T03:47:17.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[39949]: 6.4 scrub starts 2024-03-12T03:47:17.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:17 smithi169 ceph-mon[39949]: 6.4 scrub ok 2024-03-12T03:47:17.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:17.913 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:17.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:17 smithi173 ceph-mon[33544]: pgmap v669: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 783 op/s; 1320/59949 objects degraded (2.202%); 10 MiB/s, 166 objects/s recovering 2024-03-12T03:47:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:17 smithi173 ceph-mon[33544]: 3.12 scrub starts 2024-03-12T03:47:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:17 smithi173 ceph-mon[33544]: 3.12 scrub ok 2024-03-12T03:47:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:17 smithi173 ceph-mon[33544]: 5.b scrub starts 2024-03-12T03:47:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:17 smithi173 ceph-mon[33544]: 5.b scrub ok 2024-03-12T03:47:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:17 smithi173 ceph-mon[33544]: 6.4 scrub starts 2024-03-12T03:47:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:17 smithi173 ceph-mon[33544]: 6.4 scrub ok 2024-03-12T03:47:18.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:18.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:18.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:18.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:18.152 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:18.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:18.193 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:18.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:18.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:18.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:18.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:18.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:18.427 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:18.492 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:18.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:18.593 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:18.630 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:18.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:18.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:18.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:18.763 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:47:18.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[29344]: 6.3 scrub starts 2024-03-12T03:47:18.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[29344]: 5.16 scrub starts 2024-03-12T03:47:18.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[29344]: 6.3 scrub ok 2024-03-12T03:47:18.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[29344]: 5.16 scrub ok 2024-03-12T03:47:18.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3769573709' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:47:18.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:47:18.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 1320/59949 objects degraded (2.202%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:47:18.784 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:47:18.784 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[39949]: 6.3 scrub starts 2024-03-12T03:47:18.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[39949]: 5.16 scrub starts 2024-03-12T03:47:18.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[39949]: 6.3 scrub ok 2024-03-12T03:47:18.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[39949]: 5.16 scrub ok 2024-03-12T03:47:18.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3769573709' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:47:18.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:47:18.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 1320/59949 objects degraded (2.202%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:47:18.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:18 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:47:18.786 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:18.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:18.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:18.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:18.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:18.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:18.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:19.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:19.064 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:19.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:19.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:19.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:19.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:19.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:19.202 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:47:19.202 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:47:19.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:18 smithi173 ceph-mon[33544]: 6.3 scrub starts 2024-03-12T03:47:19.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:18 smithi173 ceph-mon[33544]: 5.16 scrub starts 2024-03-12T03:47:19.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:18 smithi173 ceph-mon[33544]: 6.3 scrub ok 2024-03-12T03:47:19.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:18 smithi173 ceph-mon[33544]: 5.16 scrub ok 2024-03-12T03:47:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:18 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3769573709' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:47:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:18 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:47:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:18 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 1320/59949 objects degraded (2.202%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:47:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:18 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:47:19.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:19.354 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:19.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:19.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:19.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:47:19.401 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:19.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:19.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:19.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:19.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:19.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:19.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:19.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:19.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:19.756 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:19.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:19.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:19.857 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:19.902 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:19.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:19.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:20.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:20.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:20.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:20.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:20.113 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:19 smithi169 ceph-mon[29344]: pgmap v670: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 925 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 1.14k op/s; 917/45060 objects degraded (2.035%); 5.2 MiB/s, 83 objects/s recovering 2024-03-12T03:47:20.113 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:19 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:47:20.114 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:19 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:47:20.114 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:19 smithi169 ceph-mon[29344]: osdmap e352: 8 total, 8 up, 8 in 2024-03-12T03:47:20.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:19 smithi169 ceph-mon[39949]: pgmap v670: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 925 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 1.14k op/s; 917/45060 objects degraded (2.035%); 5.2 MiB/s, 83 objects/s recovering 2024-03-12T03:47:20.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:19 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:47:20.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:19 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:47:20.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:19 smithi169 ceph-mon[39949]: osdmap e352: 8 total, 8 up, 8 in 2024-03-12T03:47:20.145 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:20.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-12T03:47:20.186 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:20.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:19 smithi173 ceph-mon[33544]: pgmap v670: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 925 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 1.14k op/s; 917/45060 objects degraded (2.035%); 5.2 MiB/s, 83 objects/s recovering 2024-03-12T03:47:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:19 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:47:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:19 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:47:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:19 smithi173 ceph-mon[33544]: osdmap e352: 8 total, 8 up, 8 in 2024-03-12T03:47:20.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:20.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:20.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:20.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:20.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:20.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:20.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:20.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:20.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:20.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:20.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:20.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:20.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:20.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:20.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:20.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:20.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:20.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:20.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:20.997 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:20.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:21.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:21.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:21.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:21.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:20 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:47:21.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:20 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:47:21.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:20 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:47:21.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:20 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:47:21.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:20 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:47:21.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:20 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:47:21.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:20 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:47:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:20 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:47:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:20 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:47:21.275 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:21.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:21.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:21.412 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:21.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:21.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:21.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:21.552 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:21.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:21.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:21.688 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:21.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:21.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:21.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:21.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:21.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:21.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:21.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:21 smithi169 ceph-mon[29344]: pgmap v672: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 821 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 1.21k op/s; 917/40092 objects degraded (2.287%); 148 KiB/s, 2 objects/s recovering 2024-03-12T03:47:21.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:21 smithi169 ceph-mon[39949]: pgmap v672: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 821 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 1.21k op/s; 917/40092 objects degraded (2.287%); 148 KiB/s, 2 objects/s recovering 2024-03-12T03:47:21.924 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:47:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:47:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:47:21.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:21.984 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:21.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:22.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:22.120 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:22.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:22.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:22.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:21 smithi173 ceph-mon[33544]: pgmap v672: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 821 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 1.21k op/s; 917/40092 objects degraded (2.287%); 148 KiB/s, 2 objects/s recovering 2024-03-12T03:47:22.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:22.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:22.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:22.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:22.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:22.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:22.543 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:22.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:22.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:22.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:22.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:22.767 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:22 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:47:22.767 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:22 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:47:22.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:22.830 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:22.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:22.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:22.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:22.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:22.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:23.052 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:22 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:47:23.052 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:22 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:47:23.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:23.106 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:23.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:23.108 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:22 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:47:23.108 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:22 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:47:23.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:23.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:23.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:23.245 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:23.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:23.393 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:23.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:23.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:47:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:47:23.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:23.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:23.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:23.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:23.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:23.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:23.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:23.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:23.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:23.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:23.808 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.5 to 1 (10000) 2024-03-12T03:47:23.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:23.913 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[29344]: pgmap v673: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 562 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 1.31k op/s; 472/27621 objects degraded (1.709%) 2024-03-12T03:47:23.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[29344]: osdmap e353: 8 total, 8 up, 8 in 2024-03-12T03:47:23.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:47:23.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:47:23.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 472/27621 objects degraded (1.709%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:47:23.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3589967923' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:47:23.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:47:23.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[39949]: pgmap v673: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 562 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 1.31k op/s; 472/27621 objects degraded (1.709%) 2024-03-12T03:47:23.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[39949]: osdmap e353: 8 total, 8 up, 8 in 2024-03-12T03:47:23.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:47:23.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:47:23.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 472/27621 objects degraded (1.709%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:47:23.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3589967923' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:47:23.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:23 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:47:23.944 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:23.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:23.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:24.046 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:24.084 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:24.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:24.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:24.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:24.227 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:24.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:23 smithi173 ceph-mon[33544]: pgmap v673: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 562 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 1.31k op/s; 472/27621 objects degraded (1.709%) 2024-03-12T03:47:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:23 smithi173 ceph-mon[33544]: osdmap e353: 8 total, 8 up, 8 in 2024-03-12T03:47:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:23 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:47:24.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:23 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:47:24.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:23 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 472/27621 objects degraded (1.709%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:47:24.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:23 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3589967923' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:47:24.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:23 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:47:24.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:24.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:24.373 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:24.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:24.383 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:24.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:24.517 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:24.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:24.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:24.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:24.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:24.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:24.707 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:47:24.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:24.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:24.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-12T03:47:24.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[29344]: osdmap e354: 8 total, 8 up, 8 in 2024-03-12T03:47:24.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:47:24.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:47:24.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:47:24.791 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:47:24.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:24.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-12T03:47:24.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[39949]: osdmap e354: 8 total, 8 up, 8 in 2024-03-12T03:47:24.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:47:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:24.811 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:24.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:24.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:24.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:47:24.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:47:24.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:24 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:47:24.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:24.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:25.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:25.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:25.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:25.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:25.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-12T03:47:25.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:25.210 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:25.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:24 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-12T03:47:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:24 smithi173 ceph-mon[33544]: osdmap e354: 8 total, 8 up, 8 in 2024-03-12T03:47:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:24 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:47:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:24 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:47:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:24 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:47:25.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:24 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:47:25.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:25.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:25.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:25.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:25.496 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:25.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:25.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:25.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:25.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:25.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:25.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:25.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:25.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:25.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:25.871 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:25.907 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:25.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:26.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:26.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:26.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:26.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:26.133 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:25 smithi169 ceph-mon[29344]: pgmap v676: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 404 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.39k op/s; 472/20055 objects degraded (2.354%) 2024-03-12T03:47:26.133 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:25 smithi169 ceph-mon[29344]: osdmap e355: 8 total, 8 up, 8 in 2024-03-12T03:47:26.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:25 smithi169 ceph-mon[39949]: pgmap v676: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 404 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.39k op/s; 472/20055 objects degraded (2.354%) 2024-03-12T03:47:26.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:25 smithi169 ceph-mon[39949]: osdmap e355: 8 total, 8 up, 8 in 2024-03-12T03:47:26.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:26.177 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:26.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:26.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:26.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:25 smithi173 ceph-mon[33544]: pgmap v676: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 404 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.39k op/s; 472/20055 objects degraded (2.354%) 2024-03-12T03:47:26.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:25 smithi173 ceph-mon[33544]: osdmap e355: 8 total, 8 up, 8 in 2024-03-12T03:47:26.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:26.319 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:26.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:26.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:26.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:26.459 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:26.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:26.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:26.598 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:26.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:26.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:26.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:26.738 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:26.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:26.839 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:26.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:26.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:26.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:26.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:26.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:27.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:27.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:27.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:27.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:27.245 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:27.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:27.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:27.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:27.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:27.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:27.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:27.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:27.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:27.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:27.624 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:27.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:27.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:27.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:27.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:27.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:27.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:27.866 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Removed 144212 objects 2024-03-12T03:47:27.866 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Clean up completed and total clean up time :121.473 2024-03-12T03:47:27.881 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:27.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:27.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:27.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:28.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:28.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:28.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:28.098 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:27 smithi169 ceph-mon[29344]: pgmap v678: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 404 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s; 472/20055 objects degraded (2.354%) 2024-03-12T03:47:28.099 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:27 smithi169 ceph-mon[29344]: osdmap e356: 8 total, 8 up, 8 in 2024-03-12T03:47:28.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:27 smithi169 ceph-mon[39949]: pgmap v678: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 404 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s; 472/20055 objects degraded (2.354%) 2024-03-12T03:47:28.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:27 smithi169 ceph-mon[39949]: osdmap e356: 8 total, 8 up, 8 in 2024-03-12T03:47:28.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:28.132 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:28.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:28.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:28.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:28.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:27 smithi173 ceph-mon[33544]: pgmap v678: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 404 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s; 472/20055 objects degraded (2.354%) 2024-03-12T03:47:28.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:27 smithi173 ceph-mon[33544]: osdmap e356: 8 total, 8 up, 8 in 2024-03-12T03:47:28.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:28.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:28.336 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:28.338 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:28.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:28.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:28.384 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:28.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:28.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:28.509 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:28.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:28.611 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:28.636 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:28.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:28.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:28.761 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:28.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:28.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:28.886 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:28.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:28.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:28 smithi169 ceph-mon[39949]: osdmap e357: 8 total, 8 up, 8 in 2024-03-12T03:47:28.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:28 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 472/20055 objects degraded (2.354%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:47:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:28 smithi169 ceph-mon[29344]: osdmap e357: 8 total, 8 up, 8 in 2024-03-12T03:47:28.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:28 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 472/20055 objects degraded (2.354%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:47:28.939 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-12T03:47:28.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-12T03:47:28.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:29.012 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:29.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:29.087 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:47:29.087 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:47:29.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:28 smithi173 ceph-mon[33544]: osdmap e357: 8 total, 8 up, 8 in 2024-03-12T03:47:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:28 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 472/20055 objects degraded (2.354%), 1 pg degraded (PG_DEGRADED) 2024-03-12T03:47:29.140 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:29.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:29.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:29.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:29.265 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:29.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:29.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:29.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:29.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:29.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:29.495 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:29.520 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:29.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:29.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:29.644 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:29.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:29.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:29.769 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:29.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:29.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:29.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:29.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:29.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:30.020 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:30.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:30.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:30.090 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:29 smithi173 ceph-mon[33544]: pgmap v681: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 96 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 59/5262 objects degraded (1.121%) 2024-03-12T03:47:30.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:30.128 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:29 smithi169 ceph-mon[29344]: pgmap v681: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 96 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 59/5262 objects degraded (1.121%) 2024-03-12T03:47:30.128 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:29 smithi169 ceph-mon[39949]: pgmap v681: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 96 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 59/5262 objects degraded (1.121%) 2024-03-12T03:47:30.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-12T03:47:30.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:30.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:30.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:30.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:30.462 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:30.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:30.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:30.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:30.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:30.640 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:47:30.660 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:47:30.684 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:47:30.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:30.707 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:47:30.713 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:30.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:30.727 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:47:30.752 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:47:30.776 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:47:30.795 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:47:30.796 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:47:30.796 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:47:30.796 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:47:30.796 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:47:30.797 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:47:30.797 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:47:30.797 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:47:30.797 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:47:30.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:30.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:30.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:30.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:30 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3609683506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:47:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:30 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3609683506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:47:30.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:30.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:30.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:31.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:31.090 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:31.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:31.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:31.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:31.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:31.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:31.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:30 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3609683506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:47:31.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:31.342 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:31.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:31.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:31.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:31.468 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:31.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:31.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:31.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:31.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:31.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:31.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:31.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:31.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:31.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:31.821 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:31.839 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:31 smithi169 ceph-mon[29344]: pgmap v682: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 21 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 59/1656 objects degraded (3.563%) 2024-03-12T03:47:31.840 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:31 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:47:31.840 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:31.840 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:47:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:47:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:47:31.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:31.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:31.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:31.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:31.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:32.072 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:32.096 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:32.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:32.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:32.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:31 smithi169 ceph-mon[39949]: pgmap v682: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 21 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 59/1656 objects degraded (3.563%) 2024-03-12T03:47:32.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:31 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:47:32.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:32.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:32.223 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:32.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:31 smithi173 ceph-mon[33544]: pgmap v682: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 167 active+clean; 21 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 59/1656 objects degraded (3.563%) 2024-03-12T03:47:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:31 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:47:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:32.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:32.347 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:32.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:32.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:32.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:32.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:32.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:32.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:32.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:32.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:32.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:32.725 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:32.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:32.825 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:32.849 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:32.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:32.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:32.974 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:32.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:33.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:33.082 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:33.101 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:33.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:33.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:32 smithi169 ceph-mon[29344]: pgmap v683: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 887 op/s; 228 KiB/s, 3 objects/s recovering 2024-03-12T03:47:33.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:32 smithi169 ceph-mon[29344]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-12T03:47:33.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:32 smithi169 ceph-mon[29344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/1656 objects degraded (3.563%), 1 pg degraded, 1 pg undersized) 2024-03-12T03:47:33.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:32 smithi169 ceph-mon[39949]: pgmap v683: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 887 op/s; 228 KiB/s, 3 objects/s recovering 2024-03-12T03:47:33.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:32 smithi169 ceph-mon[39949]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-12T03:47:33.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:32 smithi169 ceph-mon[39949]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/1656 objects degraded (3.563%), 1 pg degraded, 1 pg undersized) 2024-03-12T03:47:33.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:33.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:33.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:33.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:33.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:47:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:47:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:32 smithi173 ceph-mon[33544]: pgmap v683: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 887 op/s; 228 KiB/s, 3 objects/s recovering 2024-03-12T03:47:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:32 smithi173 ceph-mon[33544]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-12T03:47:33.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:32 smithi173 ceph-mon[33544]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/1656 objects degraded (3.563%), 1 pg degraded, 1 pg undersized) 2024-03-12T03:47:33.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:33.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:33.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:33.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:33.488 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:33.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:33.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:33.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:33.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:33.618 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:33.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:33.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:33.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:33.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:33.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:33.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:33.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:33.896 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.6 to 1 (10000) 2024-03-12T03:47:33.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:34.002 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:34.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:34.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:34.127 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:34.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:34.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:34.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:34.253 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:34.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:34.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:34.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:34.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:34.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:34.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:34.508 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:34.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:34.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:34.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:34.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:34.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:34.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:34.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:34 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:47:34.670 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:34 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:47:34.671 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:34 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/682504929' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-12T03:47:34.671 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:34 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/682504929' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-12T03:47:34.671 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:34 smithi169 ceph-mon[29344]: osdmap e358: 8 total, 8 up, 8 in 2024-03-12T03:47:34.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:34.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:34 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:47:34.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:34 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:47:34.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:34 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/682504929' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-12T03:47:34.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:34 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/682504929' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-12T03:47:34.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:34 smithi169 ceph-mon[39949]: osdmap e358: 8 total, 8 up, 8 in 2024-03-12T03:47:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:34 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:47:34.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:34 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:47:34.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:34 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/682504929' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-12T03:47:34.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:34 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/682504929' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-12T03:47:34.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:34 smithi173 ceph-mon[33544]: osdmap e358: 8 total, 8 up, 8 in 2024-03-12T03:47:34.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:34.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:34.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:34.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:34.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:34.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:34.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:34.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:47:34.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:35.020 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:35.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:35.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:35.145 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:35.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:35.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:35.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:35.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:35.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:35.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:35.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:35.401 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:35.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:35.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:35.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:35.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:35.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:35.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:35.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:35.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:35.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:35 smithi169 ceph-mon[29344]: pgmap v685: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 703 B/s rd, 0 B/s wr, 488 op/s; 229 KiB/s, 3 objects/s recovering 2024-03-12T03:47:35.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:35 smithi169 ceph-mon[39949]: pgmap v685: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 703 B/s rd, 0 B/s wr, 488 op/s; 229 KiB/s, 3 objects/s recovering 2024-03-12T03:47:35.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:35 smithi173 ceph-mon[33544]: pgmap v685: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 703 B/s rd, 0 B/s wr, 488 op/s; 229 KiB/s, 3 objects/s recovering 2024-03-12T03:47:35.760 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:35.785 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:35.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:35.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:35.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:35.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:35.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:36.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:36.041 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:36.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:36.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:36.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:36.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:36.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-12T03:47:36.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:36.295 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:36.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:36.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:36.426 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:36.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:36.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:36.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:36.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:36.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:36.680 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:36.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:36.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:36.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:36.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:36.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:36.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:36.912 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:36.935 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:36.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:36.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:36.999 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:47:36.999 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:47:36.999 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-03-12T03:47:37.000 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-03-12T03:47:37.000 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:47:37.037 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:37.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:37.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:37.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:37.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:37.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:37.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:37.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:37.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:37.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:37.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:37.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:37.452 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:37.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:37.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:37.579 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:37.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:37.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:37 smithi169 ceph-mon[39949]: pgmap v686: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 618 B/s rd, 0 B/s wr, 184 op/s; 201 KiB/s, 3 objects/s recovering 2024-03-12T03:47:37.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:37 smithi169 ceph-mon[29344]: pgmap v686: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 618 B/s rd, 0 B/s wr, 184 op/s; 201 KiB/s, 3 objects/s recovering 2024-03-12T03:47:37.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:37.704 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:37.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:37.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:37 smithi173 ceph-mon[33544]: pgmap v686: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 618 B/s rd, 0 B/s wr, 184 op/s; 201 KiB/s, 3 objects/s recovering 2024-03-12T03:47:37.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-12T03:47:37.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:37.837 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:37.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:37.940 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:37.945 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:37.961 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:37.975 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:37.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:37.994 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:38.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:38.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:38.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:38.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:38.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:38.232 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:38.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:38.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:38.360 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:38.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:38.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:38.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:38.487 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:38.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:38.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:38.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:38.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:38.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:38.741 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:38.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:38.843 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:38.866 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:38.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:38.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:38.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:38.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:39.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:39.124 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:39.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:39.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:39.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:39.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:39.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:39.354 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:39.378 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:39.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:39.431 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:47:39.480 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:39.503 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:39.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:39.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:39.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:39.629 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:39.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:39 smithi169 ceph-mon[29344]: pgmap v687: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 153 op/s; 166 KiB/s, 2 objects/s recovering 2024-03-12T03:47:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:39 smithi169 ceph-mon[29344]: osdmap e359: 8 total, 8 up, 8 in 2024-03-12T03:47:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:39 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/331690811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:47:39.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:39 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:47:39.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:39 smithi169 ceph-mon[39949]: pgmap v687: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 153 op/s; 166 KiB/s, 2 objects/s recovering 2024-03-12T03:47:39.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:39 smithi169 ceph-mon[39949]: osdmap e359: 8 total, 8 up, 8 in 2024-03-12T03:47:39.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:39 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/331690811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:47:39.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:39 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:47:39.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:39 smithi173 ceph-mon[33544]: pgmap v687: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 153 op/s; 166 KiB/s, 2 objects/s recovering 2024-03-12T03:47:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:39 smithi173 ceph-mon[33544]: osdmap e359: 8 total, 8 up, 8 in 2024-03-12T03:47:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:39 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/331690811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:47:39.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:39 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:47:39.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:39.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:39.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:39.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:39.884 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:39.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:39.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:40.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:40.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:40.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:40.235 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:40.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:40.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:40.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:40.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:40.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:40.494 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:40.517 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:40.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:40.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:40.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:40.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:40.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:40.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:40 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:47:40.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:40 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:47:40.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:40 smithi169 ceph-mon[29344]: osdmap e360: 8 total, 8 up, 8 in 2024-03-12T03:47:40.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:40 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:47:40.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:40 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:47:40.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:40 smithi169 ceph-mon[39949]: osdmap e360: 8 total, 8 up, 8 in 2024-03-12T03:47:40.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:40 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:47:40.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:40 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:47:40.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:40 smithi173 ceph-mon[33544]: osdmap e360: 8 total, 8 up, 8 in 2024-03-12T03:47:40.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:40.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:40.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:40.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:40.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:40.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:40.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:41.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:41.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:41.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:41.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:41.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:41.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:41.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:41.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:41.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:41.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:41.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:41.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:41.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:41.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:41.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:41.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:41.529 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:41.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:41.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:41.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:41.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:41.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:41.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:41 smithi169 ceph-mon[29344]: pgmap v690: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 825 B/s rd, 0 op/s 2024-03-12T03:47:41.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:41 smithi169 ceph-mon[39949]: pgmap v690: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 825 B/s rd, 0 op/s 2024-03-12T03:47:41.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:41 smithi173 ceph-mon[33544]: pgmap v690: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 825 B/s rd, 0 op/s 2024-03-12T03:47:41.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:41.780 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:41.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:41.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:41.882 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:41.907 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:41.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:42.007 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:42.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:42.033 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:42.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:42.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:42.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:42.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:42.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:47:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:47:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:47:42.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:42.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:42.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:42.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:42.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:42.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:42.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:42.453 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:42.520 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:42.543 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:42.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:42.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:42.672 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:42.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:42.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:42.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:42.799 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:42.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:42.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:42.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:42.914 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:47:42.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:42.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:42.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:43.032 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:43.056 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:43.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:43.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:43.157 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:43.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:43.181 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:43.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:43.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:43.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:43.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:43.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:43.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:47:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:47:43.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:43.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:43.465 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'unique_pool_2' removed 2024-03-12T03:47:43.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:43.565 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:43.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:43.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:43.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:43.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:43 smithi169 ceph-mon[29344]: pgmap v691: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:43 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:47:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:43 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:47:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:43 smithi169 ceph-mon[29344]: osdmap e361: 8 total, 8 up, 8 in 2024-03-12T03:47:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:43 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2955691440' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:47:43.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:43 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:47:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:43 smithi169 ceph-mon[39949]: pgmap v691: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:43 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:47:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:43 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:47:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:43 smithi169 ceph-mon[39949]: osdmap e361: 8 total, 8 up, 8 in 2024-03-12T03:47:43.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:43 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2955691440' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:47:43.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:43 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:47:43.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:43.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:43.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:43 smithi173 ceph-mon[33544]: pgmap v691: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:43 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:47:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:43 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:47:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:43 smithi173 ceph-mon[33544]: osdmap e361: 8 total, 8 up, 8 in 2024-03-12T03:47:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:43 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2955691440' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:47:43.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:43 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:47:43.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:43.824 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:43.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:43.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:43.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:43.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:44.028 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:47:44.049 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:47:44.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:44.068 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:47:44.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:44.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:44.088 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:47:44.109 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:47:44.128 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:47:44.148 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:47:44.169 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:47:44.171 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:47:44.171 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:47:44.171 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:47:44.171 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:47:44.171 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:47:44.171 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:47:44.172 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:47:44.172 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:47:44.180 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:44.205 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:44.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:44.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:44.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:44.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:44.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:44.434 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:44.459 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:44.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:44.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:44.585 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:44.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:44.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:44.687 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:44.711 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:44.711 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:44 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:47:44.711 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:44 smithi169 ceph-mon[29344]: osdmap e362: 8 total, 8 up, 8 in 2024-03-12T03:47:44.711 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:44 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4002820116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:47:44.711 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:44.712 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:44 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:47:44.712 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:44 smithi169 ceph-mon[39949]: osdmap e362: 8 total, 8 up, 8 in 2024-03-12T03:47:44.712 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:44 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4002820116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:47:44.712 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:44.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:44.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:44 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:47:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:44 smithi173 ceph-mon[33544]: osdmap e362: 8 total, 8 up, 8 in 2024-03-12T03:47:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:44 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4002820116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:47:44.814 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:44.838 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:44.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:44.940 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:44.964 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:44.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:45.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:45.090 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:45.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:45.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:45.192 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:45.215 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:45.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:45.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:45.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:45.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:45.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:45.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:45.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:45.448 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.7 to 1 (10000) 2024-03-12T03:47:45.466 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:45.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:45.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:45.591 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:45.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:45.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:45.693 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:45.717 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:45.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:45.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:45 smithi173 ceph-mon[33544]: pgmap v694: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 720 B/s rd, 0 op/s 2024-03-12T03:47:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:45 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1502748497' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:47:45.771 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:45.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:45.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:45.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:45.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:45.859 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:45 smithi169 ceph-mon[29344]: pgmap v694: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 720 B/s rd, 0 op/s 2024-03-12T03:47:45.860 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:45 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1502748497' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:47:45.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:45 smithi169 ceph-mon[39949]: pgmap v694: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 720 B/s rd, 0 op/s 2024-03-12T03:47:45.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:45 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1502748497' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:47:45.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:45.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:45.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:45.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:46.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:46.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:46.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:46.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:46.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-12T03:47:46.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:46.251 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:46.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:46.352 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:46.378 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:46.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:46.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:46.504 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:46.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:46.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:46.629 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:46.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:46.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:46 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:47:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:46 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:47:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1502748497' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-12T03:47:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:46 smithi173 ceph-mon[33544]: osdmap e363: 8 total, 8 up, 8 in 2024-03-12T03:47:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:46 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:47:46.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:46 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:46.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:46.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:46.764 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:46 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:47:46.765 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:46 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:47:46.765 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1502748497' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-12T03:47:46.765 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:46 smithi169 ceph-mon[29344]: osdmap e363: 8 total, 8 up, 8 in 2024-03-12T03:47:46.765 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:46 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:47:46.765 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:46 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:46.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:46 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:47:46.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:46 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:47:46.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1502748497' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-12T03:47:46.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:46 smithi169 ceph-mon[39949]: osdmap e363: 8 total, 8 up, 8 in 2024-03-12T03:47:46.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:46 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:47:46.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:46 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:46.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:46.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:46.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:46.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:47.013 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:47.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:47.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:47.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:47.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:47.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:47.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:47.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:47.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:47.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:47.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:47.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:47.524 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:47.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:47.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:47.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:47.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:47.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:47 smithi173 ceph-mon[33544]: pgmap v696: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:47:47.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:47.769 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:47 smithi169 ceph-mon[29344]: pgmap v696: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:47:47.769 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:47 smithi169 ceph-mon[39949]: pgmap v696: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:47:47.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:47.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:47.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:47.903 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-12T03:47:47.903 INFO:tasks.radosbench:Beginning radosbench... 2024-03-12T03:47:47.904 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-12T03:47:47.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-12T03:47:47.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:47.911 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:47.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:48.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:48.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:48.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:48.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:48.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:48.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:48.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:48.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:48.308 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:47:48.308 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:47:48.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:48.421 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:48.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:48.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:48.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:48.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:48.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:48.687 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:48.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:48.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:48.791 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:47:48.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:48.813 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:48.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:48.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:48.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:48.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:48.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:49.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-12T03:47:49.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:49.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:49.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:49.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:49.173 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:49.198 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:49.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:49.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:49.323 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:49.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:49.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:49.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:49.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:49.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:49.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:49.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:49.581 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:49.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:49.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:49.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:49.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:49.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:49.719 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:49 smithi169 ceph-mon[29344]: pgmap v697: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:47:49.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:49 smithi169 ceph-mon[39949]: pgmap v697: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:47:49.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:49 smithi173 ceph-mon[33544]: pgmap v697: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:47:49.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:49.832 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:49.832 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:49.833 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:49.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:49.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:49.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:49.970 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:49.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:49.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:50.046 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:50.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:50.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:50.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:50.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:50.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:50.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:50.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:50.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:50.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:50.357 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:50.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:50.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:50.479 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:47:50.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:50.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:50.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:50.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:50.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:50.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:50.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:50 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/193747810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:47:50.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:50 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:47:50.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:50 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/193747810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:47:50.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:50 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:47:50.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:50 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/193747810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:47:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:50 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:47:50.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:50.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:50.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:50.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:50.872 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:50.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:50.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:51.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:51.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:51.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:51.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:51.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:51.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:51.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:51.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:51.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:51.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:51.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:51.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:51.388 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:51.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:51.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:51.516 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:51.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:51.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:51.642 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:51.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:51.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:51 smithi173 ceph-mon[33544]: pgmap v698: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 534 B/s rd, 0 op/s 2024-03-12T03:47:51.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:51 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:47:51.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:51 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:47:51.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:51 smithi173 ceph-mon[33544]: osdmap e364: 8 total, 8 up, 8 in 2024-03-12T03:47:51.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:51.767 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:51.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:51 smithi169 ceph-mon[39949]: pgmap v698: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 534 B/s rd, 0 op/s 2024-03-12T03:47:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:51 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:47:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:51 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:47:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:51 smithi169 ceph-mon[39949]: osdmap e364: 8 total, 8 up, 8 in 2024-03-12T03:47:51.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:51 smithi169 ceph-mon[29344]: pgmap v698: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 534 B/s rd, 0 op/s 2024-03-12T03:47:51.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:51 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:47:51.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:51 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:47:51.824 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:51 smithi169 ceph-mon[29344]: osdmap e364: 8 total, 8 up, 8 in 2024-03-12T03:47:51.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:51.893 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:51.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:51.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:51.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:52.018 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:52.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:52.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:52.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:52.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:52.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:47:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:47:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:47:52.248 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:52.272 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:52.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:52.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:52.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:52.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:52.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:52.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:52.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:52.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:52.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:52.652 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:52.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:52.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:52 smithi169 ceph-mon[29344]: 6.7 scrub starts 2024-03-12T03:47:52.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:52 smithi169 ceph-mon[29344]: 6.7 scrub ok 2024-03-12T03:47:52.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:52 smithi169 ceph-mon[39949]: 6.7 scrub starts 2024-03-12T03:47:52.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:52 smithi169 ceph-mon[39949]: 6.7 scrub ok 2024-03-12T03:47:52.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:52 smithi173 ceph-mon[33544]: 6.7 scrub starts 2024-03-12T03:47:52.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:52 smithi173 ceph-mon[33544]: 6.7 scrub ok 2024-03-12T03:47:52.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:52.778 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:52.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:52.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:52.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:52.906 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:52.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:53.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:53.032 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:53.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:53.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:53.156 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:53.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:53.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:53.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:53.282 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:53.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:53.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:53.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:53.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:53.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:53.475 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:47:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:47:53.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:53.539 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:53.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:53.629 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:47:53.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:53.653 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:47:53.669 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:53.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:53.675 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:47:53.696 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:47:53.719 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:47:53.726 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:53 smithi169 ceph-mon[39949]: pgmap v700: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:53.726 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:53 smithi169 ceph-mon[39949]: osdmap e365: 8 total, 8 up, 8 in 2024-03-12T03:47:53.727 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:53 smithi169 ceph-mon[29344]: pgmap v700: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:53.727 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:53 smithi169 ceph-mon[29344]: osdmap e365: 8 total, 8 up, 8 in 2024-03-12T03:47:53.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:53 smithi173 ceph-mon[33544]: pgmap v700: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:53.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:53 smithi173 ceph-mon[33544]: osdmap e365: 8 total, 8 up, 8 in 2024-03-12T03:47:53.740 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:47:53.760 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:47:53.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:53.781 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:47:53.783 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:47:53.783 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:47:53.783 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:47:53.783 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:47:53.783 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:47:53.783 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:47:53.783 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:47:53.783 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:47:53.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:53.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:53.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:53.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:53.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:53.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:54.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:54.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:54.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:54.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:54.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:54.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:54.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:54.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:54.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:54.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:54.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:54.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:54.443 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:54.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:54.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:54.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:54.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:54.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:54.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:54.696 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:54.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:54.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:54 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:54.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:54 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/305400953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:47:54.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:54 smithi173 ceph-mon[33544]: 3.13 scrub starts 2024-03-12T03:47:54.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:54 smithi173 ceph-mon[33544]: 3.13 scrub ok 2024-03-12T03:47:54.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:54.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:54.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:54 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:54 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/305400953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:47:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:54 smithi169 ceph-mon[29344]: 3.13 scrub starts 2024-03-12T03:47:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:54 smithi169 ceph-mon[29344]: 3.13 scrub ok 2024-03-12T03:47:54.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:54 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:54 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/305400953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:47:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:54 smithi169 ceph-mon[39949]: 3.13 scrub starts 2024-03-12T03:47:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:54 smithi169 ceph-mon[39949]: 3.13 scrub ok 2024-03-12T03:47:54.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:54.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:55.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:55.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:55.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:55.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:55.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:55.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:55.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:55.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:55.329 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:55.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:55.431 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:55.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:55.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:55.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:55.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:55.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:55.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:55.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:55.633 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[29344]: 5.e scrub starts 2024-03-12T03:47:55.633 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[29344]: 5.e scrub ok 2024-03-12T03:47:55.633 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[29344]: 6.16 scrub starts 2024-03-12T03:47:55.633 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[29344]: 6.16 scrub ok 2024-03-12T03:47:55.633 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[29344]: pgmap v702: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:55.633 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[29344]: 6.11 scrub starts 2024-03-12T03:47:55.633 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[29344]: 6.11 scrub ok 2024-03-12T03:47:55.633 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[29344]: 6.19 scrub starts 2024-03-12T03:47:55.633 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[29344]: 6.19 scrub ok 2024-03-12T03:47:55.633 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[39949]: 5.e scrub starts 2024-03-12T03:47:55.633 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[39949]: 5.e scrub ok 2024-03-12T03:47:55.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[39949]: 6.16 scrub starts 2024-03-12T03:47:55.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[39949]: 6.16 scrub ok 2024-03-12T03:47:55.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[39949]: pgmap v702: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:55.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[39949]: 6.11 scrub starts 2024-03-12T03:47:55.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[39949]: 6.11 scrub ok 2024-03-12T03:47:55.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[39949]: 6.19 scrub starts 2024-03-12T03:47:55.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:55 smithi169 ceph-mon[39949]: 6.19 scrub ok 2024-03-12T03:47:55.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:55.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:55.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:55.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:55.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:55 smithi173 ceph-mon[33544]: 5.e scrub starts 2024-03-12T03:47:55.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:55 smithi173 ceph-mon[33544]: 5.e scrub ok 2024-03-12T03:47:55.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:55 smithi173 ceph-mon[33544]: 6.16 scrub starts 2024-03-12T03:47:55.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:55 smithi173 ceph-mon[33544]: 6.16 scrub ok 2024-03-12T03:47:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:55 smithi173 ceph-mon[33544]: pgmap v702: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:55 smithi173 ceph-mon[33544]: 6.11 scrub starts 2024-03-12T03:47:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:55 smithi173 ceph-mon[33544]: 6.11 scrub ok 2024-03-12T03:47:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:55 smithi173 ceph-mon[33544]: 6.19 scrub starts 2024-03-12T03:47:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:55 smithi173 ceph-mon[33544]: 6.19 scrub ok 2024-03-12T03:47:55.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:55.836 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:55.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:55.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:55.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:55.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:55.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:56.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:56.089 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:56.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:56.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:56.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:56.214 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:56.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:56.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:56.339 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:56.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:56.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:56.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:56.466 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:56.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:56.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:56.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:47:56.592 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:56.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:47:56.615 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:47:56.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:56.697 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'unique_pool_3' created 2024-03-12T03:47:56.721 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:56.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:56.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:56.846 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:56.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:56.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:56.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:56 smithi169 ceph-mon[29344]: 4.14 scrub starts 2024-03-12T03:47:56.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:56 smithi169 ceph-mon[29344]: 4.14 scrub ok 2024-03-12T03:47:56.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:56 smithi169 ceph-mon[29344]: 4.4 scrub starts 2024-03-12T03:47:56.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:56 smithi169 ceph-mon[29344]: 4.4 scrub ok 2024-03-12T03:47:56.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:56 smithi169 ceph-mon[39949]: 4.14 scrub starts 2024-03-12T03:47:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:56 smithi169 ceph-mon[39949]: 4.14 scrub ok 2024-03-12T03:47:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:56 smithi169 ceph-mon[39949]: 4.4 scrub starts 2024-03-12T03:47:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:56 smithi169 ceph-mon[39949]: 4.4 scrub ok 2024-03-12T03:47:56.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:56.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:56.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:56.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:56 smithi173 ceph-mon[33544]: 4.14 scrub starts 2024-03-12T03:47:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:56 smithi173 ceph-mon[33544]: 4.14 scrub ok 2024-03-12T03:47:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:56 smithi173 ceph-mon[33544]: 4.4 scrub starts 2024-03-12T03:47:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:56 smithi173 ceph-mon[33544]: 4.4 scrub ok 2024-03-12T03:47:57.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:57.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:57.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:57.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:57.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:57.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:57.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:57.336 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:57.363 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:57.363 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:47:57.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:57.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:57.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:47:57.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:57.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:57.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:47:57.591 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:57.614 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:57.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:57.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:57.693 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.2 to 1 (10000) 2024-03-12T03:47:57.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:57.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:57.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:57.800 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-12T03:47:57.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:57.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:57.868 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:57.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[29344]: 5.18 scrub starts 2024-03-12T03:47:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[29344]: 5.18 scrub ok 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[29344]: pgmap v703: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1874845042' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[29344]: 3.1c scrub starts 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[29344]: 3.1c scrub ok 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[29344]: osdmap e366: 8 total, 8 up, 8 in 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[29344]: 4.1c scrub starts 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[29344]: 4.1c scrub ok 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/850342419' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[39949]: 5.18 scrub starts 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[39949]: 5.18 scrub ok 2024-03-12T03:47:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[39949]: pgmap v703: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:47:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1874845042' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-12T03:47:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-12T03:47:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[39949]: 3.1c scrub starts 2024-03-12T03:47:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[39949]: 3.1c scrub ok 2024-03-12T03:47:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-12T03:47:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[39949]: osdmap e366: 8 total, 8 up, 8 in 2024-03-12T03:47:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[39949]: 4.1c scrub starts 2024-03-12T03:47:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[39949]: 4.1c scrub ok 2024-03-12T03:47:57.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:57 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/850342419' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:47:57.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:57.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:47:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:57 smithi173 ceph-mon[33544]: 5.18 scrub starts 2024-03-12T03:47:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:57 smithi173 ceph-mon[33544]: 5.18 scrub ok 2024-03-12T03:47:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:57 smithi173 ceph-mon[33544]: pgmap v703: 152 pgs: 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:47:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:57 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1874845042' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-12T03:47:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:57 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-12T03:47:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:57 smithi173 ceph-mon[33544]: 3.1c scrub starts 2024-03-12T03:47:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:57 smithi173 ceph-mon[33544]: 3.1c scrub ok 2024-03-12T03:47:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:57 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-12T03:47:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:57 smithi173 ceph-mon[33544]: osdmap e366: 8 total, 8 up, 8 in 2024-03-12T03:47:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:57 smithi173 ceph-mon[33544]: 4.1c scrub starts 2024-03-12T03:47:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:57 smithi173 ceph-mon[33544]: 4.1c scrub ok 2024-03-12T03:47:57.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:57 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/850342419' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:47:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:57.993 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:57.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:58.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:47:58.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:58.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:47:58.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:58.244 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:58.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:58.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:58.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:58.370 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:58.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:58.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:58.495 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:58.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:58.546 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-12T03:47:58.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:47:58.597 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:58.620 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:58.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:58.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:58.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:58.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:58.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:58.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[29344]: 4.1 scrub starts 2024-03-12T03:47:58.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[29344]: 4.1 scrub ok 2024-03-12T03:47:58.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:47:58.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:47:58.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[29344]: 6.1b scrub starts 2024-03-12T03:47:58.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[29344]: 6.1b scrub ok 2024-03-12T03:47:58.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[29344]: 4.12 scrub starts 2024-03-12T03:47:58.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[29344]: 4.12 scrub ok 2024-03-12T03:47:58.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/850342419' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-12T03:47:58.790 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[29344]: osdmap e367: 8 total, 8 up, 8 in 2024-03-12T03:47:58.790 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[39949]: 4.1 scrub starts 2024-03-12T03:47:58.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[39949]: 4.1 scrub ok 2024-03-12T03:47:58.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:47:58.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:47:58.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[39949]: 6.1b scrub starts 2024-03-12T03:47:58.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[39949]: 6.1b scrub ok 2024-03-12T03:47:58.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[39949]: 4.12 scrub starts 2024-03-12T03:47:58.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[39949]: 4.12 scrub ok 2024-03-12T03:47:58.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/850342419' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-12T03:47:58.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:58 smithi169 ceph-mon[39949]: osdmap e367: 8 total, 8 up, 8 in 2024-03-12T03:47:58.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:58.877 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:58.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:58.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:58 smithi173 ceph-mon[33544]: 4.1 scrub starts 2024-03-12T03:47:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:58 smithi173 ceph-mon[33544]: 4.1 scrub ok 2024-03-12T03:47:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:58 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:47:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:58 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:47:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:58 smithi173 ceph-mon[33544]: 6.1b scrub starts 2024-03-12T03:47:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:58 smithi173 ceph-mon[33544]: 6.1b scrub ok 2024-03-12T03:47:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:58 smithi173 ceph-mon[33544]: 4.12 scrub starts 2024-03-12T03:47:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:58 smithi173 ceph-mon[33544]: 4.12 scrub ok 2024-03-12T03:47:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:58 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/850342419' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-12T03:47:58.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:58 smithi173 ceph-mon[33544]: osdmap e367: 8 total, 8 up, 8 in 2024-03-12T03:47:59.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:59.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:59.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:47:59.135 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:59.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:47:59.236 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:59.260 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:59.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:59.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:47:59.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:47:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:59.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:47:59.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:59.517 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:59.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:59.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:59.619 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:47:59.643 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:59.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:47:59.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[29344]: 3.9 scrub starts 2024-03-12T03:47:59.654 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[29344]: 3.9 scrub ok 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[29344]: 3.1 scrub starts 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[29344]: 3.1 scrub ok 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[29344]: pgmap v706: 168 pgs: 6 creating+peering, 10 unknown, 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[29344]: 4.1d scrub starts 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[29344]: 4.1d scrub ok 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[29344]: 1.10 scrub starts 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[29344]: 1.10 scrub ok 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[29344]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[29344]: osdmap e368: 8 total, 8 up, 8 in 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[39949]: 3.9 scrub starts 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[39949]: 3.9 scrub ok 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[39949]: 3.1 scrub starts 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[39949]: 3.1 scrub ok 2024-03-12T03:47:59.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[39949]: pgmap v706: 168 pgs: 6 creating+peering, 10 unknown, 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:59.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[39949]: 4.1d scrub starts 2024-03-12T03:47:59.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[39949]: 4.1d scrub ok 2024-03-12T03:47:59.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[39949]: 1.10 scrub starts 2024-03-12T03:47:59.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[39949]: 1.10 scrub ok 2024-03-12T03:47:59.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[39949]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:47:59.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:47:59 smithi169 ceph-mon[39949]: osdmap e368: 8 total, 8 up, 8 in 2024-03-12T03:47:59.744 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:47:59.775 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:59.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:59.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:47:59.792 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:47:59.792 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:47:59.792 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-03-12T03:47:59.792 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:47:59.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:47:59.901 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:47:59.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:47:59.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:47:59.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:59 smithi173 ceph-mon[33544]: 3.9 scrub starts 2024-03-12T03:47:59.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:59 smithi173 ceph-mon[33544]: 3.9 scrub ok 2024-03-12T03:47:59.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:59 smithi173 ceph-mon[33544]: 3.1 scrub starts 2024-03-12T03:47:59.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:59 smithi173 ceph-mon[33544]: 3.1 scrub ok 2024-03-12T03:47:59.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:59 smithi173 ceph-mon[33544]: pgmap v706: 168 pgs: 6 creating+peering, 10 unknown, 152 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:59.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:59 smithi173 ceph-mon[33544]: 4.1d scrub starts 2024-03-12T03:47:59.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:59 smithi173 ceph-mon[33544]: 4.1d scrub ok 2024-03-12T03:47:59.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:59 smithi173 ceph-mon[33544]: 1.10 scrub starts 2024-03-12T03:47:59.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:59 smithi173 ceph-mon[33544]: 1.10 scrub ok 2024-03-12T03:47:59.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:59 smithi173 ceph-mon[33544]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:47:59.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:47:59 smithi173 ceph-mon[33544]: osdmap e368: 8 total, 8 up, 8 in 2024-03-12T03:48:00.003 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:00.028 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:00.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:00.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:00.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:00.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:00.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:00.283 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:00.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:00.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:00.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:00.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:00.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:00.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:00.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:00.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:00.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:00.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:00.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:00.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:00.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:00.770 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:00.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:00.795 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:00.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:00.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:00.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:00.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:00.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:01.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-12T03:48:01.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:01.026 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:01.026 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:01.041 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:01.056 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:01.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:01.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:01.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:01.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:01.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:01.316 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:01.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:01.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:01.443 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:01.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:01.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:01.569 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:01.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:01.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:01.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:01.698 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:01.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:01.799 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:01.824 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:01.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:01.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:48:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:48:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[29344]: pgmap v708: 168 pgs: 6 creating+peering, 8 unknown, 154 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[29344]: 3.18 scrub starts 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[29344]: 3.18 scrub ok 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[29344]: 4.1f scrub starts 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[29344]: 4.1f scrub ok 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[29344]: 3.4 scrub starts 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[29344]: 3.4 scrub ok 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[39949]: pgmap v708: 168 pgs: 6 creating+peering, 8 unknown, 154 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[39949]: 3.18 scrub starts 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[39949]: 3.18 scrub ok 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[39949]: 4.1f scrub starts 2024-03-12T03:48:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[39949]: 4.1f scrub ok 2024-03-12T03:48:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[39949]: 3.4 scrub starts 2024-03-12T03:48:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[39949]: 3.4 scrub ok 2024-03-12T03:48:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:01 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:01.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:01.949 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:01.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:01.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:01 smithi173 ceph-mon[33544]: pgmap v708: 168 pgs: 6 creating+peering, 8 unknown, 154 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:01 smithi173 ceph-mon[33544]: 3.18 scrub starts 2024-03-12T03:48:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:01 smithi173 ceph-mon[33544]: 3.18 scrub ok 2024-03-12T03:48:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:01 smithi173 ceph-mon[33544]: 4.1f scrub starts 2024-03-12T03:48:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:01 smithi173 ceph-mon[33544]: 4.1f scrub ok 2024-03-12T03:48:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:01 smithi173 ceph-mon[33544]: 3.4 scrub starts 2024-03-12T03:48:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:01 smithi173 ceph-mon[33544]: 3.4 scrub ok 2024-03-12T03:48:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:01 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:01 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:02.050 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:02.074 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:02.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:02.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:02.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:02.204 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:02.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:02.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:02.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:02.330 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:02.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:02.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:02.455 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:02.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:02.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:02.581 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:02.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:02.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:02.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:02.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:02.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:02.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:02.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:02.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:02 smithi169 ceph-mon[29344]: 6.1e scrub starts 2024-03-12T03:48:02.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:02 smithi169 ceph-mon[29344]: 6.1e scrub ok 2024-03-12T03:48:02.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:02 smithi169 ceph-mon[29344]: 1.2 scrub starts 2024-03-12T03:48:02.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:02 smithi169 ceph-mon[29344]: 1.2 scrub ok 2024-03-12T03:48:02.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:02 smithi169 ceph-mon[29344]: 5.a scrub starts 2024-03-12T03:48:02.914 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:02 smithi169 ceph-mon[29344]: 5.a scrub ok 2024-03-12T03:48:02.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:02 smithi169 ceph-mon[39949]: 6.1e scrub starts 2024-03-12T03:48:02.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:02 smithi169 ceph-mon[39949]: 6.1e scrub ok 2024-03-12T03:48:02.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:02 smithi169 ceph-mon[39949]: 1.2 scrub starts 2024-03-12T03:48:02.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:02 smithi169 ceph-mon[39949]: 1.2 scrub ok 2024-03-12T03:48:02.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:02 smithi169 ceph-mon[39949]: 5.a scrub starts 2024-03-12T03:48:02.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:02 smithi169 ceph-mon[39949]: 5.a scrub ok 2024-03-12T03:48:02.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:02.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:02.960 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:02.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:02 smithi173 ceph-mon[33544]: 6.1e scrub starts 2024-03-12T03:48:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:02 smithi173 ceph-mon[33544]: 6.1e scrub ok 2024-03-12T03:48:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:02 smithi173 ceph-mon[33544]: 1.2 scrub starts 2024-03-12T03:48:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:02 smithi173 ceph-mon[33544]: 1.2 scrub ok 2024-03-12T03:48:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:02 smithi173 ceph-mon[33544]: 5.a scrub starts 2024-03-12T03:48:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:02 smithi173 ceph-mon[33544]: 5.a scrub ok 2024-03-12T03:48:03.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:03.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:03.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:03.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:03.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:03.211 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:03.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:03.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:03.313 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:03.337 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:03.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:03.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:03.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:03.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:03.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:03.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:48:03.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:03.588 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:03.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:03.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:03.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:03.714 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:03.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:03.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:03.839 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:03.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:03.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:03.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[39949]: 3.1a scrub starts 2024-03-12T03:48:03.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[39949]: 3.1a scrub ok 2024-03-12T03:48:03.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[39949]: pgmap v709: 168 pgs: 3 creating+peering, 165 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:48:03.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[39949]: 6.1f scrub starts 2024-03-12T03:48:03.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[39949]: 6.1f scrub ok 2024-03-12T03:48:03.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[39949]: 3.16 scrub starts 2024-03-12T03:48:03.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[39949]: 3.16 scrub ok 2024-03-12T03:48:03.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[39949]: 6.f scrub starts 2024-03-12T03:48:03.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[39949]: 6.f scrub ok 2024-03-12T03:48:03.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[39949]: osdmap e369: 8 total, 8 up, 8 in 2024-03-12T03:48:03.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1077816928' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[29344]: 3.1a scrub starts 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[29344]: 3.1a scrub ok 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[29344]: pgmap v709: 168 pgs: 3 creating+peering, 165 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[29344]: 6.1f scrub starts 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[29344]: 6.1f scrub ok 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[29344]: 3.16 scrub starts 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[29344]: 3.16 scrub ok 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[29344]: 6.f scrub starts 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[29344]: 6.f scrub ok 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[29344]: osdmap e369: 8 total, 8 up, 8 in 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1077816928' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:48:03.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:03 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:48:03.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:03.967 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:03.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:03.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-mon[33544]: 3.1a scrub starts 2024-03-12T03:48:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-mon[33544]: 3.1a scrub ok 2024-03-12T03:48:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-mon[33544]: pgmap v709: 168 pgs: 3 creating+peering, 165 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:48:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-mon[33544]: 6.1f scrub starts 2024-03-12T03:48:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-mon[33544]: 6.1f scrub ok 2024-03-12T03:48:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-mon[33544]: 3.16 scrub starts 2024-03-12T03:48:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-mon[33544]: 3.16 scrub ok 2024-03-12T03:48:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-mon[33544]: 6.f scrub starts 2024-03-12T03:48:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-mon[33544]: 6.f scrub ok 2024-03-12T03:48:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-mon[33544]: osdmap e369: 8 total, 8 up, 8 in 2024-03-12T03:48:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1077816928' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:48:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:03 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:48:04.068 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:04.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:04.094 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:04.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:04.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:04.219 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:04.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:04.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:04.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:04.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:04.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:04.437 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:48:04.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:04.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:04.474 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:04.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:04.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:04.601 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:04.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:04.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:04.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:04.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:04.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:04.852 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:04.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:04.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[39949]: 5.1f scrub starts 2024-03-12T03:48:04.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[39949]: 5.1f scrub ok 2024-03-12T03:48:04.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:04.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[39949]: 3.d scrub starts 2024-03-12T03:48:04.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[39949]: 3.d scrub ok 2024-03-12T03:48:04.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:48:04.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:48:04.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[39949]: osdmap e370: 8 total, 8 up, 8 in 2024-03-12T03:48:04.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[29344]: 5.1f scrub starts 2024-03-12T03:48:04.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[29344]: 5.1f scrub ok 2024-03-12T03:48:04.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:04.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[29344]: 3.d scrub starts 2024-03-12T03:48:04.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[29344]: 3.d scrub ok 2024-03-12T03:48:04.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:48:04.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:48:04.927 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:04 smithi169 ceph-mon[29344]: osdmap e370: 8 total, 8 up, 8 in 2024-03-12T03:48:04.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:04.976 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:04.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:04.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:04 smithi173 ceph-mon[33544]: 5.1f scrub starts 2024-03-12T03:48:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:04 smithi173 ceph-mon[33544]: 5.1f scrub ok 2024-03-12T03:48:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:04 smithi173 ceph-mon[33544]: 3.d scrub starts 2024-03-12T03:48:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:04 smithi173 ceph-mon[33544]: 3.d scrub ok 2024-03-12T03:48:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:04 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:48:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:04 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:48:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:04 smithi173 ceph-mon[33544]: osdmap e370: 8 total, 8 up, 8 in 2024-03-12T03:48:05.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:05.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:05.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:05.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:05.229 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:05.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:05.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:05.354 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:05.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:05.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:05.391 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:48:05.418 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:48:05.437 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:48:05.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:05.457 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:48:05.476 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:48:05.480 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:05.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:05.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:05.495 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:48:05.515 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:48:05.535 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:48:05.536 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:48:05.536 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:48:05.536 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:48:05.536 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:48:05.536 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:48:05.536 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:48:05.536 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:48:05.536 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:48:05.580 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:05.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:05.606 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:05.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:05.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:05.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:05.732 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:05.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:05.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:05.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:05.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:05.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:05.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[29344]: pgmap v711: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-03-12T03:48:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[29344]: 3.11 scrub starts 2024-03-12T03:48:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[29344]: 3.11 scrub ok 2024-03-12T03:48:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[29344]: 6.b scrub starts 2024-03-12T03:48:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[29344]: 6.d scrub starts 2024-03-12T03:48:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[29344]: 6.d scrub ok 2024-03-12T03:48:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2574167805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:48:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[39949]: pgmap v711: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-03-12T03:48:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[39949]: 3.11 scrub starts 2024-03-12T03:48:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[39949]: 3.11 scrub ok 2024-03-12T03:48:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[39949]: 6.b scrub starts 2024-03-12T03:48:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[39949]: 6.d scrub starts 2024-03-12T03:48:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[39949]: 6.d scrub ok 2024-03-12T03:48:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:05 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2574167805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:48:05.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:05.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:05.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:05 smithi173 ceph-mon[33544]: pgmap v711: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-03-12T03:48:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:05 smithi173 ceph-mon[33544]: 3.11 scrub starts 2024-03-12T03:48:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:05 smithi173 ceph-mon[33544]: 3.11 scrub ok 2024-03-12T03:48:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:05 smithi173 ceph-mon[33544]: 6.b scrub starts 2024-03-12T03:48:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:05 smithi173 ceph-mon[33544]: 6.d scrub starts 2024-03-12T03:48:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:05 smithi173 ceph-mon[33544]: 6.d scrub ok 2024-03-12T03:48:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:05 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2574167805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:48:06.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:06.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:06.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:06.235 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:06.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:06.337 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:06.362 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:06.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:06.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:06.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:06.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:06.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:06.615 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:06.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:06.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:06.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:06.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:06.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:06.865 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:06.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:06 smithi169 ceph-mon[29344]: 6.b scrub ok 2024-03-12T03:48:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:06 smithi169 ceph-mon[29344]: 3.7 scrub starts 2024-03-12T03:48:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:06 smithi169 ceph-mon[29344]: 3.7 scrub ok 2024-03-12T03:48:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:06 smithi169 ceph-mon[29344]: 5.19 scrub starts 2024-03-12T03:48:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:06 smithi169 ceph-mon[29344]: 5.19 scrub ok 2024-03-12T03:48:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:06 smithi169 ceph-mon[39949]: 6.b scrub ok 2024-03-12T03:48:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:06 smithi169 ceph-mon[39949]: 3.7 scrub starts 2024-03-12T03:48:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:06 smithi169 ceph-mon[39949]: 3.7 scrub ok 2024-03-12T03:48:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:06 smithi169 ceph-mon[39949]: 5.19 scrub starts 2024-03-12T03:48:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:06 smithi169 ceph-mon[39949]: 5.19 scrub ok 2024-03-12T03:48:06.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:06 smithi173 ceph-mon[33544]: 6.b scrub ok 2024-03-12T03:48:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:06 smithi173 ceph-mon[33544]: 3.7 scrub starts 2024-03-12T03:48:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:06 smithi173 ceph-mon[33544]: 3.7 scrub ok 2024-03-12T03:48:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:06 smithi173 ceph-mon[33544]: 5.19 scrub starts 2024-03-12T03:48:06.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:06 smithi173 ceph-mon[33544]: 5.19 scrub ok 2024-03-12T03:48:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:06.991 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:06.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:07.020 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:48:07.023 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:48:07.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:07.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:07.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:07.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:07.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:07.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:07.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:07.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:07.366 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:07.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:07.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:07.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:07.492 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:07.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:07.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:07.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:07.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:07.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:07.690 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[29344]: pgmap v713: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-03-12T03:48:07.690 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[29344]: 4.16 scrub starts 2024-03-12T03:48:07.690 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[29344]: 4.16 scrub ok 2024-03-12T03:48:07.690 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[29344]: 4.10 scrub starts 2024-03-12T03:48:07.690 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[29344]: 4.10 scrub ok 2024-03-12T03:48:07.690 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[29344]: 1.1 scrub starts 2024-03-12T03:48:07.690 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[29344]: 1.1 scrub ok 2024-03-12T03:48:07.690 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[29344]: 2.1 scrub starts 2024-03-12T03:48:07.691 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[29344]: 2.1 scrub ok 2024-03-12T03:48:07.691 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[39949]: pgmap v713: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-03-12T03:48:07.691 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[39949]: 4.16 scrub starts 2024-03-12T03:48:07.691 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[39949]: 4.16 scrub ok 2024-03-12T03:48:07.691 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[39949]: 4.10 scrub starts 2024-03-12T03:48:07.691 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[39949]: 4.10 scrub ok 2024-03-12T03:48:07.691 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[39949]: 1.1 scrub starts 2024-03-12T03:48:07.691 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[39949]: 1.1 scrub ok 2024-03-12T03:48:07.691 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[39949]: 2.1 scrub starts 2024-03-12T03:48:07.691 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:07 smithi169 ceph-mon[39949]: 2.1 scrub ok 2024-03-12T03:48:07.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:07.744 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:07.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:07.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:07.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:07.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:07.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:07 smithi173 ceph-mon[33544]: pgmap v713: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-03-12T03:48:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:07 smithi173 ceph-mon[33544]: 4.16 scrub starts 2024-03-12T03:48:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:07 smithi173 ceph-mon[33544]: 4.16 scrub ok 2024-03-12T03:48:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:07 smithi173 ceph-mon[33544]: 4.10 scrub starts 2024-03-12T03:48:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:07 smithi173 ceph-mon[33544]: 4.10 scrub ok 2024-03-12T03:48:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:07 smithi173 ceph-mon[33544]: 1.1 scrub starts 2024-03-12T03:48:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:07 smithi173 ceph-mon[33544]: 1.1 scrub ok 2024-03-12T03:48:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:07 smithi173 ceph-mon[33544]: 2.1 scrub starts 2024-03-12T03:48:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:07 smithi173 ceph-mon[33544]: 2.1 scrub ok 2024-03-12T03:48:07.996 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:07.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:08.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:08.121 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:08.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:08.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:08.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:08.246 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:08.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:08.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:08.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:48:08.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:08.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:08.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:08.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:08.499 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:08.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:08.600 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:08.624 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:08.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:08.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:08.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:08.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:08.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:08.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:08.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:08.875 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:08.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:08 smithi169 ceph-mon[29344]: from='client.25448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:48:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:08 smithi169 ceph-mon[29344]: osdmap e371: 8 total, 8 up, 8 in 2024-03-12T03:48:08.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:08 smithi169 ceph-mon[39949]: from='client.25448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:48:08.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:08 smithi169 ceph-mon[39949]: osdmap e371: 8 total, 8 up, 8 in 2024-03-12T03:48:08.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:08.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:08 smithi173 ceph-mon[33544]: from='client.25448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:48:08.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:08 smithi173 ceph-mon[33544]: osdmap e371: 8 total, 8 up, 8 in 2024-03-12T03:48:09.001 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:09.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:09.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:09.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:09.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:09.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:09.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:09.252 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:09.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:09.354 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:09.377 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:09.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:09.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:48:09.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:09.529 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:09.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:09.630 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:09.641 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:09.661 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:09.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:09.663 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":713,"stamp":"2024-03-12T03:48:06.100726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":2292,"num_read_kb":2012,"num_write":914,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":530222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3743,"ondisk_log_size":3743,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":501,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4293568,"kb_used_data":18844,"kb_used_omap":24,"kb_used_meta":4274599,"kb_avail":745667648,"statfs":{"total":767960285184,"available":763563671552,"internally_reserved":0,"allocated":19296256,"data_stored":11364272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24644,"internal_metadata":4377190332},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.401581"},"pg_stats":[{"pgid":"10.2","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063607+0000","last_change":"2024-03-12T03:47:57.698728+0000","last_active":"2024-03-12T03:47:59.063607+0000","last_peered":"2024-03-12T03:47:59.063607+0000","last_clean":"2024-03-12T03:47:59.063607+0000","last_became_active":"2024-03-12T03:47:57.698438+0000","last_became_peered":"2024-03-12T03:47:57.698438+0000","last_unstale":"2024-03-12T03:47:59.063607+0000","last_undegraded":"2024-03-12T03:47:59.063607+0000","last_fullsized":"2024-03-12T03:47:59.063607+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:04.589431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":970,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065127+0000","last_change":"2024-03-12T03:46:52.574410+0000","last_active":"2024-03-12T03:47:59.065127+0000","last_peered":"2024-03-12T03:47:59.065127+0000","last_clean":"2024-03-12T03:47:59.065127+0000","last_became_active":"2024-03-12T03:46:25.289005+0000","last_became_peered":"2024-03-12T03:46:25.289005+0000","last_unstale":"2024-03-12T03:47:59.065127+0000","last_undegraded":"2024-03-12T03:47:59.065127+0000","last_fullsized":"2024-03-12T03:47:59.065127+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:46:52.574274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:46:52.574274+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:08.271467+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"344'25","reported_seq":720,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065724+0000","last_change":"2024-03-12T03:46:57.114034+0000","last_active":"2024-03-12T03:47:59.065724+0000","last_peered":"2024-03-12T03:47:59.065724+0000","last_clean":"2024-03-12T03:47:59.065724+0000","last_became_active":"2024-03-12T03:46:25.301377+0000","last_became_peered":"2024-03-12T03:46:25.301377+0000","last_unstale":"2024-03-12T03:47:59.065724+0000","last_undegraded":"2024-03-12T03:47:59.065724+0000","last_fullsized":"2024-03-12T03:47:59.065724+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'25","last_scrub_stamp":"2024-03-12T03:46:57.113869+0000","last_deep_scrub":"344'25","last_deep_scrub_stamp":"2024-03-12T03:46:57.113869+0000","last_clean_scrub_stamp":"2024-03-12T03:46:57.113869+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:05.985841+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":435,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.429783+0000","last_change":"2024-03-12T03:47:07.363992+0000","last_active":"2024-03-12T03:48:03.429783+0000","last_peered":"2024-03-12T03:48:03.429783+0000","last_clean":"2024-03-12T03:48:03.429783+0000","last_became_active":"2024-03-12T03:46:25.286247+0000","last_became_peered":"2024-03-12T03:46:25.286247+0000","last_unstale":"2024-03-12T03:48:03.429783+0000","last_undegraded":"2024-03-12T03:48:03.429783+0000","last_fullsized":"2024-03-12T03:48:03.429783+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:07.363439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:07.363439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:09.084955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":490,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:58.699794+0000","last_change":"2024-03-12T03:46:47.672447+0000","last_active":"2024-03-12T03:47:58.699794+0000","last_peered":"2024-03-12T03:47:58.699794+0000","last_clean":"2024-03-12T03:47:58.699794+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:47:58.699794+0000","last_undegraded":"2024-03-12T03:47:58.699794+0000","last_fullsized":"2024-03-12T03:47:58.699794+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:46:47.672313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:46:47.672313+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:10.348862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":666,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.067648+0000","last_change":"2024-03-12T03:47:12.937224+0000","last_active":"2024-03-12T03:47:59.067648+0000","last_peered":"2024-03-12T03:47:59.067648+0000","last_clean":"2024-03-12T03:47:59.067648+0000","last_became_active":"2024-03-12T03:39:33.265665+0000","last_became_peered":"2024-03-12T03:39:33.265665+0000","last_unstale":"2024-03-12T03:47:59.067648+0000","last_undegraded":"2024-03-12T03:47:59.067648+0000","last_fullsized":"2024-03-12T03:47:59.067648+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:12.937101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:12.937101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:26.876746+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:58.699697+0000","last_change":"2024-03-12T03:47:57.695505+0000","last_active":"2024-03-12T03:47:58.699697+0000","last_peered":"2024-03-12T03:47:58.699697+0000","last_clean":"2024-03-12T03:47:58.699697+0000","last_became_active":"2024-03-12T03:47:57.695210+0000","last_became_peered":"2024-03-12T03:47:57.695210+0000","last_unstale":"2024-03-12T03:47:58.699697+0000","last_undegraded":"2024-03-12T03:47:58.699697+0000","last_fullsized":"2024-03-12T03:47:58.699697+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:22.256335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"269'31","reported_seq":759,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.067415+0000","last_change":"2024-03-12T03:47:13.860107+0000","last_active":"2024-03-12T03:47:59.067415+0000","last_peered":"2024-03-12T03:47:59.067415+0000","last_clean":"2024-03-12T03:47:59.067415+0000","last_became_active":"2024-03-12T03:39:33.268652+0000","last_became_peered":"2024-03-12T03:39:33.268652+0000","last_unstale":"2024-03-12T03:47:59.067415+0000","last_undegraded":"2024-03-12T03:47:59.067415+0000","last_fullsized":"2024-03-12T03:47:59.067415+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:47:13.859943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.859943+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:29.361882+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":121,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065805+0000","last_change":"2024-03-12T03:45:18.109064+0000","last_active":"2024-03-12T03:47:59.065805+0000","last_peered":"2024-03-12T03:47:59.065805+0000","last_clean":"2024-03-12T03:47:59.065805+0000","last_became_active":"2024-03-12T03:45:18.108377+0000","last_became_peered":"2024-03-12T03:45:18.108377+0000","last_unstale":"2024-03-12T03:47:59.065805+0000","last_undegraded":"2024-03-12T03:47:59.065805+0000","last_fullsized":"2024-03-12T03:47:59.065805+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.582660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.582660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":658,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063446+0000","last_change":"2024-03-12T03:45:04.055127+0000","last_active":"2024-03-12T03:47:59.063446+0000","last_peered":"2024-03-12T03:47:59.063446+0000","last_clean":"2024-03-12T03:47:59.063446+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:47:59.063446+0000","last_undegraded":"2024-03-12T03:47:59.063446+0000","last_fullsized":"2024-03-12T03:47:59.063446+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.054936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.054936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":395,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:58.699057+0000","last_change":"2024-03-12T03:46:37.930461+0000","last_active":"2024-03-12T03:47:58.699057+0000","last_peered":"2024-03-12T03:47:58.699057+0000","last_clean":"2024-03-12T03:47:58.699057+0000","last_became_active":"2024-03-12T03:46:25.293293+0000","last_became_peered":"2024-03-12T03:46:25.293293+0000","last_unstale":"2024-03-12T03:47:58.699057+0000","last_undegraded":"2024-03-12T03:47:58.699057+0000","last_fullsized":"2024-03-12T03:47:58.699057+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:37.930337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:46:37.930337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":811,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063331+0000","last_change":"2024-03-12T03:47:15.793327+0000","last_active":"2024-03-12T03:47:59.063331+0000","last_peered":"2024-03-12T03:47:59.063331+0000","last_clean":"2024-03-12T03:47:59.063331+0000","last_became_active":"2024-03-12T03:46:25.297913+0000","last_became_peered":"2024-03-12T03:46:25.297913+0000","last_unstale":"2024-03-12T03:47:59.063331+0000","last_undegraded":"2024-03-12T03:47:59.063331+0000","last_fullsized":"2024-03-12T03:47:59.063331+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:15.793199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.793199+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:27.524795+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063095+0000","last_change":"2024-03-12T03:47:58.028182+0000","last_active":"2024-03-12T03:47:59.063095+0000","last_peered":"2024-03-12T03:47:59.063095+0000","last_clean":"2024-03-12T03:47:59.063095+0000","last_became_active":"2024-03-12T03:47:58.027417+0000","last_became_peered":"2024-03-12T03:47:58.027417+0000","last_unstale":"2024-03-12T03:47:59.063095+0000","last_undegraded":"2024-03-12T03:47:59.063095+0000","last_fullsized":"2024-03-12T03:47:59.063095+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:07.057165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":789,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063545+0000","last_change":"2024-03-12T03:44:53.132143+0000","last_active":"2024-03-12T03:47:59.063545+0000","last_peered":"2024-03-12T03:47:59.063545+0000","last_clean":"2024-03-12T03:47:59.063545+0000","last_became_active":"2024-03-12T03:44:24.371149+0000","last_became_peered":"2024-03-12T03:44:24.371149+0000","last_unstale":"2024-03-12T03:47:59.063545+0000","last_undegraded":"2024-03-12T03:47:59.063545+0000","last_fullsized":"2024-03-12T03:47:59.063545+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:53.131506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:53.131506+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"269'31","reported_seq":711,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.067553+0000","last_change":"2024-03-12T03:45:59.684396+0000","last_active":"2024-03-12T03:47:59.067553+0000","last_peered":"2024-03-12T03:47:59.067553+0000","last_clean":"2024-03-12T03:47:59.067553+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:47:59.067553+0000","last_undegraded":"2024-03-12T03:47:59.067553+0000","last_fullsized":"2024-03-12T03:47:59.067553+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:45:59.683848+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.683848+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":678,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065187+0000","last_change":"2024-03-12T03:47:57.920336+0000","last_active":"2024-03-12T03:47:59.065187+0000","last_peered":"2024-03-12T03:47:59.065187+0000","last_clean":"2024-03-12T03:47:59.065187+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:47:59.065187+0000","last_undegraded":"2024-03-12T03:47:59.065187+0000","last_fullsized":"2024-03-12T03:47:59.065187+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:57.920257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_clean_scrub_stamp":"2024-03-12T03:47:57.920257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:12.676781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":649,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066422+0000","last_change":"2024-03-12T03:45:52.418813+0000","last_active":"2024-03-12T03:47:59.066422+0000","last_peered":"2024-03-12T03:47:59.066422+0000","last_clean":"2024-03-12T03:47:59.066422+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:47:59.066422+0000","last_undegraded":"2024-03-12T03:47:59.066422+0000","last_fullsized":"2024-03-12T03:47:59.066422+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:52.418709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:52.418709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":445,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.431610+0000","last_change":"2024-03-12T03:47:14.301686+0000","last_active":"2024-03-12T03:48:03.431610+0000","last_peered":"2024-03-12T03:48:03.431610+0000","last_clean":"2024-03-12T03:48:03.431610+0000","last_became_active":"2024-03-12T03:46:25.300814+0000","last_became_peered":"2024-03-12T03:46:25.300814+0000","last_unstale":"2024-03-12T03:48:03.431610+0000","last_undegraded":"2024-03-12T03:48:03.431610+0000","last_fullsized":"2024-03-12T03:48:03.431610+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:14.301568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:14.301568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:19.385689+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064019+0000","last_change":"2024-03-12T03:47:57.698675+0000","last_active":"2024-03-12T03:47:59.064019+0000","last_peered":"2024-03-12T03:47:59.064019+0000","last_clean":"2024-03-12T03:47:59.064019+0000","last_became_active":"2024-03-12T03:47:57.698379+0000","last_became_peered":"2024-03-12T03:47:57.698379+0000","last_unstale":"2024-03-12T03:47:59.064019+0000","last_undegraded":"2024-03-12T03:47:59.064019+0000","last_fullsized":"2024-03-12T03:47:59.064019+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:19.979837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":785,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066169+0000","last_change":"2024-03-12T03:45:02.015686+0000","last_active":"2024-03-12T03:47:59.066169+0000","last_peered":"2024-03-12T03:47:59.066169+0000","last_clean":"2024-03-12T03:47:59.066169+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:47:59.066169+0000","last_undegraded":"2024-03-12T03:47:59.066169+0000","last_fullsized":"2024-03-12T03:47:59.066169+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:45:02.015575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:45:02.015575+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":515,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064297+0000","last_change":"2024-03-12T03:47:06.292238+0000","last_active":"2024-03-12T03:47:59.064297+0000","last_peered":"2024-03-12T03:47:59.064297+0000","last_clean":"2024-03-12T03:47:59.064297+0000","last_became_active":"2024-03-12T03:46:25.287373+0000","last_became_peered":"2024-03-12T03:46:25.287373+0000","last_unstale":"2024-03-12T03:47:59.064297+0000","last_undegraded":"2024-03-12T03:47:59.064297+0000","last_fullsized":"2024-03-12T03:47:59.064297+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:06.292142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:06.292142+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:20.472155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":148,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063928+0000","last_change":"2024-03-12T03:45:13.901275+0000","last_active":"2024-03-12T03:47:59.063928+0000","last_peered":"2024-03-12T03:47:59.063928+0000","last_clean":"2024-03-12T03:47:59.063928+0000","last_became_active":"2024-03-12T03:44:57.602127+0000","last_became_peered":"2024-03-12T03:44:57.602127+0000","last_unstale":"2024-03-12T03:47:59.063928+0000","last_undegraded":"2024-03-12T03:47:59.063928+0000","last_fullsized":"2024-03-12T03:47:59.063928+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:13.901167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:13.901167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":842,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.067590+0000","last_change":"2024-03-12T03:47:53.964322+0000","last_active":"2024-03-12T03:47:59.067590+0000","last_peered":"2024-03-12T03:47:59.067590+0000","last_clean":"2024-03-12T03:47:59.067590+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:47:59.067590+0000","last_undegraded":"2024-03-12T03:47:59.067590+0000","last_fullsized":"2024-03-12T03:47:59.067590+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:47:53.964216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:53.964216+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:15.371166+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":637,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064329+0000","last_change":"2024-03-12T03:45:08.175177+0000","last_active":"2024-03-12T03:47:59.064329+0000","last_peered":"2024-03-12T03:47:59.064329+0000","last_clean":"2024-03-12T03:47:59.064329+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:47:59.064329+0000","last_undegraded":"2024-03-12T03:47:59.064329+0000","last_fullsized":"2024-03-12T03:47:59.064329+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:08.174901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:08.174901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.067370+0000","last_change":"2024-03-12T03:47:58.028347+0000","last_active":"2024-03-12T03:47:59.067370+0000","last_peered":"2024-03-12T03:47:59.067370+0000","last_clean":"2024-03-12T03:47:59.067370+0000","last_became_active":"2024-03-12T03:47:58.027802+0000","last_became_peered":"2024-03-12T03:47:58.027802+0000","last_unstale":"2024-03-12T03:47:59.067370+0000","last_undegraded":"2024-03-12T03:47:59.067370+0000","last_fullsized":"2024-03-12T03:47:59.067370+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:14.034541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":661,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065127+0000","last_change":"2024-03-12T03:46:52.084213+0000","last_active":"2024-03-12T03:47:59.065127+0000","last_peered":"2024-03-12T03:47:59.065127+0000","last_clean":"2024-03-12T03:47:59.065127+0000","last_became_active":"2024-03-12T03:46:25.286985+0000","last_became_peered":"2024-03-12T03:46:25.286985+0000","last_unstale":"2024-03-12T03:47:59.065127+0000","last_undegraded":"2024-03-12T03:47:59.065127+0000","last_fullsized":"2024-03-12T03:47:59.065127+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:52.084134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:46:52.084134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"269'42","reported_seq":523,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.427694+0000","last_change":"2024-03-12T03:47:13.347626+0000","last_active":"2024-03-12T03:48:03.427694+0000","last_peered":"2024-03-12T03:48:03.427694+0000","last_clean":"2024-03-12T03:48:03.427694+0000","last_became_active":"2024-03-12T03:46:25.290762+0000","last_became_peered":"2024-03-12T03:46:25.290762+0000","last_unstale":"2024-03-12T03:48:03.427694+0000","last_undegraded":"2024-03-12T03:48:03.427694+0000","last_fullsized":"2024-03-12T03:48:03.427694+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:47:13.347128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.347128+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:13.358230+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":431,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.430459+0000","last_change":"2024-03-12T03:46:25.291240+0000","last_active":"2024-03-12T03:48:03.430459+0000","last_peered":"2024-03-12T03:48:03.430459+0000","last_clean":"2024-03-12T03:48:03.430459+0000","last_became_active":"2024-03-12T03:46:25.289332+0000","last_became_peered":"2024-03-12T03:46:25.289332+0000","last_unstale":"2024-03-12T03:48:03.430459+0000","last_undegraded":"2024-03-12T03:48:03.430459+0000","last_fullsized":"2024-03-12T03:48:03.430459+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:41.918538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:38:41.918538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":677,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065743+0000","last_change":"2024-03-12T03:46:56.092143+0000","last_active":"2024-03-12T03:47:59.065743+0000","last_peered":"2024-03-12T03:47:59.065743+0000","last_clean":"2024-03-12T03:47:59.065743+0000","last_became_active":"2024-03-12T03:46:25.280860+0000","last_became_peered":"2024-03-12T03:46:25.280860+0000","last_unstale":"2024-03-12T03:47:59.065743+0000","last_undegraded":"2024-03-12T03:47:59.065743+0000","last_fullsized":"2024-03-12T03:47:59.065743+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_clean_scrub_stamp":"2024-03-12T03:46:56.092011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":799,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066989+0000","last_change":"2024-03-12T03:44:55.770605+0000","last_active":"2024-03-12T03:47:59.066989+0000","last_peered":"2024-03-12T03:47:59.066989+0000","last_clean":"2024-03-12T03:47:59.066989+0000","last_became_active":"2024-03-12T03:44:19.373715+0000","last_became_peered":"2024-03-12T03:44:19.373715+0000","last_unstale":"2024-03-12T03:47:59.066989+0000","last_undegraded":"2024-03-12T03:47:59.066989+0000","last_fullsized":"2024-03-12T03:47:59.066989+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:55.770487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:55.770487+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":15,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.430217+0000","last_change":"2024-03-12T03:47:58.027791+0000","last_active":"2024-03-12T03:48:03.430217+0000","last_peered":"2024-03-12T03:48:03.430217+0000","last_clean":"2024-03-12T03:48:03.430217+0000","last_became_active":"2024-03-12T03:47:58.027347+0000","last_became_peered":"2024-03-12T03:47:58.027347+0000","last_unstale":"2024-03-12T03:48:03.430217+0000","last_undegraded":"2024-03-12T03:48:03.430217+0000","last_fullsized":"2024-03-12T03:48:03.430217+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:07.379521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"332'12","reported_seq":608,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.427771+0000","last_change":"2024-03-12T03:46:25.292527+0000","last_active":"2024-03-12T03:48:03.427771+0000","last_peered":"2024-03-12T03:48:03.427771+0000","last_clean":"2024-03-12T03:48:03.427771+0000","last_became_active":"2024-03-12T03:46:25.288674+0000","last_became_peered":"2024-03-12T03:46:25.288674+0000","last_unstale":"2024-03-12T03:48:03.427771+0000","last_undegraded":"2024-03-12T03:48:03.427771+0000","last_fullsized":"2024-03-12T03:48:03.427771+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:39:37.039925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:37.039925+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"269'42","reported_seq":725,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065456+0000","last_change":"2024-03-12T03:47:56.911844+0000","last_active":"2024-03-12T03:47:59.065456+0000","last_peered":"2024-03-12T03:47:59.065456+0000","last_clean":"2024-03-12T03:47:59.065456+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:47:59.065456+0000","last_undegraded":"2024-03-12T03:47:59.065456+0000","last_fullsized":"2024-03-12T03:47:59.065456+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:47:56.911701+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.911701+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:04.303152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"320'2","reported_seq":660,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064096+0000","last_change":"2024-03-12T03:45:56.089397+0000","last_active":"2024-03-12T03:47:59.064096+0000","last_peered":"2024-03-12T03:47:59.064096+0000","last_clean":"2024-03-12T03:47:59.064096+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:47:59.064096+0000","last_undegraded":"2024-03-12T03:47:59.064096+0000","last_fullsized":"2024-03-12T03:47:59.064096+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'2","last_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_deep_scrub":"320'2","last_deep_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_clean_scrub_stamp":"2024-03-12T03:45:56.089247+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":667,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063579+0000","last_change":"2024-03-12T03:47:17.815361+0000","last_active":"2024-03-12T03:47:59.063579+0000","last_peered":"2024-03-12T03:47:59.063579+0000","last_clean":"2024-03-12T03:47:59.063579+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:47:59.063579+0000","last_undegraded":"2024-03-12T03:47:59.063579+0000","last_fullsized":"2024-03-12T03:47:59.063579+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:17.815263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:17.815263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:38.911937+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":611,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.431300+0000","last_change":"2024-03-12T03:47:11.313289+0000","last_active":"2024-03-12T03:48:03.431300+0000","last_peered":"2024-03-12T03:48:03.431300+0000","last_clean":"2024-03-12T03:48:03.431300+0000","last_became_active":"2024-03-12T03:46:25.299964+0000","last_became_peered":"2024-03-12T03:46:25.299964+0000","last_unstale":"2024-03-12T03:48:03.431300+0000","last_undegraded":"2024-03-12T03:48:03.431300+0000","last_fullsized":"2024-03-12T03:48:03.431300+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.313179+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:34.330866+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065299+0000","last_change":"2024-03-12T03:47:57.698352+0000","last_active":"2024-03-12T03:47:59.065299+0000","last_peered":"2024-03-12T03:47:59.065299+0000","last_clean":"2024-03-12T03:47:59.065299+0000","last_became_active":"2024-03-12T03:47:57.697951+0000","last_became_peered":"2024-03-12T03:47:57.697951+0000","last_unstale":"2024-03-12T03:47:59.065299+0000","last_undegraded":"2024-03-12T03:47:59.065299+0000","last_fullsized":"2024-03-12T03:47:59.065299+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:00.553800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":835,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063897+0000","last_change":"2024-03-12T03:45:09.824960+0000","last_active":"2024-03-12T03:47:59.063897+0000","last_peered":"2024-03-12T03:47:59.063897+0000","last_clean":"2024-03-12T03:47:59.063897+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:47:59.063897+0000","last_undegraded":"2024-03-12T03:47:59.063897+0000","last_fullsized":"2024-03-12T03:47:59.063897+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:09.824819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:09.824819+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"269'26","reported_seq":705,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063485+0000","last_change":"2024-03-12T03:44:58.093469+0000","last_active":"2024-03-12T03:47:59.063485+0000","last_peered":"2024-03-12T03:47:59.063485+0000","last_clean":"2024-03-12T03:47:59.063485+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:47:59.063485+0000","last_undegraded":"2024-03-12T03:47:59.063485+0000","last_fullsized":"2024-03-12T03:47:59.063485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:44:58.093327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:44:58.093327+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":680,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064497+0000","last_change":"2024-03-12T03:47:08.836525+0000","last_active":"2024-03-12T03:47:59.064497+0000","last_peered":"2024-03-12T03:47:59.064497+0000","last_clean":"2024-03-12T03:47:59.064497+0000","last_became_active":"2024-03-12T03:46:25.294348+0000","last_became_peered":"2024-03-12T03:46:25.294348+0000","last_unstale":"2024-03-12T03:47:59.064497+0000","last_undegraded":"2024-03-12T03:47:59.064497+0000","last_fullsized":"2024-03-12T03:47:59.064497+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:08.836419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:08.836419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:24.541463+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":669,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.062498+0000","last_change":"2024-03-12T03:47:13.892905+0000","last_active":"2024-03-12T03:47:59.062498+0000","last_peered":"2024-03-12T03:47:59.062498+0000","last_clean":"2024-03-12T03:47:59.062498+0000","last_became_active":"2024-03-12T03:46:25.293236+0000","last_became_peered":"2024-03-12T03:46:25.293236+0000","last_unstale":"2024-03-12T03:47:59.062498+0000","last_undegraded":"2024-03-12T03:47:59.062498+0000","last_fullsized":"2024-03-12T03:47:59.062498+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:13.892760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.892760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:41.766338+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":398,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.431020+0000","last_change":"2024-03-12T03:47:10.307885+0000","last_active":"2024-03-12T03:48:03.431020+0000","last_peered":"2024-03-12T03:48:03.431020+0000","last_clean":"2024-03-12T03:48:03.431020+0000","last_became_active":"2024-03-12T03:46:25.287360+0000","last_became_peered":"2024-03-12T03:46:25.287360+0000","last_unstale":"2024-03-12T03:48:03.431020+0000","last_undegraded":"2024-03-12T03:48:03.431020+0000","last_fullsized":"2024-03-12T03:48:03.431020+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:10.307761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:10.307761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:25.795762+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.067259+0000","last_change":"2024-03-12T03:47:58.028314+0000","last_active":"2024-03-12T03:47:59.067259+0000","last_peered":"2024-03-12T03:47:59.067259+0000","last_clean":"2024-03-12T03:47:59.067259+0000","last_became_active":"2024-03-12T03:47:58.027648+0000","last_became_peered":"2024-03-12T03:47:58.027648+0000","last_unstale":"2024-03-12T03:47:59.067259+0000","last_undegraded":"2024-03-12T03:47:59.067259+0000","last_fullsized":"2024-03-12T03:47:59.067259+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:05.848725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":121,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063576+0000","last_change":"2024-03-12T03:45:18.106629+0000","last_active":"2024-03-12T03:47:59.063576+0000","last_peered":"2024-03-12T03:47:59.063576+0000","last_clean":"2024-03-12T03:47:59.063576+0000","last_became_active":"2024-03-12T03:45:18.106305+0000","last_became_peered":"2024-03-12T03:45:18.106305+0000","last_unstale":"2024-03-12T03:47:59.063576+0000","last_undegraded":"2024-03-12T03:47:59.063576+0000","last_fullsized":"2024-03-12T03:47:59.063576+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:05.578661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:45:05.578661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"269'35","reported_seq":718,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064151+0000","last_change":"2024-03-12T03:45:59.055468+0000","last_active":"2024-03-12T03:47:59.064151+0000","last_peered":"2024-03-12T03:47:59.064151+0000","last_clean":"2024-03-12T03:47:59.064151+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:47:59.064151+0000","last_undegraded":"2024-03-12T03:47:59.064151+0000","last_fullsized":"2024-03-12T03:47:59.064151+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-03-12T03:45:59.055318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.055318+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":687,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:48:00.710463+0000","last_change":"2024-03-12T03:48:00.710463+0000","last_active":"2024-03-12T03:48:00.710463+0000","last_peered":"2024-03-12T03:48:00.710463+0000","last_clean":"2024-03-12T03:48:00.710463+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:48:00.710463+0000","last_undegraded":"2024-03-12T03:48:00.710463+0000","last_fullsized":"2024-03-12T03:48:00.710463+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:00.710387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.710387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:17.748178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":667,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.062984+0000","last_change":"2024-03-12T03:44:54.134468+0000","last_active":"2024-03-12T03:47:59.062984+0000","last_peered":"2024-03-12T03:47:59.062984+0000","last_clean":"2024-03-12T03:47:59.062984+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:47:59.062984+0000","last_undegraded":"2024-03-12T03:47:59.062984+0000","last_fullsized":"2024-03-12T03:47:59.062984+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:54.134348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:54.134348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":428,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063959+0000","last_change":"2024-03-12T03:47:04.858096+0000","last_active":"2024-03-12T03:47:59.063959+0000","last_peered":"2024-03-12T03:47:59.063959+0000","last_clean":"2024-03-12T03:47:59.063959+0000","last_became_active":"2024-03-12T03:46:25.282167+0000","last_became_peered":"2024-03-12T03:46:25.282167+0000","last_unstale":"2024-03-12T03:47:59.063959+0000","last_undegraded":"2024-03-12T03:47:59.063959+0000","last_fullsized":"2024-03-12T03:47:59.063959+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:04.857986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:04.857986+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:14.853572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.062872+0000","last_change":"2024-03-12T03:47:57.699339+0000","last_active":"2024-03-12T03:47:59.062872+0000","last_peered":"2024-03-12T03:47:59.062872+0000","last_clean":"2024-03-12T03:47:59.062872+0000","last_became_active":"2024-03-12T03:47:57.698375+0000","last_became_peered":"2024-03-12T03:47:57.698375+0000","last_unstale":"2024-03-12T03:47:59.062872+0000","last_undegraded":"2024-03-12T03:47:59.062872+0000","last_fullsized":"2024-03-12T03:47:59.062872+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:25.005560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":649,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066276+0000","last_change":"2024-03-12T03:46:00.577914+0000","last_active":"2024-03-12T03:47:59.066276+0000","last_peered":"2024-03-12T03:47:59.066276+0000","last_clean":"2024-03-12T03:47:59.066276+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:47:59.066276+0000","last_undegraded":"2024-03-12T03:47:59.066276+0000","last_fullsized":"2024-03-12T03:47:59.066276+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:00.577693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:46:00.577693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"269'24","reported_seq":730,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064087+0000","last_change":"2024-03-12T03:45:02.759428+0000","last_active":"2024-03-12T03:47:59.064087+0000","last_peered":"2024-03-12T03:47:59.064087+0000","last_clean":"2024-03-12T03:47:59.064087+0000","last_became_active":"2024-03-12T03:39:33.277868+0000","last_became_peered":"2024-03-12T03:39:33.277868+0000","last_unstale":"2024-03-12T03:47:59.064087+0000","last_undegraded":"2024-03-12T03:47:59.064087+0000","last_fullsized":"2024-03-12T03:47:59.064087+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:45:02.759243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:02.759243+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":1553,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.503976+0000","last_change":"2024-03-12T03:47:13.591577+0000","last_active":"2024-03-12T03:47:59.503976+0000","last_peered":"2024-03-12T03:47:59.503976+0000","last_clean":"2024-03-12T03:47:59.503976+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:47:59.503976+0000","last_undegraded":"2024-03-12T03:47:59.503976+0000","last_fullsized":"2024-03-12T03:47:59.503976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:47:13.591375+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.591375+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:25.083961+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":873,"num_read_kb":868,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":669,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.062733+0000","last_change":"2024-03-12T03:47:58.051067+0000","last_active":"2024-03-12T03:47:59.062733+0000","last_peered":"2024-03-12T03:47:59.062733+0000","last_clean":"2024-03-12T03:47:59.062733+0000","last_became_active":"2024-03-12T03:46:25.282902+0000","last_became_peered":"2024-03-12T03:46:25.282902+0000","last_unstale":"2024-03-12T03:47:59.062733+0000","last_undegraded":"2024-03-12T03:47:59.062733+0000","last_fullsized":"2024-03-12T03:47:59.062733+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:58.050985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.050985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:09.583349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":672,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063109+0000","last_change":"2024-03-12T03:47:17.391210+0000","last_active":"2024-03-12T03:47:59.063109+0000","last_peered":"2024-03-12T03:47:59.063109+0000","last_clean":"2024-03-12T03:47:59.063109+0000","last_became_active":"2024-03-12T03:46:25.302256+0000","last_became_peered":"2024-03-12T03:46:25.302256+0000","last_unstale":"2024-03-12T03:47:59.063109+0000","last_undegraded":"2024-03-12T03:47:59.063109+0000","last_fullsized":"2024-03-12T03:47:59.063109+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:17.391069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:17.391069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:28.366690+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":782,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063722+0000","last_change":"2024-03-12T03:44:52.122439+0000","last_active":"2024-03-12T03:47:59.063722+0000","last_peered":"2024-03-12T03:47:59.063722+0000","last_clean":"2024-03-12T03:47:59.063722+0000","last_became_active":"2024-03-12T03:44:19.366060+0000","last_became_peered":"2024-03-12T03:44:19.366060+0000","last_unstale":"2024-03-12T03:47:59.063722+0000","last_undegraded":"2024-03-12T03:47:59.063722+0000","last_fullsized":"2024-03-12T03:47:59.063722+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:52.122312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:52.122312+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063018+0000","last_change":"2024-03-12T03:47:58.028202+0000","last_active":"2024-03-12T03:47:59.063018+0000","last_peered":"2024-03-12T03:47:59.063018+0000","last_clean":"2024-03-12T03:47:59.063018+0000","last_became_active":"2024-03-12T03:47:58.027920+0000","last_became_peered":"2024-03-12T03:47:58.027920+0000","last_unstale":"2024-03-12T03:47:59.063018+0000","last_undegraded":"2024-03-12T03:47:59.063018+0000","last_fullsized":"2024-03-12T03:47:59.063018+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:21.218558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":662,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065590+0000","last_change":"2024-03-12T03:45:16.222836+0000","last_active":"2024-03-12T03:47:59.065590+0000","last_peered":"2024-03-12T03:47:59.065590+0000","last_clean":"2024-03-12T03:47:59.065590+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:47:59.065590+0000","last_undegraded":"2024-03-12T03:47:59.065590+0000","last_fullsized":"2024-03-12T03:47:59.065590+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:16.222710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:16.222710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"315'29","reported_seq":767,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.067303+0000","last_change":"2024-03-12T03:45:18.109405+0000","last_active":"2024-03-12T03:47:59.067303+0000","last_peered":"2024-03-12T03:47:59.067303+0000","last_clean":"2024-03-12T03:47:59.067303+0000","last_became_active":"2024-03-12T03:45:18.109048+0000","last_became_peered":"2024-03-12T03:45:18.109048+0000","last_unstale":"2024-03-12T03:47:59.067303+0000","last_undegraded":"2024-03-12T03:47:59.067303+0000","last_fullsized":"2024-03-12T03:47:59.067303+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'28","last_scrub_stamp":"2024-03-12T03:44:58.632869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:44:58.632869+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":430,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.427780+0000","last_change":"2024-03-12T03:46:25.294791+0000","last_active":"2024-03-12T03:48:03.427780+0000","last_peered":"2024-03-12T03:48:03.427780+0000","last_clean":"2024-03-12T03:48:03.427780+0000","last_became_active":"2024-03-12T03:46:25.292535+0000","last_became_peered":"2024-03-12T03:46:25.292535+0000","last_unstale":"2024-03-12T03:48:03.427780+0000","last_undegraded":"2024-03-12T03:48:03.427780+0000","last_fullsized":"2024-03-12T03:48:03.427780+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:07.572642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:07.572642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":932,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063299+0000","last_change":"2024-03-12T03:44:54.092085+0000","last_active":"2024-03-12T03:47:59.063299+0000","last_peered":"2024-03-12T03:47:59.063299+0000","last_clean":"2024-03-12T03:47:59.063299+0000","last_became_active":"2024-03-12T03:44:19.379262+0000","last_became_peered":"2024-03-12T03:44:19.379262+0000","last_unstale":"2024-03-12T03:47:59.063299+0000","last_undegraded":"2024-03-12T03:47:59.063299+0000","last_fullsized":"2024-03-12T03:47:59.063299+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:54.091965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:54.091965+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":655,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065778+0000","last_change":"2024-03-12T03:47:51.670920+0000","last_active":"2024-03-12T03:47:59.065778+0000","last_peered":"2024-03-12T03:47:59.065778+0000","last_clean":"2024-03-12T03:47:59.065778+0000","last_became_active":"2024-03-12T03:46:25.295668+0000","last_became_peered":"2024-03-12T03:46:25.295668+0000","last_unstale":"2024-03-12T03:47:59.065778+0000","last_undegraded":"2024-03-12T03:47:59.065778+0000","last_fullsized":"2024-03-12T03:47:59.065778+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:51.670806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:51.670806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:59.884920+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066313+0000","last_change":"2024-03-12T03:47:57.694092+0000","last_active":"2024-03-12T03:47:59.066313+0000","last_peered":"2024-03-12T03:47:59.066313+0000","last_clean":"2024-03-12T03:47:59.066313+0000","last_became_active":"2024-03-12T03:47:57.693619+0000","last_became_peered":"2024-03-12T03:47:57.693619+0000","last_unstale":"2024-03-12T03:47:59.066313+0000","last_undegraded":"2024-03-12T03:47:59.066313+0000","last_fullsized":"2024-03-12T03:47:59.066313+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:16.156372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":393,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:58.699330+0000","last_change":"2024-03-12T03:47:03.666019+0000","last_active":"2024-03-12T03:47:58.699330+0000","last_peered":"2024-03-12T03:47:58.699330+0000","last_clean":"2024-03-12T03:47:58.699330+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:47:58.699330+0000","last_undegraded":"2024-03-12T03:47:58.699330+0000","last_fullsized":"2024-03-12T03:47:58.699330+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:03.665934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:47:03.665934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:24.141177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"269'23","reported_seq":707,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064241+0000","last_change":"2024-03-12T03:45:59.998488+0000","last_active":"2024-03-12T03:47:59.064241+0000","last_peered":"2024-03-12T03:47:59.064241+0000","last_clean":"2024-03-12T03:47:59.064241+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:47:59.064241+0000","last_undegraded":"2024-03-12T03:47:59.064241+0000","last_fullsized":"2024-03-12T03:47:59.064241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'23","last_scrub_stamp":"2024-03-12T03:45:59.998395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.998395+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":388,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:58.699190+0000","last_change":"2024-03-12T03:46:42.747615+0000","last_active":"2024-03-12T03:47:58.699190+0000","last_peered":"2024-03-12T03:47:58.699190+0000","last_clean":"2024-03-12T03:47:58.699190+0000","last_became_active":"2024-03-12T03:46:25.284867+0000","last_became_peered":"2024-03-12T03:46:25.284867+0000","last_unstale":"2024-03-12T03:47:58.699190+0000","last_undegraded":"2024-03-12T03:47:58.699190+0000","last_fullsized":"2024-03-12T03:47:58.699190+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:42.747491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:46:42.747491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:04.186064+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":679,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063149+0000","last_change":"2024-03-12T03:45:00.012141+0000","last_active":"2024-03-12T03:47:59.063149+0000","last_peered":"2024-03-12T03:47:59.063149+0000","last_clean":"2024-03-12T03:47:59.063149+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:47:59.063149+0000","last_undegraded":"2024-03-12T03:47:59.063149+0000","last_fullsized":"2024-03-12T03:47:59.063149+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:00.012064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:45:00.012064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":666,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066948+0000","last_change":"2024-03-12T03:46:51.595890+0000","last_active":"2024-03-12T03:47:59.066948+0000","last_peered":"2024-03-12T03:47:59.066948+0000","last_clean":"2024-03-12T03:47:59.066948+0000","last_became_active":"2024-03-12T03:46:25.284168+0000","last_became_peered":"2024-03-12T03:46:25.284168+0000","last_unstale":"2024-03-12T03:47:59.066948+0000","last_undegraded":"2024-03-12T03:47:59.066948+0000","last_fullsized":"2024-03-12T03:47:59.066948+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:46:51.595794+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:46:51.595794+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:15.953713+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":947,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064729+0000","last_change":"2024-03-12T03:44:53.098337+0000","last_active":"2024-03-12T03:47:59.064729+0000","last_peered":"2024-03-12T03:47:59.064729+0000","last_clean":"2024-03-12T03:47:59.064729+0000","last_became_active":"2024-03-12T03:44:19.378554+0000","last_became_peered":"2024-03-12T03:44:19.378554+0000","last_unstale":"2024-03-12T03:47:59.064729+0000","last_undegraded":"2024-03-12T03:47:59.064729+0000","last_fullsized":"2024-03-12T03:47:59.064729+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:53.097930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:53.097930+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063449+0000","last_change":"2024-03-12T03:47:58.028871+0000","last_active":"2024-03-12T03:47:59.063449+0000","last_peered":"2024-03-12T03:47:59.063449+0000","last_clean":"2024-03-12T03:47:59.063449+0000","last_became_active":"2024-03-12T03:47:58.028510+0000","last_became_peered":"2024-03-12T03:47:58.028510+0000","last_unstale":"2024-03-12T03:47:59.063449+0000","last_undegraded":"2024-03-12T03:47:59.063449+0000","last_fullsized":"2024-03-12T03:47:59.063449+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:12.997874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":650,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063678+0000","last_change":"2024-03-12T03:45:06.865102+0000","last_active":"2024-03-12T03:47:59.063678+0000","last_peered":"2024-03-12T03:47:59.063678+0000","last_clean":"2024-03-12T03:47:59.063678+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:47:59.063678+0000","last_undegraded":"2024-03-12T03:47:59.063678+0000","last_fullsized":"2024-03-12T03:47:59.063678+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:06.864708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.864708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"332'53","reported_seq":933,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066270+0000","last_change":"2024-03-12T03:47:55.621887+0000","last_active":"2024-03-12T03:47:59.066270+0000","last_peered":"2024-03-12T03:47:59.066270+0000","last_clean":"2024-03-12T03:47:59.066270+0000","last_became_active":"2024-03-12T03:46:25.301872+0000","last_became_peered":"2024-03-12T03:46:25.301872+0000","last_unstale":"2024-03-12T03:47:59.066270+0000","last_undegraded":"2024-03-12T03:47:59.066270+0000","last_fullsized":"2024-03-12T03:47:59.066270+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'53","last_scrub_stamp":"2024-03-12T03:47:55.621782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:55.621782+0000","objects_scrubbed":11,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:19.618316+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":667,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066607+0000","last_change":"2024-03-12T03:45:53.648932+0000","last_active":"2024-03-12T03:47:59.066607+0000","last_peered":"2024-03-12T03:47:59.066607+0000","last_clean":"2024-03-12T03:47:59.066607+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:47:59.066607+0000","last_undegraded":"2024-03-12T03:47:59.066607+0000","last_fullsized":"2024-03-12T03:47:59.066607+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.648824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.648824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":669,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066595+0000","last_change":"2024-03-12T03:46:04.645060+0000","last_active":"2024-03-12T03:47:59.066595+0000","last_peered":"2024-03-12T03:47:59.066595+0000","last_clean":"2024-03-12T03:47:59.066595+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:47:59.066595+0000","last_undegraded":"2024-03-12T03:47:59.066595+0000","last_fullsized":"2024-03-12T03:47:59.066595+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:46:04.644902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:46:04.644902+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":470,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.429620+0000","last_change":"2024-03-12T03:46:25.287357+0000","last_active":"2024-03-12T03:48:03.429620+0000","last_peered":"2024-03-12T03:48:03.429620+0000","last_clean":"2024-03-12T03:48:03.429620+0000","last_became_active":"2024-03-12T03:46:25.284416+0000","last_became_peered":"2024-03-12T03:46:25.284416+0000","last_unstale":"2024-03-12T03:48:03.429620+0000","last_undegraded":"2024-03-12T03:48:03.429620+0000","last_fullsized":"2024-03-12T03:48:03.429620+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:39.031166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:39.031166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":790,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064583+0000","last_change":"2024-03-12T03:44:51.991449+0000","last_active":"2024-03-12T03:47:59.064583+0000","last_peered":"2024-03-12T03:47:59.064583+0000","last_clean":"2024-03-12T03:47:59.064583+0000","last_became_active":"2024-03-12T03:44:19.371194+0000","last_became_peered":"2024-03-12T03:44:19.371194+0000","last_unstale":"2024-03-12T03:47:59.064583+0000","last_undegraded":"2024-03-12T03:47:59.064583+0000","last_fullsized":"2024-03-12T03:47:59.064583+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:51.991223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:51.991223+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:58.699837+0000","last_change":"2024-03-12T03:47:57.699330+0000","last_active":"2024-03-12T03:47:58.699837+0000","last_peered":"2024-03-12T03:47:58.699837+0000","last_clean":"2024-03-12T03:47:58.699837+0000","last_became_active":"2024-03-12T03:47:57.699023+0000","last_became_peered":"2024-03-12T03:47:57.699023+0000","last_unstale":"2024-03-12T03:47:58.699837+0000","last_undegraded":"2024-03-12T03:47:58.699837+0000","last_fullsized":"2024-03-12T03:47:58.699837+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:16.377432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":466,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.067321+0000","last_change":"2024-03-12T03:46:26.281981+0000","last_active":"2024-03-12T03:47:59.067321+0000","last_peered":"2024-03-12T03:47:59.067321+0000","last_clean":"2024-03-12T03:47:59.067321+0000","last_became_active":"2024-03-12T03:46:26.281472+0000","last_became_peered":"2024-03-12T03:46:26.281472+0000","last_unstale":"2024-03-12T03:47:59.067321+0000","last_undegraded":"2024-03-12T03:47:59.067321+0000","last_fullsized":"2024-03-12T03:47:59.067321+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:52.828725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:52.828725+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"269'26","reported_seq":696,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063969+0000","last_change":"2024-03-12T03:45:15.774203+0000","last_active":"2024-03-12T03:47:59.063969+0000","last_peered":"2024-03-12T03:47:59.063969+0000","last_clean":"2024-03-12T03:47:59.063969+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:47:59.063969+0000","last_undegraded":"2024-03-12T03:47:59.063969+0000","last_fullsized":"2024-03-12T03:47:59.063969+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:15.774076+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.774076+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":667,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066476+0000","last_change":"2024-03-12T03:47:10.855898+0000","last_active":"2024-03-12T03:47:59.066476+0000","last_peered":"2024-03-12T03:47:59.066476+0000","last_clean":"2024-03-12T03:47:59.066476+0000","last_became_active":"2024-03-12T03:46:25.303784+0000","last_became_peered":"2024-03-12T03:46:25.303784+0000","last_unstale":"2024-03-12T03:47:59.066476+0000","last_undegraded":"2024-03-12T03:47:59.066476+0000","last_fullsized":"2024-03-12T03:47:59.066476+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:10.855814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:47:10.855814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:40.854862+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":387,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:58.699951+0000","last_change":"2024-03-12T03:46:33.767004+0000","last_active":"2024-03-12T03:47:58.699951+0000","last_peered":"2024-03-12T03:47:58.699951+0000","last_clean":"2024-03-12T03:47:58.699951+0000","last_became_active":"2024-03-12T03:46:25.284556+0000","last_became_peered":"2024-03-12T03:46:25.284556+0000","last_unstale":"2024-03-12T03:47:58.699951+0000","last_undegraded":"2024-03-12T03:47:58.699951+0000","last_fullsized":"2024-03-12T03:47:58.699951+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:33.766874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:46:33.766874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":658,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.062942+0000","last_change":"2024-03-12T03:45:55.193112+0000","last_active":"2024-03-12T03:47:59.062942+0000","last_peered":"2024-03-12T03:47:59.062942+0000","last_clean":"2024-03-12T03:47:59.062942+0000","last_became_active":"2024-03-12T03:39:33.271016+0000","last_became_peered":"2024-03-12T03:39:33.271016+0000","last_unstale":"2024-03-12T03:47:59.062942+0000","last_undegraded":"2024-03-12T03:47:59.062942+0000","last_fullsized":"2024-03-12T03:47:59.062942+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:55.192996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:55.192996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063473+0000","last_change":"2024-03-12T03:47:57.699190+0000","last_active":"2024-03-12T03:47:59.063473+0000","last_peered":"2024-03-12T03:47:59.063473+0000","last_clean":"2024-03-12T03:47:59.063473+0000","last_became_active":"2024-03-12T03:47:57.698868+0000","last_became_peered":"2024-03-12T03:47:57.698868+0000","last_unstale":"2024-03-12T03:47:59.063473+0000","last_undegraded":"2024-03-12T03:47:59.063473+0000","last_fullsized":"2024-03-12T03:47:59.063473+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:24.347366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":763,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065619+0000","last_change":"2024-03-12T03:45:50.401930+0000","last_active":"2024-03-12T03:47:59.065619+0000","last_peered":"2024-03-12T03:47:59.065619+0000","last_clean":"2024-03-12T03:47:59.065619+0000","last_became_active":"2024-03-12T03:45:18.108107+0000","last_became_peered":"2024-03-12T03:45:18.108107+0000","last_unstale":"2024-03-12T03:47:59.065619+0000","last_undegraded":"2024-03-12T03:47:59.065619+0000","last_fullsized":"2024-03-12T03:47:59.065619+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:45:50.401804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:45:50.401804+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'51","reported_seq":650,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063142+0000","last_change":"2024-03-12T03:45:18.106865+0000","last_active":"2024-03-12T03:47:59.063142+0000","last_peered":"2024-03-12T03:47:59.063142+0000","last_clean":"2024-03-12T03:47:59.063142+0000","last_became_active":"2024-03-12T03:45:18.106503+0000","last_became_peered":"2024-03-12T03:45:18.106503+0000","last_unstale":"2024-03-12T03:47:59.063142+0000","last_undegraded":"2024-03-12T03:47:59.063142+0000","last_fullsized":"2024-03-12T03:47:59.063142+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'51","last_scrub_stamp":"2024-03-12T03:45:06.539548+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.539548+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":658,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066842+0000","last_change":"2024-03-12T03:44:56.159355+0000","last_active":"2024-03-12T03:47:59.066842+0000","last_peered":"2024-03-12T03:47:59.066842+0000","last_clean":"2024-03-12T03:47:59.066842+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:47:59.066842+0000","last_undegraded":"2024-03-12T03:47:59.066842+0000","last_fullsized":"2024-03-12T03:47:59.066842+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:56.159248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:44:56.159248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":385,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:58.699750+0000","last_change":"2024-03-12T03:47:16.954859+0000","last_active":"2024-03-12T03:47:58.699750+0000","last_peered":"2024-03-12T03:47:58.699750+0000","last_clean":"2024-03-12T03:47:58.699750+0000","last_became_active":"2024-03-12T03:39:33.254395+0000","last_became_peered":"2024-03-12T03:39:33.254395+0000","last_unstale":"2024-03-12T03:47:58.699750+0000","last_undegraded":"2024-03-12T03:47:58.699750+0000","last_fullsized":"2024-03-12T03:47:58.699750+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.954731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.954731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:23.659186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":676,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064958+0000","last_change":"2024-03-12T03:46:57.647557+0000","last_active":"2024-03-12T03:47:59.064958+0000","last_peered":"2024-03-12T03:47:59.064958+0000","last_clean":"2024-03-12T03:47:59.064958+0000","last_became_active":"2024-03-12T03:46:25.292122+0000","last_became_peered":"2024-03-12T03:46:25.292122+0000","last_unstale":"2024-03-12T03:47:59.064958+0000","last_undegraded":"2024-03-12T03:47:59.064958+0000","last_fullsized":"2024-03-12T03:47:59.064958+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:57.647447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:46:57.647447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":15,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.430310+0000","last_change":"2024-03-12T03:47:58.027966+0000","last_active":"2024-03-12T03:48:03.430310+0000","last_peered":"2024-03-12T03:48:03.430310+0000","last_clean":"2024-03-12T03:48:03.430310+0000","last_became_active":"2024-03-12T03:47:58.027372+0000","last_became_peered":"2024-03-12T03:47:58.027372+0000","last_unstale":"2024-03-12T03:48:03.430310+0000","last_undegraded":"2024-03-12T03:48:03.430310+0000","last_fullsized":"2024-03-12T03:48:03.430310+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:19.365947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":600,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.431197+0000","last_change":"2024-03-12T03:46:26.280678+0000","last_active":"2024-03-12T03:48:03.431197+0000","last_peered":"2024-03-12T03:48:03.431197+0000","last_clean":"2024-03-12T03:48:03.431197+0000","last_became_active":"2024-03-12T03:46:26.279796+0000","last_became_peered":"2024-03-12T03:46:26.279796+0000","last_unstale":"2024-03-12T03:48:03.431197+0000","last_undegraded":"2024-03-12T03:48:03.431197+0000","last_fullsized":"2024-03-12T03:48:03.431197+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:51.818256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:51.818256+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":393,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:58.698203+0000","last_change":"2024-03-12T03:46:57.669774+0000","last_active":"2024-03-12T03:47:58.698203+0000","last_peered":"2024-03-12T03:47:58.698203+0000","last_clean":"2024-03-12T03:47:58.698203+0000","last_became_active":"2024-03-12T03:39:33.278326+0000","last_became_peered":"2024-03-12T03:39:33.278326+0000","last_unstale":"2024-03-12T03:47:58.698203+0000","last_undegraded":"2024-03-12T03:47:58.698203+0000","last_fullsized":"2024-03-12T03:47:58.698203+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:57.669662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:46:57.669662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:13.986821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"269'30","reported_seq":737,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065564+0000","last_change":"2024-03-12T03:47:00.121826+0000","last_active":"2024-03-12T03:47:59.065564+0000","last_peered":"2024-03-12T03:47:59.065564+0000","last_clean":"2024-03-12T03:47:59.065564+0000","last_became_active":"2024-03-12T03:46:25.287885+0000","last_became_peered":"2024-03-12T03:46:25.287885+0000","last_unstale":"2024-03-12T03:47:59.065564+0000","last_undegraded":"2024-03-12T03:47:59.065564+0000","last_fullsized":"2024-03-12T03:47:59.065564+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'30","last_scrub_stamp":"2024-03-12T03:47:00.121665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:00.121665+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:04.136110+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":666,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066433+0000","last_change":"2024-03-12T03:45:57.516536+0000","last_active":"2024-03-12T03:47:59.066433+0000","last_peered":"2024-03-12T03:47:59.066433+0000","last_clean":"2024-03-12T03:47:59.066433+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:47:59.066433+0000","last_undegraded":"2024-03-12T03:47:59.066433+0000","last_fullsized":"2024-03-12T03:47:59.066433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:57.516275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:45:57.516275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":385,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.429978+0000","last_change":"2024-03-12T03:46:25.281107+0000","last_active":"2024-03-12T03:48:03.429978+0000","last_peered":"2024-03-12T03:48:03.429978+0000","last_clean":"2024-03-12T03:48:03.429978+0000","last_became_active":"2024-03-12T03:46:25.280784+0000","last_became_peered":"2024-03-12T03:46:25.280784+0000","last_unstale":"2024-03-12T03:48:03.429978+0000","last_undegraded":"2024-03-12T03:48:03.429978+0000","last_fullsized":"2024-03-12T03:48:03.429978+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:38:18.651418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:38:18.651418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065107+0000","last_change":"2024-03-12T03:47:57.698552+0000","last_active":"2024-03-12T03:47:59.065107+0000","last_peered":"2024-03-12T03:47:59.065107+0000","last_clean":"2024-03-12T03:47:59.065107+0000","last_became_active":"2024-03-12T03:47:57.698209+0000","last_became_peered":"2024-03-12T03:47:57.698209+0000","last_unstale":"2024-03-12T03:47:59.065107+0000","last_undegraded":"2024-03-12T03:47:59.065107+0000","last_fullsized":"2024-03-12T03:47:59.065107+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:05.850227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":578,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066446+0000","last_change":"2024-03-12T03:44:55.099123+0000","last_active":"2024-03-12T03:47:59.066446+0000","last_peered":"2024-03-12T03:47:59.066446+0000","last_clean":"2024-03-12T03:47:59.066446+0000","last_became_active":"2024-03-12T03:44:24.373190+0000","last_became_peered":"2024-03-12T03:44:24.373190+0000","last_unstale":"2024-03-12T03:47:59.066446+0000","last_undegraded":"2024-03-12T03:47:59.066446+0000","last_fullsized":"2024-03-12T03:47:59.066446+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:55.099010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:55.099010+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":835,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065684+0000","last_change":"2024-03-12T03:45:10.340919+0000","last_active":"2024-03-12T03:47:59.065684+0000","last_peered":"2024-03-12T03:47:59.065684+0000","last_clean":"2024-03-12T03:47:59.065684+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:47:59.065684+0000","last_undegraded":"2024-03-12T03:47:59.065684+0000","last_fullsized":"2024-03-12T03:47:59.065684+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:45:10.340796+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.340796+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"269'48","reported_seq":729,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063113+0000","last_change":"2024-03-12T03:45:14.163797+0000","last_active":"2024-03-12T03:47:59.063113+0000","last_peered":"2024-03-12T03:47:59.063113+0000","last_clean":"2024-03-12T03:47:59.063113+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:47:59.063113+0000","last_undegraded":"2024-03-12T03:47:59.063113+0000","last_fullsized":"2024-03-12T03:47:59.063113+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'48","last_scrub_stamp":"2024-03-12T03:45:14.163544+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:45:14.163544+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":670,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064258+0000","last_change":"2024-03-12T03:47:05.588686+0000","last_active":"2024-03-12T03:47:59.064258+0000","last_peered":"2024-03-12T03:47:59.064258+0000","last_clean":"2024-03-12T03:47:59.064258+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:47:59.064258+0000","last_undegraded":"2024-03-12T03:47:59.064258+0000","last_fullsized":"2024-03-12T03:47:59.064258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:47:05.588582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:05.588582+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:26.550752+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":645,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066330+0000","last_change":"2024-03-12T03:45:59.568025+0000","last_active":"2024-03-12T03:47:59.066330+0000","last_peered":"2024-03-12T03:47:59.066330+0000","last_clean":"2024-03-12T03:47:59.066330+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:47:59.066330+0000","last_undegraded":"2024-03-12T03:47:59.066330+0000","last_fullsized":"2024-03-12T03:47:59.066330+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:59.567890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.567890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":462,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065983+0000","last_change":"2024-03-12T03:44:55.920207+0000","last_active":"2024-03-12T03:47:59.065983+0000","last_peered":"2024-03-12T03:47:59.065983+0000","last_clean":"2024-03-12T03:47:59.065983+0000","last_became_active":"2024-03-12T03:44:24.365642+0000","last_became_peered":"2024-03-12T03:44:24.365642+0000","last_unstale":"2024-03-12T03:47:59.065983+0000","last_undegraded":"2024-03-12T03:47:59.065983+0000","last_fullsized":"2024-03-12T03:47:59.065983+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:55.920045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:55.920045+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"269'18","reported_seq":456,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.430552+0000","last_change":"2024-03-12T03:46:25.314522+0000","last_active":"2024-03-12T03:48:03.430552+0000","last_peered":"2024-03-12T03:48:03.430552+0000","last_clean":"2024-03-12T03:48:03.430552+0000","last_became_active":"2024-03-12T03:46:25.288573+0000","last_became_peered":"2024-03-12T03:46:25.288573+0000","last_unstale":"2024-03-12T03:48:03.430552+0000","last_undegraded":"2024-03-12T03:48:03.430552+0000","last_fullsized":"2024-03-12T03:48:03.430552+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:43:17.648603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:17.648603+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":121,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063076+0000","last_change":"2024-03-12T03:45:18.106344+0000","last_active":"2024-03-12T03:47:59.063076+0000","last_peered":"2024-03-12T03:47:59.063076+0000","last_clean":"2024-03-12T03:47:59.063076+0000","last_became_active":"2024-03-12T03:45:18.106080+0000","last_became_peered":"2024-03-12T03:45:18.106080+0000","last_unstale":"2024-03-12T03:47:59.063076+0000","last_undegraded":"2024-03-12T03:47:59.063076+0000","last_fullsized":"2024-03-12T03:47:59.063076+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:10.400135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.400135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":659,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065162+0000","last_change":"2024-03-12T03:47:17.816126+0000","last_active":"2024-03-12T03:47:59.065162+0000","last_peered":"2024-03-12T03:47:59.065162+0000","last_clean":"2024-03-12T03:47:59.065162+0000","last_became_active":"2024-03-12T03:46:25.293521+0000","last_became_peered":"2024-03-12T03:46:25.293521+0000","last_unstale":"2024-03-12T03:47:59.065162+0000","last_undegraded":"2024-03-12T03:47:59.065162+0000","last_fullsized":"2024-03-12T03:47:59.065162+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:17.816024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:17.816024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:36.683825+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":654,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066576+0000","last_change":"2024-03-12T03:47:11.641664+0000","last_active":"2024-03-12T03:47:59.066576+0000","last_peered":"2024-03-12T03:47:59.066576+0000","last_clean":"2024-03-12T03:47:59.066576+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:47:59.066576+0000","last_undegraded":"2024-03-12T03:47:59.066576+0000","last_fullsized":"2024-03-12T03:47:59.066576+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.641547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.641547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:41.604599+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":794,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063531+0000","last_change":"2024-03-12T03:46:28.316868+0000","last_active":"2024-03-12T03:47:59.063531+0000","last_peered":"2024-03-12T03:47:59.063531+0000","last_clean":"2024-03-12T03:47:59.063531+0000","last_became_active":"2024-03-12T03:46:28.316406+0000","last_became_peered":"2024-03-12T03:46:28.316406+0000","last_unstale":"2024-03-12T03:47:59.063531+0000","last_undegraded":"2024-03-12T03:47:59.063531+0000","last_fullsized":"2024-03-12T03:47:59.063531+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:56.067715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:56.067715+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"269'27","reported_seq":714,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064240+0000","last_change":"2024-03-12T03:46:25.311563+0000","last_active":"2024-03-12T03:47:59.064240+0000","last_peered":"2024-03-12T03:47:59.064240+0000","last_clean":"2024-03-12T03:47:59.064240+0000","last_became_active":"2024-03-12T03:46:25.284304+0000","last_became_peered":"2024-03-12T03:46:25.284304+0000","last_unstale":"2024-03-12T03:47:59.064240+0000","last_undegraded":"2024-03-12T03:47:59.064240+0000","last_fullsized":"2024-03-12T03:47:59.064240+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-03-12T03:43:03.583989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:03.583989+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":658,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065495+0000","last_change":"2024-03-12T03:44:58.185913+0000","last_active":"2024-03-12T03:47:59.065495+0000","last_peered":"2024-03-12T03:47:59.065495+0000","last_clean":"2024-03-12T03:47:59.065495+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:47:59.065495+0000","last_undegraded":"2024-03-12T03:47:59.065495+0000","last_fullsized":"2024-03-12T03:47:59.065495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:58.185812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:44:58.185812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":419,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.430704+0000","last_change":"2024-03-12T03:46:25.289109+0000","last_active":"2024-03-12T03:48:03.430704+0000","last_peered":"2024-03-12T03:48:03.430704+0000","last_clean":"2024-03-12T03:48:03.430704+0000","last_became_active":"2024-03-12T03:46:25.286866+0000","last_became_peered":"2024-03-12T03:46:25.286866+0000","last_unstale":"2024-03-12T03:48:03.430704+0000","last_undegraded":"2024-03-12T03:48:03.430704+0000","last_fullsized":"2024-03-12T03:48:03.430704+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:05.530729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:43:05.530729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":385,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:58.700265+0000","last_change":"2024-03-12T03:46:49.647283+0000","last_active":"2024-03-12T03:47:58.700265+0000","last_peered":"2024-03-12T03:47:58.700265+0000","last_clean":"2024-03-12T03:47:58.700265+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:47:58.700265+0000","last_undegraded":"2024-03-12T03:47:58.700265+0000","last_fullsized":"2024-03-12T03:47:58.700265+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:49.647169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:46:49.647169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:18.202463+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":470,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.062986+0000","last_change":"2024-03-12T03:47:58.612030+0000","last_active":"2024-03-12T03:47:59.062986+0000","last_peered":"2024-03-12T03:47:59.062986+0000","last_clean":"2024-03-12T03:47:59.062986+0000","last_became_active":"2024-03-12T03:44:24.376141+0000","last_became_peered":"2024-03-12T03:44:24.376141+0000","last_unstale":"2024-03-12T03:47:59.062986+0000","last_undegraded":"2024-03-12T03:47:59.062986+0000","last_fullsized":"2024-03-12T03:47:59.062986+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:58.611912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.611912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:27.419788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"269'21","reported_seq":717,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064814+0000","last_change":"2024-03-12T03:47:09.828093+0000","last_active":"2024-03-12T03:47:59.064814+0000","last_peered":"2024-03-12T03:47:59.064814+0000","last_clean":"2024-03-12T03:47:59.064814+0000","last_became_active":"2024-03-12T03:46:25.298007+0000","last_became_peered":"2024-03-12T03:46:25.298007+0000","last_unstale":"2024-03-12T03:47:59.064814+0000","last_undegraded":"2024-03-12T03:47:59.064814+0000","last_fullsized":"2024-03-12T03:47:59.064814+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'21","last_scrub_stamp":"2024-03-12T03:47:09.827966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:09.827966+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:28.032022+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":668,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063401+0000","last_change":"2024-03-12T03:47:16.849528+0000","last_active":"2024-03-12T03:47:59.063401+0000","last_peered":"2024-03-12T03:47:59.063401+0000","last_clean":"2024-03-12T03:47:59.063401+0000","last_became_active":"2024-03-12T03:46:25.276467+0000","last_became_peered":"2024-03-12T03:46:25.276467+0000","last_unstale":"2024-03-12T03:47:59.063401+0000","last_undegraded":"2024-03-12T03:47:59.063401+0000","last_fullsized":"2024-03-12T03:47:59.063401+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:23.329829+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"332'14","reported_seq":626,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.431880+0000","last_change":"2024-03-12T03:46:25.302407+0000","last_active":"2024-03-12T03:48:03.431880+0000","last_peered":"2024-03-12T03:48:03.431880+0000","last_clean":"2024-03-12T03:48:03.431880+0000","last_became_active":"2024-03-12T03:46:25.300463+0000","last_became_peered":"2024-03-12T03:46:25.300463+0000","last_unstale":"2024-03-12T03:48:03.431880+0000","last_undegraded":"2024-03-12T03:48:03.431880+0000","last_fullsized":"2024-03-12T03:48:03.431880+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'12","last_scrub_stamp":"2024-03-12T03:43:12.675070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:12.675070+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":666,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066003+0000","last_change":"2024-03-12T03:47:11.001919+0000","last_active":"2024-03-12T03:47:59.066003+0000","last_peered":"2024-03-12T03:47:59.066003+0000","last_clean":"2024-03-12T03:47:59.066003+0000","last_became_active":"2024-03-12T03:39:33.265499+0000","last_became_peered":"2024-03-12T03:39:33.265499+0000","last_unstale":"2024-03-12T03:47:59.066003+0000","last_undegraded":"2024-03-12T03:47:59.066003+0000","last_fullsized":"2024-03-12T03:47:59.066003+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.001795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.001795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:36.111531+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":482,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.062542+0000","last_change":"2024-03-12T03:46:28.302513+0000","last_active":"2024-03-12T03:47:59.062542+0000","last_peered":"2024-03-12T03:47:59.062542+0000","last_clean":"2024-03-12T03:47:59.062542+0000","last_became_active":"2024-03-12T03:46:28.301964+0000","last_became_peered":"2024-03-12T03:46:28.301964+0000","last_unstale":"2024-03-12T03:47:59.062542+0000","last_undegraded":"2024-03-12T03:47:59.062542+0000","last_fullsized":"2024-03-12T03:47:59.062542+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:55.795990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:55.795990+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"269'32","reported_seq":519,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.431512+0000","last_change":"2024-03-12T03:47:55.395679+0000","last_active":"2024-03-12T03:48:03.431512+0000","last_peered":"2024-03-12T03:48:03.431512+0000","last_clean":"2024-03-12T03:48:03.431512+0000","last_became_active":"2024-03-12T03:46:25.299798+0000","last_became_peered":"2024-03-12T03:46:25.299798+0000","last_unstale":"2024-03-12T03:48:03.431512+0000","last_undegraded":"2024-03-12T03:48:03.431512+0000","last_fullsized":"2024-03-12T03:48:03.431512+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'32","last_scrub_stamp":"2024-03-12T03:47:55.395555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:55.395555+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:18.617496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":693,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063646+0000","last_change":"2024-03-12T03:47:53.609427+0000","last_active":"2024-03-12T03:47:59.063646+0000","last_peered":"2024-03-12T03:47:59.063646+0000","last_clean":"2024-03-12T03:47:59.063646+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:47:59.063646+0000","last_undegraded":"2024-03-12T03:47:59.063646+0000","last_fullsized":"2024-03-12T03:47:59.063646+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:47:53.609317+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:47:53.609317+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:11.417787+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":833,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064649+0000","last_change":"2024-03-12T03:45:53.459029+0000","last_active":"2024-03-12T03:47:59.064649+0000","last_peered":"2024-03-12T03:47:59.064649+0000","last_clean":"2024-03-12T03:47:59.064649+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:47:59.064649+0000","last_undegraded":"2024-03-12T03:47:59.064649+0000","last_fullsized":"2024-03-12T03:47:59.064649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:53.458170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.458170+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":656,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064029+0000","last_change":"2024-03-12T03:47:54.079670+0000","last_active":"2024-03-12T03:47:59.064029+0000","last_peered":"2024-03-12T03:47:59.064029+0000","last_clean":"2024-03-12T03:47:59.064029+0000","last_became_active":"2024-03-12T03:46:25.283805+0000","last_became_peered":"2024-03-12T03:46:25.283805+0000","last_unstale":"2024-03-12T03:47:59.064029+0000","last_undegraded":"2024-03-12T03:47:59.064029+0000","last_fullsized":"2024-03-12T03:47:59.064029+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.079569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.079569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:15.554223+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"269'19","reported_seq":693,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066895+0000","last_change":"2024-03-12T03:45:15.958314+0000","last_active":"2024-03-12T03:47:59.066895+0000","last_peered":"2024-03-12T03:47:59.066895+0000","last_clean":"2024-03-12T03:47:59.066895+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:47:59.066895+0000","last_undegraded":"2024-03-12T03:47:59.066895+0000","last_fullsized":"2024-03-12T03:47:59.066895+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'19","last_scrub_stamp":"2024-03-12T03:45:15.957617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.957617+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":666,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065572+0000","last_change":"2024-03-12T03:47:07.914268+0000","last_active":"2024-03-12T03:47:59.065572+0000","last_peered":"2024-03-12T03:47:59.065572+0000","last_clean":"2024-03-12T03:47:59.065572+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:47:59.065572+0000","last_undegraded":"2024-03-12T03:47:59.065572+0000","last_fullsized":"2024-03-12T03:47:59.065572+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_clean_scrub_stamp":"2024-03-12T03:47:07.914160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:18.181685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":668,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063353+0000","last_change":"2024-03-12T03:47:05.288244+0000","last_active":"2024-03-12T03:47:59.063353+0000","last_peered":"2024-03-12T03:47:59.063353+0000","last_clean":"2024-03-12T03:47:59.063353+0000","last_became_active":"2024-03-12T03:46:25.288050+0000","last_became_peered":"2024-03-12T03:46:25.288050+0000","last_unstale":"2024-03-12T03:47:59.063353+0000","last_undegraded":"2024-03-12T03:47:59.063353+0000","last_fullsized":"2024-03-12T03:47:59.063353+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:05.288140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:05.288140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:17.919622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":443,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.428476+0000","last_change":"2024-03-12T03:47:54.433284+0000","last_active":"2024-03-12T03:48:03.428476+0000","last_peered":"2024-03-12T03:48:03.428476+0000","last_clean":"2024-03-12T03:48:03.428476+0000","last_became_active":"2024-03-12T03:46:25.296588+0000","last_became_peered":"2024-03-12T03:46:25.296588+0000","last_unstale":"2024-03-12T03:48:03.428476+0000","last_undegraded":"2024-03-12T03:48:03.428476+0000","last_fullsized":"2024-03-12T03:48:03.428476+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.433184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.433184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:00.532918+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"269'15","reported_seq":258,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066099+0000","last_change":"2024-03-12T03:47:57.579034+0000","last_active":"2024-03-12T03:47:59.066099+0000","last_peered":"2024-03-12T03:47:59.066099+0000","last_clean":"2024-03-12T03:47:59.066099+0000","last_became_active":"2024-03-12T03:43:01.037348+0000","last_became_peered":"2024-03-12T03:43:01.037348+0000","last_unstale":"2024-03-12T03:47:59.066099+0000","last_undegraded":"2024-03-12T03:47:59.066099+0000","last_fullsized":"2024-03-12T03:47:59.066099+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'15","last_scrub_stamp":"2024-03-12T03:47:57.578870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:57.578870+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:12.668435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":676,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066685+0000","last_change":"2024-03-12T03:46:53.600616+0000","last_active":"2024-03-12T03:47:59.066685+0000","last_peered":"2024-03-12T03:47:59.066685+0000","last_clean":"2024-03-12T03:47:59.066685+0000","last_became_active":"2024-03-12T03:46:25.283602+0000","last_became_peered":"2024-03-12T03:46:25.283602+0000","last_unstale":"2024-03-12T03:47:59.066685+0000","last_undegraded":"2024-03-12T03:47:59.066685+0000","last_fullsized":"2024-03-12T03:47:59.066685+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:53.600496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:46:53.600496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:02.504558+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":447,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.429694+0000","last_change":"2024-03-12T03:46:25.286580+0000","last_active":"2024-03-12T03:48:03.429694+0000","last_peered":"2024-03-12T03:48:03.429694+0000","last_clean":"2024-03-12T03:48:03.429694+0000","last_became_active":"2024-03-12T03:46:25.285418+0000","last_became_peered":"2024-03-12T03:46:25.285418+0000","last_unstale":"2024-03-12T03:48:03.429694+0000","last_undegraded":"2024-03-12T03:48:03.429694+0000","last_fullsized":"2024-03-12T03:48:03.429694+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:15.808887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:15.808887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":646,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.062464+0000","last_change":"2024-03-12T03:45:00.842486+0000","last_active":"2024-03-12T03:47:59.062464+0000","last_peered":"2024-03-12T03:47:59.062464+0000","last_clean":"2024-03-12T03:47:59.062464+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:47:59.062464+0000","last_undegraded":"2024-03-12T03:47:59.062464+0000","last_fullsized":"2024-03-12T03:47:59.062464+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:00.842351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:45:00.842351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":583,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:58.699903+0000","last_change":"2024-03-12T03:47:14.914147+0000","last_active":"2024-03-12T03:47:58.699903+0000","last_peered":"2024-03-12T03:47:58.699903+0000","last_clean":"2024-03-12T03:47:58.699903+0000","last_became_active":"2024-03-12T03:44:24.387247+0000","last_became_peered":"2024-03-12T03:44:24.387247+0000","last_unstale":"2024-03-12T03:47:58.699903+0000","last_undegraded":"2024-03-12T03:47:58.699903+0000","last_fullsized":"2024-03-12T03:47:58.699903+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:14.914077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:14.914077+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:34.091102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"269'26","reported_seq":466,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.428674+0000","last_change":"2024-03-12T03:46:25.308547+0000","last_active":"2024-03-12T03:48:03.428674+0000","last_peered":"2024-03-12T03:48:03.428674+0000","last_clean":"2024-03-12T03:48:03.428674+0000","last_became_active":"2024-03-12T03:46:25.287095+0000","last_became_peered":"2024-03-12T03:46:25.287095+0000","last_unstale":"2024-03-12T03:48:03.428674+0000","last_undegraded":"2024-03-12T03:48:03.428674+0000","last_fullsized":"2024-03-12T03:48:03.428674+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:43:06.549441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:06.549441+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":657,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065254+0000","last_change":"2024-03-12T03:45:04.074618+0000","last_active":"2024-03-12T03:47:59.065254+0000","last_peered":"2024-03-12T03:47:59.065254+0000","last_clean":"2024-03-12T03:47:59.065254+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:47:59.065254+0000","last_undegraded":"2024-03-12T03:47:59.065254+0000","last_fullsized":"2024-03-12T03:47:59.065254+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.074505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.074505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":650,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063547+0000","last_change":"2024-03-12T03:44:59.151846+0000","last_active":"2024-03-12T03:47:59.063547+0000","last_peered":"2024-03-12T03:47:59.063547+0000","last_clean":"2024-03-12T03:47:59.063547+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:47:59.063547+0000","last_undegraded":"2024-03-12T03:47:59.063547+0000","last_fullsized":"2024-03-12T03:47:59.063547+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:59.151736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:44:59.151736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":470,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.430072+0000","last_change":"2024-03-12T03:46:25.288878+0000","last_active":"2024-03-12T03:48:03.430072+0000","last_peered":"2024-03-12T03:48:03.430072+0000","last_clean":"2024-03-12T03:48:03.430072+0000","last_became_active":"2024-03-12T03:46:25.286881+0000","last_became_peered":"2024-03-12T03:46:25.286881+0000","last_unstale":"2024-03-12T03:48:03.430072+0000","last_undegraded":"2024-03-12T03:48:03.430072+0000","last_fullsized":"2024-03-12T03:48:03.430072+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"269'10","reported_seq":437,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.427467+0000","last_change":"2024-03-12T03:46:25.310336+0000","last_active":"2024-03-12T03:48:03.427467+0000","last_peered":"2024-03-12T03:48:03.427467+0000","last_clean":"2024-03-12T03:48:03.427467+0000","last_became_active":"2024-03-12T03:46:25.286315+0000","last_became_peered":"2024-03-12T03:46:25.286315+0000","last_unstale":"2024-03-12T03:48:03.427467+0000","last_undegraded":"2024-03-12T03:48:03.427467+0000","last_fullsized":"2024-03-12T03:48:03.427467+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'6","last_scrub_stamp":"2024-03-12T03:38:38.943094+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:38:38.943094+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":668,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063136+0000","last_change":"2024-03-12T03:47:11.851386+0000","last_active":"2024-03-12T03:47:59.063136+0000","last_peered":"2024-03-12T03:47:59.063136+0000","last_clean":"2024-03-12T03:47:59.063136+0000","last_became_active":"2024-03-12T03:46:25.287132+0000","last_became_peered":"2024-03-12T03:46:25.287132+0000","last_unstale":"2024-03-12T03:47:59.063136+0000","last_undegraded":"2024-03-12T03:47:59.063136+0000","last_fullsized":"2024-03-12T03:47:59.063136+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.851271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.851271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:27.985117+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":121,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064946+0000","last_change":"2024-03-12T03:45:18.110285+0000","last_active":"2024-03-12T03:47:59.064946+0000","last_peered":"2024-03-12T03:47:59.064946+0000","last_clean":"2024-03-12T03:47:59.064946+0000","last_became_active":"2024-03-12T03:45:18.109983+0000","last_became_peered":"2024-03-12T03:45:18.109983+0000","last_unstale":"2024-03-12T03:47:59.064946+0000","last_undegraded":"2024-03-12T03:47:59.064946+0000","last_fullsized":"2024-03-12T03:47:59.064946+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:08.466225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:08.466225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":658,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064833+0000","last_change":"2024-03-12T03:45:12.308511+0000","last_active":"2024-03-12T03:47:59.064833+0000","last_peered":"2024-03-12T03:47:59.064833+0000","last_clean":"2024-03-12T03:47:59.064833+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:47:59.064833+0000","last_undegraded":"2024-03-12T03:47:59.064833+0000","last_fullsized":"2024-03-12T03:47:59.064833+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:12.308394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:12.308394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"269'31","reported_seq":630,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:48:00.687749+0000","last_change":"2024-03-12T03:48:00.687798+0000","last_active":"2024-03-12T03:48:00.687749+0000","last_peered":"2024-03-12T03:48:00.687749+0000","last_clean":"2024-03-12T03:48:00.687749+0000","last_became_active":"2024-03-12T03:45:18.107047+0000","last_became_peered":"2024-03-12T03:45:18.107047+0000","last_unstale":"2024-03-12T03:48:00.687749+0000","last_undegraded":"2024-03-12T03:48:00.687749+0000","last_fullsized":"2024-03-12T03:48:00.687749+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:48:00.687677+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.687677+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:11.240840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":471,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.427582+0000","last_change":"2024-03-12T03:48:00.458553+0000","last_active":"2024-03-12T03:48:03.427582+0000","last_peered":"2024-03-12T03:48:03.427582+0000","last_clean":"2024-03-12T03:48:03.427582+0000","last_became_active":"2024-03-12T03:46:25.287635+0000","last_became_peered":"2024-03-12T03:46:25.287635+0000","last_unstale":"2024-03-12T03:48:03.427582+0000","last_undegraded":"2024-03-12T03:48:03.427582+0000","last_fullsized":"2024-03-12T03:48:03.427582+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:48:00.458460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.458460+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:13.195187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":655,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064338+0000","last_change":"2024-03-12T03:44:13.704463+0000","last_active":"2024-03-12T03:47:59.064338+0000","last_peered":"2024-03-12T03:47:59.064338+0000","last_clean":"2024-03-12T03:47:59.064338+0000","last_became_active":"2024-03-12T03:39:33.276435+0000","last_became_peered":"2024-03-12T03:39:33.276435+0000","last_unstale":"2024-03-12T03:47:59.064338+0000","last_undegraded":"2024-03-12T03:47:59.064338+0000","last_fullsized":"2024-03-12T03:47:59.064338+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:13.704331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:44:13.704331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":652,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064135+0000","last_change":"2024-03-12T03:45:01.176270+0000","last_active":"2024-03-12T03:47:59.064135+0000","last_peered":"2024-03-12T03:47:59.064135+0000","last_clean":"2024-03-12T03:47:59.064135+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:47:59.064135+0000","last_undegraded":"2024-03-12T03:47:59.064135+0000","last_fullsized":"2024-03-12T03:47:59.064135+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:01.176150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:01.176150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"269'27","reported_seq":721,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064190+0000","last_change":"2024-03-12T03:47:15.795671+0000","last_active":"2024-03-12T03:47:59.064190+0000","last_peered":"2024-03-12T03:47:59.064190+0000","last_clean":"2024-03-12T03:47:59.064190+0000","last_became_active":"2024-03-12T03:46:25.284789+0000","last_became_peered":"2024-03-12T03:46:25.284789+0000","last_unstale":"2024-03-12T03:47:59.064190+0000","last_undegraded":"2024-03-12T03:47:59.064190+0000","last_fullsized":"2024-03-12T03:47:59.064190+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:15.795515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.795515+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:32.287782+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"351'2","reported_seq":678,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.062726+0000","last_change":"2024-03-12T03:47:15.334100+0000","last_active":"2024-03-12T03:47:59.062726+0000","last_peered":"2024-03-12T03:47:59.062726+0000","last_clean":"2024-03-12T03:47:59.062726+0000","last_became_active":"2024-03-12T03:46:25.283344+0000","last_became_peered":"2024-03-12T03:46:25.283344+0000","last_unstale":"2024-03-12T03:47:59.062726+0000","last_undegraded":"2024-03-12T03:47:59.062726+0000","last_fullsized":"2024-03-12T03:47:59.062726+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'2","last_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_deep_scrub":"351'2","last_deep_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.329905+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:36.545077+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'9","reported_seq":181,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066129+0000","last_change":"2024-03-12T03:45:18.109442+0000","last_active":"2024-03-12T03:47:59.066129+0000","last_peered":"2024-03-12T03:47:59.066129+0000","last_clean":"2024-03-12T03:47:59.066129+0000","last_became_active":"2024-03-12T03:45:18.108935+0000","last_became_peered":"2024-03-12T03:45:18.108935+0000","last_unstale":"2024-03-12T03:47:59.066129+0000","last_undegraded":"2024-03-12T03:47:59.066129+0000","last_fullsized":"2024-03-12T03:47:59.066129+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:04.572791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.572791+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":674,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065476+0000","last_change":"2024-03-12T03:47:14.619080+0000","last_active":"2024-03-12T03:47:59.065476+0000","last_peered":"2024-03-12T03:47:59.065476+0000","last_clean":"2024-03-12T03:47:59.065476+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:47:59.065476+0000","last_undegraded":"2024-03-12T03:47:59.065476+0000","last_fullsized":"2024-03-12T03:47:59.065476+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:47:14.618947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:14.618947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:17.380413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"269'24","reported_seq":503,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.429061+0000","last_change":"2024-03-12T03:47:58.427568+0000","last_active":"2024-03-12T03:48:03.429061+0000","last_peered":"2024-03-12T03:48:03.429061+0000","last_clean":"2024-03-12T03:48:03.429061+0000","last_became_active":"2024-03-12T03:46:25.283193+0000","last_became_peered":"2024-03-12T03:46:25.283193+0000","last_unstale":"2024-03-12T03:48:03.429061+0000","last_undegraded":"2024-03-12T03:48:03.429061+0000","last_fullsized":"2024-03-12T03:48:03.429061+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:47:58.427463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.427463+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:09.567592+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":677,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066112+0000","last_change":"2024-03-12T03:45:08.907717+0000","last_active":"2024-03-12T03:47:59.066112+0000","last_peered":"2024-03-12T03:47:59.066112+0000","last_clean":"2024-03-12T03:47:59.066112+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:47:59.066112+0000","last_undegraded":"2024-03-12T03:47:59.066112+0000","last_fullsized":"2024-03-12T03:47:59.066112+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:08.907581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:08.907581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":679,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065822+0000","last_change":"2024-03-12T03:46:53.126237+0000","last_active":"2024-03-12T03:47:59.065822+0000","last_peered":"2024-03-12T03:47:59.065822+0000","last_clean":"2024-03-12T03:47:59.065822+0000","last_became_active":"2024-03-12T03:46:25.301691+0000","last_became_peered":"2024-03-12T03:46:25.301691+0000","last_unstale":"2024-03-12T03:47:59.065822+0000","last_undegraded":"2024-03-12T03:47:59.065822+0000","last_fullsized":"2024-03-12T03:47:59.065822+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:53.126089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:46:53.126089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:11.436021+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":472,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.430920+0000","last_change":"2024-03-12T03:48:02.624958+0000","last_active":"2024-03-12T03:48:03.430920+0000","last_peered":"2024-03-12T03:48:03.430920+0000","last_clean":"2024-03-12T03:48:03.430920+0000","last_became_active":"2024-03-12T03:46:25.296764+0000","last_became_peered":"2024-03-12T03:46:25.296764+0000","last_unstale":"2024-03-12T03:48:03.430920+0000","last_undegraded":"2024-03-12T03:48:03.430920+0000","last_fullsized":"2024-03-12T03:48:03.430920+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:02.624857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:02.624857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:06.340909+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"269'27","reported_seq":444,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:58.700203+0000","last_change":"2024-03-12T03:47:56.823434+0000","last_active":"2024-03-12T03:47:58.700203+0000","last_peered":"2024-03-12T03:47:58.700203+0000","last_clean":"2024-03-12T03:47:58.700203+0000","last_became_active":"2024-03-12T03:46:25.301700+0000","last_became_peered":"2024-03-12T03:46:25.301700+0000","last_unstale":"2024-03-12T03:47:58.700203+0000","last_undegraded":"2024-03-12T03:47:58.700203+0000","last_fullsized":"2024-03-12T03:47:58.700203+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:56.823315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.823315+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:22.394677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":657,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064068+0000","last_change":"2024-03-12T03:45:03.797399+0000","last_active":"2024-03-12T03:47:59.064068+0000","last_peered":"2024-03-12T03:47:59.064068+0000","last_clean":"2024-03-12T03:47:59.064068+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:47:59.064068+0000","last_undegraded":"2024-03-12T03:47:59.064068+0000","last_fullsized":"2024-03-12T03:47:59.064068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.797296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.797296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":656,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.063990+0000","last_change":"2024-03-12T03:45:53.016158+0000","last_active":"2024-03-12T03:47:59.063990+0000","last_peered":"2024-03-12T03:47:59.063990+0000","last_clean":"2024-03-12T03:47:59.063990+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:47:59.063990+0000","last_undegraded":"2024-03-12T03:47:59.063990+0000","last_fullsized":"2024-03-12T03:47:59.063990+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.016030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.016030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":655,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:48:00.918235+0000","last_change":"2024-03-12T03:48:00.918264+0000","last_active":"2024-03-12T03:48:00.918235+0000","last_peered":"2024-03-12T03:48:00.918235+0000","last_clean":"2024-03-12T03:48:00.918235+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:48:00.918235+0000","last_undegraded":"2024-03-12T03:48:00.918235+0000","last_fullsized":"2024-03-12T03:48:00.918235+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:00.918131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.918131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:28.624540+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"269'16","reported_seq":740,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066760+0000","last_change":"2024-03-12T03:47:04.987456+0000","last_active":"2024-03-12T03:47:59.066760+0000","last_peered":"2024-03-12T03:47:59.066760+0000","last_clean":"2024-03-12T03:47:59.066760+0000","last_became_active":"2024-03-12T03:46:25.283025+0000","last_became_peered":"2024-03-12T03:46:25.283025+0000","last_unstale":"2024-03-12T03:47:59.066760+0000","last_undegraded":"2024-03-12T03:47:59.066760+0000","last_fullsized":"2024-03-12T03:47:59.066760+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'16","last_scrub_stamp":"2024-03-12T03:47:04.987337+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:47:04.987337+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:18.199925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":656,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.066611+0000","last_change":"2024-03-12T03:47:56.585903+0000","last_active":"2024-03-12T03:47:59.066611+0000","last_peered":"2024-03-12T03:47:59.066611+0000","last_clean":"2024-03-12T03:47:59.066611+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:47:59.066611+0000","last_undegraded":"2024-03-12T03:47:59.066611+0000","last_fullsized":"2024-03-12T03:47:59.066611+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.585817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.585817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:21.082050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":649,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064630+0000","last_change":"2024-03-12T03:45:11.324897+0000","last_active":"2024-03-12T03:47:59.064630+0000","last_peered":"2024-03-12T03:47:59.064630+0000","last_clean":"2024-03-12T03:47:59.064630+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:47:59.064630+0000","last_undegraded":"2024-03-12T03:47:59.064630+0000","last_fullsized":"2024-03-12T03:47:59.064630+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:11.324666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:45:11.324666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":655,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064808+0000","last_change":"2024-03-12T03:47:54.695744+0000","last_active":"2024-03-12T03:47:59.064808+0000","last_peered":"2024-03-12T03:47:59.064808+0000","last_clean":"2024-03-12T03:47:59.064808+0000","last_became_active":"2024-03-12T03:46:25.293777+0000","last_became_peered":"2024-03-12T03:46:25.293777+0000","last_unstale":"2024-03-12T03:47:59.064808+0000","last_undegraded":"2024-03-12T03:47:59.064808+0000","last_fullsized":"2024-03-12T03:47:59.064808+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.695594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.695594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:58.456936+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"345'33","reported_seq":735,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065908+0000","last_change":"2024-03-12T03:47:02.110674+0000","last_active":"2024-03-12T03:47:59.065908+0000","last_peered":"2024-03-12T03:47:59.065908+0000","last_clean":"2024-03-12T03:47:59.065908+0000","last_became_active":"2024-03-12T03:46:25.279992+0000","last_became_peered":"2024-03-12T03:46:25.279992+0000","last_unstale":"2024-03-12T03:47:59.065908+0000","last_undegraded":"2024-03-12T03:47:59.065908+0000","last_fullsized":"2024-03-12T03:47:59.065908+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'33","last_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_deep_scrub":"345'33","last_deep_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_clean_scrub_stamp":"2024-03-12T03:47:02.110517+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:31.585583+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":656,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064551+0000","last_change":"2024-03-12T03:45:56.503256+0000","last_active":"2024-03-12T03:47:59.064551+0000","last_peered":"2024-03-12T03:47:59.064551+0000","last_clean":"2024-03-12T03:47:59.064551+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:47:59.064551+0000","last_undegraded":"2024-03-12T03:47:59.064551+0000","last_fullsized":"2024-03-12T03:47:59.064551+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:56.503146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:45:56.503146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":648,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064145+0000","last_change":"2024-03-12T03:46:02.618859+0000","last_active":"2024-03-12T03:47:59.064145+0000","last_peered":"2024-03-12T03:47:59.064145+0000","last_clean":"2024-03-12T03:47:59.064145+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:47:59.064145+0000","last_undegraded":"2024-03-12T03:47:59.064145+0000","last_fullsized":"2024-03-12T03:47:59.064145+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:02.618599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_clean_scrub_stamp":"2024-03-12T03:46:02.618599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":646,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.062686+0000","last_change":"2024-03-12T03:45:58.095317+0000","last_active":"2024-03-12T03:47:59.062686+0000","last_peered":"2024-03-12T03:47:59.062686+0000","last_clean":"2024-03-12T03:47:59.062686+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:47:59.062686+0000","last_undegraded":"2024-03-12T03:47:59.062686+0000","last_fullsized":"2024-03-12T03:47:59.062686+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_clean_scrub_stamp":"2024-03-12T03:45:58.095196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"269'62","reported_seq":622,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.428177+0000","last_change":"2024-03-12T03:46:25.315560+0000","last_active":"2024-03-12T03:48:03.428177+0000","last_peered":"2024-03-12T03:48:03.428177+0000","last_clean":"2024-03-12T03:48:03.428177+0000","last_became_active":"2024-03-12T03:46:25.285447+0000","last_became_peered":"2024-03-12T03:46:25.285447+0000","last_unstale":"2024-03-12T03:48:03.428177+0000","last_undegraded":"2024-03-12T03:48:03.428177+0000","last_fullsized":"2024-03-12T03:48:03.428177+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":489,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.429969+0000","last_change":"2024-03-12T03:46:25.281156+0000","last_active":"2024-03-12T03:48:03.429969+0000","last_peered":"2024-03-12T03:48:03.429969+0000","last_clean":"2024-03-12T03:48:03.429969+0000","last_became_active":"2024-03-12T03:46:25.279131+0000","last_became_peered":"2024-03-12T03:46:25.279131+0000","last_unstale":"2024-03-12T03:48:03.429969+0000","last_undegraded":"2024-03-12T03:48:03.429969+0000","last_fullsized":"2024-03-12T03:48:03.429969+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:26.898782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:39:26.898782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":658,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064326+0000","last_change":"2024-03-12T03:47:55.924374+0000","last_active":"2024-03-12T03:47:59.064326+0000","last_peered":"2024-03-12T03:47:59.064326+0000","last_clean":"2024-03-12T03:47:59.064326+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:47:59.064326+0000","last_undegraded":"2024-03-12T03:47:59.064326+0000","last_fullsized":"2024-03-12T03:47:59.064326+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:55.924292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:47:55.924292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:08.749956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":448,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-03-12T03:48:03.429884+0000","last_change":"2024-03-12T03:47:57.456997+0000","last_active":"2024-03-12T03:48:03.429884+0000","last_peered":"2024-03-12T03:48:03.429884+0000","last_clean":"2024-03-12T03:48:03.429884+0000","last_became_active":"2024-03-12T03:46:25.281081+0000","last_became_peered":"2024-03-12T03:46:25.281081+0000","last_unstale":"2024-03-12T03:48:03.429884+0000","last_undegraded":"2024-03-12T03:48:03.429884+0000","last_fullsized":"2024-03-12T03:48:03.429884+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:57.456889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:57.456889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:06.875503+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"269'26","reported_seq":696,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.064232+0000","last_change":"2024-03-12T03:45:10.855287+0000","last_active":"2024-03-12T03:47:59.064232+0000","last_peered":"2024-03-12T03:47:59.064232+0000","last_clean":"2024-03-12T03:47:59.064232+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:47:59.064232+0000","last_undegraded":"2024-03-12T03:47:59.064232+0000","last_fullsized":"2024-03-12T03:47:59.064232+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:10.855156+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.855156+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":663,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.062986+0000","last_change":"2024-03-12T03:44:14.713075+0000","last_active":"2024-03-12T03:47:59.062986+0000","last_peered":"2024-03-12T03:47:59.062986+0000","last_clean":"2024-03-12T03:47:59.062986+0000","last_became_active":"2024-03-12T03:39:33.276119+0000","last_became_peered":"2024-03-12T03:39:33.276119+0000","last_unstale":"2024-03-12T03:47:59.062986+0000","last_undegraded":"2024-03-12T03:47:59.062986+0000","last_fullsized":"2024-03-12T03:47:59.062986+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:44:14.712916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":658,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.062883+0000","last_change":"2024-03-12T03:43:54.998904+0000","last_active":"2024-03-12T03:47:59.062883+0000","last_peered":"2024-03-12T03:47:59.062883+0000","last_clean":"2024-03-12T03:47:59.062883+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:47:59.062883+0000","last_undegraded":"2024-03-12T03:47:59.062883+0000","last_fullsized":"2024-03-12T03:47:59.062883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:54.998817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:26.625326+0000","last_clean_scrub_stamp":"2024-03-12T03:43:54.998817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":653,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-03-12T03:47:59.065596+0000","last_change":"2024-03-12T03:45:14.952828+0000","last_active":"2024-03-12T03:47:59.065596+0000","last_peered":"2024-03-12T03:47:59.065596+0000","last_clean":"2024-03-12T03:47:59.065596+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:47:59.065596+0000","last_undegraded":"2024-03-12T03:47:59.065596+0000","last_fullsized":"2024-03-12T03:47:59.065596+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_clean_scrub_stamp":"2024-03-12T03:45:14.952550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":873,"num_read_kb":868,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1402,"num_read_kb":1127,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":2284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},"log_size":957,"ondisk_log_size":957,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070645,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":687208,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93057944,"statfs":{"total":95995035648,"available":95291334656,"internally_reserved":0,"allocated":2191360,"data_stored":1199916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":701495754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332282,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":475088,"kb_used_data":2436,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93270064,"statfs":{"total":95995035648,"available":95508545536,"internally_reserved":0,"allocated":2494464,"data_stored":1525633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":483981766},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575059,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727484,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93017668,"statfs":{"total":95995035648,"available":95250092032,"internally_reserved":0,"allocated":2211840,"data_stored":1199693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738584,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":557032,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93188120,"statfs":{"total":95995035648,"available":95424634880,"internally_reserved":0,"allocated":2519040,"data_stored":1526850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411533,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":547220,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93197932,"statfs":{"total":95995035648,"available":95434682368,"internally_reserved":0,"allocated":2428928,"data_stored":1461654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":330,"seq":1417339207701,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181692,"kb_used_data":2736,"kb_used_omap":13,"kb_used_meta":178930,"kb_avail":93563460,"statfs":{"total":95995035648,"available":95808983040,"internally_reserved":0,"allocated":2801664,"data_stored":1789648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13485,"internal_metadata":183225171},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953669,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412332,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93332820,"statfs":{"total":95995035648,"available":95572807680,"internally_reserved":0,"allocated":2195456,"data_stored":1199456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659327,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":705512,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93039640,"statfs":{"total":95995035648,"available":95272591360,"internally_reserved":0,"allocated":2453504,"data_stored":1461422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:48:09.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:48:09.677 INFO:teuthology.orchestra.run.smithi169.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-12T03:48:09.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:09.785 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:09.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:09.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:09.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:09.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:09.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[29344]: pgmap v714: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:09.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[29344]: 1.12 scrub starts 2024-03-12T03:48:09.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[29344]: 1.12 scrub ok 2024-03-12T03:48:09.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[29344]: 1.b scrub starts 2024-03-12T03:48:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[29344]: 1.b scrub ok 2024-03-12T03:48:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3020340292' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:48:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[29344]: 5.1e scrub starts 2024-03-12T03:48:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[29344]: 5.1e scrub ok 2024-03-12T03:48:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[39949]: pgmap v714: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[39949]: 1.12 scrub starts 2024-03-12T03:48:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[39949]: 1.12 scrub ok 2024-03-12T03:48:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[39949]: 1.b scrub starts 2024-03-12T03:48:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[39949]: 1.b scrub ok 2024-03-12T03:48:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3020340292' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:48:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[39949]: 5.1e scrub starts 2024-03-12T03:48:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:09 smithi169 ceph-mon[39949]: 5.1e scrub ok 2024-03-12T03:48:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:09 smithi173 ceph-mon[33544]: pgmap v714: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:09 smithi173 ceph-mon[33544]: 1.12 scrub starts 2024-03-12T03:48:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:09 smithi173 ceph-mon[33544]: 1.12 scrub ok 2024-03-12T03:48:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:09 smithi173 ceph-mon[33544]: 1.b scrub starts 2024-03-12T03:48:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:09 smithi173 ceph-mon[33544]: 1.b scrub ok 2024-03-12T03:48:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:09 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3020340292' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:48:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:09 smithi173 ceph-mon[33544]: 5.1e scrub starts 2024-03-12T03:48:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:09 smithi173 ceph-mon[33544]: 5.1e scrub ok 2024-03-12T03:48:10.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:10.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:10.041 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:10.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:10.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:10.165 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:10.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:10.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:10.267 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:10.296 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:10.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:10.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:10.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:10.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:10.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:10.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:10.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:10.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:10.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:10.681 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:10.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:10.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:10.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:10.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:10.886 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:48:10.886 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:48:10.911 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:10.936 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:10.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:10 smithi173 ceph-mon[33544]: 4.5 scrub starts 2024-03-12T03:48:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:10 smithi173 ceph-mon[33544]: 4.5 scrub ok 2024-03-12T03:48:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:10 smithi173 ceph-mon[33544]: 5.10 scrub starts 2024-03-12T03:48:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:10 smithi173 ceph-mon[33544]: 5.10 scrub ok 2024-03-12T03:48:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:10 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3020340292' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:48:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:10 smithi173 ceph-mon[33544]: osdmap e372: 8 total, 8 up, 8 in 2024-03-12T03:48:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:10 smithi173 ceph-mon[33544]: 1.3 scrub starts 2024-03-12T03:48:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:10 smithi173 ceph-mon[33544]: 1.3 scrub ok 2024-03-12T03:48:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:10 smithi173 ceph-mon[33544]: 5.13 deep-scrub starts 2024-03-12T03:48:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:10 smithi173 ceph-mon[33544]: 5.13 deep-scrub ok 2024-03-12T03:48:11.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:11.061 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:11.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:11.163 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:11.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[29344]: 4.5 scrub starts 2024-03-12T03:48:11.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[29344]: 4.5 scrub ok 2024-03-12T03:48:11.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[29344]: 5.10 scrub starts 2024-03-12T03:48:11.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[29344]: 5.10 scrub ok 2024-03-12T03:48:11.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3020340292' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:48:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[29344]: osdmap e372: 8 total, 8 up, 8 in 2024-03-12T03:48:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[29344]: 1.3 scrub starts 2024-03-12T03:48:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[29344]: 1.3 scrub ok 2024-03-12T03:48:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[29344]: 5.13 deep-scrub starts 2024-03-12T03:48:11.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[29344]: 5.13 deep-scrub ok 2024-03-12T03:48:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[39949]: 4.5 scrub starts 2024-03-12T03:48:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[39949]: 4.5 scrub ok 2024-03-12T03:48:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[39949]: 5.10 scrub starts 2024-03-12T03:48:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[39949]: 5.10 scrub ok 2024-03-12T03:48:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3020340292' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:48:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[39949]: osdmap e372: 8 total, 8 up, 8 in 2024-03-12T03:48:11.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[39949]: 1.3 scrub starts 2024-03-12T03:48:11.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[39949]: 1.3 scrub ok 2024-03-12T03:48:11.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[39949]: 5.13 deep-scrub starts 2024-03-12T03:48:11.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:10 smithi169 ceph-mon[39949]: 5.13 deep-scrub ok 2024-03-12T03:48:11.186 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:11.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:11.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:11.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:11.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:11.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:11.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:11.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:11.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:11.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:11.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:11.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:11.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:11.700 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:11.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:11.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:11.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:11.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:11.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:11.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:11.955 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:11.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:11 smithi173 ceph-mon[33544]: pgmap v717: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:11 smithi173 ceph-mon[33544]: 1.d scrub starts 2024-03-12T03:48:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:11 smithi173 ceph-mon[33544]: 1.d scrub ok 2024-03-12T03:48:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:11 smithi173 ceph-mon[33544]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:48:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:11 smithi173 ceph-mon[33544]: Cluster is now healthy 2024-03-12T03:48:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:11 smithi173 ceph-mon[33544]: 1.13 scrub starts 2024-03-12T03:48:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:11 smithi173 ceph-mon[33544]: 1.13 scrub ok 2024-03-12T03:48:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:11 smithi173 ceph-mon[33544]: 5.d scrub starts 2024-03-12T03:48:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:11 smithi173 ceph-mon[33544]: 5.d scrub ok 2024-03-12T03:48:12.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:12.058 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:12.058 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:12.085 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:12.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:12.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:12.155 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:48:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:48:12.155 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[29344]: pgmap v717: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:12.155 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[29344]: 1.d scrub starts 2024-03-12T03:48:12.155 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[29344]: 1.d scrub ok 2024-03-12T03:48:12.155 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[29344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:48:12.155 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[29344]: Cluster is now healthy 2024-03-12T03:48:12.155 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[29344]: 1.13 scrub starts 2024-03-12T03:48:12.155 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[29344]: 1.13 scrub ok 2024-03-12T03:48:12.156 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[29344]: 5.d scrub starts 2024-03-12T03:48:12.156 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[29344]: 5.d scrub ok 2024-03-12T03:48:12.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[39949]: pgmap v717: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:12.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[39949]: 1.d scrub starts 2024-03-12T03:48:12.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[39949]: 1.d scrub ok 2024-03-12T03:48:12.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[39949]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:48:12.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[39949]: Cluster is now healthy 2024-03-12T03:48:12.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[39949]: 1.13 scrub starts 2024-03-12T03:48:12.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[39949]: 1.13 scrub ok 2024-03-12T03:48:12.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[39949]: 5.d scrub starts 2024-03-12T03:48:12.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:11 smithi169 ceph-mon[39949]: 5.d scrub ok 2024-03-12T03:48:12.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:12.210 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:12.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:12.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:12.311 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:12.335 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:12.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:12.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:12.460 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:12.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:12.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-12T03:48:12.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:12.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:12.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:12.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:12.715 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:12.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:12.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:12.766 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:12 smithi169 ceph-mon[29344]: 1.0 deep-scrub starts 2024-03-12T03:48:12.767 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:12 smithi169 ceph-mon[29344]: 1.0 deep-scrub ok 2024-03-12T03:48:12.767 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:12 smithi169 ceph-mon[29344]: 3.1f scrub starts 2024-03-12T03:48:12.767 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:12 smithi169 ceph-mon[29344]: 3.1f scrub ok 2024-03-12T03:48:12.767 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:12 smithi169 ceph-mon[29344]: 3.1e scrub starts 2024-03-12T03:48:12.767 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:12 smithi169 ceph-mon[29344]: 3.1e scrub ok 2024-03-12T03:48:12.767 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:12 smithi169 ceph-mon[39949]: 1.0 deep-scrub starts 2024-03-12T03:48:12.767 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:12 smithi169 ceph-mon[39949]: 1.0 deep-scrub ok 2024-03-12T03:48:12.767 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:12 smithi169 ceph-mon[39949]: 3.1f scrub starts 2024-03-12T03:48:12.767 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:12 smithi169 ceph-mon[39949]: 3.1f scrub ok 2024-03-12T03:48:12.767 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:12 smithi169 ceph-mon[39949]: 3.1e scrub starts 2024-03-12T03:48:12.767 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:12 smithi169 ceph-mon[39949]: 3.1e scrub ok 2024-03-12T03:48:12.782 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-12T03:48:12.784 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-12T03:48:12.785 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-12T03:48:12.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:12.839 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:12.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:12.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:12.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:12.966 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:12.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:12 smithi173 ceph-mon[33544]: 1.0 deep-scrub starts 2024-03-12T03:48:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:12 smithi173 ceph-mon[33544]: 1.0 deep-scrub ok 2024-03-12T03:48:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:12 smithi173 ceph-mon[33544]: 3.1f scrub starts 2024-03-12T03:48:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:12 smithi173 ceph-mon[33544]: 3.1f scrub ok 2024-03-12T03:48:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:12 smithi173 ceph-mon[33544]: 3.1e scrub starts 2024-03-12T03:48:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:12 smithi173 ceph-mon[33544]: 3.1e scrub ok 2024-03-12T03:48:13.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:hints = 1 2024-03-12T03:48:13.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-12T03:48:13.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Object prefix: benchmark_data_smithi169_162694 2024-03-12T03:48:13.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:48:13.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 0 0 0 0 0 0 - 0 2024-03-12T03:48:13.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:13.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:13.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:13.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:13.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:13.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:13.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:13.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:13.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:13.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:13.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:13.473 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:13.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:48:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:48:13.511 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:13.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:13.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:13.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:13.658 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:13.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:13.759 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:13.799 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:13.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:13.821 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:48:13.859 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:48:13.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:13.903 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:48:13.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:13.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:13.949 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:48:13.949 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:13 smithi169 ceph-mon[29344]: 1.e deep-scrub starts 2024-03-12T03:48:13.949 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:13 smithi169 ceph-mon[29344]: 1.e deep-scrub ok 2024-03-12T03:48:13.949 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:13 smithi169 ceph-mon[29344]: pgmap v718: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T03:48:13.950 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:13 smithi169 ceph-mon[29344]: 6.10 scrub starts 2024-03-12T03:48:13.950 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:13 smithi169 ceph-mon[29344]: 6.10 scrub ok 2024-03-12T03:48:13.950 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:13 smithi169 ceph-mon[29344]: osdmap e373: 8 total, 8 up, 8 in 2024-03-12T03:48:13.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:13 smithi169 ceph-mon[39949]: 1.e deep-scrub starts 2024-03-12T03:48:13.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:13 smithi169 ceph-mon[39949]: 1.e deep-scrub ok 2024-03-12T03:48:13.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:13 smithi169 ceph-mon[39949]: pgmap v718: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T03:48:13.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:13 smithi169 ceph-mon[39949]: 6.10 scrub starts 2024-03-12T03:48:13.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:13 smithi169 ceph-mon[39949]: 6.10 scrub ok 2024-03-12T03:48:13.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:13 smithi169 ceph-mon[39949]: osdmap e373: 8 total, 8 up, 8 in 2024-03-12T03:48:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:13 smithi173 ceph-mon[33544]: 1.e deep-scrub starts 2024-03-12T03:48:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:13 smithi173 ceph-mon[33544]: 1.e deep-scrub ok 2024-03-12T03:48:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:13 smithi173 ceph-mon[33544]: pgmap v718: 168 pgs: 168 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T03:48:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:13 smithi173 ceph-mon[33544]: 6.10 scrub starts 2024-03-12T03:48:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:13 smithi173 ceph-mon[33544]: 6.10 scrub ok 2024-03-12T03:48:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:13 smithi173 ceph-mon[33544]: osdmap e373: 8 total, 8 up, 8 in 2024-03-12T03:48:13.990 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:48:14.033 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:48:14.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:14.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 1 15 3202 3187 199.176 199.188 0.00401765 0.00499731 2024-03-12T03:48:14.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:14.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:14.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:14.073 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:48:14.112 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:48:14.116 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:48:14.116 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:48:14.116 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:48:14.116 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:48:14.116 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:48:14.116 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:48:14.116 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:48:14.116 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:48:14.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:14.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:14.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:14.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:14.343 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:14.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:14.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:14.481 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:14.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:14.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:14.583 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:14.621 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:14.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:14.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:14.723 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:14.762 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:14.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:14.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:14.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:14.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:14 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:14 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2769607046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:48:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:14 smithi169 ceph-mon[29344]: 5.3 scrub starts 2024-03-12T03:48:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:14 smithi169 ceph-mon[29344]: 5.3 scrub ok 2024-03-12T03:48:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:14 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:14 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2769607046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:48:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:14 smithi169 ceph-mon[39949]: 5.3 scrub starts 2024-03-12T03:48:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:14 smithi169 ceph-mon[39949]: 5.3 scrub ok 2024-03-12T03:48:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:14 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:14 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2769607046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:48:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:14 smithi173 ceph-mon[33544]: 5.3 scrub starts 2024-03-12T03:48:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:14 smithi173 ceph-mon[33544]: 5.3 scrub ok 2024-03-12T03:48:14.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:15.039 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:15.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:15.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 2 16 6513 6497 203.017 206.875 0.00704272 0.0049152 2024-03-12T03:48:15.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:15.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:15.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:15.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:15.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:15.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:15.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:15.449 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:15.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:15.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:15.588 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:15.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:15.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:15.741 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:15.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:15.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:15.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:15.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:15.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:15.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:15 smithi169 ceph-mon[29344]: pgmap v720: 168 pgs: 168 active+clean; 10 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 1.6 MiB/s wr, 26 op/s 2024-03-12T03:48:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:15 smithi169 ceph-mon[29344]: 5.2 scrub starts 2024-03-12T03:48:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:15 smithi169 ceph-mon[29344]: 5.2 scrub ok 2024-03-12T03:48:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:15 smithi169 ceph-mon[39949]: pgmap v720: 168 pgs: 168 active+clean; 10 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 1.6 MiB/s wr, 26 op/s 2024-03-12T03:48:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:15 smithi169 ceph-mon[39949]: 5.2 scrub starts 2024-03-12T03:48:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:15 smithi169 ceph-mon[39949]: 5.2 scrub ok 2024-03-12T03:48:15.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:15 smithi173 ceph-mon[33544]: pgmap v720: 168 pgs: 168 active+clean; 10 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 1.6 MiB/s wr, 26 op/s 2024-03-12T03:48:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:15 smithi173 ceph-mon[33544]: 5.2 scrub starts 2024-03-12T03:48:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:15 smithi173 ceph-mon[33544]: 5.2 scrub ok 2024-03-12T03:48:16.019 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:16.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:16.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 3 15 9718 9703 202.13 200.375 0.00431149 0.00494018 2024-03-12T03:48:16.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:16.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:16.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:16.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:16.303 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:16.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:16.404 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:16.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:16.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:16.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:16.586 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:16.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:16.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:16.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:16.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:16.763 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:16.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:16.861 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:16.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:16.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:16.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:17.005 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:17.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:17.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 4 16 12544 12528 195.734 176.562 0.0480073 0.00506152 2024-03-12T03:48:17.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:17.147 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:17.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:17.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:17.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:17.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:17.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:17.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:17.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:17.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:17.584 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:17.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:17.609 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[29344]: 1.5 deep-scrub starts 2024-03-12T03:48:17.610 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[29344]: 1.5 deep-scrub ok 2024-03-12T03:48:17.610 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[29344]: 5.8 scrub starts 2024-03-12T03:48:17.610 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[29344]: 5.8 scrub ok 2024-03-12T03:48:17.610 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[29344]: pgmap v721: 168 pgs: 168 active+clean; 10 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 666 B/s rd, 1.2 MiB/s wr, 20 op/s 2024-03-12T03:48:17.610 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[29344]: 1.11 scrub starts 2024-03-12T03:48:17.610 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[29344]: 1.11 scrub ok 2024-03-12T03:48:17.610 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:17.610 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:17.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[39949]: 1.5 deep-scrub starts 2024-03-12T03:48:17.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[39949]: 1.5 deep-scrub ok 2024-03-12T03:48:17.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[39949]: 5.8 scrub starts 2024-03-12T03:48:17.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[39949]: 5.8 scrub ok 2024-03-12T03:48:17.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[39949]: pgmap v721: 168 pgs: 168 active+clean; 10 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 666 B/s rd, 1.2 MiB/s wr, 20 op/s 2024-03-12T03:48:17.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[39949]: 1.11 scrub starts 2024-03-12T03:48:17.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[39949]: 1.11 scrub ok 2024-03-12T03:48:17.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:17.611 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:17 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:17.685 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:17.725 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:17.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:17 smithi173 ceph-mon[33544]: 1.5 deep-scrub starts 2024-03-12T03:48:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:17 smithi173 ceph-mon[33544]: 1.5 deep-scrub ok 2024-03-12T03:48:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:17 smithi173 ceph-mon[33544]: 5.8 scrub starts 2024-03-12T03:48:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:17 smithi173 ceph-mon[33544]: 5.8 scrub ok 2024-03-12T03:48:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:17 smithi173 ceph-mon[33544]: pgmap v721: 168 pgs: 168 active+clean; 10 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 666 B/s rd, 1.2 MiB/s wr, 20 op/s 2024-03-12T03:48:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:17 smithi173 ceph-mon[33544]: 1.11 scrub starts 2024-03-12T03:48:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:17 smithi173 ceph-mon[33544]: 1.11 scrub ok 2024-03-12T03:48:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:17 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:17.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:17 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:17.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:17.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:17.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:17.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:18.000 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:18.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:18.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 5 16 14948 14932 186.634 150.25 0.00607285 0.00535061 2024-03-12T03:48:18.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:18.102 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:48:18.108 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:48:18.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:18.149 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:18.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:18.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:48:18.250 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:18.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:18.320 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:18.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:18.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:18.477 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:18.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:18.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:18.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:18.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:18.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[29344]: 4.a scrub starts 2024-03-12T03:48:18.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[29344]: 4.a scrub ok 2024-03-12T03:48:18.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[29344]: 3.8 scrub starts 2024-03-12T03:48:18.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[29344]: 3.8 scrub ok 2024-03-12T03:48:18.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[29344]: 4.6 scrub starts 2024-03-12T03:48:18.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[29344]: 4.6 scrub ok 2024-03-12T03:48:18.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[29344]: 1.a scrub starts 2024-03-12T03:48:18.634 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[29344]: 1.a scrub ok 2024-03-12T03:48:18.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[39949]: 4.a scrub starts 2024-03-12T03:48:18.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[39949]: 4.a scrub ok 2024-03-12T03:48:18.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[39949]: 3.8 scrub starts 2024-03-12T03:48:18.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[39949]: 3.8 scrub ok 2024-03-12T03:48:18.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[39949]: 4.6 scrub starts 2024-03-12T03:48:18.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[39949]: 4.6 scrub ok 2024-03-12T03:48:18.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[39949]: 1.a scrub starts 2024-03-12T03:48:18.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:18 smithi169 ceph-mon[39949]: 1.a scrub ok 2024-03-12T03:48:18.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:18 smithi173 ceph-mon[33544]: 4.a scrub starts 2024-03-12T03:48:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:18 smithi173 ceph-mon[33544]: 4.a scrub ok 2024-03-12T03:48:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:18 smithi173 ceph-mon[33544]: 3.8 scrub starts 2024-03-12T03:48:18.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:18 smithi173 ceph-mon[33544]: 3.8 scrub ok 2024-03-12T03:48:18.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:18 smithi173 ceph-mon[33544]: 4.6 scrub starts 2024-03-12T03:48:18.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:18 smithi173 ceph-mon[33544]: 4.6 scrub ok 2024-03-12T03:48:18.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:18 smithi173 ceph-mon[33544]: 1.a scrub starts 2024-03-12T03:48:18.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:18 smithi173 ceph-mon[33544]: 1.a scrub ok 2024-03-12T03:48:18.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:18.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:18.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:18.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:18.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:18.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:18.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:19.023 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:19.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:19.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:19.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 6 16 17372 17356 180.775 151.5 0.00806605 0.00552473 2024-03-12T03:48:19.125 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:19.167 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:19.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:19.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:19.269 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:19.311 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:19.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:19.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:19.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:19.447 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:48:19.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:19.458 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:19.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:19.470 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[29344]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:48:19.470 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[29344]: pgmap v722: 168 pgs: 168 active+clean; 644 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-12T03:48:19.470 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[29344]: 5.9 scrub starts 2024-03-12T03:48:19.470 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[29344]: 5.9 scrub ok 2024-03-12T03:48:19.470 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/741925291' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:48:19.470 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[29344]: 3.a scrub starts 2024-03-12T03:48:19.470 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[29344]: 3.a scrub ok 2024-03-12T03:48:19.470 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[39949]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:48:19.470 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[39949]: pgmap v722: 168 pgs: 168 active+clean; 644 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-12T03:48:19.471 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[39949]: 5.9 scrub starts 2024-03-12T03:48:19.471 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[39949]: 5.9 scrub ok 2024-03-12T03:48:19.471 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/741925291' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:48:19.471 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[39949]: 3.a scrub starts 2024-03-12T03:48:19.471 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:19 smithi169 ceph-mon[39949]: 3.a scrub ok 2024-03-12T03:48:19.559 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:19.600 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:19.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:19.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:19.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:19.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:19 smithi173 ceph-mon[33544]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:48:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:19 smithi173 ceph-mon[33544]: pgmap v722: 168 pgs: 168 active+clean; 644 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-12T03:48:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:19 smithi173 ceph-mon[33544]: 5.9 scrub starts 2024-03-12T03:48:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:19 smithi173 ceph-mon[33544]: 5.9 scrub ok 2024-03-12T03:48:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:19 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/741925291' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:48:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:19 smithi173 ceph-mon[33544]: 3.a scrub starts 2024-03-12T03:48:19.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:19 smithi173 ceph-mon[33544]: 3.a scrub ok 2024-03-12T03:48:19.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:19.879 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:19.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:19.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:19.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:20.020 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:20.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:20.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 7 15 19930 19915 177.796 159.938 0.00570673 0.00561845 2024-03-12T03:48:20.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:20.165 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:20.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:20.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:20.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:20.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:20.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:20.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:20.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:20.539 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:20.579 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:20.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:20.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:20 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:48:20.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:20 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/741925291' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:48:20.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:20 smithi169 ceph-mon[29344]: osdmap e374: 8 total, 8 up, 8 in 2024-03-12T03:48:20.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:20 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:48:20.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:20 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:48:20.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:20 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/741925291' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:48:20.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:20 smithi169 ceph-mon[39949]: osdmap e374: 8 total, 8 up, 8 in 2024-03-12T03:48:20.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:20 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:48:20.679 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:20.711 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:20.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:20 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:48:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:20 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/741925291' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:48:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:20 smithi173 ceph-mon[33544]: osdmap e374: 8 total, 8 up, 8 in 2024-03-12T03:48:20.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:20 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:48:20.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:20.850 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:20.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:20.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:21.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:21.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:21.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:21.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 8 16 22244 22228 173.64 144.562 0.00440471 0.00575256 2024-03-12T03:48:21.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:21.147 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:21.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:21.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:21.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:21.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:21.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:21.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:21.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:21.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:21.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:21.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:21.667 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:21.706 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:21.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:21.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:21 smithi173 ceph-mon[33544]: pgmap v724: 168 pgs: 168 active+clean; 756 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-12T03:48:21.740 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:21 smithi169 ceph-mon[39949]: pgmap v724: 168 pgs: 168 active+clean; 756 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-12T03:48:21.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:21 smithi169 ceph-mon[29344]: pgmap v724: 168 pgs: 168 active+clean; 756 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-12T03:48:21.742 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":721,"stamp":"2024-03-12T03:48:16.103464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10557864,"num_objects":382,"num_object_clones":0,"num_object_copies":1146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":382,"num_whiteouts":0,"num_read":2302,"num_read_kb":2022,"num_write":1066,"num_write_kb":9773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":530222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3895,"ondisk_log_size":3895,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4293960,"kb_used_data":19228,"kb_used_omap":24,"kb_used_meta":4274599,"kb_avail":745667256,"statfs":{"total":767960285184,"available":763563270144,"internally_reserved":0,"allocated":19689472,"data_stored":11587128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24644,"internal_metadata":4377190332},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9961472,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":152,"num_write_kb":9728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.683431"},"pg_stats":[{"pgid":"10.2","version":"0'0","reported_seq":21,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710694+0000","last_change":"2024-03-12T03:47:57.698728+0000","last_active":"2024-03-12T03:48:09.710694+0000","last_peered":"2024-03-12T03:48:09.710694+0000","last_clean":"2024-03-12T03:48:09.710694+0000","last_became_active":"2024-03-12T03:47:57.698438+0000","last_became_peered":"2024-03-12T03:47:57.698438+0000","last_unstale":"2024-03-12T03:48:09.710694+0000","last_undegraded":"2024-03-12T03:48:09.710694+0000","last_fullsized":"2024-03-12T03:48:09.710694+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:21.200457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":978,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706590+0000","last_change":"2024-03-12T03:46:52.574410+0000","last_active":"2024-03-12T03:48:09.706590+0000","last_peered":"2024-03-12T03:48:09.706590+0000","last_clean":"2024-03-12T03:48:09.706590+0000","last_became_active":"2024-03-12T03:46:25.289005+0000","last_became_peered":"2024-03-12T03:46:25.289005+0000","last_unstale":"2024-03-12T03:48:09.706590+0000","last_undegraded":"2024-03-12T03:48:09.706590+0000","last_fullsized":"2024-03-12T03:48:09.706590+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:46:52.574274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:46:52.574274+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"344'25","reported_seq":728,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708113+0000","last_change":"2024-03-12T03:46:57.114034+0000","last_active":"2024-03-12T03:48:09.708113+0000","last_peered":"2024-03-12T03:48:09.708113+0000","last_clean":"2024-03-12T03:48:09.708113+0000","last_became_active":"2024-03-12T03:46:25.301377+0000","last_became_peered":"2024-03-12T03:46:25.301377+0000","last_unstale":"2024-03-12T03:48:09.708113+0000","last_undegraded":"2024-03-12T03:48:09.708113+0000","last_fullsized":"2024-03-12T03:48:09.708113+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'25","last_scrub_stamp":"2024-03-12T03:46:57.113869+0000","last_deep_scrub":"344'25","last_deep_scrub_stamp":"2024-03-12T03:46:57.113869+0000","last_clean_scrub_stamp":"2024-03-12T03:46:57.113869+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":441,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709908+0000","last_change":"2024-03-12T03:47:07.363992+0000","last_active":"2024-03-12T03:48:09.709908+0000","last_peered":"2024-03-12T03:48:09.709908+0000","last_clean":"2024-03-12T03:48:09.709908+0000","last_became_active":"2024-03-12T03:46:25.286247+0000","last_became_peered":"2024-03-12T03:46:25.286247+0000","last_unstale":"2024-03-12T03:48:09.709908+0000","last_undegraded":"2024-03-12T03:48:09.709908+0000","last_fullsized":"2024-03-12T03:48:09.709908+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:07.363439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:07.363439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":508,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:10.956106+0000","last_change":"2024-03-12T03:48:10.956141+0000","last_active":"2024-03-12T03:48:10.956106+0000","last_peered":"2024-03-12T03:48:10.956106+0000","last_clean":"2024-03-12T03:48:10.956106+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:48:10.956106+0000","last_undegraded":"2024-03-12T03:48:10.956106+0000","last_fullsized":"2024-03-12T03:48:10.956106+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:48:10.956048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:10.956048+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:35.591941+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708825+0000","last_change":"2024-03-12T03:47:12.937224+0000","last_active":"2024-03-12T03:48:09.708825+0000","last_peered":"2024-03-12T03:48:09.708825+0000","last_clean":"2024-03-12T03:48:09.708825+0000","last_became_active":"2024-03-12T03:39:33.265665+0000","last_became_peered":"2024-03-12T03:39:33.265665+0000","last_unstale":"2024-03-12T03:48:09.708825+0000","last_undegraded":"2024-03-12T03:48:09.708825+0000","last_fullsized":"2024-03-12T03:48:09.708825+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:12.937101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:12.937101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:26.876746+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":22,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708203+0000","last_change":"2024-03-12T03:47:57.695505+0000","last_active":"2024-03-12T03:48:09.708203+0000","last_peered":"2024-03-12T03:48:09.708203+0000","last_clean":"2024-03-12T03:48:09.708203+0000","last_became_active":"2024-03-12T03:47:57.695210+0000","last_became_peered":"2024-03-12T03:47:57.695210+0000","last_unstale":"2024-03-12T03:48:09.708203+0000","last_undegraded":"2024-03-12T03:48:09.708203+0000","last_fullsized":"2024-03-12T03:48:09.708203+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:14.612593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"269'31","reported_seq":767,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708536+0000","last_change":"2024-03-12T03:47:13.860107+0000","last_active":"2024-03-12T03:48:09.708536+0000","last_peered":"2024-03-12T03:48:09.708536+0000","last_clean":"2024-03-12T03:48:09.708536+0000","last_became_active":"2024-03-12T03:39:33.268652+0000","last_became_peered":"2024-03-12T03:39:33.268652+0000","last_unstale":"2024-03-12T03:48:09.708536+0000","last_undegraded":"2024-03-12T03:48:09.708536+0000","last_fullsized":"2024-03-12T03:48:09.708536+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:47:13.859943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.859943+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:29.361882+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":129,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708283+0000","last_change":"2024-03-12T03:45:18.109064+0000","last_active":"2024-03-12T03:48:09.708283+0000","last_peered":"2024-03-12T03:48:09.708283+0000","last_clean":"2024-03-12T03:48:09.708283+0000","last_became_active":"2024-03-12T03:45:18.108377+0000","last_became_peered":"2024-03-12T03:45:18.108377+0000","last_unstale":"2024-03-12T03:48:09.708283+0000","last_undegraded":"2024-03-12T03:48:09.708283+0000","last_fullsized":"2024-03-12T03:48:09.708283+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.582660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.582660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":665,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711921+0000","last_change":"2024-03-12T03:45:04.055127+0000","last_active":"2024-03-12T03:48:09.711921+0000","last_peered":"2024-03-12T03:48:09.711921+0000","last_clean":"2024-03-12T03:48:09.711921+0000","last_became_active":"2024-03-12T03:33:25.183402+0000","last_became_peered":"2024-03-12T03:33:25.183402+0000","last_unstale":"2024-03-12T03:48:09.711921+0000","last_undegraded":"2024-03-12T03:48:09.711921+0000","last_fullsized":"2024-03-12T03:48:09.711921+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.054936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.054936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":411,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709028+0000","last_change":"2024-03-12T03:48:02.855262+0000","last_active":"2024-03-12T03:48:09.709028+0000","last_peered":"2024-03-12T03:48:09.709028+0000","last_clean":"2024-03-12T03:48:09.709028+0000","last_became_active":"2024-03-12T03:46:25.293293+0000","last_became_peered":"2024-03-12T03:46:25.293293+0000","last_unstale":"2024-03-12T03:48:09.709028+0000","last_undegraded":"2024-03-12T03:48:09.709028+0000","last_fullsized":"2024-03-12T03:48:09.709028+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:02.855165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:02.855165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:03.069572+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":818,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.675468+0000","last_change":"2024-03-12T03:47:15.793327+0000","last_active":"2024-03-12T03:48:09.675468+0000","last_peered":"2024-03-12T03:48:09.675468+0000","last_clean":"2024-03-12T03:48:09.675468+0000","last_became_active":"2024-03-12T03:46:25.297913+0000","last_became_peered":"2024-03-12T03:46:25.297913+0000","last_unstale":"2024-03-12T03:48:09.675468+0000","last_undegraded":"2024-03-12T03:48:09.675468+0000","last_fullsized":"2024-03-12T03:48:09.675468+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:15.793199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.793199+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:27.524795+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":21,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711185+0000","last_change":"2024-03-12T03:47:58.028182+0000","last_active":"2024-03-12T03:48:09.711185+0000","last_peered":"2024-03-12T03:48:09.711185+0000","last_clean":"2024-03-12T03:48:09.711185+0000","last_became_active":"2024-03-12T03:47:58.027417+0000","last_became_peered":"2024-03-12T03:47:58.027417+0000","last_unstale":"2024-03-12T03:48:09.711185+0000","last_undegraded":"2024-03-12T03:48:09.711185+0000","last_fullsized":"2024-03-12T03:48:09.711185+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:08.116841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":804,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711656+0000","last_change":"2024-03-12T03:48:08.710477+0000","last_active":"2024-03-12T03:48:09.711656+0000","last_peered":"2024-03-12T03:48:09.711656+0000","last_clean":"2024-03-12T03:48:09.711656+0000","last_became_active":"2024-03-12T03:44:24.371149+0000","last_became_peered":"2024-03-12T03:44:24.371149+0000","last_unstale":"2024-03-12T03:48:09.711656+0000","last_undegraded":"2024-03-12T03:48:09.711656+0000","last_fullsized":"2024-03-12T03:48:09.711656+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:08.710391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:08.710391+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:22.559677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"269'31","reported_seq":719,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708743+0000","last_change":"2024-03-12T03:45:59.684396+0000","last_active":"2024-03-12T03:48:09.708743+0000","last_peered":"2024-03-12T03:48:09.708743+0000","last_clean":"2024-03-12T03:48:09.708743+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:48:09.708743+0000","last_undegraded":"2024-03-12T03:48:09.708743+0000","last_fullsized":"2024-03-12T03:48:09.708743+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:45:59.683848+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.683848+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":686,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707490+0000","last_change":"2024-03-12T03:47:57.920336+0000","last_active":"2024-03-12T03:48:09.707490+0000","last_peered":"2024-03-12T03:48:09.707490+0000","last_clean":"2024-03-12T03:48:09.707490+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:48:09.707490+0000","last_undegraded":"2024-03-12T03:48:09.707490+0000","last_fullsized":"2024-03-12T03:48:09.707490+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:57.920257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_clean_scrub_stamp":"2024-03-12T03:47:57.920257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:12.676781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":656,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.677873+0000","last_change":"2024-03-12T03:45:52.418813+0000","last_active":"2024-03-12T03:48:09.677873+0000","last_peered":"2024-03-12T03:48:09.677873+0000","last_clean":"2024-03-12T03:48:09.677873+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:48:09.677873+0000","last_undegraded":"2024-03-12T03:48:09.677873+0000","last_fullsized":"2024-03-12T03:48:09.677873+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:52.418709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:52.418709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":451,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710801+0000","last_change":"2024-03-12T03:47:14.301686+0000","last_active":"2024-03-12T03:48:09.710801+0000","last_peered":"2024-03-12T03:48:09.710801+0000","last_clean":"2024-03-12T03:48:09.710801+0000","last_became_active":"2024-03-12T03:46:25.300814+0000","last_became_peered":"2024-03-12T03:46:25.300814+0000","last_unstale":"2024-03-12T03:48:09.710801+0000","last_undegraded":"2024-03-12T03:48:09.710801+0000","last_fullsized":"2024-03-12T03:48:09.710801+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:14.301568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:14.301568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:19.385689+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":22,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707233+0000","last_change":"2024-03-12T03:47:57.698675+0000","last_active":"2024-03-12T03:48:09.707233+0000","last_peered":"2024-03-12T03:48:09.707233+0000","last_clean":"2024-03-12T03:48:09.707233+0000","last_became_active":"2024-03-12T03:47:57.698379+0000","last_became_peered":"2024-03-12T03:47:57.698379+0000","last_unstale":"2024-03-12T03:48:09.707233+0000","last_undegraded":"2024-03-12T03:48:09.707233+0000","last_fullsized":"2024-03-12T03:48:09.707233+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:09.224301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":792,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.675586+0000","last_change":"2024-03-12T03:45:02.015686+0000","last_active":"2024-03-12T03:48:09.675586+0000","last_peered":"2024-03-12T03:48:09.675586+0000","last_clean":"2024-03-12T03:48:09.675586+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:48:09.675586+0000","last_undegraded":"2024-03-12T03:48:09.675586+0000","last_fullsized":"2024-03-12T03:48:09.675586+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:45:02.015575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:45:02.015575+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":522,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710665+0000","last_change":"2024-03-12T03:47:06.292238+0000","last_active":"2024-03-12T03:48:09.710665+0000","last_peered":"2024-03-12T03:48:09.710665+0000","last_clean":"2024-03-12T03:48:09.710665+0000","last_became_active":"2024-03-12T03:46:25.287373+0000","last_became_peered":"2024-03-12T03:46:25.287373+0000","last_unstale":"2024-03-12T03:48:09.710665+0000","last_undegraded":"2024-03-12T03:48:09.710665+0000","last_fullsized":"2024-03-12T03:48:09.710665+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:06.292142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:06.292142+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:20.472155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":156,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706666+0000","last_change":"2024-03-12T03:45:13.901275+0000","last_active":"2024-03-12T03:48:09.706666+0000","last_peered":"2024-03-12T03:48:09.706666+0000","last_clean":"2024-03-12T03:48:09.706666+0000","last_became_active":"2024-03-12T03:44:57.602127+0000","last_became_peered":"2024-03-12T03:44:57.602127+0000","last_unstale":"2024-03-12T03:48:09.706666+0000","last_undegraded":"2024-03-12T03:48:09.706666+0000","last_fullsized":"2024-03-12T03:48:09.706666+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:13.901167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:13.901167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":852,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708736+0000","last_change":"2024-03-12T03:47:53.964322+0000","last_active":"2024-03-12T03:48:09.708736+0000","last_peered":"2024-03-12T03:48:09.708736+0000","last_clean":"2024-03-12T03:48:09.708736+0000","last_became_active":"2024-03-12T03:33:25.175082+0000","last_became_peered":"2024-03-12T03:33:25.175082+0000","last_unstale":"2024-03-12T03:48:09.708736+0000","last_undegraded":"2024-03-12T03:48:09.708736+0000","last_fullsized":"2024-03-12T03:48:09.708736+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:47:53.964216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:53.964216+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:15.371166+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":652,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.676266+0000","last_change":"2024-03-12T03:48:04.770864+0000","last_active":"2024-03-12T03:48:09.676266+0000","last_peered":"2024-03-12T03:48:09.676266+0000","last_clean":"2024-03-12T03:48:09.676266+0000","last_became_active":"2024-03-12T03:33:27.190087+0000","last_became_peered":"2024-03-12T03:33:27.190087+0000","last_unstale":"2024-03-12T03:48:09.676266+0000","last_undegraded":"2024-03-12T03:48:09.676266+0000","last_fullsized":"2024-03-12T03:48:09.676266+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:04.770746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:04.770746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:28.647700+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":22,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708391+0000","last_change":"2024-03-12T03:47:58.028347+0000","last_active":"2024-03-12T03:48:09.708391+0000","last_peered":"2024-03-12T03:48:09.708391+0000","last_clean":"2024-03-12T03:48:09.708391+0000","last_became_active":"2024-03-12T03:47:58.027802+0000","last_became_peered":"2024-03-12T03:47:58.027802+0000","last_unstale":"2024-03-12T03:48:09.708391+0000","last_undegraded":"2024-03-12T03:48:09.708391+0000","last_fullsized":"2024-03-12T03:48:09.708391+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:03.770705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":669,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707427+0000","last_change":"2024-03-12T03:46:52.084213+0000","last_active":"2024-03-12T03:48:09.707427+0000","last_peered":"2024-03-12T03:48:09.707427+0000","last_clean":"2024-03-12T03:48:09.707427+0000","last_became_active":"2024-03-12T03:46:25.286985+0000","last_became_peered":"2024-03-12T03:46:25.286985+0000","last_unstale":"2024-03-12T03:48:09.707427+0000","last_undegraded":"2024-03-12T03:48:09.707427+0000","last_fullsized":"2024-03-12T03:48:09.707427+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:52.084134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:46:52.084134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"269'42","reported_seq":529,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707118+0000","last_change":"2024-03-12T03:47:13.347626+0000","last_active":"2024-03-12T03:48:09.707118+0000","last_peered":"2024-03-12T03:48:09.707118+0000","last_clean":"2024-03-12T03:48:09.707118+0000","last_became_active":"2024-03-12T03:46:25.290762+0000","last_became_peered":"2024-03-12T03:46:25.290762+0000","last_unstale":"2024-03-12T03:48:09.707118+0000","last_undegraded":"2024-03-12T03:48:09.707118+0000","last_fullsized":"2024-03-12T03:48:09.707118+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:47:13.347128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.347128+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:13.358230+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":445,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709610+0000","last_change":"2024-03-12T03:48:05.640029+0000","last_active":"2024-03-12T03:48:09.709610+0000","last_peered":"2024-03-12T03:48:09.709610+0000","last_clean":"2024-03-12T03:48:09.709610+0000","last_became_active":"2024-03-12T03:46:25.289332+0000","last_became_peered":"2024-03-12T03:46:25.289332+0000","last_unstale":"2024-03-12T03:48:09.709610+0000","last_undegraded":"2024-03-12T03:48:09.709610+0000","last_fullsized":"2024-03-12T03:48:09.709610+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:05.639950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:05.639950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:23.042803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":685,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707745+0000","last_change":"2024-03-12T03:46:56.092143+0000","last_active":"2024-03-12T03:48:09.707745+0000","last_peered":"2024-03-12T03:48:09.707745+0000","last_clean":"2024-03-12T03:48:09.707745+0000","last_became_active":"2024-03-12T03:46:25.280860+0000","last_became_peered":"2024-03-12T03:46:25.280860+0000","last_unstale":"2024-03-12T03:48:09.707745+0000","last_undegraded":"2024-03-12T03:48:09.707745+0000","last_fullsized":"2024-03-12T03:48:09.707745+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_clean_scrub_stamp":"2024-03-12T03:46:56.092011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":807,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707935+0000","last_change":"2024-03-12T03:44:55.770605+0000","last_active":"2024-03-12T03:48:09.707935+0000","last_peered":"2024-03-12T03:48:09.707935+0000","last_clean":"2024-03-12T03:48:09.707935+0000","last_became_active":"2024-03-12T03:44:19.373715+0000","last_became_peered":"2024-03-12T03:44:19.373715+0000","last_unstale":"2024-03-12T03:48:09.707935+0000","last_undegraded":"2024-03-12T03:48:09.707935+0000","last_fullsized":"2024-03-12T03:48:09.707935+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:55.770487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:55.770487+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"372'74","reported_seq":96,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:13.416962+0000","last_change":"2024-03-12T03:47:58.027791+0000","last_active":"2024-03-12T03:48:13.416962+0000","last_peered":"2024-03-12T03:48:13.416962+0000","last_clean":"2024-03-12T03:48:13.416962+0000","last_became_active":"2024-03-12T03:47:58.027347+0000","last_became_peered":"2024-03-12T03:47:58.027347+0000","last_unstale":"2024-03-12T03:48:13.416962+0000","last_undegraded":"2024-03-12T03:48:13.416962+0000","last_fullsized":"2024-03-12T03:48:13.416962+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:08.158471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4849664,"num_objects":74,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74,"num_write_kb":4736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"332'12","reported_seq":616,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707069+0000","last_change":"2024-03-12T03:46:25.292527+0000","last_active":"2024-03-12T03:48:09.707069+0000","last_peered":"2024-03-12T03:48:09.707069+0000","last_clean":"2024-03-12T03:48:09.707069+0000","last_became_active":"2024-03-12T03:46:25.288674+0000","last_became_peered":"2024-03-12T03:46:25.288674+0000","last_unstale":"2024-03-12T03:48:09.707069+0000","last_undegraded":"2024-03-12T03:48:09.707069+0000","last_fullsized":"2024-03-12T03:48:09.707069+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:39:37.039925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:39:37.039925+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"269'42","reported_seq":733,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707598+0000","last_change":"2024-03-12T03:47:56.911844+0000","last_active":"2024-03-12T03:48:09.707598+0000","last_peered":"2024-03-12T03:48:09.707598+0000","last_clean":"2024-03-12T03:48:09.707598+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:48:09.707598+0000","last_undegraded":"2024-03-12T03:48:09.707598+0000","last_fullsized":"2024-03-12T03:48:09.707598+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:47:56.911701+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.911701+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:04.303152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"320'2","reported_seq":667,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709185+0000","last_change":"2024-03-12T03:45:56.089397+0000","last_active":"2024-03-12T03:48:09.709185+0000","last_peered":"2024-03-12T03:48:09.709185+0000","last_clean":"2024-03-12T03:48:09.709185+0000","last_became_active":"2024-03-12T03:33:21.098056+0000","last_became_peered":"2024-03-12T03:33:21.098056+0000","last_unstale":"2024-03-12T03:48:09.709185+0000","last_undegraded":"2024-03-12T03:48:09.709185+0000","last_fullsized":"2024-03-12T03:48:09.709185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'2","last_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_deep_scrub":"320'2","last_deep_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_clean_scrub_stamp":"2024-03-12T03:45:56.089247+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":675,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706847+0000","last_change":"2024-03-12T03:47:17.815361+0000","last_active":"2024-03-12T03:48:09.706847+0000","last_peered":"2024-03-12T03:48:09.706847+0000","last_clean":"2024-03-12T03:48:09.706847+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:48:09.706847+0000","last_undegraded":"2024-03-12T03:48:09.706847+0000","last_fullsized":"2024-03-12T03:48:09.706847+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:17.815263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:17.815263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:38.911937+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":617,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710410+0000","last_change":"2024-03-12T03:47:11.313289+0000","last_active":"2024-03-12T03:48:09.710410+0000","last_peered":"2024-03-12T03:48:09.710410+0000","last_clean":"2024-03-12T03:48:09.710410+0000","last_became_active":"2024-03-12T03:46:25.299964+0000","last_became_peered":"2024-03-12T03:46:25.299964+0000","last_unstale":"2024-03-12T03:48:09.710410+0000","last_undegraded":"2024-03-12T03:48:09.710410+0000","last_fullsized":"2024-03-12T03:48:09.710410+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.313179+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:34.330866+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":22,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707425+0000","last_change":"2024-03-12T03:47:57.698352+0000","last_active":"2024-03-12T03:48:09.707425+0000","last_peered":"2024-03-12T03:48:09.707425+0000","last_clean":"2024-03-12T03:48:09.707425+0000","last_became_active":"2024-03-12T03:47:57.697951+0000","last_became_peered":"2024-03-12T03:47:57.697951+0000","last_unstale":"2024-03-12T03:48:09.707425+0000","last_undegraded":"2024-03-12T03:48:09.707425+0000","last_fullsized":"2024-03-12T03:48:09.707425+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:22.744943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":845,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708225+0000","last_change":"2024-03-12T03:45:09.824960+0000","last_active":"2024-03-12T03:48:09.708225+0000","last_peered":"2024-03-12T03:48:09.708225+0000","last_clean":"2024-03-12T03:48:09.708225+0000","last_became_active":"2024-03-12T03:33:25.181480+0000","last_became_peered":"2024-03-12T03:33:25.181480+0000","last_unstale":"2024-03-12T03:48:09.708225+0000","last_undegraded":"2024-03-12T03:48:09.708225+0000","last_fullsized":"2024-03-12T03:48:09.708225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:09.824819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:09.824819+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"269'26","reported_seq":712,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711025+0000","last_change":"2024-03-12T03:44:58.093469+0000","last_active":"2024-03-12T03:48:09.711025+0000","last_peered":"2024-03-12T03:48:09.711025+0000","last_clean":"2024-03-12T03:48:09.711025+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:48:09.711025+0000","last_undegraded":"2024-03-12T03:48:09.711025+0000","last_fullsized":"2024-03-12T03:48:09.711025+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:44:58.093327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:44:58.093327+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":687,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.676364+0000","last_change":"2024-03-12T03:47:08.836525+0000","last_active":"2024-03-12T03:48:09.676364+0000","last_peered":"2024-03-12T03:48:09.676364+0000","last_clean":"2024-03-12T03:48:09.676364+0000","last_became_active":"2024-03-12T03:46:25.294348+0000","last_became_peered":"2024-03-12T03:46:25.294348+0000","last_unstale":"2024-03-12T03:48:09.676364+0000","last_undegraded":"2024-03-12T03:48:09.676364+0000","last_fullsized":"2024-03-12T03:48:09.676364+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:08.836419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:08.836419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:24.541463+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":676,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706657+0000","last_change":"2024-03-12T03:47:13.892905+0000","last_active":"2024-03-12T03:48:09.706657+0000","last_peered":"2024-03-12T03:48:09.706657+0000","last_clean":"2024-03-12T03:48:09.706657+0000","last_became_active":"2024-03-12T03:46:25.293236+0000","last_became_peered":"2024-03-12T03:46:25.293236+0000","last_unstale":"2024-03-12T03:48:09.706657+0000","last_undegraded":"2024-03-12T03:48:09.706657+0000","last_fullsized":"2024-03-12T03:48:09.706657+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:13.892760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.892760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:41.766338+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":404,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710159+0000","last_change":"2024-03-12T03:47:10.307885+0000","last_active":"2024-03-12T03:48:09.710159+0000","last_peered":"2024-03-12T03:48:09.710159+0000","last_clean":"2024-03-12T03:48:09.710159+0000","last_became_active":"2024-03-12T03:46:25.287360+0000","last_became_peered":"2024-03-12T03:46:25.287360+0000","last_unstale":"2024-03-12T03:48:09.710159+0000","last_undegraded":"2024-03-12T03:48:09.710159+0000","last_fullsized":"2024-03-12T03:48:09.710159+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:10.307761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:10.307761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:25.795762+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":22,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708257+0000","last_change":"2024-03-12T03:47:58.028314+0000","last_active":"2024-03-12T03:48:09.708257+0000","last_peered":"2024-03-12T03:48:09.708257+0000","last_clean":"2024-03-12T03:48:09.708257+0000","last_became_active":"2024-03-12T03:47:58.027648+0000","last_became_peered":"2024-03-12T03:47:58.027648+0000","last_unstale":"2024-03-12T03:48:09.708257+0000","last_undegraded":"2024-03-12T03:48:09.708257+0000","last_fullsized":"2024-03-12T03:48:09.708257+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:57.071411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":128,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707440+0000","last_change":"2024-03-12T03:45:18.106629+0000","last_active":"2024-03-12T03:48:09.707440+0000","last_peered":"2024-03-12T03:48:09.707440+0000","last_clean":"2024-03-12T03:48:09.707440+0000","last_became_active":"2024-03-12T03:45:18.106305+0000","last_became_peered":"2024-03-12T03:45:18.106305+0000","last_unstale":"2024-03-12T03:48:09.707440+0000","last_undegraded":"2024-03-12T03:48:09.707440+0000","last_fullsized":"2024-03-12T03:48:09.707440+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:05.578661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:45:05.578661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"269'35","reported_seq":725,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708643+0000","last_change":"2024-03-12T03:45:59.055468+0000","last_active":"2024-03-12T03:48:09.708643+0000","last_peered":"2024-03-12T03:48:09.708643+0000","last_clean":"2024-03-12T03:48:09.708643+0000","last_became_active":"2024-03-12T03:33:23.551568+0000","last_became_peered":"2024-03-12T03:33:23.551568+0000","last_unstale":"2024-03-12T03:48:09.708643+0000","last_undegraded":"2024-03-12T03:48:09.708643+0000","last_fullsized":"2024-03-12T03:48:09.708643+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-03-12T03:45:59.055318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.055318+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":694,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711822+0000","last_change":"2024-03-12T03:48:00.710463+0000","last_active":"2024-03-12T03:48:09.711822+0000","last_peered":"2024-03-12T03:48:09.711822+0000","last_clean":"2024-03-12T03:48:09.711822+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:48:09.711822+0000","last_undegraded":"2024-03-12T03:48:09.711822+0000","last_fullsized":"2024-03-12T03:48:09.711822+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:00.710387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.710387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:17.748178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711476+0000","last_change":"2024-03-12T03:44:54.134468+0000","last_active":"2024-03-12T03:48:09.711476+0000","last_peered":"2024-03-12T03:48:09.711476+0000","last_clean":"2024-03-12T03:48:09.711476+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:48:09.711476+0000","last_undegraded":"2024-03-12T03:48:09.711476+0000","last_fullsized":"2024-03-12T03:48:09.711476+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:54.134348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:44:54.134348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":435,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708475+0000","last_change":"2024-03-12T03:47:04.858096+0000","last_active":"2024-03-12T03:48:09.708475+0000","last_peered":"2024-03-12T03:48:09.708475+0000","last_clean":"2024-03-12T03:48:09.708475+0000","last_became_active":"2024-03-12T03:46:25.282167+0000","last_became_peered":"2024-03-12T03:46:25.282167+0000","last_unstale":"2024-03-12T03:48:09.708475+0000","last_undegraded":"2024-03-12T03:48:09.708475+0000","last_fullsized":"2024-03-12T03:48:09.708475+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:04.857986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:04.857986+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:14.853572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":21,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707123+0000","last_change":"2024-03-12T03:47:57.699339+0000","last_active":"2024-03-12T03:48:09.707123+0000","last_peered":"2024-03-12T03:48:09.707123+0000","last_clean":"2024-03-12T03:48:09.707123+0000","last_became_active":"2024-03-12T03:47:57.698375+0000","last_became_peered":"2024-03-12T03:47:57.698375+0000","last_unstale":"2024-03-12T03:48:09.707123+0000","last_undegraded":"2024-03-12T03:48:09.707123+0000","last_fullsized":"2024-03-12T03:48:09.707123+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:58.420048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":656,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.677908+0000","last_change":"2024-03-12T03:46:00.577914+0000","last_active":"2024-03-12T03:48:09.677908+0000","last_peered":"2024-03-12T03:48:09.677908+0000","last_clean":"2024-03-12T03:48:09.677908+0000","last_became_active":"2024-03-12T03:33:25.177030+0000","last_became_peered":"2024-03-12T03:33:25.177030+0000","last_unstale":"2024-03-12T03:48:09.677908+0000","last_undegraded":"2024-03-12T03:48:09.677908+0000","last_fullsized":"2024-03-12T03:48:09.677908+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:00.577693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:46:00.577693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"269'24","reported_seq":737,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708693+0000","last_change":"2024-03-12T03:45:02.759428+0000","last_active":"2024-03-12T03:48:09.708693+0000","last_peered":"2024-03-12T03:48:09.708693+0000","last_clean":"2024-03-12T03:48:09.708693+0000","last_became_active":"2024-03-12T03:39:33.277868+0000","last_became_peered":"2024-03-12T03:39:33.277868+0000","last_unstale":"2024-03-12T03:48:09.708693+0000","last_undegraded":"2024-03-12T03:48:09.708693+0000","last_fullsized":"2024-03-12T03:48:09.708693+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:45:02.759243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:02.759243+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":1571,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706784+0000","last_change":"2024-03-12T03:47:13.591577+0000","last_active":"2024-03-12T03:48:09.706784+0000","last_peered":"2024-03-12T03:48:09.706784+0000","last_clean":"2024-03-12T03:48:09.706784+0000","last_became_active":"2024-03-12T03:33:20.099516+0000","last_became_peered":"2024-03-12T03:33:20.099516+0000","last_unstale":"2024-03-12T03:48:09.706784+0000","last_undegraded":"2024-03-12T03:48:09.706784+0000","last_fullsized":"2024-03-12T03:48:09.706784+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:47:13.591375+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.591375+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:25.083961+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":883,"num_read_kb":878,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":676,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707195+0000","last_change":"2024-03-12T03:47:58.051067+0000","last_active":"2024-03-12T03:48:09.707195+0000","last_peered":"2024-03-12T03:48:09.707195+0000","last_clean":"2024-03-12T03:48:09.707195+0000","last_became_active":"2024-03-12T03:46:25.282902+0000","last_became_peered":"2024-03-12T03:46:25.282902+0000","last_unstale":"2024-03-12T03:48:09.707195+0000","last_undegraded":"2024-03-12T03:48:09.707195+0000","last_fullsized":"2024-03-12T03:48:09.707195+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:58.050985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.050985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:09.583349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":680,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711587+0000","last_change":"2024-03-12T03:47:17.391210+0000","last_active":"2024-03-12T03:48:09.711587+0000","last_peered":"2024-03-12T03:48:09.711587+0000","last_clean":"2024-03-12T03:48:09.711587+0000","last_became_active":"2024-03-12T03:46:25.302256+0000","last_became_peered":"2024-03-12T03:46:25.302256+0000","last_unstale":"2024-03-12T03:48:09.711587+0000","last_undegraded":"2024-03-12T03:48:09.711587+0000","last_fullsized":"2024-03-12T03:48:09.711587+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:17.391069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:17.391069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:28.366690+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":797,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.731930+0000","last_change":"2024-03-12T03:48:09.731952+0000","last_active":"2024-03-12T03:48:09.731930+0000","last_peered":"2024-03-12T03:48:09.731930+0000","last_clean":"2024-03-12T03:48:09.731930+0000","last_became_active":"2024-03-12T03:44:19.366060+0000","last_became_peered":"2024-03-12T03:44:19.366060+0000","last_unstale":"2024-03-12T03:48:09.731930+0000","last_undegraded":"2024-03-12T03:48:09.731930+0000","last_fullsized":"2024-03-12T03:48:09.731930+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:09.731873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:09.731873+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:12.382730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":21,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711546+0000","last_change":"2024-03-12T03:47:58.028202+0000","last_active":"2024-03-12T03:48:09.711546+0000","last_peered":"2024-03-12T03:48:09.711546+0000","last_clean":"2024-03-12T03:48:09.711546+0000","last_became_active":"2024-03-12T03:47:58.027920+0000","last_became_peered":"2024-03-12T03:47:58.027920+0000","last_unstale":"2024-03-12T03:48:09.711546+0000","last_undegraded":"2024-03-12T03:48:09.711546+0000","last_fullsized":"2024-03-12T03:48:09.711546+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:12.120208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":671,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707562+0000","last_change":"2024-03-12T03:45:16.222836+0000","last_active":"2024-03-12T03:48:09.707562+0000","last_peered":"2024-03-12T03:48:09.707562+0000","last_clean":"2024-03-12T03:48:09.707562+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:48:09.707562+0000","last_undegraded":"2024-03-12T03:48:09.707562+0000","last_fullsized":"2024-03-12T03:48:09.707562+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:16.222710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:16.222710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"315'29","reported_seq":785,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708403+0000","last_change":"2024-03-12T03:48:08.875139+0000","last_active":"2024-03-12T03:48:09.708403+0000","last_peered":"2024-03-12T03:48:09.708403+0000","last_clean":"2024-03-12T03:48:09.708403+0000","last_became_active":"2024-03-12T03:45:18.109048+0000","last_became_peered":"2024-03-12T03:45:18.109048+0000","last_unstale":"2024-03-12T03:48:09.708403+0000","last_undegraded":"2024-03-12T03:48:09.708403+0000","last_fullsized":"2024-03-12T03:48:09.708403+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'29","last_scrub_stamp":"2024-03-12T03:48:08.874995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:08.874995+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:10.618617+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":436,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707091+0000","last_change":"2024-03-12T03:46:25.294791+0000","last_active":"2024-03-12T03:48:09.707091+0000","last_peered":"2024-03-12T03:48:09.707091+0000","last_clean":"2024-03-12T03:48:09.707091+0000","last_became_active":"2024-03-12T03:46:25.292535+0000","last_became_peered":"2024-03-12T03:46:25.292535+0000","last_unstale":"2024-03-12T03:48:09.707091+0000","last_undegraded":"2024-03-12T03:48:09.707091+0000","last_fullsized":"2024-03-12T03:48:09.707091+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:07.572642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:43:07.572642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":940,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706719+0000","last_change":"2024-03-12T03:44:54.092085+0000","last_active":"2024-03-12T03:48:09.706719+0000","last_peered":"2024-03-12T03:48:09.706719+0000","last_clean":"2024-03-12T03:48:09.706719+0000","last_became_active":"2024-03-12T03:44:19.379262+0000","last_became_peered":"2024-03-12T03:44:19.379262+0000","last_unstale":"2024-03-12T03:48:09.706719+0000","last_undegraded":"2024-03-12T03:48:09.706719+0000","last_fullsized":"2024-03-12T03:48:09.706719+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:54.091965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:54.091965+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":662,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.677435+0000","last_change":"2024-03-12T03:47:51.670920+0000","last_active":"2024-03-12T03:48:09.677435+0000","last_peered":"2024-03-12T03:48:09.677435+0000","last_clean":"2024-03-12T03:48:09.677435+0000","last_became_active":"2024-03-12T03:46:25.295668+0000","last_became_peered":"2024-03-12T03:46:25.295668+0000","last_unstale":"2024-03-12T03:48:09.677435+0000","last_undegraded":"2024-03-12T03:48:09.677435+0000","last_fullsized":"2024-03-12T03:48:09.677435+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:51.670806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:51.670806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:59.884920+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":21,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.675357+0000","last_change":"2024-03-12T03:47:57.694092+0000","last_active":"2024-03-12T03:48:09.675357+0000","last_peered":"2024-03-12T03:48:09.675357+0000","last_clean":"2024-03-12T03:48:09.675357+0000","last_became_active":"2024-03-12T03:47:57.693619+0000","last_became_peered":"2024-03-12T03:47:57.693619+0000","last_unstale":"2024-03-12T03:48:09.675357+0000","last_undegraded":"2024-03-12T03:48:09.675357+0000","last_fullsized":"2024-03-12T03:48:09.675357+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:11.317958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":401,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709310+0000","last_change":"2024-03-12T03:47:03.666019+0000","last_active":"2024-03-12T03:48:09.709310+0000","last_peered":"2024-03-12T03:48:09.709310+0000","last_clean":"2024-03-12T03:48:09.709310+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:48:09.709310+0000","last_undegraded":"2024-03-12T03:48:09.709310+0000","last_fullsized":"2024-03-12T03:48:09.709310+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:03.665934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:47:03.665934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:24.141177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"269'23","reported_seq":714,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710438+0000","last_change":"2024-03-12T03:45:59.998488+0000","last_active":"2024-03-12T03:48:09.710438+0000","last_peered":"2024-03-12T03:48:09.710438+0000","last_clean":"2024-03-12T03:48:09.710438+0000","last_became_active":"2024-03-12T03:33:23.163225+0000","last_became_peered":"2024-03-12T03:33:23.163225+0000","last_unstale":"2024-03-12T03:48:09.710438+0000","last_undegraded":"2024-03-12T03:48:09.710438+0000","last_fullsized":"2024-03-12T03:48:09.710438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'23","last_scrub_stamp":"2024-03-12T03:45:59.998395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.998395+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":404,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709488+0000","last_change":"2024-03-12T03:48:06.903632+0000","last_active":"2024-03-12T03:48:09.709488+0000","last_peered":"2024-03-12T03:48:09.709488+0000","last_clean":"2024-03-12T03:48:09.709488+0000","last_became_active":"2024-03-12T03:46:25.284867+0000","last_became_peered":"2024-03-12T03:46:25.284867+0000","last_unstale":"2024-03-12T03:48:09.709488+0000","last_undegraded":"2024-03-12T03:48:09.709488+0000","last_fullsized":"2024-03-12T03:48:09.709488+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:06.903506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.903506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:28.801471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":686,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711315+0000","last_change":"2024-03-12T03:45:00.012141+0000","last_active":"2024-03-12T03:48:09.711315+0000","last_peered":"2024-03-12T03:48:09.711315+0000","last_clean":"2024-03-12T03:48:09.711315+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:48:09.711315+0000","last_undegraded":"2024-03-12T03:48:09.711315+0000","last_fullsized":"2024-03-12T03:48:09.711315+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:00.012064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:45:00.012064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":675,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708659+0000","last_change":"2024-03-12T03:46:51.595890+0000","last_active":"2024-03-12T03:48:09.708659+0000","last_peered":"2024-03-12T03:48:09.708659+0000","last_clean":"2024-03-12T03:48:09.708659+0000","last_became_active":"2024-03-12T03:46:25.284168+0000","last_became_peered":"2024-03-12T03:46:25.284168+0000","last_unstale":"2024-03-12T03:48:09.708659+0000","last_undegraded":"2024-03-12T03:48:09.708659+0000","last_fullsized":"2024-03-12T03:48:09.708659+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:46:51.595794+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:34:40.663002+0000","last_clean_scrub_stamp":"2024-03-12T03:46:51.595794+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:15.953713+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":963,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707139+0000","last_change":"2024-03-12T03:48:01.456517+0000","last_active":"2024-03-12T03:48:09.707139+0000","last_peered":"2024-03-12T03:48:09.707139+0000","last_clean":"2024-03-12T03:48:09.707139+0000","last_became_active":"2024-03-12T03:44:19.378554+0000","last_became_peered":"2024-03-12T03:44:19.378554+0000","last_unstale":"2024-03-12T03:48:09.707139+0000","last_undegraded":"2024-03-12T03:48:09.707139+0000","last_fullsized":"2024-03-12T03:48:09.707139+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:01.456424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:01.456424+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:11.092454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":21,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707172+0000","last_change":"2024-03-12T03:47:58.028871+0000","last_active":"2024-03-12T03:48:09.707172+0000","last_peered":"2024-03-12T03:48:09.707172+0000","last_clean":"2024-03-12T03:48:09.707172+0000","last_became_active":"2024-03-12T03:47:58.028510+0000","last_became_peered":"2024-03-12T03:47:58.028510+0000","last_unstale":"2024-03-12T03:48:09.707172+0000","last_undegraded":"2024-03-12T03:48:09.707172+0000","last_fullsized":"2024-03-12T03:48:09.707172+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:57.150714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":658,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708917+0000","last_change":"2024-03-12T03:45:06.865102+0000","last_active":"2024-03-12T03:48:09.708917+0000","last_peered":"2024-03-12T03:48:09.708917+0000","last_clean":"2024-03-12T03:48:09.708917+0000","last_became_active":"2024-03-12T03:33:25.182882+0000","last_became_peered":"2024-03-12T03:33:25.182882+0000","last_unstale":"2024-03-12T03:48:09.708917+0000","last_undegraded":"2024-03-12T03:48:09.708917+0000","last_fullsized":"2024-03-12T03:48:09.708917+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:06.864708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.864708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"332'53","reported_seq":942,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.675820+0000","last_change":"2024-03-12T03:47:55.621887+0000","last_active":"2024-03-12T03:48:09.675820+0000","last_peered":"2024-03-12T03:48:09.675820+0000","last_clean":"2024-03-12T03:48:09.675820+0000","last_became_active":"2024-03-12T03:46:25.301872+0000","last_became_peered":"2024-03-12T03:46:25.301872+0000","last_unstale":"2024-03-12T03:48:09.675820+0000","last_undegraded":"2024-03-12T03:48:09.675820+0000","last_fullsized":"2024-03-12T03:48:09.675820+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'53","last_scrub_stamp":"2024-03-12T03:47:55.621782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:55.621782+0000","objects_scrubbed":11,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:19.618316+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":675,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707750+0000","last_change":"2024-03-12T03:45:53.648932+0000","last_active":"2024-03-12T03:48:09.707750+0000","last_peered":"2024-03-12T03:48:09.707750+0000","last_clean":"2024-03-12T03:48:09.707750+0000","last_became_active":"2024-03-12T03:33:21.106762+0000","last_became_peered":"2024-03-12T03:33:21.106762+0000","last_unstale":"2024-03-12T03:48:09.707750+0000","last_undegraded":"2024-03-12T03:48:09.707750+0000","last_fullsized":"2024-03-12T03:48:09.707750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.648824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.648824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":676,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.676133+0000","last_change":"2024-03-12T03:46:04.645060+0000","last_active":"2024-03-12T03:48:09.676133+0000","last_peered":"2024-03-12T03:48:09.676133+0000","last_clean":"2024-03-12T03:48:09.676133+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:48:09.676133+0000","last_undegraded":"2024-03-12T03:48:09.676133+0000","last_fullsized":"2024-03-12T03:48:09.676133+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:46:04.644902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:46:04.644902+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":476,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708078+0000","last_change":"2024-03-12T03:46:25.287357+0000","last_active":"2024-03-12T03:48:09.708078+0000","last_peered":"2024-03-12T03:48:09.708078+0000","last_clean":"2024-03-12T03:48:09.708078+0000","last_became_active":"2024-03-12T03:46:25.284416+0000","last_became_peered":"2024-03-12T03:46:25.284416+0000","last_unstale":"2024-03-12T03:48:09.708078+0000","last_undegraded":"2024-03-12T03:48:09.708078+0000","last_fullsized":"2024-03-12T03:48:09.708078+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:39.031166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:39:39.031166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":805,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.675552+0000","last_change":"2024-03-12T03:48:06.690954+0000","last_active":"2024-03-12T03:48:09.675552+0000","last_peered":"2024-03-12T03:48:09.675552+0000","last_clean":"2024-03-12T03:48:09.675552+0000","last_became_active":"2024-03-12T03:44:19.371194+0000","last_became_peered":"2024-03-12T03:44:19.371194+0000","last_unstale":"2024-03-12T03:48:09.675552+0000","last_undegraded":"2024-03-12T03:48:09.675552+0000","last_fullsized":"2024-03-12T03:48:09.675552+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:06.690873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.690873+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:19.323744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":22,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708547+0000","last_change":"2024-03-12T03:47:57.699330+0000","last_active":"2024-03-12T03:48:09.708547+0000","last_peered":"2024-03-12T03:48:09.708547+0000","last_clean":"2024-03-12T03:48:09.708547+0000","last_became_active":"2024-03-12T03:47:57.699023+0000","last_became_peered":"2024-03-12T03:47:57.699023+0000","last_unstale":"2024-03-12T03:48:09.708547+0000","last_undegraded":"2024-03-12T03:48:09.708547+0000","last_fullsized":"2024-03-12T03:48:09.708547+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:07.756146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":474,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708463+0000","last_change":"2024-03-12T03:46:26.281981+0000","last_active":"2024-03-12T03:48:09.708463+0000","last_peered":"2024-03-12T03:48:09.708463+0000","last_clean":"2024-03-12T03:48:09.708463+0000","last_became_active":"2024-03-12T03:46:26.281472+0000","last_became_peered":"2024-03-12T03:46:26.281472+0000","last_unstale":"2024-03-12T03:48:09.708463+0000","last_undegraded":"2024-03-12T03:48:09.708463+0000","last_fullsized":"2024-03-12T03:48:09.708463+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:52.828725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:52.828725+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"269'26","reported_seq":703,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708328+0000","last_change":"2024-03-12T03:45:15.774203+0000","last_active":"2024-03-12T03:48:09.708328+0000","last_peered":"2024-03-12T03:48:09.708328+0000","last_clean":"2024-03-12T03:48:09.708328+0000","last_became_active":"2024-03-12T03:33:23.164141+0000","last_became_peered":"2024-03-12T03:33:23.164141+0000","last_unstale":"2024-03-12T03:48:09.708328+0000","last_undegraded":"2024-03-12T03:48:09.708328+0000","last_fullsized":"2024-03-12T03:48:09.708328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:15.774076+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.774076+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.676033+0000","last_change":"2024-03-12T03:47:10.855898+0000","last_active":"2024-03-12T03:48:09.676033+0000","last_peered":"2024-03-12T03:48:09.676033+0000","last_clean":"2024-03-12T03:48:09.676033+0000","last_became_active":"2024-03-12T03:46:25.303784+0000","last_became_peered":"2024-03-12T03:46:25.303784+0000","last_unstale":"2024-03-12T03:48:09.676033+0000","last_undegraded":"2024-03-12T03:48:09.676033+0000","last_fullsized":"2024-03-12T03:48:09.676033+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:10.855814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:47:10.855814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:40.854862+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":403,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710082+0000","last_change":"2024-03-12T03:48:01.777306+0000","last_active":"2024-03-12T03:48:09.710082+0000","last_peered":"2024-03-12T03:48:09.710082+0000","last_clean":"2024-03-12T03:48:09.710082+0000","last_became_active":"2024-03-12T03:46:25.284556+0000","last_became_peered":"2024-03-12T03:46:25.284556+0000","last_unstale":"2024-03-12T03:48:09.710082+0000","last_undegraded":"2024-03-12T03:48:09.710082+0000","last_fullsized":"2024-03-12T03:48:09.710082+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:01.777199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:01.777199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:17.827622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":665,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707435+0000","last_change":"2024-03-12T03:45:55.193112+0000","last_active":"2024-03-12T03:48:09.707435+0000","last_peered":"2024-03-12T03:48:09.707435+0000","last_clean":"2024-03-12T03:48:09.707435+0000","last_became_active":"2024-03-12T03:39:33.271016+0000","last_became_peered":"2024-03-12T03:39:33.271016+0000","last_unstale":"2024-03-12T03:48:09.707435+0000","last_undegraded":"2024-03-12T03:48:09.707435+0000","last_fullsized":"2024-03-12T03:48:09.707435+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:55.192996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:55.192996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":21,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707724+0000","last_change":"2024-03-12T03:47:57.699190+0000","last_active":"2024-03-12T03:48:09.707724+0000","last_peered":"2024-03-12T03:48:09.707724+0000","last_clean":"2024-03-12T03:48:09.707724+0000","last_became_active":"2024-03-12T03:47:57.698868+0000","last_became_peered":"2024-03-12T03:47:57.698868+0000","last_unstale":"2024-03-12T03:48:09.707724+0000","last_undegraded":"2024-03-12T03:48:09.707724+0000","last_fullsized":"2024-03-12T03:48:09.707724+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:58.255402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":770,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.677277+0000","last_change":"2024-03-12T03:45:50.401930+0000","last_active":"2024-03-12T03:48:09.677277+0000","last_peered":"2024-03-12T03:48:09.677277+0000","last_clean":"2024-03-12T03:48:09.677277+0000","last_became_active":"2024-03-12T03:45:18.108107+0000","last_became_peered":"2024-03-12T03:45:18.108107+0000","last_unstale":"2024-03-12T03:48:09.677277+0000","last_undegraded":"2024-03-12T03:48:09.677277+0000","last_fullsized":"2024-03-12T03:48:09.677277+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:45:50.401804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:45:50.401804+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'51","reported_seq":657,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710534+0000","last_change":"2024-03-12T03:45:18.106865+0000","last_active":"2024-03-12T03:48:09.710534+0000","last_peered":"2024-03-12T03:48:09.710534+0000","last_clean":"2024-03-12T03:48:09.710534+0000","last_became_active":"2024-03-12T03:45:18.106503+0000","last_became_peered":"2024-03-12T03:45:18.106503+0000","last_unstale":"2024-03-12T03:48:09.710534+0000","last_undegraded":"2024-03-12T03:48:09.710534+0000","last_fullsized":"2024-03-12T03:48:09.710534+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'51","last_scrub_stamp":"2024-03-12T03:45:06.539548+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.539548+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708505+0000","last_change":"2024-03-12T03:48:03.416343+0000","last_active":"2024-03-12T03:48:09.708505+0000","last_peered":"2024-03-12T03:48:09.708505+0000","last_clean":"2024-03-12T03:48:09.708505+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:48:09.708505+0000","last_undegraded":"2024-03-12T03:48:09.708505+0000","last_fullsized":"2024-03-12T03:48:09.708505+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:03.416262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:03.416262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:17.711000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":393,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708363+0000","last_change":"2024-03-12T03:47:16.954859+0000","last_active":"2024-03-12T03:48:09.708363+0000","last_peered":"2024-03-12T03:48:09.708363+0000","last_clean":"2024-03-12T03:48:09.708363+0000","last_became_active":"2024-03-12T03:39:33.254395+0000","last_became_peered":"2024-03-12T03:39:33.254395+0000","last_unstale":"2024-03-12T03:48:09.708363+0000","last_undegraded":"2024-03-12T03:48:09.708363+0000","last_fullsized":"2024-03-12T03:48:09.708363+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.954731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.954731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:23.659186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":684,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706588+0000","last_change":"2024-03-12T03:46:57.647557+0000","last_active":"2024-03-12T03:48:09.706588+0000","last_peered":"2024-03-12T03:48:09.706588+0000","last_clean":"2024-03-12T03:48:09.706588+0000","last_became_active":"2024-03-12T03:46:25.292122+0000","last_became_peered":"2024-03-12T03:46:25.292122+0000","last_unstale":"2024-03-12T03:48:09.706588+0000","last_undegraded":"2024-03-12T03:48:09.706588+0000","last_fullsized":"2024-03-12T03:48:09.706588+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:57.647447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:46:57.647447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"372'78","reported_seq":100,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:13.414371+0000","last_change":"2024-03-12T03:47:58.027966+0000","last_active":"2024-03-12T03:48:13.414371+0000","last_peered":"2024-03-12T03:48:13.414371+0000","last_clean":"2024-03-12T03:48:13.414371+0000","last_became_active":"2024-03-12T03:47:58.027372+0000","last_became_peered":"2024-03-12T03:47:58.027372+0000","last_unstale":"2024-03-12T03:48:13.414371+0000","last_undegraded":"2024-03-12T03:48:13.414371+0000","last_fullsized":"2024-03-12T03:48:13.414371+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:09.359076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111808,"num_objects":78,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78,"num_write_kb":4992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":606,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710455+0000","last_change":"2024-03-12T03:46:26.280678+0000","last_active":"2024-03-12T03:48:09.710455+0000","last_peered":"2024-03-12T03:48:09.710455+0000","last_clean":"2024-03-12T03:48:09.710455+0000","last_became_active":"2024-03-12T03:46:26.279796+0000","last_became_peered":"2024-03-12T03:46:26.279796+0000","last_unstale":"2024-03-12T03:48:09.710455+0000","last_undegraded":"2024-03-12T03:48:09.710455+0000","last_fullsized":"2024-03-12T03:48:09.710455+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:51.818256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:51.818256+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":401,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707702+0000","last_change":"2024-03-12T03:46:57.669774+0000","last_active":"2024-03-12T03:48:09.707702+0000","last_peered":"2024-03-12T03:48:09.707702+0000","last_clean":"2024-03-12T03:48:09.707702+0000","last_became_active":"2024-03-12T03:39:33.278326+0000","last_became_peered":"2024-03-12T03:39:33.278326+0000","last_unstale":"2024-03-12T03:48:09.707702+0000","last_undegraded":"2024-03-12T03:48:09.707702+0000","last_fullsized":"2024-03-12T03:48:09.707702+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:57.669662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:46:57.669662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:13.986821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"269'30","reported_seq":745,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707700+0000","last_change":"2024-03-12T03:47:00.121826+0000","last_active":"2024-03-12T03:48:09.707700+0000","last_peered":"2024-03-12T03:48:09.707700+0000","last_clean":"2024-03-12T03:48:09.707700+0000","last_became_active":"2024-03-12T03:46:25.287885+0000","last_became_peered":"2024-03-12T03:46:25.287885+0000","last_unstale":"2024-03-12T03:48:09.707700+0000","last_undegraded":"2024-03-12T03:48:09.707700+0000","last_fullsized":"2024-03-12T03:48:09.707700+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'30","last_scrub_stamp":"2024-03-12T03:47:00.121665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:00.121665+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708094+0000","last_change":"2024-03-12T03:45:57.516536+0000","last_active":"2024-03-12T03:48:09.708094+0000","last_peered":"2024-03-12T03:48:09.708094+0000","last_clean":"2024-03-12T03:48:09.708094+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:48:09.708094+0000","last_undegraded":"2024-03-12T03:48:09.708094+0000","last_fullsized":"2024-03-12T03:48:09.708094+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:57.516275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:45:57.516275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":399,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710071+0000","last_change":"2024-03-12T03:48:04.679360+0000","last_active":"2024-03-12T03:48:09.710071+0000","last_peered":"2024-03-12T03:48:09.710071+0000","last_clean":"2024-03-12T03:48:09.710071+0000","last_became_active":"2024-03-12T03:46:25.280784+0000","last_became_peered":"2024-03-12T03:46:25.280784+0000","last_unstale":"2024-03-12T03:48:09.710071+0000","last_undegraded":"2024-03-12T03:48:09.710071+0000","last_fullsized":"2024-03-12T03:48:09.710071+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:04.679236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:04.679236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:08.406390+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":21,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.676702+0000","last_change":"2024-03-12T03:47:57.698552+0000","last_active":"2024-03-12T03:48:09.676702+0000","last_peered":"2024-03-12T03:48:09.676702+0000","last_clean":"2024-03-12T03:48:09.676702+0000","last_became_active":"2024-03-12T03:47:57.698209+0000","last_became_peered":"2024-03-12T03:47:57.698209+0000","last_unstale":"2024-03-12T03:48:09.676702+0000","last_undegraded":"2024-03-12T03:48:09.676702+0000","last_fullsized":"2024-03-12T03:48:09.676702+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:20.543054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":594,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:10.420541+0000","last_change":"2024-03-12T03:48:10.420566+0000","last_active":"2024-03-12T03:48:10.420541+0000","last_peered":"2024-03-12T03:48:10.420541+0000","last_clean":"2024-03-12T03:48:10.420541+0000","last_became_active":"2024-03-12T03:44:24.373190+0000","last_became_peered":"2024-03-12T03:44:24.373190+0000","last_unstale":"2024-03-12T03:48:10.420541+0000","last_undegraded":"2024-03-12T03:48:10.420541+0000","last_fullsized":"2024-03-12T03:48:10.420541+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:10.420466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:10.420466+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:13.935172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":845,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707134+0000","last_change":"2024-03-12T03:45:10.340919+0000","last_active":"2024-03-12T03:48:09.707134+0000","last_peered":"2024-03-12T03:48:09.707134+0000","last_clean":"2024-03-12T03:48:09.707134+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:48:09.707134+0000","last_undegraded":"2024-03-12T03:48:09.707134+0000","last_fullsized":"2024-03-12T03:48:09.707134+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:45:10.340796+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.340796+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"269'48","reported_seq":736,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.675287+0000","last_change":"2024-03-12T03:45:14.163797+0000","last_active":"2024-03-12T03:48:09.675287+0000","last_peered":"2024-03-12T03:48:09.675287+0000","last_clean":"2024-03-12T03:48:09.675287+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:48:09.675287+0000","last_undegraded":"2024-03-12T03:48:09.675287+0000","last_fullsized":"2024-03-12T03:48:09.675287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'48","last_scrub_stamp":"2024-03-12T03:45:14.163544+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:45:14.163544+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":678,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707914+0000","last_change":"2024-03-12T03:47:05.588686+0000","last_active":"2024-03-12T03:48:09.707914+0000","last_peered":"2024-03-12T03:48:09.707914+0000","last_clean":"2024-03-12T03:48:09.707914+0000","last_became_active":"2024-03-12T03:33:21.107418+0000","last_became_peered":"2024-03-12T03:33:21.107418+0000","last_unstale":"2024-03-12T03:48:09.707914+0000","last_undegraded":"2024-03-12T03:48:09.707914+0000","last_fullsized":"2024-03-12T03:48:09.707914+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:47:05.588582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:05.588582+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:26.550752+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":652,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.675768+0000","last_change":"2024-03-12T03:45:59.568025+0000","last_active":"2024-03-12T03:48:09.675768+0000","last_peered":"2024-03-12T03:48:09.675768+0000","last_clean":"2024-03-12T03:48:09.675768+0000","last_became_active":"2024-03-12T03:33:27.382583+0000","last_became_peered":"2024-03-12T03:33:27.382583+0000","last_unstale":"2024-03-12T03:48:09.675768+0000","last_undegraded":"2024-03-12T03:48:09.675768+0000","last_fullsized":"2024-03-12T03:48:09.675768+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:59.567890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.567890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":477,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.675465+0000","last_change":"2024-03-12T03:48:08.636052+0000","last_active":"2024-03-12T03:48:09.675465+0000","last_peered":"2024-03-12T03:48:09.675465+0000","last_clean":"2024-03-12T03:48:09.675465+0000","last_became_active":"2024-03-12T03:44:24.365642+0000","last_became_peered":"2024-03-12T03:44:24.365642+0000","last_unstale":"2024-03-12T03:48:09.675465+0000","last_undegraded":"2024-03-12T03:48:09.675465+0000","last_fullsized":"2024-03-12T03:48:09.675465+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:08.635956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:08.635956+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:22.433359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"269'18","reported_seq":462,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709698+0000","last_change":"2024-03-12T03:46:25.314522+0000","last_active":"2024-03-12T03:48:09.709698+0000","last_peered":"2024-03-12T03:48:09.709698+0000","last_clean":"2024-03-12T03:48:09.709698+0000","last_became_active":"2024-03-12T03:46:25.288573+0000","last_became_peered":"2024-03-12T03:46:25.288573+0000","last_unstale":"2024-03-12T03:48:09.709698+0000","last_undegraded":"2024-03-12T03:48:09.709698+0000","last_fullsized":"2024-03-12T03:48:09.709698+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-12T03:43:17.648603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:17.648603+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":128,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707296+0000","last_change":"2024-03-12T03:45:18.106344+0000","last_active":"2024-03-12T03:48:09.707296+0000","last_peered":"2024-03-12T03:48:09.707296+0000","last_clean":"2024-03-12T03:48:09.707296+0000","last_became_active":"2024-03-12T03:45:18.106080+0000","last_became_peered":"2024-03-12T03:45:18.106080+0000","last_unstale":"2024-03-12T03:48:09.707296+0000","last_undegraded":"2024-03-12T03:48:09.707296+0000","last_fullsized":"2024-03-12T03:48:09.707296+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:10.400135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.400135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":666,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.676813+0000","last_change":"2024-03-12T03:47:17.816126+0000","last_active":"2024-03-12T03:48:09.676813+0000","last_peered":"2024-03-12T03:48:09.676813+0000","last_clean":"2024-03-12T03:48:09.676813+0000","last_became_active":"2024-03-12T03:46:25.293521+0000","last_became_peered":"2024-03-12T03:46:25.293521+0000","last_unstale":"2024-03-12T03:48:09.676813+0000","last_undegraded":"2024-03-12T03:48:09.676813+0000","last_fullsized":"2024-03-12T03:48:09.676813+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:17.816024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:17.816024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:36.683825+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":662,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708110+0000","last_change":"2024-03-12T03:47:11.641664+0000","last_active":"2024-03-12T03:48:09.708110+0000","last_peered":"2024-03-12T03:48:09.708110+0000","last_clean":"2024-03-12T03:48:09.708110+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:48:09.708110+0000","last_undegraded":"2024-03-12T03:48:09.708110+0000","last_fullsized":"2024-03-12T03:48:09.708110+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.641547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.641547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:41.604599+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":809,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:10.696861+0000","last_change":"2024-03-12T03:48:10.696886+0000","last_active":"2024-03-12T03:48:10.696861+0000","last_peered":"2024-03-12T03:48:10.696861+0000","last_clean":"2024-03-12T03:48:10.696861+0000","last_became_active":"2024-03-12T03:46:28.316406+0000","last_became_peered":"2024-03-12T03:46:28.316406+0000","last_unstale":"2024-03-12T03:48:10.696861+0000","last_undegraded":"2024-03-12T03:48:10.696861+0000","last_fullsized":"2024-03-12T03:48:10.696861+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:10.696782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:10.696782+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:22.268788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"269'27","reported_seq":729,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708925+0000","last_change":"2024-03-12T03:48:06.104391+0000","last_active":"2024-03-12T03:48:09.708925+0000","last_peered":"2024-03-12T03:48:09.708925+0000","last_clean":"2024-03-12T03:48:09.708925+0000","last_became_active":"2024-03-12T03:46:25.284304+0000","last_became_peered":"2024-03-12T03:46:25.284304+0000","last_unstale":"2024-03-12T03:48:09.708925+0000","last_undegraded":"2024-03-12T03:48:09.708925+0000","last_fullsized":"2024-03-12T03:48:09.708925+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:48:06.104297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.104297+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:34.766229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706901+0000","last_change":"2024-03-12T03:48:04.420937+0000","last_active":"2024-03-12T03:48:09.706901+0000","last_peered":"2024-03-12T03:48:09.706901+0000","last_clean":"2024-03-12T03:48:09.706901+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:48:09.706901+0000","last_undegraded":"2024-03-12T03:48:09.706901+0000","last_fullsized":"2024-03-12T03:48:09.706901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:04.420853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:04.420853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:28.990567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":425,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709876+0000","last_change":"2024-03-12T03:46:25.289109+0000","last_active":"2024-03-12T03:48:09.709876+0000","last_peered":"2024-03-12T03:48:09.709876+0000","last_clean":"2024-03-12T03:48:09.709876+0000","last_became_active":"2024-03-12T03:46:25.286866+0000","last_became_peered":"2024-03-12T03:46:25.286866+0000","last_unstale":"2024-03-12T03:48:09.709876+0000","last_undegraded":"2024-03-12T03:48:09.709876+0000","last_fullsized":"2024-03-12T03:48:09.709876+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:43:05.530729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:43:05.530729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":393,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710112+0000","last_change":"2024-03-12T03:46:49.647283+0000","last_active":"2024-03-12T03:48:09.710112+0000","last_peered":"2024-03-12T03:48:09.710112+0000","last_clean":"2024-03-12T03:48:09.710112+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:48:09.710112+0000","last_undegraded":"2024-03-12T03:48:09.710112+0000","last_fullsized":"2024-03-12T03:48:09.710112+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:49.647169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:46:49.647169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:18.202463+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":477,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.675230+0000","last_change":"2024-03-12T03:47:58.612030+0000","last_active":"2024-03-12T03:48:09.675230+0000","last_peered":"2024-03-12T03:48:09.675230+0000","last_clean":"2024-03-12T03:48:09.675230+0000","last_became_active":"2024-03-12T03:44:24.376141+0000","last_became_peered":"2024-03-12T03:44:24.376141+0000","last_unstale":"2024-03-12T03:48:09.675230+0000","last_undegraded":"2024-03-12T03:48:09.675230+0000","last_fullsized":"2024-03-12T03:48:09.675230+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:58.611912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.611912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:27.419788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"269'21","reported_seq":724,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.677790+0000","last_change":"2024-03-12T03:47:09.828093+0000","last_active":"2024-03-12T03:48:09.677790+0000","last_peered":"2024-03-12T03:48:09.677790+0000","last_clean":"2024-03-12T03:48:09.677790+0000","last_became_active":"2024-03-12T03:46:25.298007+0000","last_became_peered":"2024-03-12T03:46:25.298007+0000","last_unstale":"2024-03-12T03:48:09.677790+0000","last_undegraded":"2024-03-12T03:48:09.677790+0000","last_fullsized":"2024-03-12T03:48:09.677790+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'21","last_scrub_stamp":"2024-03-12T03:47:09.827966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:09.827966+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:28.032022+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":675,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707273+0000","last_change":"2024-03-12T03:47:16.849528+0000","last_active":"2024-03-12T03:48:09.707273+0000","last_peered":"2024-03-12T03:48:09.707273+0000","last_clean":"2024-03-12T03:48:09.707273+0000","last_became_active":"2024-03-12T03:46:25.276467+0000","last_became_peered":"2024-03-12T03:46:25.276467+0000","last_unstale":"2024-03-12T03:48:09.707273+0000","last_undegraded":"2024-03-12T03:48:09.707273+0000","last_fullsized":"2024-03-12T03:48:09.707273+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:23.329829+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"332'14","reported_seq":634,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711108+0000","last_change":"2024-03-12T03:46:25.302407+0000","last_active":"2024-03-12T03:48:09.711108+0000","last_peered":"2024-03-12T03:48:09.711108+0000","last_clean":"2024-03-12T03:48:09.711108+0000","last_became_active":"2024-03-12T03:46:25.300463+0000","last_became_peered":"2024-03-12T03:46:25.300463+0000","last_unstale":"2024-03-12T03:48:09.711108+0000","last_undegraded":"2024-03-12T03:48:09.711108+0000","last_fullsized":"2024-03-12T03:48:09.711108+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'12","last_scrub_stamp":"2024-03-12T03:43:12.675070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:43:12.675070+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708953+0000","last_change":"2024-03-12T03:47:11.001919+0000","last_active":"2024-03-12T03:48:09.708953+0000","last_peered":"2024-03-12T03:48:09.708953+0000","last_clean":"2024-03-12T03:48:09.708953+0000","last_became_active":"2024-03-12T03:39:33.265499+0000","last_became_peered":"2024-03-12T03:39:33.265499+0000","last_unstale":"2024-03-12T03:48:09.708953+0000","last_undegraded":"2024-03-12T03:48:09.708953+0000","last_fullsized":"2024-03-12T03:48:09.708953+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.001795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.001795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:36.111531+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":489,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706980+0000","last_change":"2024-03-12T03:46:28.302513+0000","last_active":"2024-03-12T03:48:09.706980+0000","last_peered":"2024-03-12T03:48:09.706980+0000","last_clean":"2024-03-12T03:48:09.706980+0000","last_became_active":"2024-03-12T03:46:28.301964+0000","last_became_peered":"2024-03-12T03:46:28.301964+0000","last_unstale":"2024-03-12T03:48:09.706980+0000","last_undegraded":"2024-03-12T03:48:09.706980+0000","last_fullsized":"2024-03-12T03:48:09.706980+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:55.795990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:55.795990+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"269'32","reported_seq":525,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710709+0000","last_change":"2024-03-12T03:47:55.395679+0000","last_active":"2024-03-12T03:48:09.710709+0000","last_peered":"2024-03-12T03:48:09.710709+0000","last_clean":"2024-03-12T03:48:09.710709+0000","last_became_active":"2024-03-12T03:46:25.299798+0000","last_became_peered":"2024-03-12T03:46:25.299798+0000","last_unstale":"2024-03-12T03:48:09.710709+0000","last_undegraded":"2024-03-12T03:48:09.710709+0000","last_fullsized":"2024-03-12T03:48:09.710709+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'32","last_scrub_stamp":"2024-03-12T03:47:55.395555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:55.395555+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:18.617496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":701,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706931+0000","last_change":"2024-03-12T03:47:53.609427+0000","last_active":"2024-03-12T03:48:09.706931+0000","last_peered":"2024-03-12T03:48:09.706931+0000","last_clean":"2024-03-12T03:48:09.706931+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:48:09.706931+0000","last_undegraded":"2024-03-12T03:48:09.706931+0000","last_fullsized":"2024-03-12T03:48:09.706931+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:47:53.609317+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:47:53.609317+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:11.417787+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":842,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.677706+0000","last_change":"2024-03-12T03:45:53.459029+0000","last_active":"2024-03-12T03:48:09.677706+0000","last_peered":"2024-03-12T03:48:09.677706+0000","last_clean":"2024-03-12T03:48:09.677706+0000","last_became_active":"2024-03-12T03:33:25.182909+0000","last_became_peered":"2024-03-12T03:33:25.182909+0000","last_unstale":"2024-03-12T03:48:09.677706+0000","last_undegraded":"2024-03-12T03:48:09.677706+0000","last_fullsized":"2024-03-12T03:48:09.677706+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:53.458170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.458170+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":663,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708587+0000","last_change":"2024-03-12T03:47:54.079670+0000","last_active":"2024-03-12T03:48:09.708587+0000","last_peered":"2024-03-12T03:48:09.708587+0000","last_clean":"2024-03-12T03:48:09.708587+0000","last_became_active":"2024-03-12T03:46:25.283805+0000","last_became_peered":"2024-03-12T03:46:25.283805+0000","last_unstale":"2024-03-12T03:48:09.708587+0000","last_undegraded":"2024-03-12T03:48:09.708587+0000","last_fullsized":"2024-03-12T03:48:09.708587+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.079569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.079569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:15.554223+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"269'19","reported_seq":701,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708125+0000","last_change":"2024-03-12T03:45:15.958314+0000","last_active":"2024-03-12T03:48:09.708125+0000","last_peered":"2024-03-12T03:48:09.708125+0000","last_clean":"2024-03-12T03:48:09.708125+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:48:09.708125+0000","last_undegraded":"2024-03-12T03:48:09.708125+0000","last_fullsized":"2024-03-12T03:48:09.708125+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'19","last_scrub_stamp":"2024-03-12T03:45:15.957617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.957617+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707935+0000","last_change":"2024-03-12T03:47:07.914268+0000","last_active":"2024-03-12T03:48:09.707935+0000","last_peered":"2024-03-12T03:48:09.707935+0000","last_clean":"2024-03-12T03:48:09.707935+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:48:09.707935+0000","last_undegraded":"2024-03-12T03:48:09.707935+0000","last_fullsized":"2024-03-12T03:48:09.707935+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_clean_scrub_stamp":"2024-03-12T03:47:07.914160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:18.181685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":675,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710930+0000","last_change":"2024-03-12T03:47:05.288244+0000","last_active":"2024-03-12T03:48:09.710930+0000","last_peered":"2024-03-12T03:48:09.710930+0000","last_clean":"2024-03-12T03:48:09.710930+0000","last_became_active":"2024-03-12T03:46:25.288050+0000","last_became_peered":"2024-03-12T03:46:25.288050+0000","last_unstale":"2024-03-12T03:48:09.710930+0000","last_undegraded":"2024-03-12T03:48:09.710930+0000","last_fullsized":"2024-03-12T03:48:09.710930+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:05.288140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:05.288140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:17.919622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":449,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708813+0000","last_change":"2024-03-12T03:47:54.433284+0000","last_active":"2024-03-12T03:48:09.708813+0000","last_peered":"2024-03-12T03:48:09.708813+0000","last_clean":"2024-03-12T03:48:09.708813+0000","last_became_active":"2024-03-12T03:46:25.296588+0000","last_became_peered":"2024-03-12T03:46:25.296588+0000","last_unstale":"2024-03-12T03:48:09.708813+0000","last_undegraded":"2024-03-12T03:48:09.708813+0000","last_fullsized":"2024-03-12T03:48:09.708813+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.433184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.433184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:00.532918+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"269'15","reported_seq":265,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.675441+0000","last_change":"2024-03-12T03:47:57.579034+0000","last_active":"2024-03-12T03:48:09.675441+0000","last_peered":"2024-03-12T03:48:09.675441+0000","last_clean":"2024-03-12T03:48:09.675441+0000","last_became_active":"2024-03-12T03:43:01.037348+0000","last_became_peered":"2024-03-12T03:43:01.037348+0000","last_unstale":"2024-03-12T03:48:09.675441+0000","last_undegraded":"2024-03-12T03:48:09.675441+0000","last_fullsized":"2024-03-12T03:48:09.675441+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'15","last_scrub_stamp":"2024-03-12T03:47:57.578870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:57.578870+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:12.668435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":684,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708213+0000","last_change":"2024-03-12T03:46:53.600616+0000","last_active":"2024-03-12T03:48:09.708213+0000","last_peered":"2024-03-12T03:48:09.708213+0000","last_clean":"2024-03-12T03:48:09.708213+0000","last_became_active":"2024-03-12T03:46:25.283602+0000","last_became_peered":"2024-03-12T03:46:25.283602+0000","last_unstale":"2024-03-12T03:48:09.708213+0000","last_undegraded":"2024-03-12T03:48:09.708213+0000","last_fullsized":"2024-03-12T03:48:09.708213+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:53.600496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:46:53.600496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":461,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:10.625642+0000","last_change":"2024-03-12T03:48:10.625661+0000","last_active":"2024-03-12T03:48:10.625642+0000","last_peered":"2024-03-12T03:48:10.625642+0000","last_clean":"2024-03-12T03:48:10.625642+0000","last_became_active":"2024-03-12T03:46:25.285418+0000","last_became_peered":"2024-03-12T03:46:25.285418+0000","last_unstale":"2024-03-12T03:48:10.625642+0000","last_undegraded":"2024-03-12T03:48:10.625642+0000","last_fullsized":"2024-03-12T03:48:10.625642+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_clean_scrub_stamp":"2024-03-12T03:48:10.625582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:34.057149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":653,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708810+0000","last_change":"2024-03-12T03:45:00.842486+0000","last_active":"2024-03-12T03:48:09.708810+0000","last_peered":"2024-03-12T03:48:09.708810+0000","last_clean":"2024-03-12T03:48:09.708810+0000","last_became_active":"2024-03-12T03:33:27.192816+0000","last_became_peered":"2024-03-12T03:33:27.192816+0000","last_unstale":"2024-03-12T03:48:09.708810+0000","last_undegraded":"2024-03-12T03:48:09.708810+0000","last_fullsized":"2024-03-12T03:48:09.708810+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:00.842351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:45:00.842351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":591,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709515+0000","last_change":"2024-03-12T03:47:14.914147+0000","last_active":"2024-03-12T03:48:09.709515+0000","last_peered":"2024-03-12T03:48:09.709515+0000","last_clean":"2024-03-12T03:48:09.709515+0000","last_became_active":"2024-03-12T03:44:24.387247+0000","last_became_peered":"2024-03-12T03:44:24.387247+0000","last_unstale":"2024-03-12T03:48:09.709515+0000","last_undegraded":"2024-03-12T03:48:09.709515+0000","last_fullsized":"2024-03-12T03:48:09.709515+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:14.914077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:14.914077+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:34.091102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"269'26","reported_seq":472,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709015+0000","last_change":"2024-03-12T03:46:25.308547+0000","last_active":"2024-03-12T03:48:09.709015+0000","last_peered":"2024-03-12T03:48:09.709015+0000","last_clean":"2024-03-12T03:48:09.709015+0000","last_became_active":"2024-03-12T03:46:25.287095+0000","last_became_peered":"2024-03-12T03:46:25.287095+0000","last_unstale":"2024-03-12T03:48:09.709015+0000","last_undegraded":"2024-03-12T03:48:09.709015+0000","last_fullsized":"2024-03-12T03:48:09.709015+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-12T03:43:06.549441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:43:06.549441+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":672,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.676910+0000","last_change":"2024-03-12T03:48:02.679493+0000","last_active":"2024-03-12T03:48:09.676910+0000","last_peered":"2024-03-12T03:48:09.676910+0000","last_clean":"2024-03-12T03:48:09.676910+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:48:09.676910+0000","last_undegraded":"2024-03-12T03:48:09.676910+0000","last_fullsized":"2024-03-12T03:48:09.676910+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:02.679397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:48:02.679397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:17.853840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":666,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707169+0000","last_change":"2024-03-12T03:48:09.377483+0000","last_active":"2024-03-12T03:48:09.707169+0000","last_peered":"2024-03-12T03:48:09.707169+0000","last_clean":"2024-03-12T03:48:09.707169+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:48:09.707169+0000","last_undegraded":"2024-03-12T03:48:09.707169+0000","last_fullsized":"2024-03-12T03:48:09.707169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:09.377400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:09.377400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:36.578772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":476,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710156+0000","last_change":"2024-03-12T03:46:25.288878+0000","last_active":"2024-03-12T03:48:09.710156+0000","last_peered":"2024-03-12T03:48:09.710156+0000","last_clean":"2024-03-12T03:48:09.710156+0000","last_became_active":"2024-03-12T03:46:25.286881+0000","last_became_peered":"2024-03-12T03:46:25.286881+0000","last_unstale":"2024-03-12T03:48:09.710156+0000","last_undegraded":"2024-03-12T03:48:09.710156+0000","last_fullsized":"2024-03-12T03:48:09.710156+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"269'10","reported_seq":451,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706703+0000","last_change":"2024-03-12T03:48:06.646761+0000","last_active":"2024-03-12T03:48:09.706703+0000","last_peered":"2024-03-12T03:48:09.706703+0000","last_clean":"2024-03-12T03:48:09.706703+0000","last_became_active":"2024-03-12T03:46:25.286315+0000","last_became_peered":"2024-03-12T03:46:25.286315+0000","last_unstale":"2024-03-12T03:48:09.706703+0000","last_undegraded":"2024-03-12T03:48:09.706703+0000","last_fullsized":"2024-03-12T03:48:09.706703+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'10","last_scrub_stamp":"2024-03-12T03:48:06.646668+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.646668+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:36.138287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":675,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708164+0000","last_change":"2024-03-12T03:47:11.851386+0000","last_active":"2024-03-12T03:48:09.708164+0000","last_peered":"2024-03-12T03:48:09.708164+0000","last_clean":"2024-03-12T03:48:09.708164+0000","last_became_active":"2024-03-12T03:46:25.287132+0000","last_became_peered":"2024-03-12T03:46:25.287132+0000","last_unstale":"2024-03-12T03:48:09.708164+0000","last_undegraded":"2024-03-12T03:48:09.708164+0000","last_fullsized":"2024-03-12T03:48:09.708164+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.851271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.851271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:27.985117+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":129,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707394+0000","last_change":"2024-03-12T03:45:18.110285+0000","last_active":"2024-03-12T03:48:09.707394+0000","last_peered":"2024-03-12T03:48:09.707394+0000","last_clean":"2024-03-12T03:48:09.707394+0000","last_became_active":"2024-03-12T03:45:18.109983+0000","last_became_peered":"2024-03-12T03:45:18.109983+0000","last_unstale":"2024-03-12T03:48:09.707394+0000","last_undegraded":"2024-03-12T03:48:09.707394+0000","last_fullsized":"2024-03-12T03:48:09.707394+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:08.466225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:08.466225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":666,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707255+0000","last_change":"2024-03-12T03:45:12.308511+0000","last_active":"2024-03-12T03:48:09.707255+0000","last_peered":"2024-03-12T03:48:09.707255+0000","last_clean":"2024-03-12T03:48:09.707255+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:48:09.707255+0000","last_undegraded":"2024-03-12T03:48:09.707255+0000","last_fullsized":"2024-03-12T03:48:09.707255+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:12.308394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:12.308394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"269'31","reported_seq":637,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.677771+0000","last_change":"2024-03-12T03:48:00.687798+0000","last_active":"2024-03-12T03:48:09.677771+0000","last_peered":"2024-03-12T03:48:09.677771+0000","last_clean":"2024-03-12T03:48:09.677771+0000","last_became_active":"2024-03-12T03:45:18.107047+0000","last_became_peered":"2024-03-12T03:45:18.107047+0000","last_unstale":"2024-03-12T03:48:09.677771+0000","last_undegraded":"2024-03-12T03:48:09.677771+0000","last_fullsized":"2024-03-12T03:48:09.677771+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:48:00.687677+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.687677+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:11.240840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":477,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706875+0000","last_change":"2024-03-12T03:48:00.458553+0000","last_active":"2024-03-12T03:48:09.706875+0000","last_peered":"2024-03-12T03:48:09.706875+0000","last_clean":"2024-03-12T03:48:09.706875+0000","last_became_active":"2024-03-12T03:46:25.287635+0000","last_became_peered":"2024-03-12T03:46:25.287635+0000","last_unstale":"2024-03-12T03:48:09.706875+0000","last_undegraded":"2024-03-12T03:48:09.706875+0000","last_fullsized":"2024-03-12T03:48:09.706875+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:48:00.458460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.458460+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:13.195187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":670,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709101+0000","last_change":"2024-03-12T03:48:09.160367+0000","last_active":"2024-03-12T03:48:09.709101+0000","last_peered":"2024-03-12T03:48:09.709101+0000","last_clean":"2024-03-12T03:48:09.709101+0000","last_became_active":"2024-03-12T03:39:33.276435+0000","last_became_peered":"2024-03-12T03:39:33.276435+0000","last_unstale":"2024-03-12T03:48:09.709101+0000","last_undegraded":"2024-03-12T03:48:09.709101+0000","last_fullsized":"2024-03-12T03:48:09.709101+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:09.160277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:09.160277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:14.302739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":659,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711808+0000","last_change":"2024-03-12T03:45:01.176270+0000","last_active":"2024-03-12T03:48:09.711808+0000","last_peered":"2024-03-12T03:48:09.711808+0000","last_clean":"2024-03-12T03:48:09.711808+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:48:09.711808+0000","last_undegraded":"2024-03-12T03:48:09.711808+0000","last_fullsized":"2024-03-12T03:48:09.711808+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:01.176150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:01.176150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"269'27","reported_seq":728,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708813+0000","last_change":"2024-03-12T03:47:15.795671+0000","last_active":"2024-03-12T03:48:09.708813+0000","last_peered":"2024-03-12T03:48:09.708813+0000","last_clean":"2024-03-12T03:48:09.708813+0000","last_became_active":"2024-03-12T03:46:25.284789+0000","last_became_peered":"2024-03-12T03:46:25.284789+0000","last_unstale":"2024-03-12T03:48:09.708813+0000","last_undegraded":"2024-03-12T03:48:09.708813+0000","last_fullsized":"2024-03-12T03:48:09.708813+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:15.795515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.795515+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:32.287782+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"351'2","reported_seq":685,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711370+0000","last_change":"2024-03-12T03:47:15.334100+0000","last_active":"2024-03-12T03:48:09.711370+0000","last_peered":"2024-03-12T03:48:09.711370+0000","last_clean":"2024-03-12T03:48:09.711370+0000","last_became_active":"2024-03-12T03:46:25.283344+0000","last_became_peered":"2024-03-12T03:46:25.283344+0000","last_unstale":"2024-03-12T03:48:09.711370+0000","last_undegraded":"2024-03-12T03:48:09.711370+0000","last_fullsized":"2024-03-12T03:48:09.711370+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'2","last_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_deep_scrub":"351'2","last_deep_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.329905+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:36.545077+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'9","reported_seq":199,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709059+0000","last_change":"2024-03-12T03:48:02.854911+0000","last_active":"2024-03-12T03:48:09.709059+0000","last_peered":"2024-03-12T03:48:09.709059+0000","last_clean":"2024-03-12T03:48:09.709059+0000","last_became_active":"2024-03-12T03:45:18.108935+0000","last_became_peered":"2024-03-12T03:45:18.108935+0000","last_unstale":"2024-03-12T03:48:09.709059+0000","last_undegraded":"2024-03-12T03:48:09.709059+0000","last_fullsized":"2024-03-12T03:48:09.709059+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'9","last_scrub_stamp":"2024-03-12T03:48:02.854820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:02.854820+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:12.141406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":682,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707508+0000","last_change":"2024-03-12T03:47:14.619080+0000","last_active":"2024-03-12T03:48:09.707508+0000","last_peered":"2024-03-12T03:48:09.707508+0000","last_clean":"2024-03-12T03:48:09.707508+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:48:09.707508+0000","last_undegraded":"2024-03-12T03:48:09.707508+0000","last_fullsized":"2024-03-12T03:48:09.707508+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:47:14.618947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:14.618947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:17.380413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"269'24","reported_seq":509,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707427+0000","last_change":"2024-03-12T03:47:58.427568+0000","last_active":"2024-03-12T03:48:09.707427+0000","last_peered":"2024-03-12T03:48:09.707427+0000","last_clean":"2024-03-12T03:48:09.707427+0000","last_became_active":"2024-03-12T03:46:25.283193+0000","last_became_peered":"2024-03-12T03:46:25.283193+0000","last_unstale":"2024-03-12T03:48:09.707427+0000","last_undegraded":"2024-03-12T03:48:09.707427+0000","last_fullsized":"2024-03-12T03:48:09.707427+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:47:58.427463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.427463+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:09.567592+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":693,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708580+0000","last_change":"2024-03-12T03:48:01.902683+0000","last_active":"2024-03-12T03:48:09.708580+0000","last_peered":"2024-03-12T03:48:09.708580+0000","last_clean":"2024-03-12T03:48:09.708580+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:48:09.708580+0000","last_undegraded":"2024-03-12T03:48:09.708580+0000","last_fullsized":"2024-03-12T03:48:09.708580+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:01.902570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:01.902570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:25.239463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":687,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708261+0000","last_change":"2024-03-12T03:46:53.126237+0000","last_active":"2024-03-12T03:48:09.708261+0000","last_peered":"2024-03-12T03:48:09.708261+0000","last_clean":"2024-03-12T03:48:09.708261+0000","last_became_active":"2024-03-12T03:46:25.301691+0000","last_became_peered":"2024-03-12T03:46:25.301691+0000","last_unstale":"2024-03-12T03:48:09.708261+0000","last_undegraded":"2024-03-12T03:48:09.708261+0000","last_fullsized":"2024-03-12T03:48:09.708261+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:53.126089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:46:53.126089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:11.436021+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":478,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.710078+0000","last_change":"2024-03-12T03:48:02.624958+0000","last_active":"2024-03-12T03:48:09.710078+0000","last_peered":"2024-03-12T03:48:09.710078+0000","last_clean":"2024-03-12T03:48:09.710078+0000","last_became_active":"2024-03-12T03:46:25.296764+0000","last_became_peered":"2024-03-12T03:46:25.296764+0000","last_unstale":"2024-03-12T03:48:09.710078+0000","last_undegraded":"2024-03-12T03:48:09.710078+0000","last_fullsized":"2024-03-12T03:48:09.710078+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:02.624857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:02.624857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:06.340909+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"269'27","reported_seq":452,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709974+0000","last_change":"2024-03-12T03:47:56.823434+0000","last_active":"2024-03-12T03:48:09.709974+0000","last_peered":"2024-03-12T03:48:09.709974+0000","last_clean":"2024-03-12T03:48:09.709974+0000","last_became_active":"2024-03-12T03:46:25.301700+0000","last_became_peered":"2024-03-12T03:46:25.301700+0000","last_unstale":"2024-03-12T03:48:09.709974+0000","last_undegraded":"2024-03-12T03:48:09.709974+0000","last_fullsized":"2024-03-12T03:48:09.709974+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:56.823315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.823315+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:22.394677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":664,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708547+0000","last_change":"2024-03-12T03:45:03.797399+0000","last_active":"2024-03-12T03:48:09.708547+0000","last_peered":"2024-03-12T03:48:09.708547+0000","last_clean":"2024-03-12T03:48:09.708547+0000","last_became_active":"2024-03-12T03:33:21.103439+0000","last_became_peered":"2024-03-12T03:33:21.103439+0000","last_unstale":"2024-03-12T03:48:09.708547+0000","last_undegraded":"2024-03-12T03:48:09.708547+0000","last_fullsized":"2024-03-12T03:48:09.708547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.797296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.797296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":663,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711800+0000","last_change":"2024-03-12T03:45:53.016158+0000","last_active":"2024-03-12T03:48:09.711800+0000","last_peered":"2024-03-12T03:48:09.711800+0000","last_clean":"2024-03-12T03:48:09.711800+0000","last_became_active":"2024-03-12T03:33:25.185597+0000","last_became_peered":"2024-03-12T03:33:25.185597+0000","last_unstale":"2024-03-12T03:48:09.711800+0000","last_undegraded":"2024-03-12T03:48:09.711800+0000","last_fullsized":"2024-03-12T03:48:09.711800+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.016030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.016030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":663,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708584+0000","last_change":"2024-03-12T03:48:00.918264+0000","last_active":"2024-03-12T03:48:09.708584+0000","last_peered":"2024-03-12T03:48:09.708584+0000","last_clean":"2024-03-12T03:48:09.708584+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:48:09.708584+0000","last_undegraded":"2024-03-12T03:48:09.708584+0000","last_fullsized":"2024-03-12T03:48:09.708584+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:00.918131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.918131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:28.624540+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"269'16","reported_seq":748,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707926+0000","last_change":"2024-03-12T03:47:04.987456+0000","last_active":"2024-03-12T03:48:09.707926+0000","last_peered":"2024-03-12T03:48:09.707926+0000","last_clean":"2024-03-12T03:48:09.707926+0000","last_became_active":"2024-03-12T03:46:25.283025+0000","last_became_peered":"2024-03-12T03:46:25.283025+0000","last_unstale":"2024-03-12T03:48:09.707926+0000","last_undegraded":"2024-03-12T03:48:09.707926+0000","last_fullsized":"2024-03-12T03:48:09.707926+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'16","last_scrub_stamp":"2024-03-12T03:47:04.987337+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:47:04.987337+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:18.199925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":663,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.676138+0000","last_change":"2024-03-12T03:47:56.585903+0000","last_active":"2024-03-12T03:48:09.676138+0000","last_peered":"2024-03-12T03:48:09.676138+0000","last_clean":"2024-03-12T03:48:09.676138+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:48:09.676138+0000","last_undegraded":"2024-03-12T03:48:09.676138+0000","last_fullsized":"2024-03-12T03:48:09.676138+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.585817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.585817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:21.082050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":657,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707082+0000","last_change":"2024-03-12T03:45:11.324897+0000","last_active":"2024-03-12T03:48:09.707082+0000","last_peered":"2024-03-12T03:48:09.707082+0000","last_clean":"2024-03-12T03:48:09.707082+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:48:09.707082+0000","last_undegraded":"2024-03-12T03:48:09.707082+0000","last_fullsized":"2024-03-12T03:48:09.707082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:11.324666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:45:11.324666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":662,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.675694+0000","last_change":"2024-03-12T03:47:54.695744+0000","last_active":"2024-03-12T03:48:09.675694+0000","last_peered":"2024-03-12T03:48:09.675694+0000","last_clean":"2024-03-12T03:48:09.675694+0000","last_became_active":"2024-03-12T03:46:25.293777+0000","last_became_peered":"2024-03-12T03:46:25.293777+0000","last_unstale":"2024-03-12T03:48:09.675694+0000","last_undegraded":"2024-03-12T03:48:09.675694+0000","last_fullsized":"2024-03-12T03:48:09.675694+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.695594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.695594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:58.456936+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"345'33","reported_seq":743,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708392+0000","last_change":"2024-03-12T03:47:02.110674+0000","last_active":"2024-03-12T03:48:09.708392+0000","last_peered":"2024-03-12T03:48:09.708392+0000","last_clean":"2024-03-12T03:48:09.708392+0000","last_became_active":"2024-03-12T03:46:25.279992+0000","last_became_peered":"2024-03-12T03:46:25.279992+0000","last_unstale":"2024-03-12T03:48:09.708392+0000","last_undegraded":"2024-03-12T03:48:09.708392+0000","last_fullsized":"2024-03-12T03:48:09.708392+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'33","last_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_deep_scrub":"345'33","last_deep_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_clean_scrub_stamp":"2024-03-12T03:47:02.110517+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:48:31.585583+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":663,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.677624+0000","last_change":"2024-03-12T03:45:56.503256+0000","last_active":"2024-03-12T03:48:09.677624+0000","last_peered":"2024-03-12T03:48:09.677624+0000","last_clean":"2024-03-12T03:48:09.677624+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:48:09.677624+0000","last_undegraded":"2024-03-12T03:48:09.677624+0000","last_fullsized":"2024-03-12T03:48:09.677624+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:56.503146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:45:56.503146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":655,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.675741+0000","last_change":"2024-03-12T03:46:02.618859+0000","last_active":"2024-03-12T03:48:09.675741+0000","last_peered":"2024-03-12T03:48:09.675741+0000","last_clean":"2024-03-12T03:48:09.675741+0000","last_became_active":"2024-03-12T03:33:25.175843+0000","last_became_peered":"2024-03-12T03:33:25.175843+0000","last_unstale":"2024-03-12T03:48:09.675741+0000","last_undegraded":"2024-03-12T03:48:09.675741+0000","last_fullsized":"2024-03-12T03:48:09.675741+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:02.618599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_clean_scrub_stamp":"2024-03-12T03:46:02.618599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":653,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706847+0000","last_change":"2024-03-12T03:45:58.095317+0000","last_active":"2024-03-12T03:48:09.706847+0000","last_peered":"2024-03-12T03:48:09.706847+0000","last_clean":"2024-03-12T03:48:09.706847+0000","last_became_active":"2024-03-12T03:33:27.189230+0000","last_became_peered":"2024-03-12T03:33:27.189230+0000","last_unstale":"2024-03-12T03:48:09.706847+0000","last_undegraded":"2024-03-12T03:48:09.706847+0000","last_fullsized":"2024-03-12T03:48:09.706847+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_clean_scrub_stamp":"2024-03-12T03:45:58.095196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"269'62","reported_seq":628,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706669+0000","last_change":"2024-03-12T03:46:25.315560+0000","last_active":"2024-03-12T03:48:09.706669+0000","last_peered":"2024-03-12T03:48:09.706669+0000","last_clean":"2024-03-12T03:48:09.706669+0000","last_became_active":"2024-03-12T03:46:25.285447+0000","last_became_peered":"2024-03-12T03:46:25.285447+0000","last_unstale":"2024-03-12T03:48:09.706669+0000","last_undegraded":"2024-03-12T03:48:09.706669+0000","last_fullsized":"2024-03-12T03:48:09.706669+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":503,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:12.573571+0000","last_change":"2024-03-12T03:48:12.573604+0000","last_active":"2024-03-12T03:48:12.573571+0000","last_peered":"2024-03-12T03:48:12.573571+0000","last_clean":"2024-03-12T03:48:12.573571+0000","last_became_active":"2024-03-12T03:46:25.279131+0000","last_became_peered":"2024-03-12T03:46:25.279131+0000","last_unstale":"2024-03-12T03:48:12.573571+0000","last_undegraded":"2024-03-12T03:48:12.573571+0000","last_fullsized":"2024-03-12T03:48:12.573571+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:12.573481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:48:12.573481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:23.541780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":666,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706974+0000","last_change":"2024-03-12T03:47:55.924374+0000","last_active":"2024-03-12T03:48:09.706974+0000","last_peered":"2024-03-12T03:48:09.706974+0000","last_clean":"2024-03-12T03:48:09.706974+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:48:09.706974+0000","last_undegraded":"2024-03-12T03:48:09.706974+0000","last_fullsized":"2024-03-12T03:48:09.706974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:55.924292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:47:55.924292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:08.749956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":454,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.709987+0000","last_change":"2024-03-12T03:47:57.456997+0000","last_active":"2024-03-12T03:48:09.709987+0000","last_peered":"2024-03-12T03:48:09.709987+0000","last_clean":"2024-03-12T03:48:09.709987+0000","last_became_active":"2024-03-12T03:46:25.281081+0000","last_became_peered":"2024-03-12T03:46:25.281081+0000","last_unstale":"2024-03-12T03:48:09.709987+0000","last_undegraded":"2024-03-12T03:48:09.709987+0000","last_fullsized":"2024-03-12T03:48:09.709987+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:57.456889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:57.456889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:06.875503+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"269'26","reported_seq":704,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.706849+0000","last_change":"2024-03-12T03:45:10.855287+0000","last_active":"2024-03-12T03:48:09.706849+0000","last_peered":"2024-03-12T03:48:09.706849+0000","last_clean":"2024-03-12T03:48:09.706849+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:48:09.706849+0000","last_undegraded":"2024-03-12T03:48:09.706849+0000","last_fullsized":"2024-03-12T03:48:09.706849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:10.855156+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.855156+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":670,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.708004+0000","last_change":"2024-03-12T03:44:14.713075+0000","last_active":"2024-03-12T03:48:09.708004+0000","last_peered":"2024-03-12T03:48:09.708004+0000","last_clean":"2024-03-12T03:48:09.708004+0000","last_became_active":"2024-03-12T03:39:33.276119+0000","last_became_peered":"2024-03-12T03:39:33.276119+0000","last_unstale":"2024-03-12T03:48:09.708004+0000","last_undegraded":"2024-03-12T03:48:09.708004+0000","last_fullsized":"2024-03-12T03:48:09.708004+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:44:14.712916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":673,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.711428+0000","last_change":"2024-03-12T03:48:05.754875+0000","last_active":"2024-03-12T03:48:09.711428+0000","last_peered":"2024-03-12T03:48:09.711428+0000","last_clean":"2024-03-12T03:48:09.711428+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:48:09.711428+0000","last_undegraded":"2024-03-12T03:48:09.711428+0000","last_fullsized":"2024-03-12T03:48:09.711428+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:05.754788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:26.625326+0000","last_clean_scrub_stamp":"2024-03-12T03:48:05.754788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:49:35.424060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":661,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-03-12T03:48:09.707630+0000","last_change":"2024-03-12T03:45:14.952828+0000","last_active":"2024-03-12T03:48:09.707630+0000","last_peered":"2024-03-12T03:48:09.707630+0000","last_clean":"2024-03-12T03:48:09.707630+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:48:09.707630+0000","last_undegraded":"2024-03-12T03:48:09.707630+0000","last_fullsized":"2024-03-12T03:48:09.707630+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_clean_scrub_stamp":"2024-03-12T03:45:14.952550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":9961472,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":152,"num_write_kb":9728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":27983872,"data_stored":27983872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":883,"num_read_kb":878,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1402,"num_read_kb":1127,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":2284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},"log_size":957,"ondisk_log_size":957,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070647,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":687260,"kb_used_data":2188,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93057892,"statfs":{"total":95995035648,"available":95291281408,"internally_reserved":0,"allocated":2240512,"data_stored":1227773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":701495754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332284,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":475140,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93270012,"statfs":{"total":95995035648,"available":95508492288,"internally_reserved":0,"allocated":2543616,"data_stored":1553490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":483981766},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575061,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727528,"kb_used_data":2208,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93017624,"statfs":{"total":95995035648,"available":95250046976,"internally_reserved":0,"allocated":2260992,"data_stored":1227550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738586,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":557084,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93188068,"statfs":{"total":95995035648,"available":95424581632,"internally_reserved":0,"allocated":2568192,"data_stored":1554707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411535,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":547264,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93197888,"statfs":{"total":95995035648,"available":95434637312,"internally_reserved":0,"allocated":2478080,"data_stored":1489511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":330,"seq":1417339207703,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181736,"kb_used_data":2784,"kb_used_omap":13,"kb_used_meta":178930,"kb_avail":93563416,"statfs":{"total":95995035648,"available":95808937984,"internally_reserved":0,"allocated":2850816,"data_stored":1817505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13485,"internal_metadata":183225171},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953671,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412384,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93332768,"statfs":{"total":95995035648,"available":95572754432,"internally_reserved":0,"allocated":2244608,"data_stored":1227313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659329,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":705564,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93039588,"statfs":{"total":95995035648,"available":95272538112,"internally_reserved":0,"allocated":2502656,"data_stored":1489279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27983872,"data_stored":27983872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:48:21.749 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-12T03:48:21.749 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:48:21.749 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-12T03:48:21.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-12T03:48:21.800 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:21.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:21.865 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:21.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:21.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:21.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:22.010 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:22.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:22.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 9 16 24348 24332 168.957 131.5 0.00737782 0.0059122 2024-03-12T03:48:22.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:22.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:22.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:22.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:48:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:48:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:48:22.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:22.297 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:22.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:22.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:22.369 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:48:22.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:22.434 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:22.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:22.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:22.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:22.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:22.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:22.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:22.720 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:22.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:22.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:22.875 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:22.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:22.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:23.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:23.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:23.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:23.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 10 15 26451 26436 165.21 131.5 0.00975499 0.00604734 2024-03-12T03:48:23.119 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:23.158 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:48:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:48:23.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:23.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:23.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:23.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:23.300 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:23.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:23.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:23.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:23.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:23.537 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:23.580 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:23.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:23.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:23.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:23.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:23.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:23 smithi173 ceph-mon[33544]: pgmap v725: 168 pgs: 168 active+clean; 1.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 147 MiB/s wr, 2.35k op/s 2024-03-12T03:48:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:23 smithi173 ceph-mon[33544]: osdmap e375: 8 total, 8 up, 8 in 2024-03-12T03:48:23.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:23.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:23.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:23.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:23 smithi169 ceph-mon[29344]: pgmap v725: 168 pgs: 168 active+clean; 1.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 147 MiB/s wr, 2.35k op/s 2024-03-12T03:48:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:23 smithi169 ceph-mon[29344]: osdmap e375: 8 total, 8 up, 8 in 2024-03-12T03:48:23.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:23 smithi169 ceph-mon[39949]: pgmap v725: 168 pgs: 168 active+clean; 1.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 147 MiB/s wr, 2.35k op/s 2024-03-12T03:48:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:23 smithi169 ceph-mon[39949]: osdmap e375: 8 total, 8 up, 8 in 2024-03-12T03:48:23.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:23.991 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:23.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:24.048 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 11 15 28467 28452 161.644 126 0.009074 0.00618123 2024-03-12T03:48:24.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:24.131 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:24.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:24.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:24.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:24.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:24.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:24.406 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:24.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:24.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:24.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:24.551 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:24.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:24.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:24.691 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:24.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:24.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:24.831 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:24.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:24.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:24.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:24.934 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:24.973 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:24.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:25.049 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 12 16 30333 30317 157.886 116.562 0.00429747 0.00630775 2024-03-12T03:48:25.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:25.115 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:25.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:25.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:25.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:25.243 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:25.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:25.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:25.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:25.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:25.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:25.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:25.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:25.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:25.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:25.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:25.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:25.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:25 smithi173 ceph-mon[33544]: pgmap v727: 168 pgs: 168 active+clean; 1.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 187 MiB/s wr, 2.99k op/s 2024-03-12T03:48:25.764 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:25.804 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:25.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:25.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:25.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:25.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:25 smithi169 ceph-mon[29344]: pgmap v727: 168 pgs: 168 active+clean; 1.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 187 MiB/s wr, 2.99k op/s 2024-03-12T03:48:25.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:25 smithi169 ceph-mon[39949]: pgmap v727: 168 pgs: 168 active+clean; 1.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 187 MiB/s wr, 2.99k op/s 2024-03-12T03:48:25.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:25.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:26.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:26.049 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 13 16 32107 32091 154.266 110.875 0.00479027 0.00647622 2024-03-12T03:48:26.070 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:26.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:26.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:26.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:26.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:26.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:26.343 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:26.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:26.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:26.484 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:26.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:26.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:26.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:26.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:26.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:26.762 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:26.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:26.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:26.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:26.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:27.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:27.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:27.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:27.049 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 14 16 34045 34029 151.898 121.125 0.00383633 0.00657794 2024-03-12T03:48:27.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:27.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:27.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:27.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:27.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:27.314 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:27.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:27.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:27.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:27.454 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:27.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:27.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:27.598 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:27.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:27.624 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:27 smithi169 ceph-mon[29344]: pgmap v728: 168 pgs: 168 active+clean; 1.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-12T03:48:27.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:27 smithi169 ceph-mon[39949]: pgmap v728: 168 pgs: 168 active+clean; 1.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-12T03:48:27.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:27.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:27 smithi173 ceph-mon[33544]: pgmap v728: 168 pgs: 168 active+clean; 1.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-12T03:48:27.736 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:27.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:27.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:27.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:27.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:27.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:27.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:27.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:28.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:28.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:28.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:28.049 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 15 16 36009 35993 149.954 122.75 0.00545676 0.00666308 2024-03-12T03:48:28.132 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:28.194 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:28.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:28.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:28.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:28.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:28.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:28.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:28.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:28.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:28.528 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211058 8.7 8:efebe503:::benchmark_data_smithi169_106492_object77575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.681215+0000", 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.84002920500001, 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0584678489999999, 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.529 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211058 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681215+0000", 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681215+0000", 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681228+0000", 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3385e-05 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681233+0000", 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5700000000000003e-06 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681451+0000", 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000218364 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.681459+0000", 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5290000000000003e-06 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.683093+0000", 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001634052 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.683194+0000", 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010169400000000001 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.683333+0000", 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013846600000000001 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.688997+0000", 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056637600000000003 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.697167+0000", 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.008170198 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710173+0000", 2024-03-12T03:48:28.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013006172999999999 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710182+0000", 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4239999999999999e-06 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.739635+0000", 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0294528089999999 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.739683+0000", 2024-03-12T03:48:28.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7425000000000001e-05 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211062 8.7 8:e1bda5df:::benchmark_data_smithi169_106492_object77579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.685338+0000", 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.83590625300002, 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.054851787, 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.535 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211062 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685338+0000", 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685338+0000", 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685339+0000", 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3909999999999999e-06 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685345+0000", 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8769999999999998e-06 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685361+0000", 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5602000000000001e-05 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685369+0000", 2024-03-12T03:48:28.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3720000000000005e-06 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686032+0000", 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00066344899999999998 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686176+0000", 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000143435 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686310+0000", 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013403300000000001 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.689003+0000", 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026933310000000002 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.700621+0000", 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011617689 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.539 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710236+0000", 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0096148210000000008 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710244+0000", 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2730000000000002e-06 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740142+0000", 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.029898373 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.540 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740189+0000", 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7141000000000001e-05 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211063 8.7 8:e4c4cdd4:::benchmark_data_smithi169_106492_object77580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.685985+0000", 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.835259296, 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0542865020000001, 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.541 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211063 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685985+0000", 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685985+0000", 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685985+0000", 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4e-07 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.685991+0000", 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0719999999999999e-06 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686001+0000", 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0467e-05 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.686009+0000", 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.503e-06 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.689034+0000", 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003025659 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.689186+0000", 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000152027 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.544 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.689335+0000", 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014890899999999999 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.691991+0000", 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026562019999999999 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.701569+0000", 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0095778640000000002 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.545 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710278+0000", 2024-03-12T03:48:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087084380000000006 2024-03-12T03:48:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710285+0000", 2024-03-12T03:48:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7600000000000002e-06 2024-03-12T03:48:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740215+0000", 2024-03-12T03:48:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0299298880000001 2024-03-12T03:48:28.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740271+0000", 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5872999999999999e-05 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211068 8.7 8:e27717b5:::benchmark_data_smithi169_106492_object77585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.693940+0000", 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.82730352800002, 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0464507999999999, 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:48:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211068 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693940+0000", 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693940+0000", 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693942+0000", 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.567e-06 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693948+0000", 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2949999999999999e-06 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693964+0000", 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5559000000000001e-05 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.693973+0000", 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0480000000000004e-06 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.694033+0000", 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9882e-05 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.694152+0000", 2024-03-12T03:48:28.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011884599999999999 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.694277+0000", 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012499899999999999 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.697558+0000", 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032812539999999999 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.701613+0000", 2024-03-12T03:48:28.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040555299999999999 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710314+0000", 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087006870000000004 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.710320+0000", 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3690000000000003e-06 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740325+0000", 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0300041929999999 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740391+0000", 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6570999999999997e-05 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211087 8.7 8:e3d7750f:::benchmark_data_smithi169_106492_object77604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.732661+0000", 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.78858245999999, 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.015391752, 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.553 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.554 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.554 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211087 2024-03-12T03:48:28.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.554 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732661+0000", 2024-03-12T03:48:28.554 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732661+0000", 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732663+0000", 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6509999999999999e-06 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732670+0000", 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.562e-06 2024-03-12T03:48:28.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732685+0000", 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.543e-05 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732695+0000", 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4369999999999996e-06 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732912+0000", 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000217536 2024-03-12T03:48:28.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.732932+0000", 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9851999999999999e-05 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740436+0000", 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.007504204 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740552+0000", 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115425 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.740681+0000", 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000129512 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.747888+0000", 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072064850000000003 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.747948+0000", 2024-03-12T03:48:28.558 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0829000000000003e-05 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.747962+0000", 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3424999999999999e-05 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748006+0000", 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3905000000000001e-05 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748053+0000", 2024-03-12T03:48:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7499e-05 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211134 8.7 8:eaa19edb:::benchmark_data_smithi169_106492_object77651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.783398+0000", 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.73784583899999, 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96989582799999996, 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.560 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211134 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783398+0000", 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783398+0000", 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783400+0000", 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6080000000000001e-06 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783405+0000", 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3249999999999998e-06 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783420+0000", 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5267000000000001e-05 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783429+0000", 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5690000000000005e-06 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783496+0000", 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7441999999999996e-05 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.783510+0000", 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4178e-05 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.743998+0000", 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96048739500000002 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.744100+0000", 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101648 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.744209+0000", 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010987700000000001 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748122+0000", 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0039126300000000003 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.750470+0000", 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023481650000000001 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753230+0000", 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002759662 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753263+0000", 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3636000000000001e-05 2024-03-12T03:48:28.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753294+0000", 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0426e-05 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211149 8.7 8:e1de3961:::benchmark_data_smithi169_106492_object77666:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.803392+0000", 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.71785158199998, 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.95420632900000002, 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.566 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211149 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803392+0000", 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803392+0000", 2024-03-12T03:48:28.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803394+0000", 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4899999999999999e-06 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803400+0000", 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5250000000000002e-06 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803413+0000", 2024-03-12T03:48:28.568 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2622e-05 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803421+0000", 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9310000000000008e-06 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803476+0000", 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5510999999999998e-05 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.803483+0000", 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6989999999999997e-06 2024-03-12T03:48:28.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748175+0000", 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.94469182799999996 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748276+0000", 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100855 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.748387+0000", 2024-03-12T03:48:28.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011091600000000001 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753338+0000", 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049511879999999996 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753348+0000", 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6150000000000003e-06 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757502+0000", 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004154952 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757557+0000", 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4391000000000002e-05 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757599+0000", 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1805999999999999e-05 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211151 8.7 8:e778f11b:::benchmark_data_smithi169_106492_object77668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.804839+0000", 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.71640531200001, 2024-03-12T03:48:28.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.95295800399999997, 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211151 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804839+0000", 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.573 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804839+0000", 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804839+0000", 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4799999999999997e-07 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804844+0000", 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.899e-06 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804859+0000", 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4983e-05 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804867+0000", 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5610000000000001e-06 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804920+0000", 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2599000000000002e-05 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.804956+0000", 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6235000000000001e-05 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.750501+0000", 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.94554544200000001 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.750600+0000", 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8881000000000004e-05 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.750743+0000", 2024-03-12T03:48:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014310700000000001 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757648+0000", 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069051119999999997 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757696+0000", 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8081999999999998e-05 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757721+0000", 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4800999999999999e-05 2024-03-12T03:48:28.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757762+0000", 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0550999999999999e-05 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757797+0000", 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4903000000000001e-05 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.578 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.579 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211165 8.7 8:e0ccb79f:::benchmark_data_smithi169_106492_object77682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:48:28.579 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.852948+0000", 2024-03-12T03:48:28.579 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.66829562999999, 2024-03-12T03:48:28.579 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.90957021500000002, 2024-03-12T03:48:28.579 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.579 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.579 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.579 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:28.580 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.580 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211165 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852948+0000", 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852948+0000", 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852949+0000", 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0899999999999999e-06 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852955+0000", 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.519e-06 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852968+0000", 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2690999999999999e-05 2024-03-12T03:48:28.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.852976+0000", 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9750000000000007e-06 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.853032+0000", 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6852999999999999e-05 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.853043+0000", 2024-03-12T03:48:28.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0497e-05 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753375+0000", 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.90033189899999999 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753473+0000", 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8413999999999997e-05 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.753540+0000", 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6861999999999998e-05 2024-03-12T03:48:28.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757805+0000", 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042649519999999998 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757850+0000", 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5198999999999999e-05 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762430+0000", 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045799810000000003 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762477+0000", 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7261e-05 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762519+0000", 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1022000000000001e-05 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211182 8.7 8:e254d1c7:::benchmark_data_smithi169_106492_object77699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:48:28.586 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.870732+0000", 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.65051169600002, 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89188010699999998, 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211182 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870732+0000", 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870732+0000", 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870734+0000", 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3179999999999999e-06 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870738+0000", 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9309999999999999e-06 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870750+0000", 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1039999999999999e-05 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870755+0000", 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9070000000000004e-06 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870804+0000", 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8819999999999997e-05 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.870811+0000", 2024-03-12T03:48:28.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1790000000000002e-06 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.755458+0000", 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88464690400000001 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.755563+0000", 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000104603 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.755655+0000", 2024-03-12T03:48:28.590 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2262999999999994e-05 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.760061+0000", 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044056720000000002 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762393+0000", 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023321420000000002 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762551+0000", 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000157558 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.591 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762585+0000", 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4313999999999997e-05 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762612+0000", 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7455999999999999e-05 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.592 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211190 8.7 8:e2ab686f:::benchmark_data_smithi169_106492_object77707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.880970+0000", 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.640273751, 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89412791400000002, 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211190 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880970+0000", 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.593 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880970+0000", 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880972+0000", 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.491e-06 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880978+0000", 2024-03-12T03:48:28.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4860000000000002e-06 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.880996+0000", 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7357999999999999e-05 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.881005+0000", 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1230000000000006e-06 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.881061+0000", 2024-03-12T03:48:28.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6594000000000001e-05 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.881071+0000", 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0067e-05 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757895+0000", 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.87682368799999999 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.757991+0000", 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6284000000000002e-05 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.758109+0000", 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117265 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762403+0000", 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042943770000000003 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762648+0000", 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00024558700000000002 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.598 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.775012+0000", 2024-03-12T03:48:28.598 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012363733 2024-03-12T03:48:28.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.598 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.598 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.598 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.775060+0000", 2024-03-12T03:48:28.598 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7663000000000001e-05 2024-03-12T03:48:28.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.598 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.598 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.598 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.775098+0000", 2024-03-12T03:48:28.598 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8198000000000003e-05 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211197 8.7 8:e6e0739f:::benchmark_data_smithi169_106492_object77714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.890205+0000", 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.63103894, 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88828141800000004, 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.599 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211197 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890205+0000", 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890205+0000", 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890206+0000", 2024-03-12T03:48:28.600 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.61e-07 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890209+0000", 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7299999999999999e-06 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890216+0000", 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8040000000000004e-06 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890221+0000", 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9200000000000003e-06 2024-03-12T03:48:28.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890270+0000", 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8677999999999997e-05 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.890278+0000", 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1249999999999993e-06 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.760085+0000", 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.869807471 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.760173+0000", 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7177999999999998e-05 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.760276+0000", 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010333299999999999 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762620+0000", 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023441180000000001 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.766983+0000", 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043625510000000001 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778399+0000", 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011416608999999999 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778446+0000", 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6796999999999999e-05 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.605 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778486+0000", 2024-03-12T03:48:28.605 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0442999999999999e-05 2024-03-12T03:48:28.605 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.605 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.605 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.605 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.605 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211199 8.7 8:e601f7fc:::benchmark_data_smithi169_106492_object77716:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:48:28.605 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.893694+0000", 2024-03-12T03:48:28.605 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.62755028200002, 2024-03-12T03:48:28.606 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88494738100000003, 2024-03-12T03:48:28.606 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.606 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.606 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.606 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.606 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211199 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893694+0000", 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893694+0000", 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893695+0000", 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.173e-06 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893700+0000", 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3380000000000004e-06 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893713+0000", 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2649e-05 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893721+0000", 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7689999999999996e-06 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893767+0000", 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5949999999999999e-05 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.893776+0000", 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0680000000000003e-06 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762678+0000", 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.86890198600000002 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762763+0000", 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5915999999999995e-05 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.762862+0000", 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8690999999999994e-05 2024-03-12T03:48:28.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.767024+0000", 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041618280000000002 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771291+0000", 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042668070000000001 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778547+0000", 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072558479999999996 2024-03-12T03:48:28.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778601+0000", 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4703000000000001e-05 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778641+0000", 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9654999999999999e-05 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.612 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211211 8.7 8:e437348b:::benchmark_data_smithi169_106492_object77728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.933217+0000", 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.58802695999998, 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.84557843200000005, 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211211 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933217+0000", 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933217+0000", 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933218+0000", 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1030000000000001e-06 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933223+0000", 2024-03-12T03:48:28.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6909999999999996e-06 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933232+0000", 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8740000000000001e-06 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933237+0000", 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6790000000000002e-06 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933283+0000", 2024-03-12T03:48:28.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5688e-05 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.933290+0000", 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.007e-06 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.764859+0000", 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.83156881400000005 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.764956+0000", 2024-03-12T03:48:28.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7408e-05 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.765060+0000", 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103928 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771299+0000", 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006238609 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771351+0000", 2024-03-12T03:48:28.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.257e-05 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778700+0000", 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0073482479999999999 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778758+0000", 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8440999999999998e-05 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778795+0000", 2024-03-12T03:48:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7372000000000001e-05 2024-03-12T03:48:28.619 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.619 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.619 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.619 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211219 8.7 8:ec912678:::benchmark_data_smithi169_106492_object77736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:48:28.619 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.959590+0000", 2024-03-12T03:48:28.619 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.56165351300001, 2024-03-12T03:48:28.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81933082300000004, 2024-03-12T03:48:28.619 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211219 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959590+0000", 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.620 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959590+0000", 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959592+0000", 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.156e-06 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959596+0000", 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6450000000000004e-06 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959608+0000", 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1902000000000001e-05 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959616+0000", 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0779999999999996e-06 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959674+0000", 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7806000000000002e-05 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.959685+0000", 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0841e-05 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.767051+0000", 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80736609000000004 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.767150+0000", 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9516000000000001e-05 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.767246+0000", 2024-03-12T03:48:28.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5678000000000001e-05 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771306+0000", 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040594740000000004 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.775173+0000", 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038675329999999998 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778846+0000", 2024-03-12T03:48:28.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036730550000000002 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778887+0000", 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1253e-05 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778921+0000", 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3796e-05 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24914.0:211222 8.7 8:e61e560a:::benchmark_data_smithi169_106492_object77739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:42:29.969003+0000", 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.55224067300003, 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81401453999999995, 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24914", 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 211222 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969003+0000", 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969003+0000", 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969005+0000", 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2640000000000001e-06 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969009+0000", 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3549999999999998e-06 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969019+0000", 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0091e-05 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969026+0000", 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4869999999999996e-06 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969072+0000", 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5597999999999999e-05 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:29.969078+0000", 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.003e-06 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.769181+0000", 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80010249300000003 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.769269+0000", 2024-03-12T03:48:28.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8134999999999996e-05 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.769375+0000", 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000106086 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.771421+0000", 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0020459520000000002 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.778959+0000", 2024-03-12T03:48:28.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075385310000000002 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.782940+0000", 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003980946 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.782981+0000", 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1287e-05 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:42:30.783018+0000", 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6312000000000001e-05 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.631 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.632 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:203860 9.0 9:011d0815:::benchmark_data_smithi169_135229_object59645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:46:13.367720+0000", 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 135.15352354300001, 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.146631541, 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 203860 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.367720+0000", 2024-03-12T03:48:28.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.367720+0000", 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.367721+0000", 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8199999999999998e-07 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.367726+0000", 2024-03-12T03:48:28.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8099999999999997e-06 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.367736+0000", 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0175e-05 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.367744+0000", 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2590000000000003e-06 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.367796+0000", 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2085e-05 2024-03-12T03:48:28.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.367914+0000", 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000118347 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2,6", 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.368031+0000", 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116983 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.450053+0000", 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.082022164999999994 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.455679+0000", 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056257809999999998 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.514265+0000", 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.058586472000000001 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.514297+0000", 2024-03-12T03:48:28.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1637000000000002e-05 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:13.514352+0000", 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4945000000000003e-05 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:208263 9.0 9:087f3375:::benchmark_data_smithi169_135229_object64048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:46:17.101766+0000", 2024-03-12T03:48:28.638 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 131.41947838799999, 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.147881702, 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 208263 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.101766+0000", 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.101766+0000", 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.101767+0000", 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3009999999999999e-06 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.101772+0000", 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.04e-06 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.101787+0000", 2024-03-12T03:48:28.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5418e-05 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.101796+0000", 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6670000000000002e-06 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.159315+0000", 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.057519176999999998 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.159394+0000", 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8621999999999994e-05 2024-03-12T03:48:28.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2,6", 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.159468+0000", 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4612999999999999e-05 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.162162+0000", 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026939360000000001 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.165642+0000", 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00347985 2024-03-12T03:48:28.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.249553+0000", 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.083910544000000004 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.249588+0000", 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5373000000000002e-05 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:17.249647+0000", 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9160999999999998e-05 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.643 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:29254 10.2 10:482cf204:::benchmark_data_smithi169_162694_object29253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:48:24.393291+0000", 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.1279528110000001, 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.080492948999999994, 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 29254 2024-03-12T03:48:28.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.393291+0000", 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.393291+0000", 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.393293+0000", 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6199999999999999e-06 2024-03-12T03:48:28.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.393327+0000", 2024-03-12T03:48:28.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.413e-05 2024-03-12T03:48:28.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.393349+0000", 2024-03-12T03:48:28.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1963e-05 2024-03-12T03:48:28.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.393358+0000", 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1300000000000007e-06 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.393430+0000", 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1804999999999994e-05 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.393560+0000", 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013036999999999999 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,4", 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.393691+0000", 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013082299999999999 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.395173+0000", 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001482108 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.395547+0000", 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037413399999999998 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.473420+0000", 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.077872565000000005 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.473734+0000", 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031409 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:24.473784+0000", 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0210999999999997e-05 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.649 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:31202 10.2 10:4cc0de60:::benchmark_data_smithi169_162694_object31201:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:48:25.523215+0000", 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 2.9980291019999998, 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.070821462000000002, 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 31202 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.650 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.523215+0000", 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.523215+0000", 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.523216+0000", 2024-03-12T03:48:28.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5e-06 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.523268+0000", 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1724000000000001e-05 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.523279+0000", 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0929e-05 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.523287+0000", 2024-03-12T03:48:28.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8280000000000003e-06 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.523344+0000", 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7266999999999999e-05 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.523478+0000", 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000134256 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,4", 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.523612+0000", 2024-03-12T03:48:28.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013319699999999999 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.525000+0000", 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013883439999999999 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.526460+0000", 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001460528 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.593941+0000", 2024-03-12T03:48:28.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.067480377999999994 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.593996+0000", 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5081000000000003e-05 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:25.594036+0000", 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0429999999999997e-05 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:28.655 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:28.656 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:48:28.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:28.656 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:28.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:28.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:28.807 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:28.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:28.909 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:28.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:29.049 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 16 16 37589 37573 146.753 98.75 0.0151287 0.00680874 2024-03-12T03:48:29.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:29.099 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:29.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:29.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:29.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:29.245 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:29.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:29.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:29.380 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:29.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:29.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:29.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:29.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:29.620 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:29.708 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:29.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:29.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:29 smithi173 ceph-mon[33544]: pgmap v729: 168 pgs: 168 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 149 MiB/s wr, 2.38k op/s 2024-03-12T03:48:29.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:29.818 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:29 smithi169 ceph-mon[29344]: pgmap v729: 168 pgs: 168 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 149 MiB/s wr, 2.38k op/s 2024-03-12T03:48:29.818 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:29 smithi169 ceph-mon[39949]: pgmap v729: 168 pgs: 168 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 149 MiB/s wr, 2.38k op/s 2024-03-12T03:48:29.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:29.856 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:29.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:29.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:29.997 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:29.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:30.050 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 17 15 39323 39308 144.499 108.438 0.00626055 0.00691503 2024-03-12T03:48:30.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:30.140 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:30.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:30.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:30.258 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:30.258 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:30.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:30.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:30.318 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-12T03:48:30.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:30.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:30.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:30.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:30.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:30.580 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:30.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:30.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:30.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:30.730 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:30.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:30.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:30.876 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:30.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:30 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:30 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:30 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:30 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:30.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:31.024 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:31.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:31.049 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 18 15 40905 40890 141.964 98.875 0.00924801 0.00703846 2024-03-12T03:48:31.056 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:48:31.103 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:48:31.126 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:31.153 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:48:31.164 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:31.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:31.195 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:48:31.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:30 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:31.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:30 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:31.245 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:48:31.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:31.291 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:48:31.307 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:31.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:31.333 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:48:31.372 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:48:31.376 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:48:31.377 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:48:31.377 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:48:31.377 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:48:31.377 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:48:31.377 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:48:31.377 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:48:31.377 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:48:31.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:31.450 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:31.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:31.552 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:31.590 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:31.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:31.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:31.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:31.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:31.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:31.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:31.882 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:31.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:31.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:32.030 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:32.031 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:31 smithi169 ceph-mon[29344]: pgmap v730: 168 pgs: 168 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-12T03:48:32.031 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:31 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/149361253' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:48:32.031 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:31 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:32.031 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:32.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:32.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:32.032 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:48:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:48:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:48:32.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:31 smithi169 ceph-mon[39949]: pgmap v730: 168 pgs: 168 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-12T03:48:32.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:31 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/149361253' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:48:32.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:31 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:32.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:32.049 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 19 16 42502 42486 139.742 99.75 0.00468228 0.00715091 2024-03-12T03:48:32.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:32.183 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:32.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:32.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:31 smithi173 ceph-mon[33544]: pgmap v730: 168 pgs: 168 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-12T03:48:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:31 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/149361253' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:48:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:31 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:32.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:32.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:32.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:32.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:32.427 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:32.466 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:32.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:32.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:32.601 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:32.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:32.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:32.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:32.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:32.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:32.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:32.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:32.880 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:32.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:32.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:33.021 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:33.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:33.042 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:33.050 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:48:33.048712+0000 min lat: 0.00147924 max lat: 0.0880292 avg lat: 0.00721733 2024-03-12T03:48:33.050 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:48:33.050 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 20 16 44322 44306 138.441 113.75 0.0124031 0.00721733 2024-03-12T03:48:33.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:33.159 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:33.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:33.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:33.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:33.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:33.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:33.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:33.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:33.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:33.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:48:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:48:33.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:33.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:33.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:33.664 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:33.710 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:33.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:33.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:33.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:33.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:33.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:33.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:33.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:33.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:33.986 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:33 smithi169 ceph-mon[29344]: pgmap v731: 168 pgs: 168 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-12T03:48:33.986 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:33 smithi169 ceph-mon[39949]: pgmap v731: 168 pgs: 168 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-12T03:48:34.050 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 21 16 46040 46024 136.962 107.375 0.0064319 0.00729632 2024-03-12T03:48:34.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:34.136 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:34.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:34.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:33 smithi173 ceph-mon[33544]: pgmap v731: 168 pgs: 168 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-12T03:48:34.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:34.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:34.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:34.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:34.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:34.410 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:34.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:34.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:34.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:34.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:34.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:34.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:34.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:34.703 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:34.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:34.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:34.833 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:34.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:34.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:34.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:34.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:34.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:34.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:35.050 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 22 16 47704 47688 135.463 104 0.011208 0.00737769 2024-03-12T03:48:35.071 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:35.072 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:48:35.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:35.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:34 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:48:35.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:34 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:48:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:35.116 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:35.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:35.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:35.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:34 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:48:35.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:34 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:48:35.168 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:48:35.169 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:48:35.169 INFO:teuthology.orchestra.run.smithi169.stdout: "epoch": 376, 2024-03-12T03:48:35.169 INFO:teuthology.orchestra.run.smithi169.stdout: "fsid": "677d624a-e020-11ee-95c8-87774f69a715", 2024-03-12T03:48:35.169 INFO:teuthology.orchestra.run.smithi169.stdout: "created": "2024-03-12T03:27:53.319986+0000", 2024-03-12T03:48:35.169 INFO:teuthology.orchestra.run.smithi169.stdout: "modified": "2024-03-12T03:48:34.783964+0000", 2024-03-12T03:48:35.169 INFO:teuthology.orchestra.run.smithi169.stdout: "last_up_change": "2024-03-12T03:46:24.244643+0000", 2024-03-12T03:48:35.169 INFO:teuthology.orchestra.run.smithi169.stdout: "last_in_change": "2024-03-12T03:39:32.320120+0000", 2024-03-12T03:48:35.169 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_num": 5801984, 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_set": [ 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: "noscrub", 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: "pglog_hardlimit", 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: "purged_snapdirs", 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_deletes", 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: "sortbitwise" 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_version": 20, 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: "full_ratio": 0.94999998807907104, 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-12T03:48:35.170 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_snapshot": "", 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_max": 10, 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "max_osd": 8, 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "require_min_compat_client": "luminous", 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "min_compat_client": "luminous", 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "require_osd_release": "squid", 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "allow_crimson": false, 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "pools": [ 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 1, 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": ".mgr", 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:31:04.863493+0000", 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:48:35.171 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 21, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 21, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 21, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 21, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 21, 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:48:35.172 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "293", 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "291", 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:48:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:48:35.174 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:48:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: "options": { 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_max": 32, 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_min": 1 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr": {} 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 2.0099999904632568, 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 2.0099999904632568, 2024-03-12T03:48:35.176 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1.1399999856948853, 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 2.2899999618530273, 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 2.2899999618530273, 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.875, 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 2, 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "datapool", 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:18.983144+0000", 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 8193, 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:48:35.177 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 3, 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 3, 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 3, 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 3, 2024-03-12T03:48:35.178 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 3, 2024-03-12T03:48:35.179 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:48:35.179 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:48:35.179 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:48:35.179 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:48:35.179 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:48:35.179 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:48:35.179 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "70", 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 2, 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 70, 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:48:35.180 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:48:35.181 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd": {} 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:48:35.182 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 2.3299999237060547, 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 2.3299999237060547, 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1, 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.875, 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 3, 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": ".rgw.root", 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:19.343301+0000", 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:48:35.183 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:48:35.184 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:48:35.184 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:48:35.184 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:48:35.184 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:48:35.184 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:48:35.184 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:48:35.184 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:48:35.184 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:48:35.184 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:48:35.184 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 32, 2024-03-12T03:48:35.184 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 32, 2024-03-12T03:48:35.185 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 32, 2024-03-12T03:48:35.185 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 32, 2024-03-12T03:48:35.185 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 32, 2024-03-12T03:48:35.185 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:48:35.185 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:48:35.185 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:48:35.185 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:48:35.185 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:48:35.185 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:48:35.186 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:48:35.186 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.186 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "68", 2024-03-12T03:48:35.186 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:48:35.186 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:48:35.186 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:48:35.187 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:48:35.187 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:48:35.187 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:48:35.187 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:48:35.187 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:48:35.187 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:48:35.187 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:48:35.187 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:48:35.187 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:48:35.187 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:48:35.187 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:48:35.187 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:48:35.188 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw": {} 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:48:35.189 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 1.5399999618530273, 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 1.5399999618530273, 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1.1399999856948853, 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.75, 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.75, 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.875, 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 4, 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "default.rgw.log", 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:21.864621+0000", 2024-03-12T03:48:35.190 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 32, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 32, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 32, 2024-03-12T03:48:35.191 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 32, 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 32, 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "70", 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:48:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:48:35.193 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:48:35.194 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw": {} 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 2.190000057220459, 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 2.190000057220459, 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1.1399999856948853, 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 2.5, 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 2.5, 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.875, 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:48:35.195 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 5, 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "default.rgw.control", 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:23.416265+0000", 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:48:35.196 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 32, 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 32, 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 32, 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 32, 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 32, 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:48:35.197 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "72", 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:48:35.198 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.199 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw": {} 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.200 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 1.3200000524520874, 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 1.3200000524520874, 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1.1399999856948853, 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.5, 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.5, 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.875, 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 6, 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "default.rgw.meta", 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:25.476511+0000", 2024-03-12T03:48:35.201 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 32, 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 32, 2024-03-12T03:48:35.202 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 32, 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 32, 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 32, 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "74", 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:48:35.203 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:48:35.204 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:48:35.205 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: "options": { 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_bias": 4 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw": {} 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 1.5399999618530273, 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 1.5399999618530273, 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1.1399999856948853, 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.75, 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.75, 2024-03-12T03:48:35.206 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.875, 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 10, 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "unique_pool_3", 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:47:56.631010+0000", 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:48:35.207 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 16, 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 16, 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 16, 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 16, 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 16, 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:48:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "372", 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:48:35.209 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:48:35.210 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: "rados": {} 2024-03-12T03:48:35.211 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 1.3200000524520874, 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 1.3200000524520874, 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1.1399999856948853, 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.5, 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.5, 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.875, 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 1, 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 0, 2024-03-12T03:48:35.212 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "c44466b0-3112-44dc-b5f3-d6494b958761", 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 8, 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 366, 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6801", 2024-03-12T03:48:35.213 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 676779916 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6802", 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 676779916 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.214 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6804", 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 676779916 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6803", 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 676779916 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.215 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.216 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.169:6801/676779916", 2024-03-12T03:48:35.216 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.169:6802/676779916", 2024-03-12T03:48:35.216 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.169:6804/676779916", 2024-03-12T03:48:35.216 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.169:6803/676779916", 2024-03-12T03:48:35.216 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:48:35.216 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:48:35.216 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:48:35.216 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.216 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.216 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.216 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 1, 2024-03-12T03:48:35.217 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "184c5c97-a992-4be9-bcd3-c08cb58a0e56", 2024-03-12T03:48:35.217 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:48:35.217 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:48:35.217 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:48:35.217 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:48:35.217 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:48:35.217 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:48:35.217 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 13, 2024-03-12T03:48:35.217 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 366, 2024-03-12T03:48:35.217 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:48:35.217 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:48:35.217 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6805", 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 130042595 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6806", 2024-03-12T03:48:35.218 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 130042595 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6808", 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 130042595 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.219 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6807", 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 130042595 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.169:6805/130042595", 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.169:6806/130042595", 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.169:6808/130042595", 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.169:6807/130042595", 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.220 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 2, 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "beb15e06-1ca4-46a7-b640-efd1991367e8", 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 18, 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 366, 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.221 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6809", 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2751816702 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6810", 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2751816702 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:48:35.222 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6812", 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2751816702 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6811", 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2751816702 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.223 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.169:6809/2751816702", 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.169:6810/2751816702", 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.169:6812/2751816702", 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.169:6811/2751816702", 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 3, 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "a00b4c50-e064-4c9e-85dc-b0a75ea5bf37", 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:48:35.224 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 222, 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 264, 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 330, 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 366, 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 265, 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6813", 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3056769071 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.225 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6814", 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3056769071 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6816", 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3056769071 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6815", 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3056769071 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.227 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.169:6813/3056769071", 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.169:6814/3056769071", 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.169:6816/3056769071", 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.169:6815/3056769071", 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 4, 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "3a6121ef-6309-41b7-a964-6772c010b3a0", 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:48:35.228 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 32, 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 366, 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6800", 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1397212305 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.229 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.230 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:48:35.230 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.230 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.230 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.230 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6801", 2024-03-12T03:48:35.230 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1397212305 2024-03-12T03:48:35.230 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.230 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.230 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.230 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6803", 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1397212305 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6802", 2024-03-12T03:48:35.231 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1397212305 2024-03-12T03:48:35.232 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.232 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.232 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.232 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.173:6800/1397212305", 2024-03-12T03:48:35.232 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.173:6801/1397212305", 2024-03-12T03:48:35.232 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.173:6803/1397212305", 2024-03-12T03:48:35.232 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.173:6802/1397212305", 2024-03-12T03:48:35.232 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:48:35.232 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:48:35.232 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:48:35.232 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.232 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 5, 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "3c2f6dd9-9784-4a07-8710-55ac3de82964", 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 41, 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 366, 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:48:35.233 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6804", 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3525888417 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6805", 2024-03-12T03:48:35.234 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3525888417 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6807", 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3525888417 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.235 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:48:35.236 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.236 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.236 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.236 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6806", 2024-03-12T03:48:35.236 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3525888417 2024-03-12T03:48:35.236 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.236 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.236 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.237 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.173:6804/3525888417", 2024-03-12T03:48:35.237 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.173:6805/3525888417", 2024-03-12T03:48:35.237 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.173:6807/3525888417", 2024-03-12T03:48:35.237 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.173:6806/3525888417", 2024-03-12T03:48:35.237 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:48:35.237 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:48:35.237 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:48:35.237 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.237 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.237 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 6, 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "d7759df7-1978-4d70-94a2-721540804f59", 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 0, 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 51, 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 345, 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:48:35.238 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6808", 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2147192629 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6809", 2024-03-12T03:48:35.239 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2147192629 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6811", 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2147192629 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:48:35.240 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6810", 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2147192629 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.173:6808/2147192629", 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.173:6809/2147192629", 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.173:6811/2147192629", 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.173:6810/2147192629", 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:48:35.241 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 7, 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "8843039e-2b17-4230-9cca-5788a2294d8c", 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 59, 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 366, 2024-03-12T03:48:35.242 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:48:35.243 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:48:35.243 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:48:35.243 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.243 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.243 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.243 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6812", 2024-03-12T03:48:35.243 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 4014976876 2024-03-12T03:48:35.243 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.243 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6813", 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 4014976876 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.244 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6815", 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 4014976876 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6814", 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 4014976876 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.245 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.173:6812/4014976876", 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.173:6813/4014976876", 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.173:6815/4014976876", 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.173:6814/4014976876", 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_xinfo": [ 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 0, 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:48:35.246 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:47:23.142986+0000", 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 1, 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:48:34.070089+0000", 2024-03-12T03:48:35.247 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 2, 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:47:21.998626+0000", 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 3, 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:48:35.248 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:47:24.657624+0000", 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 264 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 4, 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:47:42.293309+0000", 2024-03-12T03:48:35.249 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 5, 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:47:24.205126+0000", 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 6, 2024-03-12T03:48:35.250 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:47:45.138670+0000", 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 7, 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:48:35.251 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:47:57.003850+0000", 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_upmap": [ 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "3.8", 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: 4, 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: 5, 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: 0 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_upmap_items": [ 2024-03-12T03:48:35.252 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "4.4", 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: "mappings": [ 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: "from": 1, 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: "to": 5 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "4.12", 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: "mappings": [ 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: "from": 1, 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: "to": 5 2024-03-12T03:48:35.253 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "4.14", 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: "mappings": [ 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: "from": 1, 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: "to": 2 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_upmap_primaries": [], 2024-03-12T03:48:35.254 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_temp": [], 2024-03-12T03:48:35.255 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_temp": [], 2024-03-12T03:48:35.255 INFO:teuthology.orchestra.run.smithi169.stdout: "blocklist": { 2024-03-12T03:48:35.255 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/4041747161": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:48:35.255 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:6800/2779395456": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:48:35.255 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:35.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:35.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:34 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:48:35.256 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:34 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:48:35.256 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/555866474": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:48:35.256 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/342292643": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:48:35.256 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/197210445": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:48:35.256 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/1430250501": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/3918553041": "2024-03-13T03:28:48.518732+0000", 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/1020514981": "2024-03-13T03:28:21.868923+0000", 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/284191239": "2024-03-13T03:28:21.868923+0000", 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/1212459137": "2024-03-13T03:28:48.518732+0000", 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:6800/204892396": "2024-03-13T03:28:21.868923+0000", 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/2751007072": "2024-03-13T03:28:48.518732+0000", 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/2349376510": "2024-03-13T03:28:21.868923+0000", 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:6800/108891420": "2024-03-13T03:28:48.518732+0000" 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "range_blocklist": {}, 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profiles": { 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "default": { 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "crush-failure-domain": "osd", 2024-03-12T03:48:35.257 INFO:teuthology.orchestra.run.smithi169.stdout: "k": "2", 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "m": "1", 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "plugin": "jerasure", 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "technique": "reed_sol_van" 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps_queue": [], 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "new_removed_snaps": [], 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "new_purged_snaps": [], 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_node_flags": {}, 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "device_class_flags": {}, 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "stretch_mode": { 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "stretch_mode_enabled": false, 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "stretch_bucket_count": 0, 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "degraded_stretch_mode": 0, 2024-03-12T03:48:35.258 INFO:teuthology.orchestra.run.smithi169.stdout: "recovering_stretch_mode": 0, 2024-03-12T03:48:35.259 INFO:teuthology.orchestra.run.smithi169.stdout: "stretch_mode_bucket": 0 2024-03-12T03:48:35.259 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:48:35.259 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:48:35.303 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:35.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:35.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:35.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:35.451 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:35.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:35.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:35.594 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:35.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:35.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:35.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:35.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:35.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:35.867 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:35.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:35.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:36.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:36.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:36.050 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 23 16 49269 49253 133.825 97.8125 0.0112644 0.00746104 2024-03-12T03:48:36.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:36.143 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:36.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:36.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:36.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:35 smithi169 ceph-mon[29344]: pgmap v732: 168 pgs: 168 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 959 B/s rd, 115 MiB/s wr, 1.85k op/s 2024-03-12T03:48:36.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:35 smithi169 ceph-mon[29344]: osdmap e376: 8 total, 8 up, 8 in 2024-03-12T03:48:36.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:35 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2999162205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-12T03:48:36.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:35 smithi169 ceph-mon[39949]: pgmap v732: 168 pgs: 168 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 959 B/s rd, 115 MiB/s wr, 1.85k op/s 2024-03-12T03:48:36.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:35 smithi169 ceph-mon[39949]: osdmap e376: 8 total, 8 up, 8 in 2024-03-12T03:48:36.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:35 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2999162205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-12T03:48:36.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:35 smithi173 ceph-mon[33544]: pgmap v732: 168 pgs: 168 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 959 B/s rd, 115 MiB/s wr, 1.85k op/s 2024-03-12T03:48:36.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:35 smithi173 ceph-mon[33544]: osdmap e376: 8 total, 8 up, 8 in 2024-03-12T03:48:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:35 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2999162205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-12T03:48:36.245 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:36.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:36.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:36.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:36.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:36.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:36.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:36.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:36.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:36.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:36.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:36.694 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:36.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:36.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:36.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:36.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:36.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:36.965 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:36.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:37.050 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 24 16 50919 50903 132.546 103.125 0.0078783 0.00753957 2024-03-12T03:48:37.067 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:37.106 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:37.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:37.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:37.208 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:37.247 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:37.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:37.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:37.380 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:37.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:37.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:37.521 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:37.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:37.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:37.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:37.660 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:37.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:37.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:37.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:37.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:37.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:37.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:37.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:37.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:37.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:38.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:38.050 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 25 16 52595 52579 131.434 104.75 0.00767003 0.00760302 2024-03-12T03:48:38.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:38.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:38.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:37 smithi169 ceph-mon[29344]: pgmap v734: 168 pgs: 168 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-12T03:48:38.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:37 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:48:38.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:37 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:48:38.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:37 smithi169 ceph-mon[39949]: pgmap v734: 168 pgs: 168 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-12T03:48:38.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:37 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:48:38.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:37 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:48:38.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:38.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:38.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:38.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:37 smithi173 ceph-mon[33544]: pgmap v734: 168 pgs: 168 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-12T03:48:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:37 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:48:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:37 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:48:38.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:38.355 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:38.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:38.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:38.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:38.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:38.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:38.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:38.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:38.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:38.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:38.730 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:38.774 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:38.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:38.875 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:38.923 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:38.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:38.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:39.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:39.027 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:38 smithi169 ceph-mon[29344]: osdmap e377: 8 total, 8 up, 8 in 2024-03-12T03:48:39.027 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:38 smithi169 ceph-mon[39949]: osdmap e377: 8 total, 8 up, 8 in 2024-03-12T03:48:39.030 INFO:tasks.thrashosds.thrasher:j is {'epoch': 376, 'fsid': '677d624a-e020-11ee-95c8-87774f69a715', 'created': '2024-03-12T03:27:53.319986+0000', 'modified': '2024-03-12T03:48:34.783964+0000', 'last_up_change': '2024-03-12T03:46:24.244643+0000', 'last_in_change': '2024-03-12T03:39:32.320120+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T03:31:04.863493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '293', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '291', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.009999990463257, 'score_stable': 2.009999990463257, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.2899999618530273, 'raw_score_stable': 2.2899999618530273, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-12T03:33:18.983144+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-12T03:33:19.343301+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-12T03:33:21.864621+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-12T03:33:23.416265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-12T03:33:25.476511+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-12T03:47:56.631010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '372', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'c44466b0-3112-44dc-b5f3-d6494b958761', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6801', 'nonce': 676779916}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6802', 'nonce': 676779916}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6804', 'nonce': 676779916}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6803', 'nonce': 676779916}]}, 'public_addr': '172.21.15.169:6801/676779916', 'cluster_addr': '172.21.15.169:6802/676779916', 'heartbeat_back_addr': '172.21.15.169:6804/676779916', 'heartbeat_front_addr': '172.21.15.169:6803/676779916', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '184c5c97-a992-4be9-bcd3-c08cb58a0e56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6805', 'nonce': 130042595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6806', 'nonce': 130042595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6808', 'nonce': 130042595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6807', 'nonce': 130042595}]}, 'public_addr': '172.21.15.169:6805/130042595', 'cluster_addr': '172.21.15.169:6806/130042595', 'heartbeat_back_addr': '172.21.15.169:6808/130042595', 'heartbeat_front_addr': '172.21.15.169:6807/130042595', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'beb15e06-1ca4-46a7-b640-efd1991367e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6809', 'nonce': 2751816702}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6810', 'nonce': 2751816702}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6812', 'nonce': 2751816702}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6811', 'nonce': 2751816702}]}, 'public_addr': '172.21.15.169:6809/2751816702', 'cluster_addr': '172.21.15.169:6810/2751816702', 'heartbeat_back_addr': '172.21.15.169:6812/2751816702', 'heartbeat_front_addr': '172.21.15.169:6811/2751816702', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a00b4c50-e064-4c9e-85dc-b0a75ea5bf37', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 222, 'last_clean_end': 264, 'up_from': 330, 'up_thru': 366, 'down_at': 265, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6813', 'nonce': 3056769071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6814', 'nonce': 3056769071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6816', 'nonce': 3056769071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6815', 'nonce': 3056769071}]}, 'public_addr': '172.21.15.169:6813/3056769071', 'cluster_addr': '172.21.15.169:6814/3056769071', 'heartbeat_back_addr': '172.21.15.169:6816/3056769071', 'heartbeat_front_addr': '172.21.15.169:6815/3056769071', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '3a6121ef-6309-41b7-a964-6772c010b3a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 1397212305}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6801', 'nonce': 1397212305}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6803', 'nonce': 1397212305}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 1397212305}]}, 'public_addr': '172.21.15.173:6800/1397212305', 'cluster_addr': '172.21.15.173:6801/1397212305', 'heartbeat_back_addr': '172.21.15.173:6803/1397212305', 'heartbeat_front_addr': '172.21.15.173:6802/1397212305', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '3c2f6dd9-9784-4a07-8710-55ac3de82964', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 3525888417}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6805', 'nonce': 3525888417}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6807', 'nonce': 3525888417}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 3525888417}]}, 'public_addr': '172.21.15.173:6804/3525888417', 'cluster_addr': '172.21.15.173:6805/3525888417', 'heartbeat_back_addr': '172.21.15.173:6807/3525888417', 'heartbeat_front_addr': '172.21.15.173:6806/3525888417', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd7759df7-1978-4d70-94a2-721540804f59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 345, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 2147192629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6809', 'nonce': 2147192629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6811', 'nonce': 2147192629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 2147192629}]}, 'public_addr': '172.21.15.173:6808/2147192629', 'cluster_addr': '172.21.15.173:6809/2147192629', 'heartbeat_back_addr': '172.21.15.173:6811/2147192629', 'heartbeat_front_addr': '172.21.15.173:6810/2147192629', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '8843039e-2b17-4230-9cca-5788a2294d8c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 4014976876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6813', 'nonce': 4014976876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6815', 'nonce': 4014976876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 4014976876}]}, 'public_addr': '172.21.15.173:6812/4014976876', 'cluster_addr': '172.21.15.173:6813/4014976876', 'heartbeat_back_addr': '172.21.15.173:6815/4014976876', 'heartbeat_front_addr': '172.21.15.173:6814/4014976876', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:47:23.142986+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:48:34.070089+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:47:21.998626+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:47:24.657624+0000', 'dead_epoch': 264}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:47:42.293309+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:47:24.205126+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:47:45.138670+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:47:57.003850+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.8', 'osds': [4, 5, 0]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.169:0/4041747161': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:6800/2779395456': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/555866474': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/342292643': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/197210445': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/1430250501': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/3918553041': '2024-03-13T03:28:48.518732+0000', '172.21.15.169:0/1020514981': '2024-03-13T03:28:21.868923+0000', '172.21.15.169:0/284191239': '2024-03-13T03:28:21.868923+0000', '172.21.15.169:0/1212459137': '2024-03-13T03:28:48.518732+0000', '172.21.15.169:6800/204892396': '2024-03-13T03:28:21.868923+0000', '172.21.15.169:0/2751007072': '2024-03-13T03:28:48.518732+0000', '172.21.15.169:0/2349376510': '2024-03-13T03:28:21.868923+0000', '172.21.15.169:6800/108891420': '2024-03-13T03:28:48.518732+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-12T03:48:39.030 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.4 2024-03-12T03:48:39.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd rm-pg-upmap-items 4.4 2024-03-12T03:48:39.051 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 26 16 54242 54226 130.332 102.938 0.0139318 0.00766726 2024-03-12T03:48:39.067 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:39.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:39.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:39.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:39.221 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:39.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:38 smithi173 ceph-mon[33544]: osdmap e377: 8 total, 8 up, 8 in 2024-03-12T03:48:39.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:39.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:39.355 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:39.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:39.456 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:39.488 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:39.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:39.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:39.628 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:39.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:39.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:39.770 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:39.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:39.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:39.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:39.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:39.923 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:39.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:40.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:40.052 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 27 15 55724 55709 128.935 92.6875 0.013862 0.00775083 2024-03-12T03:48:40.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:40.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:40.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:40.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:39 smithi169 ceph-mon[29344]: pgmap v736: 168 pgs: 168 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-12T03:48:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:39 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:48:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:39 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:48:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:39 smithi169 ceph-mon[39949]: pgmap v736: 168 pgs: 168 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-12T03:48:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:39 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:48:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:39 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:48:40.204 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:40.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:40.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:40.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:39 smithi173 ceph-mon[33544]: pgmap v736: 168 pgs: 168 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-12T03:48:40.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:39 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:48:40.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:39 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:48:40.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:40.342 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:40.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:40.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:40.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:40.481 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:40.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:40.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:40.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:40.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:40.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:40.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:40.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:40.792 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:40.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:40.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:40.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:40.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:41.031 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:41.052 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 28 16 57256 57240 127.748 95.6875 0.0121839 0.00782194 2024-03-12T03:48:41.075 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:41.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:41.116 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:40 smithi169 ceph-mon[29344]: osdmap e378: 8 total, 8 up, 8 in 2024-03-12T03:48:41.116 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:40 smithi169 ceph-mon[39949]: osdmap e378: 8 total, 8 up, 8 in 2024-03-12T03:48:41.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:41.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:41.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:40 smithi173 ceph-mon[33544]: osdmap e378: 8 total, 8 up, 8 in 2024-03-12T03:48:41.251 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:41.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:41.335 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:48:41.335 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:48:41.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:41.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:41.358 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:41.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:41.496 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:41.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:41.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:41.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:41.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:41.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:41.772 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:41.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:41.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:41.893 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:48:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:41.906 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:41.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:41.950 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:41 smithi169 ceph-mon[29344]: pgmap v738: 168 pgs: 168 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-12T03:48:41.950 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:41 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1070417283' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:48:41.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:41 smithi169 ceph-mon[39949]: pgmap v738: 168 pgs: 168 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-12T03:48:41.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:41 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1070417283' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:48:41.951 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:48:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:48:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:48:42.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:42.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:42.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:42.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:42.052 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 29 16 58575 58559 126.185 82.4375 0.00668857 0.00792012 2024-03-12T03:48:42.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:42.186 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:42.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:42.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:41 smithi173 ceph-mon[33544]: pgmap v738: 168 pgs: 168 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-12T03:48:42.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:41 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1070417283' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:48:42.287 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:42.333 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:42.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:42.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:42.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:42.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:42.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:42.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:42.612 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:42.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:42.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:42.757 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:42.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:42.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:42.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:42.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:42.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:43.034 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:43.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:43.052 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 30 16 60124 60108 125.206 96.8125 0.0106214 0.00798134 2024-03-12T03:48:43.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:42 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:48:43.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:42 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1070417283' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:48:43.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:42 smithi173 ceph-mon[33544]: osdmap e379: 8 total, 8 up, 8 in 2024-03-12T03:48:43.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:43.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:42 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:48:43.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:42 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1070417283' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:48:43.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:42 smithi169 ceph-mon[29344]: osdmap e379: 8 total, 8 up, 8 in 2024-03-12T03:48:43.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:42 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:48:43.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:42 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1070417283' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:48:43.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:42 smithi169 ceph-mon[39949]: osdmap e379: 8 total, 8 up, 8 in 2024-03-12T03:48:43.166 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:43.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:43.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:43.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:43.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:43.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:43.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-12T03:48:43.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:43.442 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:43.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:43.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:48:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:48:43.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:43.585 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:43.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:43.687 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:43.731 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:43.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:43.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:43.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:43.888 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:43.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:43.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:43.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:44.021 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:44.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:44.054 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 31 16 61541 61525 124.017 88.5625 0.0061055 0.00805829 2024-03-12T03:48:44.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:44.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:43 smithi169 ceph-mon[29344]: pgmap v740: 168 pgs: 168 active+clean; 3.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 144 MiB/s wr, 2.30k op/s 2024-03-12T03:48:44.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:43 smithi169 ceph-mon[39949]: pgmap v740: 168 pgs: 168 active+clean; 3.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 144 MiB/s wr, 2.30k op/s 2024-03-12T03:48:44.178 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:44.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:44.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:43 smithi173 ceph-mon[33544]: pgmap v740: 168 pgs: 168 active+clean; 3.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 144 MiB/s wr, 2.30k op/s 2024-03-12T03:48:44.280 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:44.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:44.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:44.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:44.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:44.454 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:44.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:44.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:44.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:44.599 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:44.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:44.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:44.738 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:44.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:44.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:44.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:44.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:44.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:45.020 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:45.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:45.042 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:45.042 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:45.054 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 32 16 63054 63038 123.097 94.5625 0.00896712 0.0081182 2024-03-12T03:48:45.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:45.172 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:45.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:45.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:45.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:45.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:45.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:45.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:45.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:45.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:45.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:45.593 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:45.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:45.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:45.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:45.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:45.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:45.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:45.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:45.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:45.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:46.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:46.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:46.054 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 33 16 64526 64510 122.154 92 0.011509 0.00818091 2024-03-12T03:48:46.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:46.122 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:45 smithi169 ceph-mon[39949]: pgmap v741: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 813 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-12T03:48:46.123 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:45 smithi169 ceph-mon[29344]: pgmap v741: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 813 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-12T03:48:46.144 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:46.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:45 smithi173 ceph-mon[33544]: pgmap v741: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 813 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-12T03:48:46.237 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0: {} 2024-03-12T03:48:46.245 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:46.278 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:46.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:46.282 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:48:46.322 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:48:46.366 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:48:46.380 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:46.409 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:48:46.431 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:46.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:46.461 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:48:46.505 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:48:46.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:46.543 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:48:46.552 INFO:teuthology.orchestra.run.smithi169.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-12T03:48:46.552 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:48:46.552 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:48:46.552 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:48:46.552 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:48:46.552 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:48:46.552 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:48:46.552 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:48:46.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:46.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:46.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:46.668 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:46.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:46.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:46.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:46.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:46.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:46.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:46.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:46.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:46.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:46.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:46.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:47.054 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 34 16 65804 65788 120.911 79.875 0.00863644 0.00826526 2024-03-12T03:48:47.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:47.095 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:48:47.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:47.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:47.122 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:47.167 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:46 smithi169 ceph-mon[39949]: pgmap v742: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-12T03:48:47.167 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1776564296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:48:47.168 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:46 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:47.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:46 smithi169 ceph-mon[29344]: pgmap v742: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-12T03:48:47.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1776564296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:48:47.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:46 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:47.201 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:47.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:47.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:47.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:47.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:46 smithi173 ceph-mon[33544]: pgmap v742: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-12T03:48:47.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1776564296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:48:47.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:46 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:47.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:47.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:47.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:47.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:47.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:47.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:47.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:47.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:47.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:47.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:47.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:47.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:47.786 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:47.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:47.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:47.888 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:47.929 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:47.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:47.949 INFO:teuthology.orchestra.run.smithi169.stderr:clear 4.4 pg_upmap_items mapping 2024-03-12T03:48:48.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:48.054 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 35 15 67104 67089 119.779 81.3125 0.0169014 0.00834421 2024-03-12T03:48:48.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:48.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:48.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:48.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:47 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/385271915' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-03-12T03:48:48.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:47 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-03-12T03:48:48.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:47 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/385271915' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-03-12T03:48:48.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:47 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-03-12T03:48:48.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:48.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:48.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:47 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/385271915' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-03-12T03:48:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:47 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-03-12T03:48:48.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:48.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:48.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:48.490 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:48.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:48.592 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:48.626 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:48.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:48.726 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:48.771 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:48.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:48.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:48.912 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:48.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:48.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:49.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:49.054 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:49.054 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 36 16 68462 68446 118.808 84.8125 0.0127986 0.00841247 2024-03-12T03:48:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:49.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:49.156 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:49.188 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:49.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:48 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]': finished 2024-03-12T03:48:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:48 smithi173 ceph-mon[33544]: osdmap e380: 8 total, 8 up, 8 in 2024-03-12T03:48:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:48 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:48:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:48 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:48:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:48 smithi173 ceph-mon[33544]: pgmap v744: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-03-12T03:48:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:48 smithi173 ceph-mon[33544]: osdmap e381: 8 total, 8 up, 8 in 2024-03-12T03:48:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:48 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:48:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:48 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:48:49.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:49.332 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:49.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]': finished 2024-03-12T03:48:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[29344]: osdmap e380: 8 total, 8 up, 8 in 2024-03-12T03:48:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:48:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:48:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[29344]: pgmap v744: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-03-12T03:48:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[29344]: osdmap e381: 8 total, 8 up, 8 in 2024-03-12T03:48:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:48:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:48:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]': finished 2024-03-12T03:48:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[39949]: osdmap e380: 8 total, 8 up, 8 in 2024-03-12T03:48:49.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:48:49.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:48:49.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[39949]: pgmap v744: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-03-12T03:48:49.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[39949]: osdmap e381: 8 total, 8 up, 8 in 2024-03-12T03:48:49.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:48:49.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:48 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:48:49.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:49.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:49.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:49.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:49.613 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:49.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:49.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:49.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:49.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:49.854 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:49.899 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:49.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:49.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:50.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:50.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:50.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:50.054 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 37 16 69889 69873 118.007 89.1875 0.00482322 0.00846905 2024-03-12T03:48:50.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:50.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:50.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:50.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:50.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:50.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:50.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:50.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:50.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:50.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:50.584 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:50.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:50.686 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:50.723 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:50.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:50 smithi173 ceph-mon[33544]: osdmap e382: 8 total, 8 up, 8 in 2024-03-12T03:48:50.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:50.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:50.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:50.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:50 smithi169 ceph-mon[29344]: osdmap e382: 8 total, 8 up, 8 in 2024-03-12T03:48:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:50 smithi169 ceph-mon[39949]: osdmap e382: 8 total, 8 up, 8 in 2024-03-12T03:48:50.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:50.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:50.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:51.054 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 38 16 71248 71232 117.137 84.9375 0.00815974 0.00853206 2024-03-12T03:48:51.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:51.142 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:51.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:51.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:51.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:51.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:51.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:51.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:51.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:51.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:51.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:51.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:51.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:51.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:51.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:51.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:51 smithi169 ceph-mon[39949]: pgmap v747: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-12T03:48:51.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:51 smithi169 ceph-mon[29344]: pgmap v747: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-12T03:48:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:51.701 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:51.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:51.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:51 smithi173 ceph-mon[33544]: pgmap v747: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-12T03:48:51.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:51.846 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:51.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:51.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:51.992 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:51.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:52.054 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 39 16 72642 72626 116.367 87.125 0.0121368 0.00858858 2024-03-12T03:48:52.095 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:52.135 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:52.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:52.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:48:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:48:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:48:52.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:52.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:52.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:52.368 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:52.419 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:52.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:52.520 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:52.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:52.562 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:52.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:52.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:52.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:52.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:52.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:52.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:52.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:52.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:52.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:52.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:52.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:52.993 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:53.056 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:48:53.054840+0000 min lat: 0.00147924 max lat: 0.0880292 avg lat: 0.00865131 2024-03-12T03:48:53.056 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:48:53.056 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 40 16 73965 73949 115.521 82.6875 0.0269822 0.00865131 2024-03-12T03:48:53.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:53.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:53.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:53.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:53.243 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:53.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:53.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:53.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:53.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:53.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:53.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:53.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:48:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:48:53.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:53.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:53.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:53.661 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:53.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:53.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:53.805 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:53.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:53.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:53.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:53 smithi169 ceph-mon[39949]: pgmap v748: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-12T03:48:53.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:53 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:48:53.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:53 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:48:53.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:53 smithi169 ceph-mon[39949]: osdmap e383: 8 total, 8 up, 8 in 2024-03-12T03:48:53.835 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:53 smithi169 ceph-mon[29344]: pgmap v748: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-12T03:48:53.835 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:53 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:48:53.835 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:53 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:48:53.835 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:53 smithi169 ceph-mon[29344]: osdmap e383: 8 total, 8 up, 8 in 2024-03-12T03:48:53.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:53.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:53.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:53.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:53 smithi173 ceph-mon[33544]: pgmap v748: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-12T03:48:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:53 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:48:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:53 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:48:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:53 smithi173 ceph-mon[33544]: osdmap e383: 8 total, 8 up, 8 in 2024-03-12T03:48:54.033 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:54.056 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 41 16 75212 75196 114.604 77.9375 0.0126439 0.00872031 2024-03-12T03:48:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:54.075 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:54.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:54.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:54.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:54.210 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:54.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:54.260 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:48:54.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:54.344 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:54.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:54.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:54.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:54.468 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:48:54.468 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:48:54.468 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-03-12T03:48:54.468 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-03-12T03:48:54.468 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:48:54.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:54.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:54.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:54.506 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:54 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:54.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:54.628 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:54.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:54.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:54.781 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:54.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:54.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:54.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:54.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:54.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:54 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:54 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:55.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:55.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:55.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:55.056 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 42 16 76455 76439 113.725 77.6875 0.00554385 0.00878792 2024-03-12T03:48:55.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:55.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:55.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:55.292 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0 2024-03-12T03:48:55.347 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:55.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:55.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.0' 2024-03-12T03:48:55.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:55.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:55.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:55.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:55.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:55.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:55.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:55.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:55.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:55 smithi169 ceph-mon[29344]: pgmap v750: 168 pgs: 168 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 682 B/s rd, 80 MiB/s wr, 1.28k op/s; 59 B/s, 0 objects/s recovering 2024-03-12T03:48:55.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:55 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:55.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:55 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:55.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:55 smithi169 ceph-mon[39949]: pgmap v750: 168 pgs: 168 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 682 B/s rd, 80 MiB/s wr, 1.28k op/s; 59 B/s, 0 objects/s recovering 2024-03-12T03:48:55.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:55 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:55.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:55 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:55.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:55.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:55.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:55.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:55.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:55.933 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:55.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:55 smithi173 ceph-mon[33544]: pgmap v750: 168 pgs: 168 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 682 B/s rd, 80 MiB/s wr, 1.28k op/s; 59 B/s, 0 objects/s recovering 2024-03-12T03:48:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:55 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:55 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:56.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-12T03:48:56.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:56.057 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 43 16 77634 77618 112.794 73.6875 0.0205927 0.00886108 2024-03-12T03:48:56.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:56.140 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:56.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:56.146 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:48:56.146 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:48:56.146 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:48:56.146 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-12T03:48:56.147 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.0 2024-03-12T03:48:56.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:56.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:56.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:56.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:56.422 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:56.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:56.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:56.514 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:48:56 smithi169 systemd[1]: Stopping Ceph osd.0 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:48:56.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:56.565 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:56.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:56.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:56.667 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:56.705 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:56.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:56.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:56.784 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:48:56.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:56.876 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:56.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:56.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:56 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:48:56.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:56 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:48:56.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:56 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:56 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:48:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:56 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:48:56.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:56 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:56.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:56 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:48:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:56 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:48:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:56 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:48:57.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:57.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:57.056 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 44 16 78678 78662 111.714 65.25 0.0072344 0.0089468 2024-03-12T03:48:57.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:57.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:57.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:57.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:57.208 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:48:56 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0[43969]: 2024-03-12T03:48:56.979+0000 7f48d2141700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:48:57.208 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:48:56 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0[43969]: 2024-03-12T03:48:56.979+0000 7f48d2141700 -1 osd.0 383 *** Got signal Terminated *** 2024-03-12T03:48:57.209 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:48:56 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0[43969]: 2024-03-12T03:48:56.979+0000 7f48d2141700 -1 osd.0 383 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:48:57.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:57.287 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:57.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:57.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:57.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:57.436 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:57.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:57.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:57.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:57.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:57.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:48:57.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:57.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:57.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:57.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:57 smithi169 ceph-mon[29344]: pgmap v751: 168 pgs: 168 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 533 B/s rd, 63 MiB/s wr, 1.00k op/s; 46 B/s, 0 objects/s recovering 2024-03-12T03:48:57.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:57 smithi169 ceph-mon[29344]: osd.0 marked itself down and dead 2024-03-12T03:48:57.741 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:57 smithi169 ceph-mon[39949]: pgmap v751: 168 pgs: 168 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 533 B/s rd, 63 MiB/s wr, 1.00k op/s; 46 B/s, 0 objects/s recovering 2024-03-12T03:48:57.741 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:57 smithi169 ceph-mon[39949]: osd.0 marked itself down and dead 2024-03-12T03:48:57.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:57.824 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:57.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:57.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:57.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:57.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:57 smithi173 ceph-mon[33544]: pgmap v751: 168 pgs: 168 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 533 B/s rd, 63 MiB/s wr, 1.00k op/s; 46 B/s, 0 objects/s recovering 2024-03-12T03:48:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:57 smithi173 ceph-mon[33544]: osd.0 marked itself down and dead 2024-03-12T03:48:58.052 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:58.056 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 45 16 79320 79304 110.123 40.125 0.00350103 0.00897938 2024-03-12T03:48:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:58.077 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:58.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:58.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:48:58.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:58.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:48:58.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:58.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:58.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:58.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:58.454 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:48:58.466 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:58.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:58.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:58.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:48:58.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:58.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:58.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:48:58.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:58.742 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:58.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:58.844 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:58.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:58.874 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:58.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:58.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:58 smithi169 ceph-mon[39949]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T03:48:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:58 smithi169 ceph-mon[39949]: osdmap e384: 8 total, 7 up, 8 in 2024-03-12T03:48:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:58 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3552337643' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:48:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:58 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:48:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:58 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3552337643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:48:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:58 smithi169 ceph-mon[39949]: osdmap e385: 8 total, 7 up, 8 in 2024-03-12T03:48:58.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:58 smithi169 ceph-mon[29344]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T03:48:58.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:58 smithi169 ceph-mon[29344]: osdmap e384: 8 total, 7 up, 8 in 2024-03-12T03:48:58.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:58 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3552337643' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:48:58.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:58 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:48:58.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:58 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3552337643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:48:58.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:58 smithi169 ceph-mon[29344]: osdmap e385: 8 total, 7 up, 8 in 2024-03-12T03:48:58.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:58.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:58 smithi173 ceph-mon[33544]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T03:48:58.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:58 smithi173 ceph-mon[33544]: osdmap e384: 8 total, 7 up, 8 in 2024-03-12T03:48:58.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:58 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3552337643' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:48:58.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:58 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:48:58.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:58 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3552337643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:48:58.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:58 smithi173 ceph-mon[33544]: osdmap e385: 8 total, 7 up, 8 in 2024-03-12T03:48:59.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:59.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:59.056 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 46 16 80170 80154 108.884 53.125 0.00967121 0.00917928 2024-03-12T03:48:59.117 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:59.151 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:59.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:59.253 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:59.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:59.292 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:59.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:59.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:48:59.428 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:59.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:48:59.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:48:59.564 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:59.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:48:59.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:48:59.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:59.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:59.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:48:59.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:48:59.841 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:48:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:59.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:48:59.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:59 smithi173 ceph-mon[33544]: pgmap v753: 168 pgs: 11 peering, 21 stale+active+clean, 136 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 895 B/s rd, 95 MiB/s wr, 1.52k op/s; 44 B/s, 0 objects/s recovering 2024-03-12T03:48:59.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:48:59 smithi173 ceph-mon[33544]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-12T03:48:59.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:59 smithi169 ceph-mon[29344]: pgmap v753: 168 pgs: 11 peering, 21 stale+active+clean, 136 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 895 B/s rd, 95 MiB/s wr, 1.52k op/s; 44 B/s, 0 objects/s recovering 2024-03-12T03:48:59.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:48:59 smithi169 ceph-mon[29344]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-12T03:48:59.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:59 smithi169 ceph-mon[39949]: pgmap v753: 168 pgs: 11 peering, 21 stale+active+clean, 136 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 895 B/s rd, 95 MiB/s wr, 1.52k op/s; 44 B/s, 0 objects/s recovering 2024-03-12T03:48:59.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:48:59 smithi169 ceph-mon[39949]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-12T03:48:59.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:48:59.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:48:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:48:59.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:00.057 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 47 16 81739 81723 108.653 98.0625 0.0128544 0.00919858 2024-03-12T03:49:00.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:00.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:00.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:00.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:00.232 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:00.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:00.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:00.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:00.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:00.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:00.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:00.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:00.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:00.590 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:49:00 smithi169 podman[167445]: 2024-03-12 03:49:00.24599158 +0000 UTC m=+3.443533421 container died a2017972caee979e4991ebd4c2050e6b11ed80a3953bf4357ddf0c4f323e9f4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-12T03:49:00.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:00.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:00.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:00.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:00.840 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:00.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:00.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:00 smithi169 ceph-mon[29344]: Health check failed: Degraded data redundancy: 29/236094 objects degraded (0.012%), 5 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:00.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:00 smithi169 ceph-mon[39949]: Health check failed: Degraded data redundancy: 29/236094 objects degraded (0.012%), 5 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:00.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:00.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:00.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:00.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:00.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:00 smithi173 ceph-mon[33544]: Health check failed: Degraded data redundancy: 29/236094 objects degraded (0.012%), 5 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:01.057 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 48 16 83226 83210 108.326 92.9375 0.00885202 0.00922709 2024-03-12T03:49:01.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:01.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:01.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:01.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:01.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:01.253 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:01.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:01.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:01.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:01.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:01.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:01.527 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:01.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:01.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:01.663 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:01.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:01.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:01.765 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:01.797 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:01.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:01.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:01.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:01.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:49:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:49:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-mon[29344]: pgmap v755: 168 pgs: 5 active+undersized+degraded, 7 active+undersized, 11 peering, 18 stale+active+clean, 127 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 2.4 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 29/236094 objects degraded (0.012%); 12 KiB/s, 0 objects/s recovering 2024-03-12T03:49:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-mon[29344]: 5.11 scrub starts 2024-03-12T03:49:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-mon[29344]: 5.11 scrub ok 2024-03-12T03:49:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-mon[29344]: 4.8 scrub starts 2024-03-12T03:49:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-mon[39949]: pgmap v755: 168 pgs: 5 active+undersized+degraded, 7 active+undersized, 11 peering, 18 stale+active+clean, 127 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 2.4 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 29/236094 objects degraded (0.012%); 12 KiB/s, 0 objects/s recovering 2024-03-12T03:49:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-mon[39949]: 5.11 scrub starts 2024-03-12T03:49:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-mon[39949]: 5.11 scrub ok 2024-03-12T03:49:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-mon[39949]: 4.8 scrub starts 2024-03-12T03:49:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:01 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:01.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:01.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:01.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:01 smithi173 ceph-mon[33544]: pgmap v755: 168 pgs: 5 active+undersized+degraded, 7 active+undersized, 11 peering, 18 stale+active+clean, 127 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 2.4 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 29/236094 objects degraded (0.012%); 12 KiB/s, 0 objects/s recovering 2024-03-12T03:49:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:01 smithi173 ceph-mon[33544]: 5.11 scrub starts 2024-03-12T03:49:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:01 smithi173 ceph-mon[33544]: 5.11 scrub ok 2024-03-12T03:49:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:01 smithi173 ceph-mon[33544]: 4.8 scrub starts 2024-03-12T03:49:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:01 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:01 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:02.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:02.057 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 49 16 84755 84739 108.065 95.5625 0.00888954 0.00924863 2024-03-12T03:49:02.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:02.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:02.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:02.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:02.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:02.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:02.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:02.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:02.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:02.536 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:02.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:02.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:02.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:02.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:02.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:02.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:02.813 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:02.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:02.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:02.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:02.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:02 smithi169 ceph-mon[29344]: 4.8 scrub ok 2024-03-12T03:49:02.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:02 smithi169 ceph-mon[29344]: 4.18 scrub starts 2024-03-12T03:49:02.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:02 smithi169 ceph-mon[29344]: 4.18 scrub ok 2024-03-12T03:49:02.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:02 smithi169 ceph-mon[29344]: 6.14 scrub starts 2024-03-12T03:49:02.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:02 smithi169 ceph-mon[29344]: 6.14 scrub ok 2024-03-12T03:49:02.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:02 smithi169 ceph-mon[39949]: 4.8 scrub ok 2024-03-12T03:49:02.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:02 smithi169 ceph-mon[39949]: 4.18 scrub starts 2024-03-12T03:49:02.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:02 smithi169 ceph-mon[39949]: 4.18 scrub ok 2024-03-12T03:49:02.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:02 smithi169 ceph-mon[39949]: 6.14 scrub starts 2024-03-12T03:49:02.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:02 smithi169 ceph-mon[39949]: 6.14 scrub ok 2024-03-12T03:49:02.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:02.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:02 smithi173 ceph-mon[33544]: 4.8 scrub ok 2024-03-12T03:49:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:02 smithi173 ceph-mon[33544]: 4.18 scrub starts 2024-03-12T03:49:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:02 smithi173 ceph-mon[33544]: 4.18 scrub ok 2024-03-12T03:49:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:02 smithi173 ceph-mon[33544]: 6.14 scrub starts 2024-03-12T03:49:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:02 smithi173 ceph-mon[33544]: 6.14 scrub ok 2024-03-12T03:49:03.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:03.057 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 50 16 86214 86198 107.728 91.1875 0.00570723 0.00927792 2024-03-12T03:49:03.083 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:03.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:03.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:03.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:03.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:03.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:03.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:03.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:03.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:03.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:03.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:49:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:49:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:03.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:03.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:03.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:03.640 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:03.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:03.742 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:03.782 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:03.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:03.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:03.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:03.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[29344]: pgmap v756: 168 pgs: 21 active+undersized+degraded, 31 active+undersized, 11 peering, 105 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 2.1 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 41706/247773 objects degraded (16.832%); 18 KiB/s, 0 objects/s recovering 2024-03-12T03:49:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[29344]: 5.1a scrub starts 2024-03-12T03:49:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[29344]: 5.1a scrub ok 2024-03-12T03:49:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[29344]: 6.1 scrub starts 2024-03-12T03:49:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[29344]: 6.1 scrub ok 2024-03-12T03:49:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[29344]: 1.f scrub starts 2024-03-12T03:49:03.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[29344]: 1.f scrub ok 2024-03-12T03:49:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[39949]: pgmap v756: 168 pgs: 21 active+undersized+degraded, 31 active+undersized, 11 peering, 105 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 2.1 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 41706/247773 objects degraded (16.832%); 18 KiB/s, 0 objects/s recovering 2024-03-12T03:49:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[39949]: 5.1a scrub starts 2024-03-12T03:49:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[39949]: 5.1a scrub ok 2024-03-12T03:49:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[39949]: 6.1 scrub starts 2024-03-12T03:49:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[39949]: 6.1 scrub ok 2024-03-12T03:49:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[39949]: 1.f scrub starts 2024-03-12T03:49:03.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:03 smithi169 ceph-mon[39949]: 1.f scrub ok 2024-03-12T03:49:03.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:03.926 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:49:03 smithi169 podman[167445]: 2024-03-12 03:49:03.572331734 +0000 UTC m=+6.769873577 container cleanup a2017972caee979e4991ebd4c2050e6b11ed80a3953bf4357ddf0c4f323e9f4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-12T03:49:03.926 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:49:03 smithi169 bash[167445]: ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0 2024-03-12T03:49:03.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:03 smithi173 ceph-mon[33544]: pgmap v756: 168 pgs: 21 active+undersized+degraded, 31 active+undersized, 11 peering, 105 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 2.1 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 41706/247773 objects degraded (16.832%); 18 KiB/s, 0 objects/s recovering 2024-03-12T03:49:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:03 smithi173 ceph-mon[33544]: 5.1a scrub starts 2024-03-12T03:49:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:03 smithi173 ceph-mon[33544]: 5.1a scrub ok 2024-03-12T03:49:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:03 smithi173 ceph-mon[33544]: 6.1 scrub starts 2024-03-12T03:49:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:03 smithi173 ceph-mon[33544]: 6.1 scrub ok 2024-03-12T03:49:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:03 smithi173 ceph-mon[33544]: 1.f scrub starts 2024-03-12T03:49:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:03 smithi173 ceph-mon[33544]: 1.f scrub ok 2024-03-12T03:49:04.026 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:04.058 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 51 16 87754 87738 107.5 96.25 0.00558909 0.00929802 2024-03-12T03:49:04.060 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:04.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:04.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:04.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:04.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:04.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:04.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:04.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:04.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:04.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:04.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:04.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:04.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:04.612 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:04.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:04.672 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:49:04 smithi169 podman[167626]: 2024-03-12 03:49:04.552636693 +0000 UTC m=+6.181703453 container remove a2017972caee979e4991ebd4c2050e6b11ed80a3953bf4357ddf0c4f323e9f4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-12T03:49:04.672 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:04.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[29344]: 6.6 scrub starts 2024-03-12T03:49:04.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[29344]: 4.2 scrub starts 2024-03-12T03:49:04.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[29344]: 4.2 scrub ok 2024-03-12T03:49:04.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[29344]: 6.6 scrub ok 2024-03-12T03:49:04.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[29344]: 6.1a scrub starts 2024-03-12T03:49:04.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[29344]: 6.1a scrub ok 2024-03-12T03:49:04.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[29344]: 5.6 scrub starts 2024-03-12T03:49:04.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[29344]: 5.6 scrub ok 2024-03-12T03:49:04.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[29344]: 3.1d scrub starts 2024-03-12T03:49:04.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[29344]: 3.1d scrub ok 2024-03-12T03:49:04.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:04.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[39949]: 6.6 scrub starts 2024-03-12T03:49:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[39949]: 4.2 scrub starts 2024-03-12T03:49:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[39949]: 4.2 scrub ok 2024-03-12T03:49:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[39949]: 6.6 scrub ok 2024-03-12T03:49:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[39949]: 6.1a scrub starts 2024-03-12T03:49:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[39949]: 6.1a scrub ok 2024-03-12T03:49:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[39949]: 5.6 scrub starts 2024-03-12T03:49:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[39949]: 5.6 scrub ok 2024-03-12T03:49:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[39949]: 3.1d scrub starts 2024-03-12T03:49:04.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:04 smithi169 ceph-mon[39949]: 3.1d scrub ok 2024-03-12T03:49:04.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:04.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:04.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:04.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:04.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:04.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:04.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:04.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:04 smithi173 ceph-mon[33544]: 6.6 scrub starts 2024-03-12T03:49:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:04 smithi173 ceph-mon[33544]: 4.2 scrub starts 2024-03-12T03:49:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:04 smithi173 ceph-mon[33544]: 4.2 scrub ok 2024-03-12T03:49:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:04 smithi173 ceph-mon[33544]: 6.6 scrub ok 2024-03-12T03:49:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:04 smithi173 ceph-mon[33544]: 6.1a scrub starts 2024-03-12T03:49:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:04 smithi173 ceph-mon[33544]: 6.1a scrub ok 2024-03-12T03:49:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:04 smithi173 ceph-mon[33544]: 5.6 scrub starts 2024-03-12T03:49:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:04 smithi173 ceph-mon[33544]: 5.6 scrub ok 2024-03-12T03:49:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:04 smithi173 ceph-mon[33544]: 3.1d scrub starts 2024-03-12T03:49:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:04 smithi173 ceph-mon[33544]: 3.1d scrub ok 2024-03-12T03:49:05.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:05.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:05.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:49:05.058 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 52 16 89234 89218 107.211 92.5 0.0138149 0.00932291 2024-03-12T03:49:05.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:05.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:05.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:05.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:05.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:05.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:05.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:05.420 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:49:05.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:05.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:05.453 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:49:05.490 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:49:05.527 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:49:05.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:05.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:05.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:05.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:05.569 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:49:05.603 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:49:05.644 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:49:05.644 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:49:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:49:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:49:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:49:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:49:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:49:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:49:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:49:05.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:05.699 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:05.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:05.800 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:05.835 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:05.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:05.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:05 smithi169 ceph-mon[29344]: pgmap v757: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 2.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 52332/254145 objects degraded (20.591%); 18 KiB/s, 0 objects/s recovering 2024-03-12T03:49:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:05 smithi169 ceph-mon[29344]: 6.12 scrub starts 2024-03-12T03:49:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:05 smithi169 ceph-mon[29344]: 6.12 scrub ok 2024-03-12T03:49:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:05 smithi169 ceph-mon[29344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 5 pgs peering) 2024-03-12T03:49:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:05 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/303070787' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:05 smithi169 ceph-mon[39949]: pgmap v757: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 2.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 52332/254145 objects degraded (20.591%); 18 KiB/s, 0 objects/s recovering 2024-03-12T03:49:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:05 smithi169 ceph-mon[39949]: 6.12 scrub starts 2024-03-12T03:49:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:05 smithi169 ceph-mon[39949]: 6.12 scrub ok 2024-03-12T03:49:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:05 smithi169 ceph-mon[39949]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 5 pgs peering) 2024-03-12T03:49:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:05 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/303070787' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:05.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:05.972 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:05.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:05 smithi173 ceph-mon[33544]: pgmap v757: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 2.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 52332/254145 objects degraded (20.591%); 18 KiB/s, 0 objects/s recovering 2024-03-12T03:49:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:05 smithi173 ceph-mon[33544]: 6.12 scrub starts 2024-03-12T03:49:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:05 smithi173 ceph-mon[33544]: 6.12 scrub ok 2024-03-12T03:49:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:05 smithi173 ceph-mon[33544]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 5 pgs peering) 2024-03-12T03:49:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:05 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/303070787' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:06.058 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 53 16 90723 90707 106.944 93.0625 0.00975782 0.00934612 2024-03-12T03:49:06.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:06.107 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:06.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:06.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:06.243 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:06.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:06.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:06.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:06.378 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:06.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:06.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:06.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:06.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:06.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:06.619 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:49:06.630 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:49:06.660 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:06.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:06.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:06.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:06 smithi169 ceph-mon[29344]: 3.0 scrub starts 2024-03-12T03:49:06.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:06 smithi169 ceph-mon[29344]: 3.0 scrub ok 2024-03-12T03:49:06.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:06 smithi169 ceph-mon[29344]: 5.f scrub starts 2024-03-12T03:49:06.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:06 smithi169 ceph-mon[29344]: 5.f scrub ok 2024-03-12T03:49:06.803 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:06.803 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:06 smithi169 ceph-mon[39949]: 3.0 scrub starts 2024-03-12T03:49:06.803 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:06 smithi169 ceph-mon[39949]: 3.0 scrub ok 2024-03-12T03:49:06.803 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:06 smithi169 ceph-mon[39949]: 5.f scrub starts 2024-03-12T03:49:06.804 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:06 smithi169 ceph-mon[39949]: 5.f scrub ok 2024-03-12T03:49:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:06.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:06.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:06.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:06.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:06 smithi173 ceph-mon[33544]: 3.0 scrub starts 2024-03-12T03:49:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:06 smithi173 ceph-mon[33544]: 3.0 scrub ok 2024-03-12T03:49:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:06 smithi173 ceph-mon[33544]: 5.f scrub starts 2024-03-12T03:49:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:06 smithi173 ceph-mon[33544]: 5.f scrub ok 2024-03-12T03:49:07.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:07.058 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 54 16 92221 92205 106.698 93.625 0.00401551 0.00936764 2024-03-12T03:49:07.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:07.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:07.087 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:07.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:07.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:07.221 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:07.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:07.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:07.352 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:07.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:07.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:07.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:07.487 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:07.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:07.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:07.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:07.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:07.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:07.672 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:07 smithi169 ceph-mon[29344]: pgmap v758: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 57 MiB/s wr, 907 op/s; 52332/254145 objects degraded (20.591%); 17 KiB/s, 0 objects/s recovering 2024-03-12T03:49:07.672 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:07 smithi169 ceph-mon[39949]: pgmap v758: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 57 MiB/s wr, 907 op/s; 52332/254145 objects degraded (20.591%); 17 KiB/s, 0 objects/s recovering 2024-03-12T03:49:07.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:07.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:07.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:07.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:07.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:07.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:07 smithi173 ceph-mon[33544]: pgmap v758: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 57 MiB/s wr, 907 op/s; 52332/254145 objects degraded (20.591%); 17 KiB/s, 0 objects/s recovering 2024-03-12T03:49:08.007 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:08.043 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:08.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:08.058 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 55 15 93720 93705 106.462 93.75 0.00727037 0.00938915 2024-03-12T03:49:08.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:08.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:08.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:08.283 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:08.321 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:08.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:08.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:08.455 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:08.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:08.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:08.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:08.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:08.630 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:49:08.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:08.724 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:08.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:08.785 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[29344]: 5.0 scrub starts 2024-03-12T03:49:08.785 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[29344]: 5.4 scrub starts 2024-03-12T03:49:08.785 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[29344]: 5.0 scrub ok 2024-03-12T03:49:08.785 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[29344]: 5.4 scrub ok 2024-03-12T03:49:08.785 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[29344]: 6.1d scrub starts 2024-03-12T03:49:08.785 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[29344]: 6.1d scrub ok 2024-03-12T03:49:08.785 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[29344]: 2.2 scrub starts 2024-03-12T03:49:08.785 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[29344]: 2.2 scrub ok 2024-03-12T03:49:08.786 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 52332/254145 objects degraded (20.591%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:08.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[39949]: 5.0 scrub starts 2024-03-12T03:49:08.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[39949]: 5.4 scrub starts 2024-03-12T03:49:08.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[39949]: 5.0 scrub ok 2024-03-12T03:49:08.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[39949]: 5.4 scrub ok 2024-03-12T03:49:08.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[39949]: 6.1d scrub starts 2024-03-12T03:49:08.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[39949]: 6.1d scrub ok 2024-03-12T03:49:08.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[39949]: 2.2 scrub starts 2024-03-12T03:49:08.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[39949]: 2.2 scrub ok 2024-03-12T03:49:08.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:08 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 52332/254145 objects degraded (20.591%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:08.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:08.850 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:08.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:08.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:08.952 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:08 smithi173 ceph-mon[33544]: 5.0 scrub starts 2024-03-12T03:49:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:08 smithi173 ceph-mon[33544]: 5.4 scrub starts 2024-03-12T03:49:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:08 smithi173 ceph-mon[33544]: 5.0 scrub ok 2024-03-12T03:49:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:08 smithi173 ceph-mon[33544]: 5.4 scrub ok 2024-03-12T03:49:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:08 smithi173 ceph-mon[33544]: 6.1d scrub starts 2024-03-12T03:49:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:08 smithi173 ceph-mon[33544]: 6.1d scrub ok 2024-03-12T03:49:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:08 smithi173 ceph-mon[33544]: 2.2 scrub starts 2024-03-12T03:49:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:08 smithi173 ceph-mon[33544]: 2.2 scrub ok 2024-03-12T03:49:08.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:08 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 52332/254145 objects degraded (20.591%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:09.007 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:09.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:09.061 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 56 16 95193 95177 106.201 92 0.00484962 0.00941127 2024-03-12T03:49:09.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:09.143 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:09.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:09.245 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:09.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:09.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:09.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:09.395 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:49:09.396 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:49:09.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:09.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:09.524 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:09.564 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:09.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:09.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:09.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:09.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:09.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:09.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:09.841 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:09.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:09.923 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:49:09 smithi169 podman[168583]: 2024-03-12 03:49:09.545383079 +0000 UTC m=+0.218005998 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: pgmap v759: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.9 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 56980/273678 objects degraded (20.820%); 14 KiB/s, 0 objects/s recovering 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: 3.2 scrub starts 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: 3.2 scrub ok 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: 3.e scrub starts 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: 3.e scrub ok 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: 5.12 scrub starts 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: 5.12 scrub ok 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: 4.13 scrub starts 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: 4.13 scrub ok 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: 1.14 scrub starts 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: 1.14 scrub ok 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: 3.5 scrub starts 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[29344]: 3.5 scrub ok 2024-03-12T03:49:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: pgmap v759: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.9 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 56980/273678 objects degraded (20.820%); 14 KiB/s, 0 objects/s recovering 2024-03-12T03:49:09.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: 3.2 scrub starts 2024-03-12T03:49:09.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: 3.2 scrub ok 2024-03-12T03:49:09.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: 3.e scrub starts 2024-03-12T03:49:09.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: 3.e scrub ok 2024-03-12T03:49:09.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: 5.12 scrub starts 2024-03-12T03:49:09.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: 5.12 scrub ok 2024-03-12T03:49:09.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: 4.13 scrub starts 2024-03-12T03:49:09.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: 4.13 scrub ok 2024-03-12T03:49:09.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: 1.14 scrub starts 2024-03-12T03:49:09.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: 1.14 scrub ok 2024-03-12T03:49:09.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: 3.5 scrub starts 2024-03-12T03:49:09.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:09 smithi169 ceph-mon[39949]: 3.5 scrub ok 2024-03-12T03:49:09.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:09.975 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:09.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:09.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: pgmap v759: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.9 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 56980/273678 objects degraded (20.820%); 14 KiB/s, 0 objects/s recovering 2024-03-12T03:49:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: 3.2 scrub starts 2024-03-12T03:49:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: 3.2 scrub ok 2024-03-12T03:49:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: 3.e scrub starts 2024-03-12T03:49:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: 3.e scrub ok 2024-03-12T03:49:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: 5.12 scrub starts 2024-03-12T03:49:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: 5.12 scrub ok 2024-03-12T03:49:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: 4.13 scrub starts 2024-03-12T03:49:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: 4.13 scrub ok 2024-03-12T03:49:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: 1.14 scrub starts 2024-03-12T03:49:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: 1.14 scrub ok 2024-03-12T03:49:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: 3.5 scrub starts 2024-03-12T03:49:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:09 smithi173 ceph-mon[33544]: 3.5 scrub ok 2024-03-12T03:49:10.060 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 57 16 96678 96662 105.966 92.8125 0.00612978 0.00943233 2024-03-12T03:49:10.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:10.114 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:10.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:10.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:10.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:10.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:10.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:10.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:10.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:10.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:10.485 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:10.519 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:10.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:10.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:10.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:10.657 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:10.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:10.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:10.662 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:49:10.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:10.794 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:10.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:10.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:10.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:10.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:10 smithi169 ceph-mon[29344]: 4.e scrub starts 2024-03-12T03:49:10.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:10 smithi169 ceph-mon[29344]: 4.e scrub ok 2024-03-12T03:49:10.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:10 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2340586102' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:49:10.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:10 smithi169 ceph-mon[39949]: 4.e scrub starts 2024-03-12T03:49:10.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:10 smithi169 ceph-mon[39949]: 4.e scrub ok 2024-03-12T03:49:10.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:10 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2340586102' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:49:10.924 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:49:10 smithi169 podman[168583]: 2024-03-12 03:49:10.880109964 +0000 UTC m=+1.552732869 container create f7c7f556f29b9184d93fa55253f5cfa609083dc301aa0f607c64925f73c30283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-12T03:49:10.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:10.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:10 smithi173 ceph-mon[33544]: 4.e scrub starts 2024-03-12T03:49:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:10 smithi173 ceph-mon[33544]: 4.e scrub ok 2024-03-12T03:49:10.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:10 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2340586102' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:49:11.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:11.060 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 58 16 98128 98112 105.701 90.625 0.0107929 0.00945631 2024-03-12T03:49:11.086 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:11.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:11.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:11.223 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:11.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:11.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:11.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:11.353 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:11.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:11.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:11.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:11.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:11.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:11.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:11.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:11.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:11.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:11.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:11.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:11.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:11.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:11.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:11.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:11.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:49:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:49:11.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[29344]: pgmap v760: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 75 MiB/s wr, 1.21k op/s; 56980/276534 objects degraded (20.605%); 12 KiB/s, 0 objects/s recovering 2024-03-12T03:49:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[29344]: 6.8 scrub starts 2024-03-12T03:49:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[29344]: 6.8 scrub ok 2024-03-12T03:49:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[29344]: 5.14 scrub starts 2024-03-12T03:49:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[29344]: 5.14 scrub ok 2024-03-12T03:49:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:49:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2340586102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:49:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[29344]: osdmap e386: 8 total, 7 up, 8 in 2024-03-12T03:49:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[39949]: pgmap v760: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 75 MiB/s wr, 1.21k op/s; 56980/276534 objects degraded (20.605%); 12 KiB/s, 0 objects/s recovering 2024-03-12T03:49:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[39949]: 6.8 scrub starts 2024-03-12T03:49:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[39949]: 6.8 scrub ok 2024-03-12T03:49:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[39949]: 5.14 scrub starts 2024-03-12T03:49:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[39949]: 5.14 scrub ok 2024-03-12T03:49:11.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:49:11.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2340586102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:49:11.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:11 smithi169 ceph-mon[39949]: osdmap e386: 8 total, 7 up, 8 in 2024-03-12T03:49:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:11 smithi173 ceph-mon[33544]: pgmap v760: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 75 MiB/s wr, 1.21k op/s; 56980/276534 objects degraded (20.605%); 12 KiB/s, 0 objects/s recovering 2024-03-12T03:49:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:11 smithi173 ceph-mon[33544]: 6.8 scrub starts 2024-03-12T03:49:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:11 smithi173 ceph-mon[33544]: 6.8 scrub ok 2024-03-12T03:49:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:11 smithi173 ceph-mon[33544]: 5.14 scrub starts 2024-03-12T03:49:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:11 smithi173 ceph-mon[33544]: 5.14 scrub ok 2024-03-12T03:49:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:11 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:49:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:11 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2340586102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:49:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:11 smithi173 ceph-mon[33544]: osdmap e386: 8 total, 7 up, 8 in 2024-03-12T03:49:12.012 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:12.048 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:12.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:12.060 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 59 16 99615 99599 105.485 92.9375 0.0163255 0.00947494 2024-03-12T03:49:12.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:12.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:12.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:12.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:12.322 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:12.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:12.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:12.456 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:12.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:12.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:12.589 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:12.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:12.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:12.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-12T03:49:12.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:12.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:12.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:12.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:12.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:12.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:12.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:12 smithi169 ceph-mon[29344]: 5.1 scrub starts 2024-03-12T03:49:12.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:12 smithi169 ceph-mon[29344]: 5.1 scrub ok 2024-03-12T03:49:12.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:12 smithi169 ceph-mon[39949]: 5.1 scrub starts 2024-03-12T03:49:12.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:12 smithi169 ceph-mon[39949]: 5.1 scrub ok 2024-03-12T03:49:12.924 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:49:12 smithi169 podman[168583]: 2024-03-12 03:49:12.797623897 +0000 UTC m=+3.470246812 container init f7c7f556f29b9184d93fa55253f5cfa609083dc301aa0f607c64925f73c30283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-12T03:49:12.924 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:49:12 smithi169 podman[168583]: 2024-03-12 03:49:12.801983796 +0000 UTC m=+3.474606698 container start f7c7f556f29b9184d93fa55253f5cfa609083dc301aa0f607c64925f73c30283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:49:12.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:12 smithi173 ceph-mon[33544]: 5.1 scrub starts 2024-03-12T03:49:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:12 smithi173 ceph-mon[33544]: 5.1 scrub ok 2024-03-12T03:49:13.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:13.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:13.060 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:49:13.059192+0000 min lat: 0.00147924 max lat: 0.937563 avg lat: 0.0094935 2024-03-12T03:49:13.060 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:49:13.060 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 60 16 101110 101094 105.284 93.4375 0.00735859 0.0094935 2024-03-12T03:49:13.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:13.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:13.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:13.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:13.256 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:13.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:13.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:13.381 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:49:12 smithi169 podman[168583]: 2024-03-12 03:49:12.922724131 +0000 UTC m=+3.595347032 container attach f7c7f556f29b9184d93fa55253f5cfa609083dc301aa0f607c64925f73c30283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-12T03:49:13.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:13.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:13.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:13.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:13.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:49:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:49:13.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:13.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:13.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:13.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:13.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:13.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:13.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:13.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:13.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:13.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[29344]: pgmap v762: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 92 MiB/s wr, 1.47k op/s; 60708/292008 objects degraded (20.790%) 2024-03-12T03:49:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[29344]: 3.15 scrub starts 2024-03-12T03:49:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[29344]: 3.15 scrub ok 2024-03-12T03:49:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[29344]: 4.f scrub starts 2024-03-12T03:49:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[29344]: 4.f scrub ok 2024-03-12T03:49:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 60708/292008 objects degraded (20.790%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[29344]: osdmap e387: 8 total, 7 up, 8 in 2024-03-12T03:49:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[39949]: pgmap v762: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 92 MiB/s wr, 1.47k op/s; 60708/292008 objects degraded (20.790%) 2024-03-12T03:49:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[39949]: 3.15 scrub starts 2024-03-12T03:49:13.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[39949]: 3.15 scrub ok 2024-03-12T03:49:13.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[39949]: 4.f scrub starts 2024-03-12T03:49:13.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[39949]: 4.f scrub ok 2024-03-12T03:49:13.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 60708/292008 objects degraded (20.790%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:13.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:13 smithi169 ceph-mon[39949]: osdmap e387: 8 total, 7 up, 8 in 2024-03-12T03:49:13.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:13 smithi173 ceph-mon[33544]: pgmap v762: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 5.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 92 MiB/s wr, 1.47k op/s; 60708/292008 objects degraded (20.790%) 2024-03-12T03:49:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:13 smithi173 ceph-mon[33544]: 3.15 scrub starts 2024-03-12T03:49:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:13 smithi173 ceph-mon[33544]: 3.15 scrub ok 2024-03-12T03:49:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:13 smithi173 ceph-mon[33544]: 4.f scrub starts 2024-03-12T03:49:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:13 smithi173 ceph-mon[33544]: 4.f scrub ok 2024-03-12T03:49:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:13 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 60708/292008 objects degraded (20.790%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:13 smithi173 ceph-mon[33544]: osdmap e387: 8 total, 7 up, 8 in 2024-03-12T03:49:14.009 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:49:14.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:14.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:14.060 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 61 16 102492 102476 104.974 86.375 0.0108133 0.00952166 2024-03-12T03:49:14.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:14.149 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:14.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:14.250 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:14.285 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:14.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:14.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:14.423 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:49:14 smithi169 podman[168583]: 2024-03-12 03:49:14.008326826 +0000 UTC m=+4.680949730 container died f7c7f556f29b9184d93fa55253f5cfa609083dc301aa0f607c64925f73c30283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e) 2024-03-12T03:49:14.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:14.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:14.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:14.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:14.570 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:14.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:14.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:14.707 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:14.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:14.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:14.840 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:14.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:14 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:14 smithi169 ceph-mon[29344]: 6.4 scrub starts 2024-03-12T03:49:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:14 smithi169 ceph-mon[29344]: 6.4 scrub ok 2024-03-12T03:49:14.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:14 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:14 smithi169 ceph-mon[39949]: 6.4 scrub starts 2024-03-12T03:49:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:14 smithi169 ceph-mon[39949]: 6.4 scrub ok 2024-03-12T03:49:14.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:14.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:14.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:14 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:14 smithi173 ceph-mon[33544]: 6.4 scrub starts 2024-03-12T03:49:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:14 smithi173 ceph-mon[33544]: 6.4 scrub ok 2024-03-12T03:49:15.060 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 62 16 103946 103930 104.747 90.875 0.00591786 0.00954254 2024-03-12T03:49:15.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:15.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:15.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:15.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:15.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:15.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:15.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:15.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:15.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:15.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:15.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:15.496 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:15.549 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:15.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:15.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:15.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.0.service' 2024-03-12T03:49:15.694 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:15.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:15.727 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:49:15.727 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-12T03:49:15.727 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-03-12T03:49:15.727 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi169.front.sepia.ceph.com 2024-03-12T03:49:15.728 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.0 --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-03-12T03:49:15.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:15.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:15.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:15.907 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:15 smithi173 ceph-mon[33544]: pgmap v764: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s; 61600/298716 objects degraded (20.622%) 2024-03-12T03:49:15.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:15 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:49:15.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:15 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:49:15.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:15 smithi173 ceph-mon[33544]: 3.19 scrub starts 2024-03-12T03:49:15.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:15 smithi173 ceph-mon[33544]: 3.19 scrub ok 2024-03-12T03:49:15.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:15 smithi173 ceph-mon[33544]: osdmap e388: 8 total, 7 up, 8 in 2024-03-12T03:49:15.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:15 smithi173 ceph-mon[33544]: 6.2 scrub starts 2024-03-12T03:49:15.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:15 smithi173 ceph-mon[33544]: 6.2 scrub ok 2024-03-12T03:49:15.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[29344]: pgmap v764: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s; 61600/298716 objects degraded (20.622%) 2024-03-12T03:49:15.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[29344]: 3.19 scrub starts 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[29344]: 3.19 scrub ok 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[29344]: osdmap e388: 8 total, 7 up, 8 in 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[29344]: 6.2 scrub starts 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[29344]: 6.2 scrub ok 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[39949]: pgmap v764: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s; 61600/298716 objects degraded (20.622%) 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[39949]: 3.19 scrub starts 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[39949]: 3.19 scrub ok 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[39949]: osdmap e388: 8 total, 7 up, 8 in 2024-03-12T03:49:15.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[39949]: 6.2 scrub starts 2024-03-12T03:49:15.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:15 smithi169 ceph-mon[39949]: 6.2 scrub ok 2024-03-12T03:49:15.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:16.005 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:16.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:16.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:16.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:49:16.060 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 63 16 105354 105338 104.48 88 0.00856646 0.009567 2024-03-12T03:49:16.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:16.122 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:49:16.122 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:49:16.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:16.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:16.175 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/osd.0/config 2024-03-12T03:49:16.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:16.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:16.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:16.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:16.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:16.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:16.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:16.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:16.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:16.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:16.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:16.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:16.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:16.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:16.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:16.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:16.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:16.949 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:16 smithi169 ceph-mon[29344]: 1.9 scrub starts 2024-03-12T03:49:16.949 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:16 smithi169 ceph-mon[29344]: 1.9 scrub ok 2024-03-12T03:49:16.949 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:16 smithi169 ceph-mon[29344]: 4.4 scrub starts 2024-03-12T03:49:16.949 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:16 smithi169 ceph-mon[29344]: 4.4 scrub ok 2024-03-12T03:49:16.949 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:16 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:16.949 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:16 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:16.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:16 smithi169 ceph-mon[39949]: 1.9 scrub starts 2024-03-12T03:49:16.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:16 smithi169 ceph-mon[39949]: 1.9 scrub ok 2024-03-12T03:49:16.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:16 smithi169 ceph-mon[39949]: 4.4 scrub starts 2024-03-12T03:49:16.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:16 smithi169 ceph-mon[39949]: 4.4 scrub ok 2024-03-12T03:49:16.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:16 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:16.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:16 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:16.981 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:16 smithi173 ceph-mon[33544]: 1.9 scrub starts 2024-03-12T03:49:16.981 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:16 smithi173 ceph-mon[33544]: 1.9 scrub ok 2024-03-12T03:49:16.981 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:16 smithi173 ceph-mon[33544]: 4.4 scrub starts 2024-03-12T03:49:16.981 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:16 smithi173 ceph-mon[33544]: 4.4 scrub ok 2024-03-12T03:49:16.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:16 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:16.982 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:16 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:16.982 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:16.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:16.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-12T03:49:17.061 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 64 16 106711 106695 104.173 84.8125 0.00755165 0.0095949 2024-03-12T03:49:17.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:17.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:17.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:17.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:17.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:17.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:17.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:17.387 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:17.420 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:17.421 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:49:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:17.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:17.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:17.557 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:17.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:17.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:17.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:17.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:17.738 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:17.805 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:17.834 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:17.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:17.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:17.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:17.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:17 smithi173 ceph-mon[33544]: pgmap v766: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s; 61600/298716 objects degraded (20.622%) 2024-03-12T03:49:17.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:17 smithi173 ceph-mon[33544]: 4.17 scrub starts 2024-03-12T03:49:17.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:17 smithi173 ceph-mon[33544]: 4.17 scrub ok 2024-03-12T03:49:17.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:17 smithi173 ceph-mon[33544]: 6.f scrub starts 2024-03-12T03:49:17.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:17 smithi173 ceph-mon[33544]: 6.f scrub ok 2024-03-12T03:49:17.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:17 smithi173 ceph-mon[33544]: 1.8 deep-scrub starts 2024-03-12T03:49:17.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:17 smithi173 ceph-mon[33544]: 1.8 deep-scrub ok 2024-03-12T03:49:17.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:17.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:18.034 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[29344]: pgmap v766: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s; 61600/298716 objects degraded (20.622%) 2024-03-12T03:49:18.034 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[29344]: 4.17 scrub starts 2024-03-12T03:49:18.034 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[29344]: 4.17 scrub ok 2024-03-12T03:49:18.034 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[29344]: 6.f scrub starts 2024-03-12T03:49:18.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[29344]: 6.f scrub ok 2024-03-12T03:49:18.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[29344]: 1.8 deep-scrub starts 2024-03-12T03:49:18.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[29344]: 1.8 deep-scrub ok 2024-03-12T03:49:18.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[39949]: pgmap v766: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s; 61600/298716 objects degraded (20.622%) 2024-03-12T03:49:18.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[39949]: 4.17 scrub starts 2024-03-12T03:49:18.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[39949]: 4.17 scrub ok 2024-03-12T03:49:18.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[39949]: 6.f scrub starts 2024-03-12T03:49:18.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[39949]: 6.f scrub ok 2024-03-12T03:49:18.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[39949]: 1.8 deep-scrub starts 2024-03-12T03:49:18.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:17 smithi169 ceph-mon[39949]: 1.8 deep-scrub ok 2024-03-12T03:49:18.061 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 65 16 108024 108008 103.833 82.0625 0.00420625 0.00962517 2024-03-12T03:49:18.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:18.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:18.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:18.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:18.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:18.245 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:18.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:18.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:18.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:18.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:18.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:18.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:18.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:18.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:18.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:18.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:18.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:18.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:18.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:18.800 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:18.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:18.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:18.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:18.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:18 smithi173 ceph-mon[33544]: 4.9 scrub starts 2024-03-12T03:49:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:18 smithi173 ceph-mon[33544]: 4.9 scrub ok 2024-03-12T03:49:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:18 smithi173 ceph-mon[33544]: 5.17 scrub starts 2024-03-12T03:49:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:18 smithi173 ceph-mon[33544]: 5.17 scrub ok 2024-03-12T03:49:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:18 smithi173 ceph-mon[33544]: 4.c deep-scrub starts 2024-03-12T03:49:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:18 smithi173 ceph-mon[33544]: 4.c deep-scrub ok 2024-03-12T03:49:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:18 smithi173 ceph-mon[33544]: 10.3 deep-scrub starts 2024-03-12T03:49:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:18 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 61600/298716 objects degraded (20.622%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:18 smithi173 ceph-mon[33544]: osdmap e389: 8 total, 7 up, 8 in 2024-03-12T03:49:19.037 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:19.061 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 66 16 109136 109120 103.313 69.5 0.0236331 0.00967312 2024-03-12T03:49:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:19.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:19.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:19.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[29344]: 4.9 scrub starts 2024-03-12T03:49:19.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[29344]: 4.9 scrub ok 2024-03-12T03:49:19.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[29344]: 5.17 scrub starts 2024-03-12T03:49:19.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[29344]: 5.17 scrub ok 2024-03-12T03:49:19.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[29344]: 4.c deep-scrub starts 2024-03-12T03:49:19.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[29344]: 4.c deep-scrub ok 2024-03-12T03:49:19.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[29344]: 10.3 deep-scrub starts 2024-03-12T03:49:19.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 61600/298716 objects degraded (20.622%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:19.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[29344]: osdmap e389: 8 total, 7 up, 8 in 2024-03-12T03:49:19.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[39949]: 4.9 scrub starts 2024-03-12T03:49:19.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[39949]: 4.9 scrub ok 2024-03-12T03:49:19.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[39949]: 5.17 scrub starts 2024-03-12T03:49:19.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[39949]: 5.17 scrub ok 2024-03-12T03:49:19.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[39949]: 4.c deep-scrub starts 2024-03-12T03:49:19.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[39949]: 4.c deep-scrub ok 2024-03-12T03:49:19.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[39949]: 10.3 deep-scrub starts 2024-03-12T03:49:19.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 61600/298716 objects degraded (20.622%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:19.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:18 smithi169 ceph-mon[39949]: osdmap e389: 8 total, 7 up, 8 in 2024-03-12T03:49:19.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:19.198 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:19.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:19.299 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:19.338 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:19.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:19.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:19.476 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:19.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:19.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:19.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:19.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:19.664 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:49:19.701 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:49:19.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:19.740 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[39949]: pgmap v767: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 115 MiB/s wr, 1.83k op/s; 66052/317661 objects degraded (20.793%) 2024-03-12T03:49:19.740 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[39949]: 6.5 deep-scrub starts 2024-03-12T03:49:19.740 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[39949]: 6.5 deep-scrub ok 2024-03-12T03:49:19.740 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[39949]: 10.0 scrub starts 2024-03-12T03:49:19.740 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[39949]: 4.15 scrub starts 2024-03-12T03:49:19.740 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[39949]: 4.15 scrub ok 2024-03-12T03:49:19.740 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/851908292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:19.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[29344]: pgmap v767: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 115 MiB/s wr, 1.83k op/s; 66052/317661 objects degraded (20.793%) 2024-03-12T03:49:19.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[29344]: 6.5 deep-scrub starts 2024-03-12T03:49:19.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[29344]: 6.5 deep-scrub ok 2024-03-12T03:49:19.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[29344]: 10.0 scrub starts 2024-03-12T03:49:19.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[29344]: 4.15 scrub starts 2024-03-12T03:49:19.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[29344]: 4.15 scrub ok 2024-03-12T03:49:19.741 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:19 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/851908292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:19.741 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:49:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:19.764 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:19.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:19.780 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:49:19.818 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:49:19.855 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:49:19.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:19.893 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:49:19.893 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:49:19.893 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:49:19.893 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:49:19.893 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:49:19.893 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:49:19.893 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:49:19.893 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:49:19.894 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:49:19.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:19.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:19 smithi173 ceph-mon[33544]: pgmap v767: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 115 MiB/s wr, 1.83k op/s; 66052/317661 objects degraded (20.793%) 2024-03-12T03:49:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:19 smithi173 ceph-mon[33544]: 6.5 deep-scrub starts 2024-03-12T03:49:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:19 smithi173 ceph-mon[33544]: 6.5 deep-scrub ok 2024-03-12T03:49:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:19 smithi173 ceph-mon[33544]: 10.0 scrub starts 2024-03-12T03:49:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:19 smithi173 ceph-mon[33544]: 4.15 scrub starts 2024-03-12T03:49:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:19 smithi173 ceph-mon[33544]: 4.15 scrub ok 2024-03-12T03:49:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:19 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/851908292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:20.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:20.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:20.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:20.061 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 67 16 110190 110174 102.754 65.875 0.0079144 0.00972702 2024-03-12T03:49:20.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:20.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:20.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:20.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:20.313 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:20.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:20.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:20.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:20.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:20.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:20.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:20.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:20.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:20.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:20.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:20.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:20.753 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:20.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:20.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:20.874 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:20 smithi173 ceph-mon[33544]: 5.a scrub starts 2024-03-12T03:49:20.874 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:20 smithi173 ceph-mon[33544]: 5.a scrub ok 2024-03-12T03:49:20.874 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:20 smithi173 ceph-mon[33544]: 5.16 scrub starts 2024-03-12T03:49:20.874 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:20 smithi173 ceph-mon[33544]: 5.16 scrub ok 2024-03-12T03:49:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:20.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:20.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:21.000 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:21.028 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:21.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:21.032 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:20 smithi169 ceph-mon[29344]: 5.a scrub starts 2024-03-12T03:49:21.032 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:20 smithi169 ceph-mon[29344]: 5.a scrub ok 2024-03-12T03:49:21.032 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:20 smithi169 ceph-mon[29344]: 5.16 scrub starts 2024-03-12T03:49:21.032 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:20 smithi169 ceph-mon[29344]: 5.16 scrub ok 2024-03-12T03:49:21.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:20 smithi169 ceph-mon[39949]: 5.a scrub starts 2024-03-12T03:49:21.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:20 smithi169 ceph-mon[39949]: 5.a scrub ok 2024-03-12T03:49:21.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:20 smithi169 ceph-mon[39949]: 5.16 scrub starts 2024-03-12T03:49:21.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:20 smithi169 ceph-mon[39949]: 5.16 scrub ok 2024-03-12T03:49:21.061 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 68 16 111342 111326 102.302 72 0.00494944 0.00976704 2024-03-12T03:49:21.083 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:49:21.083 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:49:21.083 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:49:21.083 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:49:21.083 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:49:21.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:21.166 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:21.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:21.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:21.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:21.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:21.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:21.440 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:21.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:21.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:21.574 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:21.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:21.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:21.701 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:49:21.701 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:49:21.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:21.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:21.730 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:49:21.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:21.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:21.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:21.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:21 smithi169 ceph-mon[39949]: 5.1c scrub starts 2024-03-12T03:49:21.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:21 smithi169 ceph-mon[39949]: 5.1c scrub ok 2024-03-12T03:49:21.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:21 smithi169 ceph-mon[39949]: pgmap v769: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 765 B/s rd, 75 MiB/s wr, 1.21k op/s; 66052/320223 objects degraded (20.627%) 2024-03-12T03:49:21.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:21 smithi169 ceph-mon[39949]: 10.0 scrub ok 2024-03-12T03:49:21.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:21 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2311654688' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:49:21.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:21 smithi169 ceph-mon[29344]: 5.1c scrub starts 2024-03-12T03:49:21.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:21 smithi169 ceph-mon[29344]: 5.1c scrub ok 2024-03-12T03:49:21.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:21 smithi169 ceph-mon[29344]: pgmap v769: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 765 B/s rd, 75 MiB/s wr, 1.21k op/s; 66052/320223 objects degraded (20.627%) 2024-03-12T03:49:21.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:21 smithi169 ceph-mon[29344]: 10.0 scrub ok 2024-03-12T03:49:21.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:21 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2311654688' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:49:21.924 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:49:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:49:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:49:21.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:21.976 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:21.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:21 smithi173 ceph-mon[33544]: 5.1c scrub starts 2024-03-12T03:49:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:21 smithi173 ceph-mon[33544]: 5.1c scrub ok 2024-03-12T03:49:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:21 smithi173 ceph-mon[33544]: pgmap v769: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 765 B/s rd, 75 MiB/s wr, 1.21k op/s; 66052/320223 objects degraded (20.627%) 2024-03-12T03:49:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:21 smithi173 ceph-mon[33544]: 10.0 scrub ok 2024-03-12T03:49:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:21 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2311654688' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:49:22.051 INFO:teuthology.orchestra.run.smithi169.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-12T03:49:22.061 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 69 16 112443 112427 101.816 68.8125 0.00903856 0.0098176 2024-03-12T03:49:22.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:22.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:22.114 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:22.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:22.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:22.250 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:22.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:22.352 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:22.383 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:22.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:22.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:22.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:22.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:22.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:22.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:22.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:22.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:22.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:22.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:22.787 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:22.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:22.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:22.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:22.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:22.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:22.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:22 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:49:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:22 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2311654688' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:49:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:22 smithi173 ceph-mon[33544]: osdmap e390: 8 total, 7 up, 8 in 2024-03-12T03:49:23.061 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 70 15 113692 113677 101.478 78.125 0.00965205 0.0098491 2024-03-12T03:49:23.069 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:23.104 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:23.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:23.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:22 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:49:23.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:22 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2311654688' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:49:23.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:22 smithi169 ceph-mon[39949]: osdmap e390: 8 total, 7 up, 8 in 2024-03-12T03:49:23.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:22 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:49:23.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:22 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2311654688' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:49:23.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:22 smithi169 ceph-mon[29344]: osdmap e390: 8 total, 7 up, 8 in 2024-03-12T03:49:23.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:23.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:23.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:23.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:23.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:23.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:23.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:23.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:49:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:49:23.523 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:23.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:23.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:23.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:23.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:23.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:23.702 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:49:23.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:23.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-12T03:49:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:23.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:23.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:23.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:23 smithi173 ceph-mon[33544]: pgmap v771: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 69003/332406 objects degraded (20.759%) 2024-03-12T03:49:23.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:23 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 69003/332406 objects degraded (20.759%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:23.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:23 smithi173 ceph-mon[33544]: osdmap e391: 8 total, 7 up, 8 in 2024-03-12T03:49:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:24.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:24.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:24.061 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 71 16 114790 114774 101.014 68.5625 0.0131459 0.00989507 2024-03-12T03:49:24.077 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:24.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:24.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:24.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:24.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:23 smithi169 ceph-mon[29344]: pgmap v771: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 69003/332406 objects degraded (20.759%) 2024-03-12T03:49:24.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:23 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 69003/332406 objects degraded (20.759%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:24.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:23 smithi169 ceph-mon[29344]: osdmap e391: 8 total, 7 up, 8 in 2024-03-12T03:49:24.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:23 smithi169 ceph-mon[39949]: pgmap v771: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 69003/332406 objects degraded (20.759%) 2024-03-12T03:49:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:23 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 69003/332406 objects degraded (20.759%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:23 smithi169 ceph-mon[39949]: osdmap e391: 8 total, 7 up, 8 in 2024-03-12T03:49:24.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:24.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:24.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:24.333 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:49:24.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:24.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:24.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:24.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:24.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:24.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:24.725 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:24.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:24.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:24.860 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:24.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:24.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:24.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:25.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:25.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:25.012 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:25.012 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:25.065 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 72 15 115960 115945 100.623 73.1875 0.00782765 0.00993177 2024-03-12T03:49:25.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:25.142 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:25.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:25.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:25.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:25.282 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:25.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:25.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:25.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:25.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:25.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:25.520 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:25.520 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:49:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:25.564 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:25.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:25.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:25.735 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:25.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:25.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:25.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:25.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:25.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:25.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:25.901 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:25 smithi169 ceph-mon[29344]: pgmap v773: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.11k op/s; 69712/337704 objects degraded (20.643%) 2024-03-12T03:49:25.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:25 smithi169 ceph-mon[39949]: pgmap v773: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.11k op/s; 69712/337704 objects degraded (20.643%) 2024-03-12T03:49:25.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:25 smithi173 ceph-mon[33544]: pgmap v773: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.11k op/s; 69712/337704 objects degraded (20.643%) 2024-03-12T03:49:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:26.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:26.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:26.065 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 73 16 117183 117167 100.29 76.375 0.00837854 0.00996535 2024-03-12T03:49:26.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:26.129 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:49:26.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:26.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:26.169 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:49:26.172 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:26.206 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:49:26.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:26.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:26.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:26.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:26.302 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:49:26.341 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:49:26.379 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:49:26.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:26.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:26.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:26.422 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:49:26.422 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:49:26.423 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:49:26.423 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:49:26.423 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:49:26.423 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:49:26.423 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:49:26.423 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:49:26.423 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:49:26.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:26.557 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:26.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:26.559 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:49:26.559 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:49:26.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:26.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:26.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:26.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:26.844 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:26.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:26.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:26.949 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:26 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4038421492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:26.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:26.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:26.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:27.065 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 74 16 118367 118351 99.9348 74 0.0100169 0.0100009 2024-03-12T03:49:27.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:27.117 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:27.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:27.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:26 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4038421492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:27.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:26 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4038421492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:27.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:27.253 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:27.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:27.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-12T03:49:27.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:27.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:27.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:27.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:27.501 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:27.540 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:27.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:27.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:27.674 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:27.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:27.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:27.701 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:49:27.702 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:49:27.715 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:49:27.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:27.814 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:27.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:27.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:27.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:27 smithi169 ceph-mon[39949]: pgmap v774: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 665 B/s rd, 54 MiB/s wr, 869 op/s; 69712/337704 objects degraded (20.643%) 2024-03-12T03:49:27.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:27 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:49:27.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:27 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:49:27.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:27 smithi169 ceph-mon[39949]: osdmap e392: 8 total, 7 up, 8 in 2024-03-12T03:49:27.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:27 smithi169 ceph-mon[29344]: pgmap v774: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 665 B/s rd, 54 MiB/s wr, 869 op/s; 69712/337704 objects degraded (20.643%) 2024-03-12T03:49:27.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:27 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:49:27.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:27 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:49:27.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:27 smithi169 ceph-mon[29344]: osdmap e392: 8 total, 7 up, 8 in 2024-03-12T03:49:27.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:27.954 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:27.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:28.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:28.065 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 75 16 119619 119603 99.6456 78.25 0.00858886 0.0100297 2024-03-12T03:49:28.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:28.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:28.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:28.224 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:28.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:28.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:28.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:27 smithi173 ceph-mon[33544]: pgmap v774: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 665 B/s rd, 54 MiB/s wr, 869 op/s; 69712/337704 objects degraded (20.643%) 2024-03-12T03:49:28.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:27 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:49:28.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:27 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:49:28.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:27 smithi173 ceph-mon[33544]: osdmap e392: 8 total, 7 up, 8 in 2024-03-12T03:49:28.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:28.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:28.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:28.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:28.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:28.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:28.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:28.646 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:28.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:28.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:28.791 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:28.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:28.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:28.922 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:28.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:28 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 69712/337704 objects degraded (20.643%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:28 smithi169 ceph-mon[29344]: osdmap e393: 8 total, 7 up, 8 in 2024-03-12T03:49:28.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:28 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 69712/337704 objects degraded (20.643%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:28.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:28 smithi169 ceph-mon[39949]: osdmap e393: 8 total, 7 up, 8 in 2024-03-12T03:49:28.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:28 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 69712/337704 objects degraded (20.643%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:28.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:28 smithi173 ceph-mon[33544]: osdmap e393: 8 total, 7 up, 8 in 2024-03-12T03:49:29.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:29.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:29.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:29.065 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 76 16 120870 120854 99.3632 78.1875 0.00482495 0.0100562 2024-03-12T03:49:29.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:29.193 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:29.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:29.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:29.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:29.332 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:29.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:29.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:29.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:29.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:29.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:29.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:29.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:29.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:29.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:29.703 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:49:29.716 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:29.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:29.806 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:29.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:29.862 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:29 smithi173 ceph-mon[33544]: pgmap v776: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 798 B/s rd, 67 MiB/s wr, 1.07k op/s; 73368/353001 objects degraded (20.784%) 2024-03-12T03:49:29.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:29 smithi169 ceph-mon[29344]: pgmap v776: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 798 B/s rd, 67 MiB/s wr, 1.07k op/s; 73368/353001 objects degraded (20.784%) 2024-03-12T03:49:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:29 smithi169 ceph-mon[39949]: pgmap v776: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 798 B/s rd, 67 MiB/s wr, 1.07k op/s; 73368/353001 objects degraded (20.784%) 2024-03-12T03:49:29.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:29.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:30.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:30.065 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 77 16 121956 121940 98.9542 67.875 0.0210822 0.0101001 2024-03-12T03:49:30.111 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:30.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:30.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:30.199 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:30.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:30.270 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:30.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:30.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:30.418 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:30.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:30.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:30.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:30.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:30.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:30.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:30.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:30.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:30.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:30.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:30.802 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:49:30.837 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:30.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:30.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:30.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:30.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:30 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2100422380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:49:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:30 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:49:31.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 78 16 123072 123056 98.5798 69.75 0.00776046 0.0101397 2024-03-12T03:49:31.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:31.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:31.144 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:31.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:31.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:30 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2100422380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:49:31.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:30 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:49:31.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:30 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2100422380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:49:31.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:30 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:49:31.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:31.284 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:31.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:31.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:31.386 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:31.420 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:31.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:31.436 INFO:teuthology.orchestra.run.smithi169.stdout:6.d 2024-03-12T03:49:31.436 INFO:teuthology.orchestra.run.smithi169.stdout:6.18 2024-03-12T03:49:31.436 INFO:teuthology.orchestra.run.smithi169.stdout:6.16 2024-03-12T03:49:31.436 INFO:teuthology.orchestra.run.smithi169.stdout:6.13 2024-03-12T03:49:31.436 INFO:teuthology.orchestra.run.smithi169.stdout:6.11 2024-03-12T03:49:31.436 INFO:teuthology.orchestra.run.smithi169.stdout:6.10 2024-03-12T03:49:31.436 INFO:teuthology.orchestra.run.smithi169.stdout:5.c 2024-03-12T03:49:31.436 INFO:teuthology.orchestra.run.smithi169.stdout:5.7 2024-03-12T03:49:31.436 INFO:teuthology.orchestra.run.smithi169.stdout:5.b 2024-03-12T03:49:31.436 INFO:teuthology.orchestra.run.smithi169.stdout:5.5 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:6.a 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:5.2 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:5.1d 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:5.1b 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:5.1e 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:5.15 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:6.0 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:5.13 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:4.b 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:4.3 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:4.7 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:1.6 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:4.1e 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:10.a 2024-03-12T03:49:31.437 INFO:teuthology.orchestra.run.smithi169.stdout:5.3 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:10.5 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:10.c 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:3.17 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:10.4 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:4.0 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:1.2 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:5.e 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:3.8 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:10.6 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:1.0 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:10.1 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:10.8 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:4.16 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:2.1 2024-03-12T03:49:31.438 INFO:teuthology.orchestra.run.smithi169.stdout:10.b 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:1.c 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:3.f 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:6.9 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:1.b 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:1.11 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:1.1 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:4.6 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:10.d 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:3.12 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:2.0 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:3.10 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:3.18 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:5.8 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:4.19 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:3.1 2024-03-12T03:49:31.439 INFO:teuthology.orchestra.run.smithi169.stdout:3.1f 2024-03-12T03:49:31.440 INFO:teuthology.orchestra.run.smithi169.stdout:3.3 2024-03-12T03:49:31.440 INFO:teuthology.orchestra.run.smithi169.stdout:3.1b 2024-03-12T03:49:31.440 INFO:teuthology.orchestra.run.smithi169.stdout:3.6 2024-03-12T03:49:31.440 INFO:teuthology.orchestra.run.smithi169.stdout:3.b 2024-03-12T03:49:31.440 INFO:teuthology.orchestra.run.smithi169.stdout:10.2 2024-03-12T03:49:31.440 INFO:teuthology.orchestra.run.smithi169.stdout:3.7 2024-03-12T03:49:31.440 INFO:teuthology.orchestra.run.smithi169.stdout:4.1a 2024-03-12T03:49:31.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:31.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:31.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:31.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:31.696 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:31.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:31.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:31.774 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:49:31.774 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:49:31.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:31.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:31.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:31.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:31.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:31.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:31 smithi173 ceph-mon[33544]: pgmap v778: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 765 B/s rd, 62 MiB/s wr, 989 op/s; 73368/355407 objects degraded (20.643%) 2024-03-12T03:49:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:31 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:49:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:31 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:49:31.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:31 smithi173 ceph-mon[33544]: osdmap e394: 8 total, 7 up, 8 in 2024-03-12T03:49:31.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:31 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:31.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:32.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 79 16 124359 124343 98.3501 80.4375 0.00976587 0.010163 2024-03-12T03:49:32.080 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:32.116 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:32.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:32.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:32.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-mon[29344]: pgmap v778: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 765 B/s rd, 62 MiB/s wr, 989 op/s; 73368/355407 objects degraded (20.643%) 2024-03-12T03:49:32.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:49:32.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:49:32.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-mon[29344]: osdmap e394: 8 total, 7 up, 8 in 2024-03-12T03:49:32.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:32.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:32.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:49:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:49:32.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-mon[39949]: pgmap v778: 168 pgs: 1 active+clean+scrubbing+deep, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 765 B/s rd, 62 MiB/s wr, 989 op/s; 73368/355407 objects degraded (20.643%) 2024-03-12T03:49:32.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:49:32.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:49:32.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-mon[39949]: osdmap e394: 8 total, 7 up, 8 in 2024-03-12T03:49:32.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:32.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:32.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:32.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:32.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:32.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:32.418 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:32.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:32.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:32.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:32.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:32.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:32.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-12T03:49:32.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:32.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:32.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:32.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:32.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:32.848 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:32.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:32.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:32.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:32.982 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:32.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:33.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:49:33.064742+0000 min lat: 0.00147924 max lat: 0.937563 avg lat: 0.0101888 2024-03-12T03:49:33.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:49:33.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 80 16 125613 125597 98.1004 78.375 0.00513302 0.0101888 2024-03-12T03:49:33.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:33.113 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:33.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:33.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:49:33.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:33.266 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:33.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:33.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:33.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:33.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:33.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:49:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:49:33.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:33.544 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:33.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:33.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:33.676 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:33.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:33.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:33.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:33.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:33.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:33.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:33 smithi169 ceph-mon[29344]: pgmap v780: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 76391/367737 objects degraded (20.773%) 2024-03-12T03:49:33.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:33 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 76391/367737 objects degraded (20.773%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:33.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:33 smithi169 ceph-mon[29344]: osdmap e395: 8 total, 7 up, 8 in 2024-03-12T03:49:33.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:33 smithi169 ceph-mon[39949]: pgmap v780: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 76391/367737 objects degraded (20.773%) 2024-03-12T03:49:33.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:33 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 76391/367737 objects degraded (20.773%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:33.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:33 smithi169 ceph-mon[39949]: osdmap e395: 8 total, 7 up, 8 in 2024-03-12T03:49:33.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:33.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:33.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:34.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:34.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 81 16 126882 126866 97.8684 79.3125 0.00331579 0.0102133 2024-03-12T03:49:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:34.094 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:34.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:34.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:34.230 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:34.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:34.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:33 smithi173 ceph-mon[33544]: pgmap v780: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 76391/367737 objects degraded (20.773%) 2024-03-12T03:49:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:33 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 76391/367737 objects degraded (20.773%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:33 smithi173 ceph-mon[33544]: osdmap e395: 8 total, 7 up, 8 in 2024-03-12T03:49:34.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:34.369 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:34.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:34.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:34.494 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:34.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:34.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:34.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:34.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:34.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:34.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:34.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:34.780 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:34.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:34.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:34.906 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:34.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:34.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:35.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:35.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:35.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:35.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 82 16 127967 127951 97.5018 67.8125 0.00752045 0.010252 2024-03-12T03:49:35.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:35.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:35.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:35.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:35.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:35.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:35.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:35.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:35.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:35.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:35.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:35.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:35.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:35.553 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:35.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:35.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:35.718 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:35.747 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:35.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:35.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:35.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:35.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:35.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:36.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:36.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:36.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 83 16 129339 129323 97.3602 85.75 0.0106896 0.010267 2024-03-12T03:49:36.111 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:35 smithi173 ceph-mon[33544]: pgmap v782: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.13k op/s; 77111/373329 objects degraded (20.655%) 2024-03-12T03:49:36.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:36.143 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:36.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:36.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:36.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:35 smithi169 ceph-mon[29344]: pgmap v782: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.13k op/s; 77111/373329 objects degraded (20.655%) 2024-03-12T03:49:36.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:35 smithi169 ceph-mon[39949]: pgmap v782: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.13k op/s; 77111/373329 objects degraded (20.655%) 2024-03-12T03:49:36.245 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:36.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:36.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:36.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:36.308 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:49:36.308 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:49:36.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:36.414 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:36.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:36.516 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:36.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:36.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:36.694 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:36.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:36.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:36.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:36.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:36.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:36.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:36.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:36.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:36.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:37.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-12T03:49:37.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 84 16 130706 130690 97.2182 85.4375 0.0140114 0.0102821 2024-03-12T03:49:37.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:37.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:37.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:37.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:37.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:37.278 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:37.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:37.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:37.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:37.430 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:37.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:37.439 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:49:37.466 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:49:37.469 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:49:37.502 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:49:37.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:37.547 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:49:37.563 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:37.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:37.587 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:49:37.629 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:49:37.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:37.665 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:49:37.666 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:49:37.666 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:49:37.667 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:49:37.667 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:49:37.667 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:49:37.667 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:49:37.667 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:49:37.667 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:49:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:37.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:37.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:37.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:37.837 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:37.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:37.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:37.928 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:37 smithi173 ceph-mon[33544]: pgmap v783: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 665 B/s rd, 55 MiB/s wr, 881 op/s; 77111/373329 objects degraded (20.655%) 2024-03-12T03:49:37.928 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:37 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4036984441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:37.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:37.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:37.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:37.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.0 --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 5.1d --file /var/log/ceph/exp.5.1d.0 2024-03-12T03:49:38.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 85 16 132069 132053 97.0766 85.1875 0.00873627 0.0102973 2024-03-12T03:49:38.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:38.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:38.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:38.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:37 smithi169 ceph-mon[29344]: pgmap v783: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 665 B/s rd, 55 MiB/s wr, 881 op/s; 77111/373329 objects degraded (20.655%) 2024-03-12T03:49:38.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:37 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4036984441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:38.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:37 smithi169 ceph-mon[39949]: pgmap v783: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 665 B/s rd, 55 MiB/s wr, 881 op/s; 77111/373329 objects degraded (20.655%) 2024-03-12T03:49:38.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:37 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4036984441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:38.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:38.269 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:38.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:38.273 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/osd.0/config 2024-03-12T03:49:38.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:38.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:38.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:38.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:38.542 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:38.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:38.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:38.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:49:38.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:38.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:38.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:38.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:38.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:38.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:38.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:38.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:38.965 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:38.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:38.983 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:38 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:49:38.983 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:38 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:49:38.983 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:38 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 77111/373329 objects degraded (20.655%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:39.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 86 16 133490 133474 96.9804 88.8125 0.0121152 0.0103072 2024-03-12T03:49:39.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:39.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:39.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:39.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:38 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:49:39.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:38 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:49:39.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:38 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 77111/373329 objects degraded (20.655%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:39.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:38 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:49:39.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:38 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:49:39.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:38 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 77111/373329 objects degraded (20.655%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:39.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:39.245 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:39.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:39.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:39.380 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:39.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:39.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:39.513 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:39.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:39.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:39.627 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:49:39.627 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:49:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:39.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:39.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:39.787 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:39.819 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:39.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:39.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:39.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:39.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:39 smithi169 ceph-mon[29344]: pgmap v784: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 81000/389739 objects degraded (20.783%) 2024-03-12T03:49:39.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:39 smithi169 ceph-mon[29344]: osdmap e396: 8 total, 7 up, 8 in 2024-03-12T03:49:39.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:39 smithi169 ceph-mon[39949]: pgmap v784: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 81000/389739 objects degraded (20.783%) 2024-03-12T03:49:39.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:39 smithi169 ceph-mon[39949]: osdmap e396: 8 total, 7 up, 8 in 2024-03-12T03:49:39.950 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:39.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:39.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:39 smithi173 ceph-mon[33544]: pgmap v784: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 81000/389739 objects degraded (20.783%) 2024-03-12T03:49:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:39 smithi173 ceph-mon[33544]: osdmap e396: 8 total, 7 up, 8 in 2024-03-12T03:49:40.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:40.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 87 15 134924 134909 96.8965 89.6875 0.0248118 0.0103158 2024-03-12T03:49:40.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:40.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:40.097 INFO:teuthology.orchestra.run.smithi169.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-12T03:49:40.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:40.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:40.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:40.270 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:49:40.270 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:49:40.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:49:40.270 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:49:40.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.270 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:207226 8.5 8:afc51c0f:::benchmark_data_smithi169_106492_object73743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-03-12T03:49:40.270 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:25.224290+0000", 2024-03-12T03:49:40.270 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 435.04010810699998, 2024-03-12T03:49:40.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18384750899999999, 2024-03-12T03:49:40.270 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.270 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 207226 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224290+0000", 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224290+0000", 2024-03-12T03:49:40.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224291+0000", 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.204e-06 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224295+0000", 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.296e-06 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224303+0000", 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0160000000000008e-06 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224311+0000", 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4599999999999997e-06 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.230995+0000", 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0066846989999999997 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.231191+0000", 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000196202 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.231333+0000", 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000141361 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.235484+0000", 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041509989999999998 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.256368+0000", 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020884171 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.407984+0000", 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.151616267 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.408078+0000", 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3574999999999998e-05 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.408137+0000", 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9259000000000003e-05 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:210664 8.5 8:ac07f159:::benchmark_data_smithi169_106492_object77181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:29.292061+0000", 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 430.97233717, 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19662268099999999, 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.276 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 210664 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292061+0000", 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292061+0000", 2024-03-12T03:49:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292062+0000", 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.483e-06 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292066+0000", 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1309999999999999e-06 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292076+0000", 2024-03-12T03:49:40.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5689999999999994e-06 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292082+0000", 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6760000000000001e-06 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.359067+0000", 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.066984848 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.359154+0000", 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7153999999999999e-05 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.359237+0000", 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3061999999999999e-05 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.365384+0000", 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006146744 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:49:40.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.368955+0000", 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035707500000000001 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.488598+0000", 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.119642817 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.488634+0000", 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6406999999999999e-05 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.488683+0000", 2024-03-12T03:49:40.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9039999999999998e-05 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:210936 8.5 8:a7cf8e7a:::benchmark_data_smithi169_106492_object77453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:29.560667+0000", 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 430.70373104100003, 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17761102000000001, 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 210936 2024-03-12T03:49:40.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560667+0000", 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560667+0000", 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560668+0000", 2024-03-12T03:49:40.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.201e-06 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560673+0000", 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8840000000000002e-06 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560685+0000", 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1907e-05 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560691+0000", 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4670000000000001e-06 2024-03-12T03:49:40.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.626200+0000", 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.065508937000000003 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.626307+0000", 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107222 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.626416+0000", 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108634 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.632883+0000", 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00646724 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.634125+0000", 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012422920000000001 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.738168+0000", 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104042204 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.738216+0000", 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8018e-05 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.738278+0000", 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2014000000000006e-05 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:258442 9.1 9:8a150de2:::benchmark_data_smithi169_135229_object114227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:47:00.807043+0000", 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 159.45735429600001, 2024-03-12T03:49:40.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.030208796, 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 258442 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807043+0000", 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807043+0000", 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807045+0000", 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8300000000000001e-06 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807051+0000", 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.728e-06 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807062+0000", 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1297e-05 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807078+0000", 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5784999999999999e-05 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807943+0000", 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086520399999999995 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.808076+0000", 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000132724 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.808206+0000", 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013005100000000001 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.810938+0000", 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027315830000000001 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.813484+0000", 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025461030000000001 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.836790+0000", 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.023306473000000001 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.836901+0000", 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011044699999999999 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.837252+0000", 2024-03-12T03:49:40.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035157100000000001 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78615 10.e e383/366 10:7dfe4819:::benchmark_data_smithi169_162694_object78614:head v 383'4854, mlcod=383'4852)", 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.988349+0000", 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.276048692000003, 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.042519055, 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988349+0000", 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988349+0000", 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988350+0000", 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4499999999999995e-07 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988374+0000", 2024-03-12T03:49:40.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4346999999999999e-05 2024-03-12T03:49:40.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988385+0000", 2024-03-12T03:49:40.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0669000000000001e-05 2024-03-12T03:49:40.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.296 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:40.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988393+0000", 2024-03-12T03:49:40.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1259999999999998e-06 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988481+0000", 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7696999999999997e-05 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988506+0000", 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4791999999999999e-05 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.990980+0000", 2024-03-12T03:49:40.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024742309999999999 2024-03-12T03:49:40.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030868+0000", 2024-03-12T03:49:40.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039888248000000001 2024-03-12T03:49:40.298 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.298 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.298 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78612 10.d e383/366 10:b3a4751d:::benchmark_data_smithi169_162694_object78611:head v 383'4904, mlcod=383'4902)", 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.988423+0000", 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.275974900999998, 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.042485090000000003, 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988423+0000", 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988423+0000", 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988423+0000", 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2500000000000001e-07 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988443+0000", 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9653e-05 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988449+0000", 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5939999999999999e-06 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988455+0000", 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2120000000000002e-06 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988543+0000", 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8616999999999995e-05 2024-03-12T03:49:40.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988562+0000", 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8893999999999999e-05 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991063+0000", 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025009260000000001 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030908+0000", 2024-03-12T03:49:40.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039844769000000002 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78613 10.a e383/366 10:5ef8fbae:::benchmark_data_smithi169_162694_object78612:head v 383'4848, mlcod=383'4846)", 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.988636+0000", 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.275761637000002, 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.042224314999999998, 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988636+0000", 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988636+0000", 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988637+0000", 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4600000000000005e-07 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988733+0000", 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6521000000000006e-05 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988742+0000", 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1600000000000004e-06 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988749+0000", 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6409999999999996e-06 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988805+0000", 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6424000000000001e-05 2024-03-12T03:49:40.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988823+0000", 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7884e-05 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030828+0000", 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.042004527999999999 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030860+0000", 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2611000000000003e-05 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.306 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78614 10.f e383/366 10:fb4a5bd3:::benchmark_data_smithi169_162694_object78613:head v 383'4934, mlcod=383'4932)", 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.989287+0000", 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.275110294000001, 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041609650999999998, 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989287+0000", 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989287+0000", 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989288+0000", 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7199999999999998e-07 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989336+0000", 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7648999999999997e-05 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989345+0000", 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3889999999999994e-06 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989352+0000", 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7410000000000003e-06 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989426+0000", 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3819999999999995e-05 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989444+0000", 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8717000000000001e-05 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030864+0000", 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041419639000000001 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030897+0000", 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3024000000000001e-05 2024-03-12T03:49:40.310 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78618 10.7 e383/366 10:e87822aa:::benchmark_data_smithi169_162694_object78617:head v 383'4971, mlcod=383'4969)", 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.991518+0000", 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.272879195999998, 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.051062718999999999, 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991518+0000", 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991518+0000", 2024-03-12T03:49:40.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991519+0000", 2024-03-12T03:49:40.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1699999999999997e-07 2024-03-12T03:49:40.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991552+0000", 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2230999999999997e-05 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030814+0000", 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039262496000000001 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030828+0000", 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4209000000000001e-05 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030876+0000", 2024-03-12T03:49:40.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8000000000000001e-05 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030898+0000", 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2101000000000001e-05 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.042488+0000", 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011589868 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.042581+0000", 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2997000000000004e-05 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78634 10.a e383/366 10:57b48c2c:::benchmark_data_smithi169_162694_object78633:head v 383'4849, mlcod=383'4846)", 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.005400+0000", 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.258997657000002, 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.029465791000000002, 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.005400+0000", 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.005400+0000", 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.005401+0000", 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1039999999999999e-06 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.005443+0000", 2024-03-12T03:49:40.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2264000000000001e-05 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030824+0000", 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025381048 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030835+0000", 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0369e-05 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030890+0000", 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4744999999999998e-05 2024-03-12T03:49:40.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030908+0000", 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8261e-05 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.034823+0000", 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039151730000000001 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.034866+0000", 2024-03-12T03:49:40.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2827000000000002e-05 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:79202 10.b e383/366 10:dbb2f93d:::benchmark_data_smithi169_162694_object79201:head v 383'4898, mlcod=383'4896)", 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.472218+0000", 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 42.792179546, 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026937955999999999, 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.472218+0000", 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.472218+0000", 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.472219+0000", 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.54e-07 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491702+0000", 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.019482816999999999 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491709+0000", 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7909999999999999e-06 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491716+0000", 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1609999999999997e-06 2024-03-12T03:49:40.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491764+0000", 2024-03-12T03:49:40.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8424000000000003e-05 2024-03-12T03:49:40.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491783+0000", 2024-03-12T03:49:40.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8453000000000001e-05 2024-03-12T03:49:40.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.499104+0000", 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0073216410000000003 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.499156+0000", 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1915e-05 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:82510 10.0 e385/366 10:04d7ff8c:::benchmark_data_smithi169_162694_object82509:head v 385'5237, mlcod=385'5235)", 2024-03-12T03:49:40.323 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:00.587558+0000", 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 39.676839792999999, 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.023921832000000001, 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:00.587558+0000", 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:00.587558+0000", 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:00.587559+0000", 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1800000000000005e-07 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:00.587589+0000", 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0395e-05 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:00.587597+0000", 2024-03-12T03:49:40.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7849999999999992e-06 2024-03-12T03:49:40.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:00.587602+0000", 2024-03-12T03:49:40.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9060000000000001e-06 2024-03-12T03:49:40.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:00.587652+0000", 2024-03-12T03:49:40.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0689000000000002e-05 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:00.587678+0000", 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5304e-05 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:00.611300+0000", 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.023621855000000001 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:00.611480+0000", 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018008 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:83773 10.b e385/384 10:d2f1df5b:::benchmark_data_smithi169_162694_object83772:head v 385'5166, mlcod=385'5163)", 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:01.422884+0000", 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.841513456999998, 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025626627999999999, 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.328 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:01.422884+0000", 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:01.422884+0000", 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:01.422885+0000", 2024-03-12T03:49:40.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3e-07 2024-03-12T03:49:40.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:01.422926+0000", 2024-03-12T03:49:40.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0846999999999997e-05 2024-03-12T03:49:40.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:01.424077+0000", 2024-03-12T03:49:40.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011511970000000001 2024-03-12T03:49:40.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:01.424087+0000", 2024-03-12T03:49:40.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0044000000000001e-05 2024-03-12T03:49:40.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:01.427232+0000", 2024-03-12T03:49:40.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0031453269999999998 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:01.429581+0000", 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002348396 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:01.448444+0000", 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.018863464 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:01.448511+0000", 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6623000000000002e-05 2024-03-12T03:49:40.332 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:88933 10.b e385/384 10:d3c05714:::benchmark_data_smithi169_162694_object88932:head v 385'5501, mlcod=385'5499)", 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:04.839297+0000", 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.425100608999998, 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024457083000000001, 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:04.839297+0000", 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:04.839297+0000", 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:04.839298+0000", 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2300000000000001e-07 2024-03-12T03:49:40.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:04.857655+0000", 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.018357040000000002 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:04.857669+0000", 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4049999999999999e-05 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:04.857678+0000", 2024-03-12T03:49:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1640000000000007e-06 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:04.857752+0000", 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4406000000000006e-05 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:04.857775+0000", 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.315e-05 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:04.863672+0000", 2024-03-12T03:49:40.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058969519999999996 2024-03-12T03:49:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:04.863754+0000", 2024-03-12T03:49:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1798000000000005e-05 2024-03-12T03:49:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:110617 10.a e389/384 10:5aa4436c:::benchmark_data_smithi169_162694_object110616:head v 389'6831, mlcod=389'6828)", 2024-03-12T03:49:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:20.439153+0000", 2024-03-12T03:49:40.338 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 19.825244886, 2024-03-12T03:49:40.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027448368000000001, 2024-03-12T03:49:40.338 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.338 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.338 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439153+0000", 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439153+0000", 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439154+0000", 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1799999999999999e-06 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439180+0000", 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6109999999999999e-05 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439191+0000", 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1099e-05 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439203+0000", 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2242000000000001e-05 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.440972+0000", 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001768146 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.440997+0000", 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4956999999999998e-05 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.466546+0000", 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025549471000000001 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.466601+0000", 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5163e-05 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:110621 10.0 e389/366 10:06bc38d8:::benchmark_data_smithi169_162694_object110620:head v 389'6993, mlcod=389'6991)", 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:20.441460+0000", 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 19.822938098000002, 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026238656999999999, 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.441460+0000", 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.441460+0000", 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.441460+0000", 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8500000000000002e-07 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.441487+0000", 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6925000000000001e-05 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.441494+0000", 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4669999999999998e-06 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.441500+0000", 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2220000000000001e-06 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.447497+0000", 2024-03-12T03:49:40.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0059972589999999996 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.447517+0000", 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9636e-05 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.467654+0000", 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020137502000000002 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.467698+0000", 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4161e-05 2024-03-12T03:49:40.345 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:110618 10.a e389/384 10:5986ca61:::benchmark_data_smithi169_162694_object110617:head v 389'6832, mlcod=389'6828)", 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:20.443978+0000", 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 19.820419422000001, 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025328272999999998, 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.443978+0000", 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.443978+0000", 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.443979+0000", 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.85e-07 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.444006+0000", 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7274e-05 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.444017+0000", 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.077e-05 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.444025+0000", 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4680000000000003e-06 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.452329+0000", 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0083045510000000003 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.452391+0000", 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1865999999999993e-05 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.469257+0000", 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.016865897000000001 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.469307+0000", 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9561999999999997e-05 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:110619 10.a e389/384 10:5c645395:::benchmark_data_smithi169_162694_object110618:head v 389'6833, mlcod=389'6828)", 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:20.446155+0000", 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 19.818242936000001, 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.023180471000000001, 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.350 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.446155+0000", 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.446155+0000", 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.446156+0000", 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.252e-06 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.446181+0000", 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.495e-05 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.446196+0000", 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5113000000000001e-05 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.446204+0000", 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9170000000000006e-06 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.457026+0000", 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010821538 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.457048+0000", 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2436999999999999e-05 2024-03-12T03:49:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.469310+0000", 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012262096 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.469335+0000", 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5168e-05 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:114672 10.f e391/366 10:fe0e19ae:::benchmark_data_smithi169_162694_object114671:head v 391'7222, mlcod=391'7219)", 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:23.949154+0000", 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 16.315243892000002, 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026819050000000001, 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949154+0000", 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949154+0000", 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949155+0000", 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.540000000000001e-07 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949190+0000", 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5561000000000001e-05 2024-03-12T03:49:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949201+0000", 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0206e-05 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949208+0000", 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8769999999999993e-06 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.950125+0000", 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091698799999999998 2024-03-12T03:49:40.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.950147+0000", 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1537000000000001e-05 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.975879+0000", 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025731878 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.975973+0000", 2024-03-12T03:49:40.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4048999999999993e-05 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:133700 10.b e396/384 10:dd1e0ee5:::benchmark_data_smithi169_162694_object133699:head v 396'8221, mlcod=396'8218)", 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:39.213740+0000", 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.050657486, 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024144766000000002, 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:40.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:39.213740+0000", 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:39.213740+0000", 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:39.213741+0000", 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3399999999999998e-07 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:39.213800+0000", 2024-03-12T03:49:40.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9231999999999998e-05 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:39.213811+0000", 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0321e-05 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:39.213817+0000", 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8909999999999998e-06 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:39.218391+0000", 2024-03-12T03:49:40.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045736479999999996 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:39.218412+0000", 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1160000000000001e-05 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:39.237663+0000", 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.019250469999999999 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:40.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:49:40.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:39.237885+0000", 2024-03-12T03:49:40.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022221 2024-03-12T03:49:40.363 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.363 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.363 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.363 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:40.363 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:40.363 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:49:40.363 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:40.395 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:40.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:40.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:40.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:40.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:40.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:40.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:40.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:40.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:40.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:40.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:40.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:40.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:40.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:40.958 INFO:teuthology.orchestra.run.smithi169.stderr:Exporting 5.1d info 5.1d( empty local-lis/les=70/71 n=0 ec=70/70 lis/c=70/70 les/c/f=71/71/0 sis=70) 2024-03-12T03:49:40.972 INFO:teuthology.orchestra.run.smithi169.stderr:Export successful 2024-03-12T03:49:41.043 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:41.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 88 16 136234 136218 96.725 81.8125 0.0049983 0.0103344 2024-03-12T03:49:41.082 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:41.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:41.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:41.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:41.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:41.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:41.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:41.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:41.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:41.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:41.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:41.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:41.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:41.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:41.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:41.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:49:41.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:41.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:41.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:41.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:41.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:41.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:41.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:41.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-12T03:49:41.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:41.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:42.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:42.052 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:42.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:42.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:42.066 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 89 16 137630 137614 96.6185 87.25 0.0189267 0.0103459 2024-03-12T03:49:42.124 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:41 smithi173 ceph-mon[33544]: pgmap v786: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.0 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s; 81000/392307 objects degraded (20.647%) 2024-03-12T03:49:42.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:42.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:41 smithi169 ceph-mon[29344]: pgmap v786: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.0 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s; 81000/392307 objects degraded (20.647%) 2024-03-12T03:49:42.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:49:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:49:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:49:42.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:41 smithi169 ceph-mon[39949]: pgmap v786: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.0 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s; 81000/392307 objects degraded (20.647%) 2024-03-12T03:49:42.200 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:42.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:42.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:42.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:42.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:42.362 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:49:42.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:42.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:42.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:42.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:42.600 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:42.634 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:42.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:42.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:42.771 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:42.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:42.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:42.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:42.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:43.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:43.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:43.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Total time run: 90.0053 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Total writes made: 138992 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Write size: 65536 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Object size: 65536 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Bandwidth (MB/sec): 96.5166 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev Bandwidth: 29.7474 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max bandwidth (MB/sec): 206.875 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min bandwidth (MB/sec): 40.125 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Average IOPS: 1544 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev IOPS: 475.958 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max IOPS: 3310 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min IOPS: 642 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Average Latency(s): 0.0103573 2024-03-12T03:49:43.067 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev Latency(s): 0.0124691 2024-03-12T03:49:43.068 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max latency(s): 0.937563 2024-03-12T03:49:43.068 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min latency(s): 0.00147924 2024-03-12T03:49:43.072 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-03-12T03:49:43.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:43.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.0 --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 5.1d 2024-03-12T03:49:43.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:43.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:43.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:43.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:43.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:43.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:43.455 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/osd.0/config 2024-03-12T03:49:43.476 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:43.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:43.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:43.479 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:49:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:49:43.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:43.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:43.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:43.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:43.747 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:43.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:43.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:43.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:43.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:43.904 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:43.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:44.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:44.011 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:43 smithi169 ceph-mon[29344]: pgmap v787: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s; 84505/406740 objects degraded (20.776%) 2024-03-12T03:49:44.011 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:43 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:49:44.011 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:43 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:49:44.011 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:43 smithi169 ceph-mon[29344]: osdmap e397: 8 total, 7 up, 8 in 2024-03-12T03:49:44.011 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:43 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 84505/406740 objects degraded (20.776%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:44.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:43 smithi169 ceph-mon[39949]: pgmap v787: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s; 84505/406740 objects degraded (20.776%) 2024-03-12T03:49:44.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:43 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:49:44.012 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:43 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:49:44.012 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:43 smithi169 ceph-mon[39949]: osdmap e397: 8 total, 7 up, 8 in 2024-03-12T03:49:44.012 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:43 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 84505/406740 objects degraded (20.776%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:44.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:44.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:44.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:44.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:44.194 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:44.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:44.212 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:43 smithi173 ceph-mon[33544]: pgmap v787: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s; 84505/406740 objects degraded (20.776%) 2024-03-12T03:49:44.212 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:43 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:49:44.212 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:43 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:49:44.212 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:43 smithi173 ceph-mon[33544]: osdmap e397: 8 total, 7 up, 8 in 2024-03-12T03:49:44.212 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:43 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 84505/406740 objects degraded (20.776%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:44.212 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:140660 10.c 10:371a3b28:::benchmark_data_smithi169_162694_object1665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:44.200664+0000", 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0089831040000000004, 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0090181610000000002, 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 140660 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.213 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.200664+0000", 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.200664+0000", 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:44.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.200665+0000", 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4869999999999999e-06 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.200672+0000", 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2600000000000002e-06 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.200683+0000", 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1642e-05 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.200691+0000", 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9640000000000003e-06 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.200749+0000", 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8078999999999997e-05 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.200941+0000", 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019179000000000001 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4", 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.201364+0000", 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00042297500000000002 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.208088+0000", 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0067244200000000001 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:140661 10.c 10:36d02171:::benchmark_data_smithi169_162694_object1666:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-03-12T03:49:44.217 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:44.202567+0000", 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0070802829999999997, 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0072029099999999999, 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 140661 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:44.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.202567+0000", 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.202567+0000", 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.202568+0000", 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.353e-06 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.202573+0000", 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2379999999999997e-06 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.202589+0000", 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5670000000000001e-05 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.202597+0000", 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4719999999999995e-06 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.204356+0000", 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001758744 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.204643+0000", 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028666100000000001 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4", 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.205048+0000", 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040533600000000001 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:49:44.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.208310+0000", 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032623610000000001 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:140664 10.9 e397/366 10:9aa16427:::benchmark_data_smithi169_162694_object1669:head v 397'8794, mlcod=397'8791)", 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:44.208160+0000", 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.001486802, 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016884179999999999, 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:49:44.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.208160+0000", 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.208160+0000", 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.208161+0000", 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8999999999999995e-07 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.208168+0000", 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.579e-06 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.208179+0000", 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1083999999999999e-05 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.208187+0000", 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0739999999999993e-06 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.208499+0000", 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031231299999999999 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.208554+0000", 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5435e-05 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 3 2024-03-12T03:49:44.225 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:49:44.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:44.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:44.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:44.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:44.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:44.472 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:44.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:44.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:44.617 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:44.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:44.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:44.755 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:44.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:44.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:44.881 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:49:44.891 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:44.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:44.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-12T03:49:44.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:45.027 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:45.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:45.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:45.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:45.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:45.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:45.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:45.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:44 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3344788161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:49:45.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:44 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:49:45.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:45.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:44 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3344788161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:49:45.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:44 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:49:45.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:45.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:44 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3344788161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:49:45.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:44 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:49:45.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:45.275 INFO:teuthology.orchestra.run.smithi169.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-12T03:49:45.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:45.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:45.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:45.393 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:49:45.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:45.468 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:45.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:45.569 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:45.600 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:45.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:45.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:45.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:45.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:45.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:45.882 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:45.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:45.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:45.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:46.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:46.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:46.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:46.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:46.161 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:46.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:46.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:46.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:45 smithi169 ceph-mon[29344]: pgmap v789: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.4 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 85338/412620 objects degraded (20.682%) 2024-03-12T03:49:46.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:45 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:49:46.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:45 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:49:46.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:45 smithi169 ceph-mon[29344]: osdmap e398: 8 total, 7 up, 8 in 2024-03-12T03:49:46.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:45 smithi169 ceph-mon[29344]: 3.c scrub starts 2024-03-12T03:49:46.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:45 smithi169 ceph-mon[29344]: 3.c scrub ok 2024-03-12T03:49:46.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:45 smithi169 ceph-mon[39949]: pgmap v789: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.4 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 85338/412620 objects degraded (20.682%) 2024-03-12T03:49:46.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:45 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:49:46.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:45 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:49:46.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:45 smithi169 ceph-mon[39949]: osdmap e398: 8 total, 7 up, 8 in 2024-03-12T03:49:46.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:45 smithi169 ceph-mon[39949]: 3.c scrub starts 2024-03-12T03:49:46.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:45 smithi169 ceph-mon[39949]: 3.c scrub ok 2024-03-12T03:49:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:45 smithi173 ceph-mon[33544]: pgmap v789: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.4 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 85338/412620 objects degraded (20.682%) 2024-03-12T03:49:46.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:45 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:49:46.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:45 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:49:46.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:45 smithi173 ceph-mon[33544]: osdmap e398: 8 total, 7 up, 8 in 2024-03-12T03:49:46.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:45 smithi173 ceph-mon[33544]: 3.c scrub starts 2024-03-12T03:49:46.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:45 smithi173 ceph-mon[33544]: 3.c scrub ok 2024-03-12T03:49:46.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:46.293 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:46.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:46.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:46.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:46.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:46.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:46.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:46.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:46.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:46.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:46.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:46.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:46.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:46.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:46.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:46.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:46.988 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:46.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:46.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:47.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:47.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:47.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:47.140 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:46 smithi173 ceph-mon[33544]: 6.19 scrub starts 2024-03-12T03:49:47.140 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:46 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:47.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:46 smithi169 ceph-mon[29344]: 6.19 scrub starts 2024-03-12T03:49:47.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:46 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:47.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:46 smithi169 ceph-mon[39949]: 6.19 scrub starts 2024-03-12T03:49:47.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:46 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:47.230 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:47.261 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:47.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:47.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:47.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:47.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:47.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:47.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:47.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:47.614 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:49:47.614 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:49:47.614 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:49:47.614 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:49:47.614 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:49:47.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:47.676 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:47.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:47.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:47.805 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:47.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:47.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:47.908 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:47.937 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:47.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:47.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:49:48.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:48.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:48.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:47 smithi169 ceph-mon[29344]: pgmap v791: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.4 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 699 B/s rd, 59 MiB/s wr, 948 op/s; 85338/412620 objects degraded (20.682%) 2024-03-12T03:49:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:47 smithi169 ceph-mon[29344]: 6.19 scrub ok 2024-03-12T03:49:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:47 smithi169 ceph-mon[29344]: 4.11 scrub starts 2024-03-12T03:49:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:47 smithi169 ceph-mon[29344]: 4.11 scrub ok 2024-03-12T03:49:48.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:47 smithi169 ceph-mon[39949]: pgmap v791: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.4 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 699 B/s rd, 59 MiB/s wr, 948 op/s; 85338/412620 objects degraded (20.682%) 2024-03-12T03:49:48.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:47 smithi169 ceph-mon[39949]: 6.19 scrub ok 2024-03-12T03:49:48.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:47 smithi169 ceph-mon[39949]: 4.11 scrub starts 2024-03-12T03:49:48.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:47 smithi169 ceph-mon[39949]: 4.11 scrub ok 2024-03-12T03:49:48.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:48.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:48.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:47 smithi173 ceph-mon[33544]: pgmap v791: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.4 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 699 B/s rd, 59 MiB/s wr, 948 op/s; 85338/412620 objects degraded (20.682%) 2024-03-12T03:49:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:47 smithi173 ceph-mon[33544]: 6.19 scrub ok 2024-03-12T03:49:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:47 smithi173 ceph-mon[33544]: 4.11 scrub starts 2024-03-12T03:49:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:47 smithi173 ceph-mon[33544]: 4.11 scrub ok 2024-03-12T03:49:48.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:48.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:48.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:48.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:48.440 INFO:teuthology.orchestra.run.smithi169.stdout: marking collection for removal 2024-03-12T03:49:48.441 INFO:teuthology.orchestra.run.smithi169.stdout:setting '_remove' omap key 2024-03-12T03:49:48.441 INFO:teuthology.orchestra.run.smithi169.stdout:finish_remove_pgs 5.1d_head removing 5.1d 2024-03-12T03:49:48.450 INFO:teuthology.orchestra.run.smithi169.stdout:Remove successful 2024-03-12T03:49:48.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:48.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-12T03:49:48.511 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:48.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:48.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:48.644 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:48.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:48.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:48.786 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:48.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:48.799 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:48.810 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:49:48.841 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:49:48.872 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:49:48.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:48.907 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:49:48.916 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:48.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:48.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:48.925 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:49:48.957 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:49:48.996 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:48 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 85338/412620 objects degraded (20.682%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:48.997 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:48 smithi169 ceph-mon[29344]: osdmap e399: 8 total, 7 up, 8 in 2024-03-12T03:49:48.997 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:48 smithi169 ceph-mon[29344]: 6.c scrub starts 2024-03-12T03:49:48.997 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:48 smithi169 ceph-mon[29344]: 6.c scrub ok 2024-03-12T03:49:48.997 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:48 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4127508536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:48.997 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:48 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 85338/412620 objects degraded (20.682%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:48.997 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:48 smithi169 ceph-mon[39949]: osdmap e399: 8 total, 7 up, 8 in 2024-03-12T03:49:48.997 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:48 smithi169 ceph-mon[39949]: 6.c scrub starts 2024-03-12T03:49:48.997 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:48 smithi169 ceph-mon[39949]: 6.c scrub ok 2024-03-12T03:49:48.997 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:48 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4127508536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:48.998 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:49:49.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:49.028 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:49:49.028 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:49:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:49:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:49:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:49:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:49:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:49:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:49:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:49:49.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:49.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:49.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:49.164 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:49.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:49.196 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:49.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:49.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:48 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 85338/412620 objects degraded (20.682%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:49.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:48 smithi173 ceph-mon[33544]: osdmap e399: 8 total, 7 up, 8 in 2024-03-12T03:49:49.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:48 smithi173 ceph-mon[33544]: 6.c scrub starts 2024-03-12T03:49:49.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:48 smithi173 ceph-mon[33544]: 6.c scrub ok 2024-03-12T03:49:49.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:48 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4127508536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:49:49.219 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:49:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=sqlstore.transactions t=2024-03-12T03:49:49.065993796Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-12T03:49:49.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:49.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:49.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:49.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:49.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:49.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:49.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:49.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:49.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:49.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:49.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:49.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:49.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:49.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:49.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:49.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:49.881 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:49.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:49.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:50.021 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:50.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:50.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:50.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:50.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:50.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:50.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:50.172 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:49 smithi173 ceph-mon[33544]: pgmap v792: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 66 MiB/s wr, 1.67k op/s; 83104/402927 objects degraded (20.625%) 2024-03-12T03:49:50.172 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:49 smithi173 ceph-mon[33544]: 6.1c scrub starts 2024-03-12T03:49:50.172 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:49 smithi173 ceph-mon[33544]: 6.1c scrub ok 2024-03-12T03:49:50.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:49 smithi173 ceph-mon[33544]: 5.d scrub starts 2024-03-12T03:49:50.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:49 smithi173 ceph-mon[33544]: 5.d scrub ok 2024-03-12T03:49:50.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:49 smithi169 ceph-mon[29344]: pgmap v792: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 66 MiB/s wr, 1.67k op/s; 83104/402927 objects degraded (20.625%) 2024-03-12T03:49:50.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:49 smithi169 ceph-mon[29344]: 6.1c scrub starts 2024-03-12T03:49:50.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:49 smithi169 ceph-mon[29344]: 6.1c scrub ok 2024-03-12T03:49:50.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:49 smithi169 ceph-mon[29344]: 5.d scrub starts 2024-03-12T03:49:50.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:49 smithi169 ceph-mon[29344]: 5.d scrub ok 2024-03-12T03:49:50.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:49 smithi169 ceph-mon[39949]: pgmap v792: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 66 MiB/s wr, 1.67k op/s; 83104/402927 objects degraded (20.625%) 2024-03-12T03:49:50.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:49 smithi169 ceph-mon[39949]: 6.1c scrub starts 2024-03-12T03:49:50.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:49 smithi169 ceph-mon[39949]: 6.1c scrub ok 2024-03-12T03:49:50.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:49 smithi169 ceph-mon[39949]: 5.d scrub starts 2024-03-12T03:49:50.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:49 smithi169 ceph-mon[39949]: 5.d scrub ok 2024-03-12T03:49:50.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:50.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:50.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:50.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:50.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:50.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:50.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:50.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:50.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:50.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:50.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:50.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:50.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:50.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:50.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:50.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:50.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:50.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:50.957 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:50.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.0 --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.1d.0 2024-03-12T03:49:51.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:51.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:51.078 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:50 smithi173 ceph-mon[33544]: 6.7 scrub starts 2024-03-12T03:49:51.078 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:50 smithi173 ceph-mon[33544]: 1.7 scrub starts 2024-03-12T03:49:51.078 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:50 smithi173 ceph-mon[33544]: 1.7 scrub ok 2024-03-12T03:49:51.078 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:50 smithi173 ceph-mon[33544]: 1.3 scrub starts 2024-03-12T03:49:51.078 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:50 smithi173 ceph-mon[33544]: 1.3 scrub ok 2024-03-12T03:49:51.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:51.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:51.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:51.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:50 smithi169 ceph-mon[29344]: 6.7 scrub starts 2024-03-12T03:49:51.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:50 smithi169 ceph-mon[29344]: 1.7 scrub starts 2024-03-12T03:49:51.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:50 smithi169 ceph-mon[29344]: 1.7 scrub ok 2024-03-12T03:49:51.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:50 smithi169 ceph-mon[29344]: 1.3 scrub starts 2024-03-12T03:49:51.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:50 smithi169 ceph-mon[29344]: 1.3 scrub ok 2024-03-12T03:49:51.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:50 smithi169 ceph-mon[39949]: 6.7 scrub starts 2024-03-12T03:49:51.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:50 smithi169 ceph-mon[39949]: 1.7 scrub starts 2024-03-12T03:49:51.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:50 smithi169 ceph-mon[39949]: 1.7 scrub ok 2024-03-12T03:49:51.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:50 smithi169 ceph-mon[39949]: 1.3 scrub starts 2024-03-12T03:49:51.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:50 smithi169 ceph-mon[39949]: 1.3 scrub ok 2024-03-12T03:49:51.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:51.284 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:51.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:51.386 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:51.418 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:51.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:51.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:51.520 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:51.554 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:51.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:51.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:51.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:51.690 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:51.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:51.722 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:49:51.723 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:49:51.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:51.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:51.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:51.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:51.924 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:49:51.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:51.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:52.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:52.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:52.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:52.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:52.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:49:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:49:52.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[29344]: pgmap v794: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 842 B/s rd, 20 MiB/s wr, 1.12k op/s; 83104/400257 objects degraded (20.763%) 2024-03-12T03:49:52.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[29344]: 6.7 scrub ok 2024-03-12T03:49:52.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[29344]: 3.14 scrub starts 2024-03-12T03:49:52.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[29344]: 3.14 scrub ok 2024-03-12T03:49:52.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[29344]: 3.4 scrub starts 2024-03-12T03:49:52.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[29344]: 3.4 scrub ok 2024-03-12T03:49:52.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3288223005' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:49:52.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[39949]: pgmap v794: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 842 B/s rd, 20 MiB/s wr, 1.12k op/s; 83104/400257 objects degraded (20.763%) 2024-03-12T03:49:52.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[39949]: 6.7 scrub ok 2024-03-12T03:49:52.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[39949]: 3.14 scrub starts 2024-03-12T03:49:52.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[39949]: 3.14 scrub ok 2024-03-12T03:49:52.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[39949]: 3.4 scrub starts 2024-03-12T03:49:52.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[39949]: 3.4 scrub ok 2024-03-12T03:49:52.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:51 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3288223005' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:49:52.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:52.234 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:52.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:51 smithi173 ceph-mon[33544]: pgmap v794: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 842 B/s rd, 20 MiB/s wr, 1.12k op/s; 83104/400257 objects degraded (20.763%) 2024-03-12T03:49:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:51 smithi173 ceph-mon[33544]: 6.7 scrub ok 2024-03-12T03:49:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:51 smithi173 ceph-mon[33544]: 3.14 scrub starts 2024-03-12T03:49:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:51 smithi173 ceph-mon[33544]: 3.14 scrub ok 2024-03-12T03:49:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:51 smithi173 ceph-mon[33544]: 3.4 scrub starts 2024-03-12T03:49:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:51 smithi173 ceph-mon[33544]: 3.4 scrub ok 2024-03-12T03:49:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:51 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3288223005' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:49:52.264 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:49:52.264 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:49:52.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:52.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:52.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:52.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:52.487 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/osd.0/config 2024-03-12T03:49:52.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:52.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:52.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:52.632 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-12T03:49:52.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:52.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:52.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:52.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:52.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:52.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:52.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:52.924 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:52.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:52.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:53.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:53.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:53.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:53.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:53.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:53.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:52 smithi169 ceph-mon[29344]: 10.3 scrub starts 2024-03-12T03:49:53.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:52 smithi169 ceph-mon[29344]: 4.1b scrub starts 2024-03-12T03:49:53.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:52 smithi169 ceph-mon[29344]: 4.1b scrub ok 2024-03-12T03:49:53.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:52 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:49:53.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:52 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3288223005' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:49:53.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:52 smithi169 ceph-mon[29344]: osdmap e400: 8 total, 7 up, 8 in 2024-03-12T03:49:53.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:52 smithi169 ceph-mon[39949]: 10.3 scrub starts 2024-03-12T03:49:53.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:52 smithi169 ceph-mon[39949]: 4.1b scrub starts 2024-03-12T03:49:53.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:52 smithi169 ceph-mon[39949]: 4.1b scrub ok 2024-03-12T03:49:53.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:52 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:49:53.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:52 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3288223005' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:49:53.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:52 smithi169 ceph-mon[39949]: osdmap e400: 8 total, 7 up, 8 in 2024-03-12T03:49:53.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:53.194 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:53.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:53.209 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:52 smithi173 ceph-mon[33544]: 10.3 scrub starts 2024-03-12T03:49:53.209 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:52 smithi173 ceph-mon[33544]: 4.1b scrub starts 2024-03-12T03:49:53.210 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:52 smithi173 ceph-mon[33544]: 4.1b scrub ok 2024-03-12T03:49:53.210 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:52 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:49:53.210 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:52 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3288223005' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:49:53.210 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:52 smithi173 ceph-mon[33544]: osdmap e400: 8 total, 7 up, 8 in 2024-03-12T03:49:53.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:53.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:53.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:53.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:53.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:53.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:53.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:53.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:49:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:49:53.569 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:53.602 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:53.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:53.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:53.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:53.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:53.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:53.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:53.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:53.880 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:53.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:53.972 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[29344]: pgmap v796: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.5 KiB/s rd, 14 MiB/s wr, 1.67k op/s; 79599/385746 objects degraded (20.635%) 2024-03-12T03:49:53.972 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:53.972 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[29344]: 4.1f scrub starts 2024-03-12T03:49:53.972 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[29344]: 4.1f scrub ok 2024-03-12T03:49:53.972 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[29344]: 1.4 scrub starts 2024-03-12T03:49:53.972 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[29344]: 1.4 scrub ok 2024-03-12T03:49:53.972 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 79599/385746 objects degraded (20.635%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:53.972 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[29344]: osdmap e401: 8 total, 7 up, 8 in 2024-03-12T03:49:53.972 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[29344]: 4.a scrub starts 2024-03-12T03:49:53.973 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[29344]: 4.a scrub ok 2024-03-12T03:49:53.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[39949]: pgmap v796: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.5 KiB/s rd, 14 MiB/s wr, 1.67k op/s; 79599/385746 objects degraded (20.635%) 2024-03-12T03:49:53.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:53.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[39949]: 4.1f scrub starts 2024-03-12T03:49:53.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[39949]: 4.1f scrub ok 2024-03-12T03:49:53.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[39949]: 1.4 scrub starts 2024-03-12T03:49:53.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[39949]: 1.4 scrub ok 2024-03-12T03:49:53.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 79599/385746 objects degraded (20.635%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:53.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[39949]: osdmap e401: 8 total, 7 up, 8 in 2024-03-12T03:49:53.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[39949]: 4.a scrub starts 2024-03-12T03:49:53.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:53 smithi169 ceph-mon[39949]: 4.a scrub ok 2024-03-12T03:49:53.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:54.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:54.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:54.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:54.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:54.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:54.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:54.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:53 smithi173 ceph-mon[33544]: pgmap v796: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.5 KiB/s rd, 14 MiB/s wr, 1.67k op/s; 79599/385746 objects degraded (20.635%) 2024-03-12T03:49:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:53 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:53 smithi173 ceph-mon[33544]: 4.1f scrub starts 2024-03-12T03:49:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:53 smithi173 ceph-mon[33544]: 4.1f scrub ok 2024-03-12T03:49:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:53 smithi173 ceph-mon[33544]: 1.4 scrub starts 2024-03-12T03:49:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:53 smithi173 ceph-mon[33544]: 1.4 scrub ok 2024-03-12T03:49:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:53 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 79599/385746 objects degraded (20.635%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:53 smithi173 ceph-mon[33544]: osdmap e401: 8 total, 7 up, 8 in 2024-03-12T03:49:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:53 smithi173 ceph-mon[33544]: 4.a scrub starts 2024-03-12T03:49:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:53 smithi173 ceph-mon[33544]: 4.a scrub ok 2024-03-12T03:49:54.260 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:54.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:49:54.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:54.292 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:54.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:54.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:54.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:54.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:54.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:54.560 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:54.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:54.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:54.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:54.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:54.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:54.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:54.712 INFO:teuthology.orchestra.run.smithi169.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-12T03:49:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:54.787 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:54.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:54.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:54.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:54.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:54.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:54.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:54.979 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:54.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:55.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:55.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:55.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:55.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[29344]: 10.3 scrub ok 2024-03-12T03:49:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[29344]: pgmap v798: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s; 78708/379029 objects degraded (20.766%) 2024-03-12T03:49:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[29344]: 10.9 scrub starts 2024-03-12T03:49:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[29344]: 6.1f scrub starts 2024-03-12T03:49:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[29344]: 6.1f scrub ok 2024-03-12T03:49:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[29344]: 6.e scrub starts 2024-03-12T03:49:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[29344]: 6.e scrub ok 2024-03-12T03:49:55.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[39949]: 10.3 scrub ok 2024-03-12T03:49:55.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[39949]: pgmap v798: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s; 78708/379029 objects degraded (20.766%) 2024-03-12T03:49:55.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[39949]: 10.9 scrub starts 2024-03-12T03:49:55.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[39949]: 6.1f scrub starts 2024-03-12T03:49:55.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[39949]: 6.1f scrub ok 2024-03-12T03:49:55.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[39949]: 6.e scrub starts 2024-03-12T03:49:55.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:54 smithi169 ceph-mon[39949]: 6.e scrub ok 2024-03-12T03:49:55.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:54 smithi173 ceph-mon[33544]: 10.3 scrub ok 2024-03-12T03:49:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:54 smithi173 ceph-mon[33544]: pgmap v798: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s; 78708/379029 objects degraded (20.766%) 2024-03-12T03:49:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:54 smithi173 ceph-mon[33544]: 10.9 scrub starts 2024-03-12T03:49:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:54 smithi173 ceph-mon[33544]: 6.1f scrub starts 2024-03-12T03:49:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:54 smithi173 ceph-mon[33544]: 6.1f scrub ok 2024-03-12T03:49:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:54 smithi173 ceph-mon[33544]: 6.e scrub starts 2024-03-12T03:49:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:54 smithi173 ceph-mon[33544]: 6.e scrub ok 2024-03-12T03:49:55.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:55.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:55.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:55.384 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:55.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:55.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:55.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:49:55.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:55.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:55.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:55.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:55.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:55.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:55.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:55.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:55.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:55.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:55.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:55.923 INFO:teuthology.orchestra.run.smithi169.stdout:get_pg_num_history pg_num_history pg_num_history(e383 pg_nums {1={149=11,291=21},2={65=3},3={66=32},4={68=32},5={70=32},6={72=32},8={198=16},9={278=16},10={366=16}} deleted_pools 272,8,362,9) 2024-03-12T03:49:55.923 INFO:teuthology.orchestra.run.smithi169.stdout:Importing pgid 5.1d 2024-03-12T03:49:55.923 INFO:teuthology.orchestra.run.smithi169.stdout:write_pg epoch 360 info 5.1d( empty local-lis/les=70/71 n=0 ec=70/70 lis/c=70/70 les/c/f=71/71/0 sis=70) 2024-03-12T03:49:55.924 INFO:teuthology.orchestra.run.smithi169.stdout:Import successful 2024-03-12T03:49:55.949 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:55.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:56.051 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:56.088 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:56.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:56.189 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:55 smithi169 ceph-mon[29344]: 6.3 deep-scrub starts 2024-03-12T03:49:56.189 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:55 smithi169 ceph-mon[29344]: 6.3 deep-scrub ok 2024-03-12T03:49:56.189 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:55 smithi169 ceph-mon[29344]: 6.1b scrub starts 2024-03-12T03:49:56.189 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:55 smithi169 ceph-mon[29344]: 6.1b scrub ok 2024-03-12T03:49:56.189 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:55 smithi169 ceph-mon[39949]: 6.3 deep-scrub starts 2024-03-12T03:49:56.189 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:55 smithi169 ceph-mon[39949]: 6.3 deep-scrub ok 2024-03-12T03:49:56.189 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:55 smithi169 ceph-mon[39949]: 6.1b scrub starts 2024-03-12T03:49:56.189 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:55 smithi169 ceph-mon[39949]: 6.1b scrub ok 2024-03-12T03:49:56.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:56.229 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:56.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:55 smithi173 ceph-mon[33544]: 6.3 deep-scrub starts 2024-03-12T03:49:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:55 smithi173 ceph-mon[33544]: 6.3 deep-scrub ok 2024-03-12T03:49:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:55 smithi173 ceph-mon[33544]: 6.1b scrub starts 2024-03-12T03:49:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:55 smithi173 ceph-mon[33544]: 6.1b scrub ok 2024-03-12T03:49:56.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:56.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:56.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:56.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:56.472 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:56.512 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:56.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:56.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:56.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:56.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:56.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:56.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:56.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:56.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:56.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:57.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:57.082 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:57.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:57.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:57.144 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:56 smithi169 ceph-mon[29344]: pgmap v799: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 665 B/s rd, 0 B/s wr, 1.03k op/s; 78708/379029 objects degraded (20.766%) 2024-03-12T03:49:57.144 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:56 smithi169 ceph-mon[29344]: 10.f scrub starts 2024-03-12T03:49:57.144 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:56 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:49:57.144 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:56 smithi169 ceph-mon[39949]: pgmap v799: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 665 B/s rd, 0 B/s wr, 1.03k op/s; 78708/379029 objects degraded (20.766%) 2024-03-12T03:49:57.144 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:56 smithi169 ceph-mon[39949]: 10.f scrub starts 2024-03-12T03:49:57.144 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:56 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:49:57.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:57.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:56 smithi173 ceph-mon[33544]: pgmap v799: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 665 B/s rd, 0 B/s wr, 1.03k op/s; 78708/379029 objects degraded (20.766%) 2024-03-12T03:49:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:56 smithi173 ceph-mon[33544]: 10.f scrub starts 2024-03-12T03:49:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:56 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:49:57.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:57.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:57.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:57.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:57.517 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:57.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:57.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:57.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:57.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:57.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:57.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:57.791 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:57.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:57.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:57.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:57.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:58.028 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:49:58.062 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:58.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:49:58.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:58.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:58.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:58.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:57 smithi173 ceph-mon[33544]: 4.12 scrub starts 2024-03-12T03:49:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:57 smithi173 ceph-mon[33544]: 4.12 scrub ok 2024-03-12T03:49:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:57 smithi173 ceph-mon[33544]: 10.9 scrub ok 2024-03-12T03:49:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:57 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:49:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:57 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:49:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:57 smithi173 ceph-mon[33544]: osdmap e402: 8 total, 7 up, 8 in 2024-03-12T03:49:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:57 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:57 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:58.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:58.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:58.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:58.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:58.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[29344]: 4.12 scrub starts 2024-03-12T03:49:58.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[29344]: 4.12 scrub ok 2024-03-12T03:49:58.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[29344]: 10.9 scrub ok 2024-03-12T03:49:58.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:49:58.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:49:58.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[29344]: osdmap e402: 8 total, 7 up, 8 in 2024-03-12T03:49:58.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:58.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:58.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[39949]: 4.12 scrub starts 2024-03-12T03:49:58.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[39949]: 4.12 scrub ok 2024-03-12T03:49:58.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[39949]: 10.9 scrub ok 2024-03-12T03:49:58.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:49:58.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:49:58.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[39949]: osdmap e402: 8 total, 7 up, 8 in 2024-03-12T03:49:58.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:58.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:57 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:49:58.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:58.458 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:58.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:58.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:58.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:58.596 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:58.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:58.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:49:58.724 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:58.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:58.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:49:58.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:58.857 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:58.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:58.889 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:49:58.959 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:49:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:58.985 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:49:58.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:49:59.087 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:59.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:59.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:59.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:59.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:59.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:59.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:59.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:59.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:49:59.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:59.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:59.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:49:59.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:59.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:59 smithi173 ceph-mon[33544]: 4.1c scrub starts 2024-03-12T03:49:59.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:59 smithi173 ceph-mon[33544]: 4.1c scrub ok 2024-03-12T03:49:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:59 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:49:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:59 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:49:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:59 smithi173 ceph-mon[33544]: pgmap v801: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 822 B/s rd, 0 B/s wr, 1.36k op/s; 74263/360366 objects degraded (20.608%) 2024-03-12T03:49:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:59 smithi173 ceph-mon[33544]: osdmap e403: 8 total, 7 up, 8 in 2024-03-12T03:49:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:59 smithi173 ceph-mon[33544]: 6.15 deep-scrub starts 2024-03-12T03:49:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:59 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 78708/379029 objects degraded (20.766%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:59.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:49:59 smithi173 ceph-mon[33544]: 6.15 deep-scrub ok 2024-03-12T03:49:59.504 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:59.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:49:59.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:49:59.643 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:59.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:49:59.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[29344]: 4.1c scrub starts 2024-03-12T03:49:59.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[29344]: 4.1c scrub ok 2024-03-12T03:49:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:49:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:49:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[29344]: pgmap v801: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 822 B/s rd, 0 B/s wr, 1.36k op/s; 74263/360366 objects degraded (20.608%) 2024-03-12T03:49:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[29344]: osdmap e403: 8 total, 7 up, 8 in 2024-03-12T03:49:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[29344]: 6.15 deep-scrub starts 2024-03-12T03:49:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 78708/379029 objects degraded (20.766%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:59.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[29344]: 6.15 deep-scrub ok 2024-03-12T03:49:59.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[39949]: 4.1c scrub starts 2024-03-12T03:49:59.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[39949]: 4.1c scrub ok 2024-03-12T03:49:59.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:49:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:49:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[39949]: pgmap v801: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 822 B/s rd, 0 B/s wr, 1.36k op/s; 74263/360366 objects degraded (20.608%) 2024-03-12T03:49:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[39949]: osdmap e403: 8 total, 7 up, 8 in 2024-03-12T03:49:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[39949]: 6.15 deep-scrub starts 2024-03-12T03:49:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 78708/379029 objects degraded (20.766%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T03:49:59.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:49:59 smithi169 ceph-mon[39949]: 6.15 deep-scrub ok 2024-03-12T03:49:59.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:49:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:59.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:59.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:49:59.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:49:59.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:49:59.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:49:59.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:00.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:00.052 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:00.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:00.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:00.186 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:00.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:00.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:00.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:00.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:00.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:00.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:00.452 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:00.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:00.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:00.553 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:00.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:00.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:00.644 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: 10.f scrub ok 2024-03-12T03:50:00.644 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: 1.13 scrub starts 2024-03-12T03:50:00.644 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: 1.13 scrub ok 2024-03-12T03:50:00.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 74263/360366 objects degraded (20.608%), 27 pgs degraded 2024-03-12T03:50:00.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: [WRN] OSD_DOWN: 1 osds down 2024-03-12T03:50:00.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: osd.0 (root=default,host=smithi169) is down 2024-03-12T03:50:00.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: [WRN] PG_DEGRADED: Degraded data redundancy: 74263/360366 objects degraded (20.608%), 27 pgs degraded 2024-03-12T03:50:00.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 1.c is active+undersized+degraded, acting [3,6] 2024-03-12T03:50:00.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-03-12T03:50:00.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-03-12T03:50:00.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 3.f is active+undersized+degraded, acting [7,4] 2024-03-12T03:50:00.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-03-12T03:50:00.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-03-12T03:50:00.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-03-12T03:50:00.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 4.6 is active+undersized+degraded, acting [1,2] 2024-03-12T03:50:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 4.7 is active+undersized+degraded, acting [1,5] 2024-03-12T03:50:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 4.b is active+undersized+degraded, acting [1,4] 2024-03-12T03:50:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 4.16 is active+undersized+degraded, acting [3,7] 2024-03-12T03:50:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 4.19 is active+undersized+degraded, acting [3,2] 2024-03-12T03:50:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 4.1a is active+undersized+degraded, acting [4,3] 2024-03-12T03:50:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 4.1e is active+undersized+degraded, acting [7,3] 2024-03-12T03:50:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 5.3 is active+undersized+degraded, acting [5,6] 2024-03-12T03:50:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 5.e is active+undersized+degraded, acting [4,5] 2024-03-12T03:50:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 5.15 is active+undersized+degraded, acting [5,1] 2024-03-12T03:50:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 10.1 is active+undersized+degraded, acting [4,7] 2024-03-12T03:50:00.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 10.2 is active+undersized+degraded, acting [1,4] 2024-03-12T03:50:00.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 10.4 is active+undersized+degraded, acting [4,3] 2024-03-12T03:50:00.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 10.5 is active+undersized+degraded, acting [2,3] 2024-03-12T03:50:00.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 10.6 is active+undersized+degraded, acting [5,4] 2024-03-12T03:50:00.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 10.8 is active+undersized+degraded, acting [2,7] 2024-03-12T03:50:00.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 10.a is active+undersized+degraded, acting [5,6] 2024-03-12T03:50:00.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 10.b is active+undersized+degraded, acting [1,6] 2024-03-12T03:50:00.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 10.c is active+undersized+degraded, acting [7,4] 2024-03-12T03:50:00.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[39949]: pg 10.d is active+undersized+degraded, acting [4,6] 2024-03-12T03:50:00.647 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: 10.f scrub ok 2024-03-12T03:50:00.647 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: 1.13 scrub starts 2024-03-12T03:50:00.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: 1.13 scrub ok 2024-03-12T03:50:00.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 74263/360366 objects degraded (20.608%), 27 pgs degraded 2024-03-12T03:50:00.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: [WRN] OSD_DOWN: 1 osds down 2024-03-12T03:50:00.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: osd.0 (root=default,host=smithi169) is down 2024-03-12T03:50:00.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: [WRN] PG_DEGRADED: Degraded data redundancy: 74263/360366 objects degraded (20.608%), 27 pgs degraded 2024-03-12T03:50:00.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 1.c is active+undersized+degraded, acting [3,6] 2024-03-12T03:50:00.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-03-12T03:50:00.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-03-12T03:50:00.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 3.f is active+undersized+degraded, acting [7,4] 2024-03-12T03:50:00.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-03-12T03:50:00.648 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 4.6 is active+undersized+degraded, acting [1,2] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 4.7 is active+undersized+degraded, acting [1,5] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 4.b is active+undersized+degraded, acting [1,4] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 4.16 is active+undersized+degraded, acting [3,7] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 4.19 is active+undersized+degraded, acting [3,2] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 4.1a is active+undersized+degraded, acting [4,3] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 4.1e is active+undersized+degraded, acting [7,3] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 5.3 is active+undersized+degraded, acting [5,6] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 5.e is active+undersized+degraded, acting [4,5] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 5.15 is active+undersized+degraded, acting [5,1] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 10.1 is active+undersized+degraded, acting [4,7] 2024-03-12T03:50:00.649 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 10.2 is active+undersized+degraded, acting [1,4] 2024-03-12T03:50:00.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 10.4 is active+undersized+degraded, acting [4,3] 2024-03-12T03:50:00.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 10.5 is active+undersized+degraded, acting [2,3] 2024-03-12T03:50:00.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 10.6 is active+undersized+degraded, acting [5,4] 2024-03-12T03:50:00.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 10.8 is active+undersized+degraded, acting [2,7] 2024-03-12T03:50:00.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 10.a is active+undersized+degraded, acting [5,6] 2024-03-12T03:50:00.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 10.b is active+undersized+degraded, acting [1,6] 2024-03-12T03:50:00.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 10.c is active+undersized+degraded, acting [7,4] 2024-03-12T03:50:00.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:00 smithi169 ceph-mon[29344]: pg 10.d is active+undersized+degraded, acting [4,6] 2024-03-12T03:50:00.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:00.725 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:00.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: 10.f scrub ok 2024-03-12T03:50:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: 1.13 scrub starts 2024-03-12T03:50:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: 1.13 scrub ok 2024-03-12T03:50:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 74263/360366 objects degraded (20.608%), 27 pgs degraded 2024-03-12T03:50:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: [WRN] OSD_DOWN: 1 osds down 2024-03-12T03:50:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: osd.0 (root=default,host=smithi169) is down 2024-03-12T03:50:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: [WRN] PG_DEGRADED: Degraded data redundancy: 74263/360366 objects degraded (20.608%), 27 pgs degraded 2024-03-12T03:50:00.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 1.c is active+undersized+degraded, acting [3,6] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 3.f is active+undersized+degraded, acting [7,4] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 4.6 is active+undersized+degraded, acting [1,2] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 4.7 is active+undersized+degraded, acting [1,5] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 4.b is active+undersized+degraded, acting [1,4] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 4.16 is active+undersized+degraded, acting [3,7] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 4.19 is active+undersized+degraded, acting [3,2] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 4.1a is active+undersized+degraded, acting [4,3] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 4.1e is active+undersized+degraded, acting [7,3] 2024-03-12T03:50:00.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 5.3 is active+undersized+degraded, acting [5,6] 2024-03-12T03:50:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 5.e is active+undersized+degraded, acting [4,5] 2024-03-12T03:50:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 5.15 is active+undersized+degraded, acting [5,1] 2024-03-12T03:50:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 10.1 is active+undersized+degraded, acting [4,7] 2024-03-12T03:50:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 10.2 is active+undersized+degraded, acting [1,4] 2024-03-12T03:50:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 10.4 is active+undersized+degraded, acting [4,3] 2024-03-12T03:50:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 10.5 is active+undersized+degraded, acting [2,3] 2024-03-12T03:50:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 10.6 is active+undersized+degraded, acting [5,4] 2024-03-12T03:50:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 10.8 is active+undersized+degraded, acting [2,7] 2024-03-12T03:50:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 10.a is active+undersized+degraded, acting [5,6] 2024-03-12T03:50:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 10.b is active+undersized+degraded, acting [1,6] 2024-03-12T03:50:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 10.c is active+undersized+degraded, acting [7,4] 2024-03-12T03:50:00.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:00 smithi173 ceph-mon[33544]: pg 10.d is active+undersized+degraded, acting [4,6] 2024-03-12T03:50:00.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:00.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:00.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:00.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:00.996 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:00.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:00.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:01.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:01.120 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:01.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:01.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:01.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:01.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:01.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:01.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:01.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:01.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:01.532 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:01.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:01.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:01.658 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:01.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:01.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:01 smithi169 ceph-mon[29344]: pgmap v803: 168 pgs: 2 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 103 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 1.20k op/s; 74263/357708 objects degraded (20.761%) 2024-03-12T03:50:01.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:01 smithi169 ceph-mon[29344]: 3.16 scrub starts 2024-03-12T03:50:01.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:01 smithi169 ceph-mon[29344]: 3.16 scrub ok 2024-03-12T03:50:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:01 smithi169 ceph-mon[29344]: 3.13 scrub starts 2024-03-12T03:50:01.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:01 smithi169 ceph-mon[29344]: 3.13 scrub ok 2024-03-12T03:50:01.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:01 smithi169 ceph-mon[39949]: pgmap v803: 168 pgs: 2 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 103 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 1.20k op/s; 74263/357708 objects degraded (20.761%) 2024-03-12T03:50:01.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:01 smithi169 ceph-mon[39949]: 3.16 scrub starts 2024-03-12T03:50:01.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:01 smithi169 ceph-mon[39949]: 3.16 scrub ok 2024-03-12T03:50:01.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:01 smithi169 ceph-mon[39949]: 3.13 scrub starts 2024-03-12T03:50:01.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:01 smithi169 ceph-mon[39949]: 3.13 scrub ok 2024-03-12T03:50:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:01 smithi173 ceph-mon[33544]: pgmap v803: 168 pgs: 2 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 103 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 1.20k op/s; 74263/357708 objects degraded (20.761%) 2024-03-12T03:50:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:01 smithi173 ceph-mon[33544]: 3.16 scrub starts 2024-03-12T03:50:01.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:01 smithi173 ceph-mon[33544]: 3.16 scrub ok 2024-03-12T03:50:01.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:01 smithi173 ceph-mon[33544]: 3.13 scrub starts 2024-03-12T03:50:01.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:01 smithi173 ceph-mon[33544]: 3.13 scrub ok 2024-03-12T03:50:01.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:01.793 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:01.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:01.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:01.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:01.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:01.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:02.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:02.088 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:02.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:02.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:02.138 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:50:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:50:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:50:02.180 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/exp.5.1d.0 2024-03-12T03:50:02.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:02.223 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:02.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:02.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:02.355 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:02.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:02.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:02.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:02.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:02.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:02.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:02.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:02.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[29344]: 1.d scrub starts 2024-03-12T03:50:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[29344]: 1.d scrub ok 2024-03-12T03:50:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[29344]: 6.b scrub starts 2024-03-12T03:50:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:50:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[29344]: 6.b scrub ok 2024-03-12T03:50:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[29344]: 4.d scrub starts 2024-03-12T03:50:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[29344]: 4.d scrub ok 2024-03-12T03:50:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[39949]: 1.d scrub starts 2024-03-12T03:50:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[39949]: 1.d scrub ok 2024-03-12T03:50:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[39949]: 6.b scrub starts 2024-03-12T03:50:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:50:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[39949]: 6.b scrub ok 2024-03-12T03:50:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[39949]: 4.d scrub starts 2024-03-12T03:50:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:02 smithi169 ceph-mon[39949]: 4.d scrub ok 2024-03-12T03:50:02.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:02.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:02.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:02.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:02.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:02.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:02 smithi173 ceph-mon[33544]: 1.d scrub starts 2024-03-12T03:50:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:02 smithi173 ceph-mon[33544]: 1.d scrub ok 2024-03-12T03:50:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:02 smithi173 ceph-mon[33544]: 6.b scrub starts 2024-03-12T03:50:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:02 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:50:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:02 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:02 smithi173 ceph-mon[33544]: 6.b scrub ok 2024-03-12T03:50:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:02 smithi173 ceph-mon[33544]: 4.d scrub starts 2024-03-12T03:50:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:02 smithi173 ceph-mon[33544]: 4.d scrub ok 2024-03-12T03:50:02.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:03.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:03.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:03.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:03.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:03.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:03.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:03.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:03.307 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:03.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:03.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:03.409 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:03.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:03.443 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:03.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:03.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:50:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:50:03.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:03.576 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:03.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:03 smithi169 ceph-mon[29344]: pgmap v804: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 71216/345057 objects degraded (20.639%) 2024-03-12T03:50:03.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:03 smithi169 ceph-mon[29344]: 3.1c scrub starts 2024-03-12T03:50:03.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:03 smithi169 ceph-mon[29344]: 3.1c scrub ok 2024-03-12T03:50:03.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:03 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 71216/345057 objects degraded (20.639%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:03 smithi169 ceph-mon[39949]: pgmap v804: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 71216/345057 objects degraded (20.639%) 2024-03-12T03:50:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:03 smithi169 ceph-mon[39949]: 3.1c scrub starts 2024-03-12T03:50:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:03 smithi169 ceph-mon[39949]: 3.1c scrub ok 2024-03-12T03:50:03.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:03 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 71216/345057 objects degraded (20.639%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:03.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:03.710 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:03.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:03.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:03.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:03.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:03.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:03.986 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:03.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:03.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:03.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:03 smithi173 ceph-mon[33544]: pgmap v804: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 71216/345057 objects degraded (20.639%) 2024-03-12T03:50:03.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:03 smithi173 ceph-mon[33544]: 3.1c scrub starts 2024-03-12T03:50:03.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:03 smithi173 ceph-mon[33544]: 3.1c scrub ok 2024-03-12T03:50:03.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:03 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 71216/345057 objects degraded (20.639%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:04.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:04.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:04.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:04.232 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:04.272 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:04.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:04.340 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:50:04.340 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:50:04.374 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:04.407 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:04.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:04.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:04.509 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:04.542 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:04.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:04.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:04.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:04.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:04.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:04.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:04.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:04.813 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:04.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:04.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:04.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:04.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[29344]: 3.d scrub starts 2024-03-12T03:50:04.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[29344]: 3.d scrub ok 2024-03-12T03:50:04.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[29344]: 10.7 scrub starts 2024-03-12T03:50:04.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[29344]: 6.17 scrub starts 2024-03-12T03:50:04.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:50:04.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:50:04.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:50:04.863 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:04.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[39949]: 3.d scrub starts 2024-03-12T03:50:04.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[39949]: 3.d scrub ok 2024-03-12T03:50:04.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[39949]: 10.7 scrub starts 2024-03-12T03:50:04.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[39949]: 6.17 scrub starts 2024-03-12T03:50:04.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:50:04.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:50:04.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:04 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:50:04.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:04.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:04.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:04 smithi173 ceph-mon[33544]: 3.d scrub starts 2024-03-12T03:50:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:04 smithi173 ceph-mon[33544]: 3.d scrub ok 2024-03-12T03:50:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:04 smithi173 ceph-mon[33544]: 10.7 scrub starts 2024-03-12T03:50:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:04 smithi173 ceph-mon[33544]: 6.17 scrub starts 2024-03-12T03:50:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:04 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:50:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:04 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:50:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:04 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:50:05.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:05.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:05.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:05.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:05.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:05.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:05.225 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:05.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:05.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:05.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:05.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:05.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:05.507 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:05.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:05.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:05.617 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:05 smithi169 ceph-mon[29344]: 6.17 scrub ok 2024-03-12T03:50:05.618 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:05 smithi169 ceph-mon[39949]: 6.17 scrub ok 2024-03-12T03:50:05.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:05.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:05.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:05.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:05.766 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:05.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:05.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:05.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:05.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:05 smithi169 ceph-mon[29344]: pgmap v805: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 70413/338985 objects degraded (20.772%) 2024-03-12T03:50:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:05 smithi169 ceph-mon[29344]: 5.19 deep-scrub starts 2024-03-12T03:50:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:05 smithi169 ceph-mon[29344]: 5.19 deep-scrub ok 2024-03-12T03:50:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:05 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1996492188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:50:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:05 smithi169 ceph-mon[39949]: pgmap v805: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 70413/338985 objects degraded (20.772%) 2024-03-12T03:50:05.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:05 smithi169 ceph-mon[39949]: 5.19 deep-scrub starts 2024-03-12T03:50:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:05 smithi169 ceph-mon[39949]: 5.19 deep-scrub ok 2024-03-12T03:50:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:05 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1996492188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:50:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:05 smithi173 ceph-mon[33544]: 6.17 scrub ok 2024-03-12T03:50:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:05 smithi173 ceph-mon[33544]: pgmap v805: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 70413/338985 objects degraded (20.772%) 2024-03-12T03:50:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:05 smithi173 ceph-mon[33544]: 5.19 deep-scrub starts 2024-03-12T03:50:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:05 smithi173 ceph-mon[33544]: 5.19 deep-scrub ok 2024-03-12T03:50:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:05 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1996492188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:50:06.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:06.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:06.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:06.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:06.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:06.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:06.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:06.277 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:50:06.277 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:50:06.277 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:50:06.277 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:50:06.277 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:50:06.277 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:50:06.277 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:50:06.277 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:50:06.278 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:50:06.278 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:50:06.278 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:50:06.278 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:50:06.278 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:50:06.278 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:50:06.278 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:50:06.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:06.311 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:06.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:06.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:06.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:06.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:06.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:06.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:06.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:06.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:06.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:06.596 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:50:06.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:06.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:06.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:06.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:06.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:06.867 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:06.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:06 smithi169 ceph-mon[29344]: 4.1d scrub starts 2024-03-12T03:50:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:06 smithi169 ceph-mon[29344]: 4.1d scrub ok 2024-03-12T03:50:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:06 smithi169 ceph-mon[29344]: 10.7 scrub ok 2024-03-12T03:50:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:06 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/134132627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:50:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:06 smithi169 ceph-mon[39949]: 4.1d scrub starts 2024-03-12T03:50:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:06 smithi169 ceph-mon[39949]: 4.1d scrub ok 2024-03-12T03:50:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:06 smithi169 ceph-mon[39949]: 10.7 scrub ok 2024-03-12T03:50:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:06 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/134132627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:50:06.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:06 smithi173 ceph-mon[33544]: 4.1d scrub starts 2024-03-12T03:50:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:06 smithi173 ceph-mon[33544]: 4.1d scrub ok 2024-03-12T03:50:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:06 smithi173 ceph-mon[33544]: 10.7 scrub ok 2024-03-12T03:50:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:06 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/134132627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:50:07.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:07.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:07.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:07.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:07.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:07.225 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:50:07.225 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:50:07.226 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-12T03:50:07.226 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-12T03:50:07.226 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-12T03:50:07.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-03-12T03:50:07.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:07.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:07.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:07.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:07.444 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:07.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:07.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:07.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:07.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:07.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:07.631 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:50:07.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:07.741 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:07.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:07.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:07.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:07.882 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:07.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[29344]: pgmap v806: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 556 B/s rd, 0 B/s wr, 916 op/s; 70413/338985 objects degraded (20.772%) 2024-03-12T03:50:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[29344]: 1.12 deep-scrub starts 2024-03-12T03:50:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[29344]: 1.12 deep-scrub ok 2024-03-12T03:50:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[29344]: 4.14 deep-scrub starts 2024-03-12T03:50:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[29344]: 4.14 deep-scrub ok 2024-03-12T03:50:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:50:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/134132627' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:50:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[29344]: osdmap e404: 8 total, 7 up, 8 in 2024-03-12T03:50:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[39949]: pgmap v806: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 556 B/s rd, 0 B/s wr, 916 op/s; 70413/338985 objects degraded (20.772%) 2024-03-12T03:50:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[39949]: 1.12 deep-scrub starts 2024-03-12T03:50:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[39949]: 1.12 deep-scrub ok 2024-03-12T03:50:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[39949]: 4.14 deep-scrub starts 2024-03-12T03:50:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[39949]: 4.14 deep-scrub ok 2024-03-12T03:50:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:50:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/134132627' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:50:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:07 smithi169 ceph-mon[39949]: osdmap e404: 8 total, 7 up, 8 in 2024-03-12T03:50:07.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:07.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:07 smithi173 ceph-mon[33544]: pgmap v806: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 556 B/s rd, 0 B/s wr, 916 op/s; 70413/338985 objects degraded (20.772%) 2024-03-12T03:50:07.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:07 smithi173 ceph-mon[33544]: 1.12 deep-scrub starts 2024-03-12T03:50:07.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:07 smithi173 ceph-mon[33544]: 1.12 deep-scrub ok 2024-03-12T03:50:07.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:07 smithi173 ceph-mon[33544]: 4.14 deep-scrub starts 2024-03-12T03:50:07.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:07 smithi173 ceph-mon[33544]: 4.14 deep-scrub ok 2024-03-12T03:50:07.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:07 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:50:07.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:07 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/134132627' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:50:07.989 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:07 smithi173 ceph-mon[33544]: osdmap e404: 8 total, 7 up, 8 in 2024-03-12T03:50:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:08.023 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:08.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:08.124 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:08.152 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:08.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:08.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:08.253 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:08.277 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:08.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:08.378 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:08.402 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:08.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:08.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:08.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:08.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:08.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:08.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:08.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:08.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:08.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:08.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:08.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:08 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:50:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:08 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:50:08.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:08 smithi169 ceph-mon[29344]: osdmap e405: 8 total, 7 up, 8 in 2024-03-12T03:50:08.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:08 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 70413/338985 objects degraded (20.772%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:08.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:08 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:50:08.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:08 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:50:08.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:08 smithi169 ceph-mon[39949]: osdmap e405: 8 total, 7 up, 8 in 2024-03-12T03:50:08.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:08 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 70413/338985 objects degraded (20.772%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:08.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:08.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:08.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:08.971 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:08 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:50:08.971 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:08 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:50:08.971 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:08 smithi173 ceph-mon[33544]: osdmap e405: 8 total, 7 up, 8 in 2024-03-12T03:50:08.971 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:08 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 70413/338985 objects degraded (20.772%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:50:09.062 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:50:09.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:09.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:09.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:09.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:09.214 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:09.245 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:09.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:09.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:09.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:09.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:09.485 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:09.519 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:09.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:09.558 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-12T03:50:09.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:09.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:09.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:09.754 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:50:09.755 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-12T03:50:09.755 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:50:09.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:09.806 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:09.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:09.908 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:09.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:09 smithi169 ceph-mon[29344]: pgmap v809: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s; 65786/319479 objects degraded (20.592%) 2024-03-12T03:50:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:09 smithi169 ceph-mon[39949]: pgmap v809: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s; 65786/319479 objects degraded (20.592%) 2024-03-12T03:50:09.939 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:09.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:09 smithi173 ceph-mon[33544]: pgmap v809: 168 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 104 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s; 65786/319479 objects degraded (20.592%) 2024-03-12T03:50:10.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:10.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:10.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:10.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:10.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:10.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:10.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:10.345 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:10.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:10.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:10.477 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:10.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:10.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:10.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:10.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:10.627 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:10.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:10.728 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:10.761 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:10.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:10.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:10.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:10.894 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:10.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:10.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:10.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:11.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:11.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:11.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:11.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:50:11.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:11.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:11.169 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:11.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:11.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:11.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:11.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:11.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:11.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:11.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:11.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:11.570 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:11.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:11.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:11.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:11.712 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:11.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:11.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:11.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:11.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:11 smithi169 ceph-mon[29344]: pgmap v810: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.17k op/s; 65786/316941 objects degraded (20.757%) 2024-03-12T03:50:11.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:11 smithi169 ceph-mon[39949]: pgmap v810: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.17k op/s; 65786/316941 objects degraded (20.757%) 2024-03-12T03:50:11.925 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:50:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:50:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:50:11.937 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:11.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:11.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:11.985 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:11.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:11 smithi173 ceph-mon[33544]: pgmap v810: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.17k op/s; 65786/316941 objects degraded (20.757%) 2024-03-12T03:50:12.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:12.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:12.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:12.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:12.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:12.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:12.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:12.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:12.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:12.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:12.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:12.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:50:12.494 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:12.528 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:12.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:12.546 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:50:12.546 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:50:12.546 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-03-12T03:50:12.546 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-03-12T03:50:12.546 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:50:12.630 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:12.680 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:12.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:12.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:12.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:12.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:12.920 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:12.955 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:12.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:13.057 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:13.091 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:13.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:13.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:13.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:13.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:13.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:13.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:13.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:13.465 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-12T03:50:13.467 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:13.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:50:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:50:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:13.515 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:13.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:13.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:13.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:13.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:13.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:13.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:13.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:13.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:13.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:13.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:13.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:13 smithi173 ceph-mon[33544]: pgmap v811: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 62261/302379 objects degraded (20.590%) 2024-03-12T03:50:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:13 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 62261/302379 objects degraded (20.590%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:14.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:14.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:14.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:14.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:13 smithi169 ceph-mon[39949]: pgmap v811: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 62261/302379 objects degraded (20.590%) 2024-03-12T03:50:14.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:13 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 62261/302379 objects degraded (20.590%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:14.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:13 smithi169 ceph-mon[29344]: pgmap v811: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 62261/302379 objects degraded (20.590%) 2024-03-12T03:50:14.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:13 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 62261/302379 objects degraded (20.590%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:14.178 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:14.215 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:14.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:14.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:14.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:14.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:14.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:14.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:14.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:14.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:14.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:14.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:14.635 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:14.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:14.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:14.792 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:14.796 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:14.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:14.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:14 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:14.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:14 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:14.930 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:14.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:14.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:14 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:15.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:15.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:15.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:15.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:15.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:15.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:15.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:15.317 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:15.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:15.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:15.386 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:50:15.417 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:50:15.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:15.456 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:50:15.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:15.485 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:15.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:15.497 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:50:15.534 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:50:15.569 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:50:15.586 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:15.609 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:50:15.609 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:50:15.609 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:50:15.609 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:50:15.610 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:50:15.610 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:50:15.610 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:50:15.610 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:50:15.611 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:50:15.622 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:15.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:15.676 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:50:15.676 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:50:15.676 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_3 2024-03-12T03:50:15.676 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_3 by 10 min 8 2024-03-12T03:50:15.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:50:15.724 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:15.761 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:15.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:15.820 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:15 smithi169 ceph-mon[29344]: pgmap v812: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.82k op/s; 61337/295350 objects degraded (20.768%) 2024-03-12T03:50:15.820 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:15 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/962750278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:50:15.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:15 smithi169 ceph-mon[39949]: pgmap v812: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.82k op/s; 61337/295350 objects degraded (20.768%) 2024-03-12T03:50:15.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:15 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/962750278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:50:15.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:15.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:15.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:15.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:15 smithi173 ceph-mon[33544]: pgmap v812: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.82k op/s; 61337/295350 objects degraded (20.768%) 2024-03-12T03:50:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:15 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/962750278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:50:16.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:16.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:16.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:16.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:16.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:16.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:16.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:16.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:16.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:16.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:16.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:16.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:16.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:16.588 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:16.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:16.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:16.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:16.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:16.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:16.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:16.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:16.861 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:16.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:16.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:16.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:16.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:16.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:16.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:17.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:17.129 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:17.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:17.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:17.231 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:17.269 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:17.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:17.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:17.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:17.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:17.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:17.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:17.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:17.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:17.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:17.680 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:17.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:17.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:17.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:17.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:17.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:17.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:17 smithi169 ceph-mon[29344]: pgmap v813: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 535 B/s rd, 0 B/s wr, 1.27k op/s; 61337/295350 objects degraded (20.768%) 2024-03-12T03:50:17.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:17 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:50:17.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:17 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:17 smithi169 ceph-mon[39949]: pgmap v813: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 535 B/s rd, 0 B/s wr, 1.27k op/s; 61337/295350 objects degraded (20.768%) 2024-03-12T03:50:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:17 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:50:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:17 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:17.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:17.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:17 smithi173 ceph-mon[33544]: pgmap v813: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 535 B/s rd, 0 B/s wr, 1.27k op/s; 61337/295350 objects degraded (20.768%) 2024-03-12T03:50:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:17 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:50:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:17 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:18.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:18.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:18.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:18.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:18.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:18.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:18.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:18.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:18.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:18.369 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:18.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:18.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:18.511 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:18.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:18.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:18.642 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:18.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:18.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:50:18.762 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:50:18.780 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:18.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:18.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:18.883 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:18 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 61337/295350 objects degraded (20.768%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:18.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:18 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 61337/295350 objects degraded (20.768%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:18.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:18.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:18.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:18 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 61337/295350 objects degraded (20.768%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:19.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:19.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:19.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:19.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:19.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:19.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:19.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:19.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:19.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:19.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:19.475 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:19.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:19.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:19.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:19.611 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:19.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:19.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:19.740 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:19.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:19.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:19.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:19.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:19.876 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:19 smithi169 ceph-mon[29344]: pgmap v814: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 975 B/s rd, 0 B/s wr, 1.55k op/s; 56413/274674 objects degraded (20.538%) 2024-03-12T03:50:19.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:19 smithi169 ceph-mon[39949]: pgmap v814: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 975 B/s rd, 0 B/s wr, 1.55k op/s; 56413/274674 objects degraded (20.538%) 2024-03-12T03:50:19.882 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:19.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:19.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:19 smithi173 ceph-mon[33544]: pgmap v814: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 975 B/s rd, 0 B/s wr, 1.55k op/s; 56413/274674 objects degraded (20.538%) 2024-03-12T03:50:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:20.014 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:20.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:20.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:20.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:20.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:20.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:20.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:20.283 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:20.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:20.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:20.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:20.415 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:20.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:20.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:20.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:20.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:20.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:20.627 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:50:20.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:20.694 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:20.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:20.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:50:20.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:20.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:20.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:20 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3231636019' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:50:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:20 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:50:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:20 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3231636019' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:50:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:20 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:50:20.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:20.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:20.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:20 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3231636019' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:50:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:20 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:50:21.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:21.109 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:21.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:21.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:21.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:21.246 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:21.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:21.338 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:50:21.338 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:50:21.348 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:21.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:21.384 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:21.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:21.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:21.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:21.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:21.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:21.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:21.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:21.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:21.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:21.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:21 smithi169 ceph-mon[29344]: pgmap v815: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.33k op/s; 56413/271713 objects degraded (20.762%) 2024-03-12T03:50:21.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:21 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:50:21.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:21 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:50:21.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:21 smithi169 ceph-mon[29344]: osdmap e406: 8 total, 7 up, 8 in 2024-03-12T03:50:21.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:21 smithi169 ceph-mon[39949]: pgmap v815: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.33k op/s; 56413/271713 objects degraded (20.762%) 2024-03-12T03:50:21.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:21 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:50:21.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:21 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:50:21.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:21 smithi169 ceph-mon[39949]: osdmap e406: 8 total, 7 up, 8 in 2024-03-12T03:50:21.759 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:21.794 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:21.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:21.896 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:21.925 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:21.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:21.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:21 smithi173 ceph-mon[33544]: pgmap v815: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.33k op/s; 56413/271713 objects degraded (20.762%) 2024-03-12T03:50:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:21 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:50:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:21 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:50:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:21 smithi173 ceph-mon[33544]: osdmap e406: 8 total, 7 up, 8 in 2024-03-12T03:50:22.027 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:22.061 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:22.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:22.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:22.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:50:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:50:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:50:22.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:22.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:22.299 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:22.331 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:22.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:22.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:22.433 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:22.468 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:22.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:22.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:22.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:22.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:22.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:22.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:22.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:22.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:22.883 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:22.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:22.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:23.030 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:23.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:23.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:23.166 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:23.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:23.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:23.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:23.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:23.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:23.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:23.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:23.442 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:23.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:23.540 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:50:23.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:23.553 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:50:23.586 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:23.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:23.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:50:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:50:23.688 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:23.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:23.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:23.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:23.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:23.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:23 smithi169 ceph-mon[29344]: pgmap v817: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.57k op/s; 52457/255363 objects degraded (20.542%) 2024-03-12T03:50:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:23 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 52457/255363 objects degraded (20.542%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:23 smithi169 ceph-mon[29344]: osdmap e407: 8 total, 7 up, 8 in 2024-03-12T03:50:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:23 smithi169 ceph-mon[39949]: pgmap v817: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.57k op/s; 52457/255363 objects degraded (20.542%) 2024-03-12T03:50:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:23 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 52457/255363 objects degraded (20.542%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:23 smithi169 ceph-mon[39949]: osdmap e407: 8 total, 7 up, 8 in 2024-03-12T03:50:23.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:23 smithi173 ceph-mon[33544]: pgmap v817: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.57k op/s; 52457/255363 objects degraded (20.542%) 2024-03-12T03:50:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:23 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 52457/255363 objects degraded (20.542%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:23 smithi173 ceph-mon[33544]: osdmap e407: 8 total, 7 up, 8 in 2024-03-12T03:50:23.999 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:24.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:24.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:24.126 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:24.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:24.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:24.260 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:24.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:24.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:24.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-12T03:50:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:24.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:24.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:24.501 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:24.536 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:24.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:24.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:24.621 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:24.623 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:24.623 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:24 smithi169 ceph-mon[39949]: from='client.25580 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:50:24.638 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:24.671 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:24.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:24.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:24.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:24.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:24.911 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:24.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:50:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:24 smithi169 ceph-mon[29344]: from='client.25580 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:50:24.945 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:24.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:24 smithi173 ceph-mon[33544]: from='client.25580 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:50:25.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:25.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:25.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:25.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:25.202 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":818,"stamp":"2024-03-12T03:50:23.471472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5469536915,"num_objects":83680,"num_object_clones":0,"num_object_copies":251040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":51529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83680,"num_whiteouts":0,"num_read":2435,"num_read_kb":2159,"num_write":195458,"num_write_kb":8895644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":676620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80932,"ondisk_log_size":80932,"up":441,"acting":441,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21090840,"kb_used_data":15949508,"kb_used_omap":24,"kb_used_meta":5140903,"kb_avail":728870376,"statfs":{"total":767960285184,"available":746363265024,"internally_reserved":0,"allocated":16332296192,"data_stored":16322856237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24644,"internal_metadata":5264285628},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-967966720,"num_objects":-14770,"num_object_clones":0,"num_object_copies":-44310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-9808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-14770,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14770,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.315071"},"pg_stats":[{"pgid":"10.2","version":"406'12184","reported_seq":12277,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:22.782915+0000","last_change":"2024-03-12T03:48:58.458222+0000","last_active":"2024-03-12T03:50:22.782915+0000","last_peered":"2024-03-12T03:50:22.782915+0000","last_clean":"2024-03-12T03:48:57.540498+0000","last_became_active":"2024-03-12T03:48:58.458222+0000","last_became_peered":"2024-03-12T03:48:58.458222+0000","last_unstale":"2024-03-12T03:50:22.782915+0000","last_undegraded":"2024-03-12T03:48:58.441694+0000","last_fullsized":"2024-03-12T03:48:58.441567+0000","mapping_epoch":384,"log_start":"392'7300","ondisk_log_start":"392'7300","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4884,"log_dups_size":0,"ondisk_log_size":4884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323223552,"num_objects":4932,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":547712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":4932}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1055,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.618827+0000","last_change":"2024-03-12T03:49:15.453480+0000","last_active":"2024-03-12T03:50:20.618827+0000","last_peered":"2024-03-12T03:50:20.618827+0000","last_clean":"2024-03-12T03:50:20.618827+0000","last_became_active":"2024-03-12T03:46:25.289005+0000","last_became_peered":"2024-03-12T03:46:25.289005+0000","last_unstale":"2024-03-12T03:50:20.618827+0000","last_undegraded":"2024-03-12T03:50:20.618827+0000","last_fullsized":"2024-03-12T03:50:20.618827+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:15.453179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:15.453179+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:29.976062+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"344'25","reported_seq":805,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627342+0000","last_change":"2024-03-12T03:49:17.840270+0000","last_active":"2024-03-12T03:50:20.627342+0000","last_peered":"2024-03-12T03:50:20.627342+0000","last_clean":"2024-03-12T03:50:20.627342+0000","last_became_active":"2024-03-12T03:46:25.301377+0000","last_became_peered":"2024-03-12T03:46:25.301377+0000","last_unstale":"2024-03-12T03:50:20.627342+0000","last_undegraded":"2024-03-12T03:50:20.627342+0000","last_fullsized":"2024-03-12T03:50:20.627342+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'25","last_scrub_stamp":"2024-03-12T03:49:17.839910+0000","last_deep_scrub":"344'25","last_deep_scrub_stamp":"2024-03-12T03:49:17.839910+0000","last_clean_scrub_stamp":"2024-03-12T03:49:17.839910+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:34.771372+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":511,"reported_epoch":405,"state":"active+undersized","last_fresh":"2024-03-12T03:50:07.669676+0000","last_change":"2024-03-12T03:48:58.454102+0000","last_active":"2024-03-12T03:50:07.669676+0000","last_peered":"2024-03-12T03:50:07.669676+0000","last_clean":"2024-03-12T03:48:53.448575+0000","last_became_active":"2024-03-12T03:48:58.454102+0000","last_became_peered":"2024-03-12T03:48:58.454102+0000","last_unstale":"2024-03-12T03:50:07.669676+0000","last_undegraded":"2024-03-12T03:50:07.669676+0000","last_fullsized":"2024-03-12T03:48:58.443116+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:07.363439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:07.363439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:11.523365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":611,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.622150+0000","last_change":"2024-03-12T03:49:48.662261+0000","last_active":"2024-03-12T03:50:20.622150+0000","last_peered":"2024-03-12T03:50:20.622150+0000","last_clean":"2024-03-12T03:50:20.622150+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:50:20.622150+0000","last_undegraded":"2024-03-12T03:50:20.622150+0000","last_fullsized":"2024-03-12T03:50:20.622150+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:49:48.662179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:48.662179+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:02.935042+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":751,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.629274+0000","last_change":"2024-03-12T03:49:54.872090+0000","last_active":"2024-03-12T03:50:20.629274+0000","last_peered":"2024-03-12T03:50:20.629274+0000","last_clean":"2024-03-12T03:50:20.629274+0000","last_became_active":"2024-03-12T03:39:33.265665+0000","last_became_peered":"2024-03-12T03:39:33.265665+0000","last_unstale":"2024-03-12T03:50:20.629274+0000","last_undegraded":"2024-03-12T03:50:20.629274+0000","last_fullsized":"2024-03-12T03:50:20.629274+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:54.871895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.871895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:21.677087+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"406'13991","reported_seq":14270,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:21.763232+0000","last_change":"2024-03-12T03:49:54.072615+0000","last_active":"2024-03-12T03:50:21.763232+0000","last_peered":"2024-03-12T03:50:21.763232+0000","last_clean":"2024-03-12T03:50:21.763232+0000","last_became_active":"2024-03-12T03:47:57.695210+0000","last_became_peered":"2024-03-12T03:47:57.695210+0000","last_unstale":"2024-03-12T03:50:21.763232+0000","last_undegraded":"2024-03-12T03:50:21.763232+0000","last_fullsized":"2024-03-12T03:50:21.763232+0000","mapping_epoch":366,"log_start":"390'7801","ondisk_log_start":"390'7801","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'11483","last_scrub_stamp":"2024-03-12T03:49:54.072402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.072402+0000","objects_scrubbed":7964,"log_size":6190,"log_dups_size":0,"ondisk_log_size":6190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:04.899343+0000","scrub_duration":2423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350224384,"num_objects":5344,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12440,"num_write_kb":569088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"269'31","reported_seq":844,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.628477+0000","last_change":"2024-03-12T03:50:01.871630+0000","last_active":"2024-03-12T03:50:20.628477+0000","last_peered":"2024-03-12T03:50:20.628477+0000","last_clean":"2024-03-12T03:50:20.628477+0000","last_became_active":"2024-03-12T03:39:33.268652+0000","last_became_peered":"2024-03-12T03:39:33.268652+0000","last_unstale":"2024-03-12T03:50:20.628477+0000","last_undegraded":"2024-03-12T03:50:20.628477+0000","last_fullsized":"2024-03-12T03:50:20.628477+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:50:01.871412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:01.871412+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:05.340467+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":206,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.629064+0000","last_change":"2024-03-12T03:48:18.820400+0000","last_active":"2024-03-12T03:50:20.629064+0000","last_peered":"2024-03-12T03:50:20.629064+0000","last_clean":"2024-03-12T03:50:20.629064+0000","last_became_active":"2024-03-12T03:45:18.108377+0000","last_became_peered":"2024-03-12T03:45:18.108377+0000","last_unstale":"2024-03-12T03:50:20.629064+0000","last_undegraded":"2024-03-12T03:50:20.629064+0000","last_fullsized":"2024-03-12T03:50:20.629064+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:18.820292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:48:18.820292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949785+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":737,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.618729+0000","last_change":"2024-03-12T03:48:58.457284+0000","last_active":"2024-03-12T03:50:20.618729+0000","last_peered":"2024-03-12T03:50:20.618729+0000","last_clean":"2024-03-12T03:48:53.449180+0000","last_became_active":"2024-03-12T03:48:58.457284+0000","last_became_peered":"2024-03-12T03:48:58.457284+0000","last_unstale":"2024-03-12T03:50:20.618729+0000","last_undegraded":"2024-03-12T03:50:20.618729+0000","last_fullsized":"2024-03-12T03:48:58.441935+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.054936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.054936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":488,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.622589+0000","last_change":"2024-03-12T03:49:17.061923+0000","last_active":"2024-03-12T03:50:20.622589+0000","last_peered":"2024-03-12T03:50:20.622589+0000","last_clean":"2024-03-12T03:50:20.622589+0000","last_became_active":"2024-03-12T03:46:25.293293+0000","last_became_peered":"2024-03-12T03:46:25.293293+0000","last_unstale":"2024-03-12T03:50:20.622589+0000","last_undegraded":"2024-03-12T03:50:20.622589+0000","last_fullsized":"2024-03-12T03:50:20.622589+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:17.061809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:17.061809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:45.913368+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"388'129","reported_seq":896,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626450+0000","last_change":"2024-03-12T03:49:17.168749+0000","last_active":"2024-03-12T03:50:20.626450+0000","last_peered":"2024-03-12T03:50:20.626450+0000","last_clean":"2024-03-12T03:50:20.626450+0000","last_became_active":"2024-03-12T03:46:25.297913+0000","last_became_peered":"2024-03-12T03:46:25.297913+0000","last_unstale":"2024-03-12T03:50:20.626450+0000","last_undegraded":"2024-03-12T03:50:20.626450+0000","last_fullsized":"2024-03-12T03:50:20.626450+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"388'129","last_scrub_stamp":"2024-03-12T03:49:17.168603+0000","last_deep_scrub":"388'129","last_deep_scrub_stamp":"2024-03-12T03:49:17.168603+0000","last_clean_scrub_stamp":"2024-03-12T03:49:17.168603+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:46.165998+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"406'12329","reported_seq":12498,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:22.790078+0000","last_change":"2024-03-12T03:49:20.727415+0000","last_active":"2024-03-12T03:50:22.790078+0000","last_peered":"2024-03-12T03:50:22.790078+0000","last_clean":"2024-03-12T03:50:22.790078+0000","last_became_active":"2024-03-12T03:47:58.027417+0000","last_became_peered":"2024-03-12T03:47:58.027417+0000","last_unstale":"2024-03-12T03:50:22.790078+0000","last_undegraded":"2024-03-12T03:50:22.790078+0000","last_fullsized":"2024-03-12T03:50:22.790078+0000","mapping_epoch":366,"log_start":"391'7400","ondisk_log_start":"391'7400","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'7006","last_scrub_stamp":"2024-03-12T03:49:20.727145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:49:20.727145+0000","objects_scrubbed":6935,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:27.589288+0000","scrub_duration":2177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332857344,"num_objects":5079,"num_object_clones":0,"num_object_copies":15237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":557056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":876,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.618086+0000","last_change":"2024-03-12T03:48:58.456582+0000","last_active":"2024-03-12T03:50:20.618086+0000","last_peered":"2024-03-12T03:50:20.618086+0000","last_clean":"2024-03-12T03:48:53.447572+0000","last_became_active":"2024-03-12T03:48:58.456582+0000","last_became_peered":"2024-03-12T03:48:58.456582+0000","last_unstale":"2024-03-12T03:50:20.618086+0000","last_undegraded":"2024-03-12T03:50:20.618086+0000","last_fullsized":"2024-03-12T03:48:58.441708+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:08.710391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:08.710391+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"269'31","reported_seq":796,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.629000+0000","last_change":"2024-03-12T03:49:09.755599+0000","last_active":"2024-03-12T03:50:20.629000+0000","last_peered":"2024-03-12T03:50:20.629000+0000","last_clean":"2024-03-12T03:50:20.629000+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:50:20.629000+0000","last_undegraded":"2024-03-12T03:50:20.629000+0000","last_fullsized":"2024-03-12T03:50:20.629000+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:49:09.755469+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:49:09.755469+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:35.253531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":755,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626834+0000","last_change":"2024-03-12T03:47:57.920336+0000","last_active":"2024-03-12T03:50:20.626834+0000","last_peered":"2024-03-12T03:50:20.626834+0000","last_clean":"2024-03-12T03:50:20.626834+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:50:20.626834+0000","last_undegraded":"2024-03-12T03:50:20.626834+0000","last_fullsized":"2024-03-12T03:50:20.626834+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:57.920257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_clean_scrub_stamp":"2024-03-12T03:47:57.920257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":733,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627382+0000","last_change":"2024-03-12T03:49:06.194181+0000","last_active":"2024-03-12T03:50:20.627382+0000","last_peered":"2024-03-12T03:50:20.627382+0000","last_clean":"2024-03-12T03:50:20.627382+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:50:20.627382+0000","last_undegraded":"2024-03-12T03:50:20.627382+0000","last_fullsized":"2024-03-12T03:50:20.627382+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:06.192749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:06.192749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:34.863444+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":526,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.670147+0000","last_change":"2024-03-12T03:49:48.536574+0000","last_active":"2024-03-12T03:50:07.670147+0000","last_peered":"2024-03-12T03:50:07.670147+0000","last_clean":"2024-03-12T03:50:07.670147+0000","last_became_active":"2024-03-12T03:46:25.300814+0000","last_became_peered":"2024-03-12T03:46:25.300814+0000","last_unstale":"2024-03-12T03:50:07.670147+0000","last_undegraded":"2024-03-12T03:50:07.670147+0000","last_fullsized":"2024-03-12T03:50:07.670147+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:48.536435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:48.536435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:04.740323+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"406'11972","reported_seq":12066,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:21.240999+0000","last_change":"2024-03-12T03:48:58.455008+0000","last_active":"2024-03-12T03:50:21.240999+0000","last_peered":"2024-03-12T03:50:21.240999+0000","last_clean":"2024-03-12T03:48:57.553732+0000","last_became_active":"2024-03-12T03:48:58.455008+0000","last_became_peered":"2024-03-12T03:48:58.455008+0000","last_unstale":"2024-03-12T03:50:21.240999+0000","last_undegraded":"2024-03-12T03:48:58.446599+0000","last_fullsized":"2024-03-12T03:48:58.446493+0000","mapping_epoch":384,"log_start":"395'7900","ondisk_log_start":"395'7900","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4072,"log_dups_size":0,"ondisk_log_size":4072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335544320,"num_objects":5120,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11972,"num_write_kb":546944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":5120}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":869,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627472+0000","last_change":"2024-03-12T03:48:17.651575+0000","last_active":"2024-03-12T03:50:20.627472+0000","last_peered":"2024-03-12T03:50:20.627472+0000","last_clean":"2024-03-12T03:50:20.627472+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:50:20.627472+0000","last_undegraded":"2024-03-12T03:50:20.627472+0000","last_fullsized":"2024-03-12T03:50:20.627472+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:17.651463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:17.651463+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":599,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.618413+0000","last_change":"2024-03-12T03:49:12.640242+0000","last_active":"2024-03-12T03:50:20.618413+0000","last_peered":"2024-03-12T03:50:20.618413+0000","last_clean":"2024-03-12T03:50:20.618413+0000","last_became_active":"2024-03-12T03:46:25.287373+0000","last_became_peered":"2024-03-12T03:46:25.287373+0000","last_unstale":"2024-03-12T03:50:20.618413+0000","last_undegraded":"2024-03-12T03:50:20.618413+0000","last_fullsized":"2024-03-12T03:50:20.618413+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:49:12.640132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:12.640132+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":236,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.625782+0000","last_change":"2024-03-12T03:48:58.456372+0000","last_active":"2024-03-12T03:50:20.625782+0000","last_peered":"2024-03-12T03:50:20.625782+0000","last_clean":"2024-03-12T03:48:53.446320+0000","last_became_active":"2024-03-12T03:48:58.456372+0000","last_became_peered":"2024-03-12T03:48:58.456372+0000","last_unstale":"2024-03-12T03:50:20.625782+0000","last_undegraded":"2024-03-12T03:50:20.625782+0000","last_fullsized":"2024-03-12T03:48:58.444143+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:16.876204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:16.876204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949730+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"385'9","reported_seq":951,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:20.629110+0000","last_change":"2024-03-12T03:48:58.455645+0000","last_active":"2024-03-12T03:50:20.629110+0000","last_peered":"2024-03-12T03:50:20.629110+0000","last_clean":"2024-03-12T03:48:56.818397+0000","last_became_active":"2024-03-12T03:48:58.455645+0000","last_became_peered":"2024-03-12T03:48:58.455645+0000","last_unstale":"2024-03-12T03:50:20.629110+0000","last_undegraded":"2024-03-12T03:48:58.448928+0000","last_fullsized":"2024-03-12T03:48:58.448807+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:47:53.964216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:53.964216+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":724,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.626360+0000","last_change":"2024-03-12T03:48:58.448113+0000","last_active":"2024-03-12T03:50:20.626360+0000","last_peered":"2024-03-12T03:50:20.626360+0000","last_clean":"2024-03-12T03:48:53.472090+0000","last_became_active":"2024-03-12T03:48:58.448113+0000","last_became_peered":"2024-03-12T03:48:58.448113+0000","last_unstale":"2024-03-12T03:50:20.626360+0000","last_undegraded":"2024-03-12T03:50:20.626360+0000","last_fullsized":"2024-03-12T03:48:58.441990+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:04.770746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:04.770746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"406'12129","reported_seq":12220,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:21.251421+0000","last_change":"2024-03-12T03:47:58.028347+0000","last_active":"2024-03-12T03:50:21.251421+0000","last_peered":"2024-03-12T03:50:21.251421+0000","last_clean":"2024-03-12T03:50:21.251421+0000","last_became_active":"2024-03-12T03:47:58.027802+0000","last_became_peered":"2024-03-12T03:47:58.027802+0000","last_unstale":"2024-03-12T03:50:21.251421+0000","last_undegraded":"2024-03-12T03:50:21.251421+0000","last_fullsized":"2024-03-12T03:50:21.251421+0000","mapping_epoch":366,"log_start":"395'8100","ondisk_log_start":"395'8100","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4029,"log_dups_size":0,"ondisk_log_size":4029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344391680,"num_objects":5255,"num_object_clones":0,"num_object_copies":15765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12129,"num_write_kb":556288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":746,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626745+0000","last_change":"2024-03-12T03:49:11.818292+0000","last_active":"2024-03-12T03:50:20.626745+0000","last_peered":"2024-03-12T03:50:20.626745+0000","last_clean":"2024-03-12T03:50:20.626745+0000","last_became_active":"2024-03-12T03:46:25.286985+0000","last_became_peered":"2024-03-12T03:46:25.286985+0000","last_unstale":"2024-03-12T03:50:20.626745+0000","last_undegraded":"2024-03-12T03:50:20.626745+0000","last_fullsized":"2024-03-12T03:50:20.626745+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:11.818116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:11.818116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949894+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"269'42","reported_seq":599,"reported_epoch":405,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:07.670318+0000","last_change":"2024-03-12T03:48:58.463377+0000","last_active":"2024-03-12T03:50:07.670318+0000","last_peered":"2024-03-12T03:50:07.670318+0000","last_clean":"2024-03-12T03:48:53.447338+0000","last_became_active":"2024-03-12T03:48:58.463377+0000","last_became_peered":"2024-03-12T03:48:58.463377+0000","last_unstale":"2024-03-12T03:50:07.670318+0000","last_undegraded":"2024-03-12T03:48:58.440183+0000","last_fullsized":"2024-03-12T03:48:58.440082+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:47:13.347128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.347128+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:11.523391+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":515,"reported_epoch":405,"state":"active+undersized","last_fresh":"2024-03-12T03:50:07.672710+0000","last_change":"2024-03-12T03:48:58.464970+0000","last_active":"2024-03-12T03:50:07.672710+0000","last_peered":"2024-03-12T03:50:07.672710+0000","last_clean":"2024-03-12T03:48:53.448054+0000","last_became_active":"2024-03-12T03:48:58.464970+0000","last_became_peered":"2024-03-12T03:48:58.464970+0000","last_unstale":"2024-03-12T03:50:07.672710+0000","last_undegraded":"2024-03-12T03:50:07.672710+0000","last_fullsized":"2024-03-12T03:48:58.441302+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:05.639950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:05.639950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:11.523415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":762,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.628893+0000","last_change":"2024-03-12T03:49:14.836471+0000","last_active":"2024-03-12T03:50:20.628893+0000","last_peered":"2024-03-12T03:50:20.628893+0000","last_clean":"2024-03-12T03:50:20.628893+0000","last_became_active":"2024-03-12T03:46:25.280860+0000","last_became_peered":"2024-03-12T03:46:25.280860+0000","last_unstale":"2024-03-12T03:50:20.628893+0000","last_undegraded":"2024-03-12T03:50:20.628893+0000","last_fullsized":"2024-03-12T03:50:20.628893+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:14.836179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_clean_scrub_stamp":"2024-03-12T03:49:14.836179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:41.107430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":884,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627388+0000","last_change":"2024-03-12T03:48:15.895744+0000","last_active":"2024-03-12T03:50:20.627388+0000","last_peered":"2024-03-12T03:50:20.627388+0000","last_clean":"2024-03-12T03:50:20.627388+0000","last_became_active":"2024-03-12T03:44:19.373715+0000","last_became_peered":"2024-03-12T03:44:19.373715+0000","last_unstale":"2024-03-12T03:50:20.627388+0000","last_undegraded":"2024-03-12T03:50:20.627388+0000","last_fullsized":"2024-03-12T03:50:20.627388+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:15.895638+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:15.895638+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.895638+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"405'11966","reported_seq":12137,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:18.550350+0000","last_change":"2024-03-12T03:49:58.948258+0000","last_active":"2024-03-12T03:50:18.550350+0000","last_peered":"2024-03-12T03:50:18.550350+0000","last_clean":"2024-03-12T03:50:18.550350+0000","last_became_active":"2024-03-12T03:47:58.027347+0000","last_became_peered":"2024-03-12T03:47:58.027347+0000","last_unstale":"2024-03-12T03:50:18.550350+0000","last_undegraded":"2024-03-12T03:50:18.550350+0000","last_fullsized":"2024-03-12T03:50:18.550350+0000","mapping_epoch":366,"log_start":"393'7600","ondisk_log_start":"393'7600","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'10207","last_scrub_stamp":"2024-03-12T03:49:58.947894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:49:58.947894+0000","objects_scrubbed":7488,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:01.246152+0000","scrub_duration":2515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368443392,"num_objects":5622,"num_object_clones":0,"num_object_copies":16866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11966,"num_write_kb":562816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"332'12","reported_seq":716,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:11.829302+0000","last_change":"2024-03-12T03:49:07.516290+0000","last_active":"2024-03-12T03:50:11.829302+0000","last_peered":"2024-03-12T03:50:11.829302+0000","last_clean":"2024-03-12T03:50:11.829302+0000","last_became_active":"2024-03-12T03:46:25.288674+0000","last_became_peered":"2024-03-12T03:46:25.288674+0000","last_unstale":"2024-03-12T03:50:11.829302+0000","last_undegraded":"2024-03-12T03:50:11.829302+0000","last_fullsized":"2024-03-12T03:50:11.829302+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'12","last_scrub_stamp":"2024-03-12T03:49:07.516156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:07.516156+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:32.306891+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"269'42","reported_seq":802,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627253+0000","last_change":"2024-03-12T03:47:56.911844+0000","last_active":"2024-03-12T03:50:20.627253+0000","last_peered":"2024-03-12T03:50:20.627253+0000","last_clean":"2024-03-12T03:50:20.627253+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:50:20.627253+0000","last_undegraded":"2024-03-12T03:50:20.627253+0000","last_fullsized":"2024-03-12T03:50:20.627253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:47:56.911701+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.911701+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"320'2","reported_seq":608,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:20.618229+0000","last_change":"2024-03-12T03:48:58.457502+0000","last_active":"2024-03-12T03:50:20.618229+0000","last_peered":"2024-03-12T03:50:20.618229+0000","last_clean":"2024-03-12T03:45:56.084290+0000","last_became_active":"2024-03-12T03:48:58.457502+0000","last_became_peered":"2024-03-12T03:48:58.457502+0000","last_unstale":"2024-03-12T03:50:20.618229+0000","last_undegraded":"2024-03-12T03:48:58.439950+0000","last_fullsized":"2024-03-12T03:48:58.439792+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'2","last_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_deep_scrub":"320'2","last_deep_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_clean_scrub_stamp":"2024-03-12T03:45:56.089247+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":752,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617177+0000","last_change":"2024-03-12T03:49:54.471443+0000","last_active":"2024-03-12T03:50:20.617177+0000","last_peered":"2024-03-12T03:50:20.617177+0000","last_clean":"2024-03-12T03:50:20.617177+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:50:20.617177+0000","last_undegraded":"2024-03-12T03:50:20.617177+0000","last_fullsized":"2024-03-12T03:50:20.617177+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:54.471078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:49:54.471078+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.471078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:55.755964+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":692,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.661082+0000","last_change":"2024-03-12T03:49:53.442784+0000","last_active":"2024-03-12T03:50:07.661082+0000","last_peered":"2024-03-12T03:50:07.661082+0000","last_clean":"2024-03-12T03:50:07.661082+0000","last_became_active":"2024-03-12T03:46:25.299964+0000","last_became_peered":"2024-03-12T03:46:25.299964+0000","last_unstale":"2024-03-12T03:50:07.661082+0000","last_undegraded":"2024-03-12T03:50:07.661082+0000","last_fullsized":"2024-03-12T03:50:07.661082+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:53.442638+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_clean_scrub_stamp":"2024-03-12T03:49:53.442638+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:06.708718+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"406'11972","reported_seq":12066,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:21.434362+0000","last_change":"2024-03-12T03:48:58.456284+0000","last_active":"2024-03-12T03:50:21.434362+0000","last_peered":"2024-03-12T03:50:21.434362+0000","last_clean":"2024-03-12T03:48:57.535679+0000","last_became_active":"2024-03-12T03:48:58.456284+0000","last_became_peered":"2024-03-12T03:48:58.456284+0000","last_unstale":"2024-03-12T03:50:21.434362+0000","last_undegraded":"2024-03-12T03:48:58.446353+0000","last_fullsized":"2024-03-12T03:48:58.446227+0000","mapping_epoch":384,"log_start":"391'7200","ondisk_log_start":"391'7200","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340918272,"num_objects":5202,"num_object_clones":0,"num_object_copies":15606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11972,"num_write_kb":549568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":5202}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"385'9","reported_seq":92,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:20.627973+0000","last_change":"2024-03-12T03:48:58.451360+0000","last_active":"2024-03-12T03:50:20.627973+0000","last_peered":"2024-03-12T03:50:20.627973+0000","last_clean":"2024-03-12T03:33:28.360558+0000","last_became_active":"2024-03-12T03:48:58.451360+0000","last_became_peered":"2024-03-12T03:48:58.451360+0000","last_unstale":"2024-03-12T03:50:20.627973+0000","last_undegraded":"2024-03-12T03:48:58.442329+0000","last_fullsized":"2024-03-12T03:48:58.441949+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:48:14.138048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:14.138048+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"269'26","reported_seq":789,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.618599+0000","last_change":"2024-03-12T03:49:03.562283+0000","last_active":"2024-03-12T03:50:20.618599+0000","last_peered":"2024-03-12T03:50:20.618599+0000","last_clean":"2024-03-12T03:50:20.618599+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:50:20.618599+0000","last_undegraded":"2024-03-12T03:50:20.618599+0000","last_fullsized":"2024-03-12T03:50:20.618599+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:49:03.562112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:03.562112+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":764,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626535+0000","last_change":"2024-03-12T03:49:09.155688+0000","last_active":"2024-03-12T03:50:20.626535+0000","last_peered":"2024-03-12T03:50:20.626535+0000","last_clean":"2024-03-12T03:50:20.626535+0000","last_became_active":"2024-03-12T03:46:25.294348+0000","last_became_peered":"2024-03-12T03:46:25.294348+0000","last_unstale":"2024-03-12T03:50:20.626535+0000","last_undegraded":"2024-03-12T03:50:20.626535+0000","last_fullsized":"2024-03-12T03:50:20.626535+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:09.155572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:49:09.155572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:33.600543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":626,"reported_epoch":405,"state":"active+undersized","last_fresh":"2024-03-12T03:50:07.660799+0000","last_change":"2024-03-12T03:48:58.455443+0000","last_active":"2024-03-12T03:50:07.660799+0000","last_peered":"2024-03-12T03:50:07.660799+0000","last_clean":"2024-03-12T03:43:13.330724+0000","last_became_active":"2024-03-12T03:48:58.455443+0000","last_became_peered":"2024-03-12T03:48:58.455443+0000","last_unstale":"2024-03-12T03:50:07.660799+0000","last_undegraded":"2024-03-12T03:50:07.660799+0000","last_fullsized":"2024-03-12T03:48:58.439578+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:13.892760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.892760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:11.523378+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":479,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.673259+0000","last_change":"2024-03-12T03:49:50.458935+0000","last_active":"2024-03-12T03:50:07.673259+0000","last_peered":"2024-03-12T03:50:07.673259+0000","last_clean":"2024-03-12T03:50:07.673259+0000","last_became_active":"2024-03-12T03:46:25.287360+0000","last_became_peered":"2024-03-12T03:46:25.287360+0000","last_unstale":"2024-03-12T03:50:07.673259+0000","last_undegraded":"2024-03-12T03:50:07.673259+0000","last_fullsized":"2024-03-12T03:50:07.673259+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:50.458811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.458811+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:59.838393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"406'12203","reported_seq":12296,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:21.259151+0000","last_change":"2024-03-12T03:48:58.455501+0000","last_active":"2024-03-12T03:50:21.259151+0000","last_peered":"2024-03-12T03:50:21.259151+0000","last_clean":"2024-03-12T03:48:57.549912+0000","last_became_active":"2024-03-12T03:48:58.455501+0000","last_became_peered":"2024-03-12T03:48:58.455501+0000","last_unstale":"2024-03-12T03:50:21.259151+0000","last_undegraded":"2024-03-12T03:48:58.444776+0000","last_fullsized":"2024-03-12T03:48:58.444662+0000","mapping_epoch":384,"log_start":"395'8200","ondisk_log_start":"395'8200","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4003,"log_dups_size":0,"ondisk_log_size":4003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344981528,"num_objects":5265,"num_object_clones":0,"num_object_copies":15795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5265,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12203,"num_write_kb":558913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5265}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":53,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.627896+0000","last_change":"2024-03-12T03:48:58.449166+0000","last_active":"2024-03-12T03:50:20.627896+0000","last_peered":"2024-03-12T03:50:20.627896+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.449166+0000","last_became_peered":"2024-03-12T03:48:58.449166+0000","last_unstale":"2024-03-12T03:50:20.627896+0000","last_undegraded":"2024-03-12T03:50:20.627896+0000","last_fullsized":"2024-03-12T03:48:58.441301+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.179416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.179416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"269'35","reported_seq":551,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:20.628241+0000","last_change":"2024-03-12T03:48:58.454600+0000","last_active":"2024-03-12T03:50:20.628241+0000","last_peered":"2024-03-12T03:50:20.628241+0000","last_clean":"2024-03-12T03:43:28.848207+0000","last_became_active":"2024-03-12T03:48:58.454600+0000","last_became_peered":"2024-03-12T03:48:58.454600+0000","last_unstale":"2024-03-12T03:50:20.628241+0000","last_undegraded":"2024-03-12T03:48:58.443785+0000","last_fullsized":"2024-03-12T03:48:58.443677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-03-12T03:45:59.055318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.055318+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":47,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":771,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.618642+0000","last_change":"2024-03-12T03:49:51.465759+0000","last_active":"2024-03-12T03:50:20.618642+0000","last_peered":"2024-03-12T03:50:20.618642+0000","last_clean":"2024-03-12T03:50:20.618642+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:50:20.618642+0000","last_undegraded":"2024-03-12T03:50:20.618642+0000","last_fullsized":"2024-03-12T03:50:20.618642+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:51.465633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:49:51.465633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:55.149499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":751,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617430+0000","last_change":"2024-03-12T03:49:02.614970+0000","last_active":"2024-03-12T03:50:20.617430+0000","last_peered":"2024-03-12T03:50:20.617430+0000","last_clean":"2024-03-12T03:50:20.617430+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:50:20.617430+0000","last_undegraded":"2024-03-12T03:50:20.617430+0000","last_fullsized":"2024-03-12T03:50:20.617430+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:02.614847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:02.614847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:30.546437+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":349,"reported_epoch":405,"state":"active+undersized","last_fresh":"2024-03-12T03:50:07.672197+0000","last_change":"2024-03-12T03:48:58.454186+0000","last_active":"2024-03-12T03:50:07.672197+0000","last_peered":"2024-03-12T03:50:07.672197+0000","last_clean":"2024-03-12T03:34:01.860694+0000","last_became_active":"2024-03-12T03:48:58.454186+0000","last_became_peered":"2024-03-12T03:48:58.454186+0000","last_unstale":"2024-03-12T03:50:07.672197+0000","last_undegraded":"2024-03-12T03:50:07.672197+0000","last_fullsized":"2024-03-12T03:48:58.443407+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:04.857986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:04.857986+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:10.548127+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"406'12273","reported_seq":12365,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:21.727519+0000","last_change":"2024-03-12T03:48:58.462488+0000","last_active":"2024-03-12T03:50:21.727519+0000","last_peered":"2024-03-12T03:50:21.727519+0000","last_clean":"2024-03-12T03:48:57.504096+0000","last_became_active":"2024-03-12T03:48:58.462488+0000","last_became_peered":"2024-03-12T03:48:58.462488+0000","last_unstale":"2024-03-12T03:50:21.727519+0000","last_undegraded":"2024-03-12T03:48:58.445111+0000","last_fullsized":"2024-03-12T03:48:58.444978+0000","mapping_epoch":384,"log_start":"385'6100","ondisk_log_start":"385'6100","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":6173,"log_dups_size":0,"ondisk_log_size":6173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343474176,"num_objects":5241,"num_object_clones":0,"num_object_copies":15723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12273,"num_write_kb":560448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":5241}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":728,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.627469+0000","last_change":"2024-03-12T03:48:58.467253+0000","last_active":"2024-03-12T03:50:20.627469+0000","last_peered":"2024-03-12T03:50:20.627469+0000","last_clean":"2024-03-12T03:48:53.472859+0000","last_became_active":"2024-03-12T03:48:58.467253+0000","last_became_peered":"2024-03-12T03:48:58.467253+0000","last_unstale":"2024-03-12T03:50:20.627469+0000","last_undegraded":"2024-03-12T03:50:20.627469+0000","last_fullsized":"2024-03-12T03:48:58.444374+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:00.577693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:46:00.577693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"269'24","reported_seq":572,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:20.619245+0000","last_change":"2024-03-12T03:48:58.449415+0000","last_active":"2024-03-12T03:50:20.619245+0000","last_peered":"2024-03-12T03:50:20.619245+0000","last_clean":"2024-03-12T03:43:28.788554+0000","last_became_active":"2024-03-12T03:48:58.449415+0000","last_became_peered":"2024-03-12T03:48:58.449415+0000","last_unstale":"2024-03-12T03:50:20.619245+0000","last_undegraded":"2024-03-12T03:48:58.441112+0000","last_fullsized":"2024-03-12T03:48:58.441001+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:48:17.191968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:17.191968+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":1772,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:20.619335+0000","last_change":"2024-03-12T03:48:58.462253+0000","last_active":"2024-03-12T03:50:20.619335+0000","last_peered":"2024-03-12T03:50:20.619335+0000","last_clean":"2024-03-12T03:48:56.580904+0000","last_became_active":"2024-03-12T03:48:58.462253+0000","last_became_peered":"2024-03-12T03:48:58.462253+0000","last_unstale":"2024-03-12T03:50:20.619335+0000","last_undegraded":"2024-03-12T03:48:58.445825+0000","last_fullsized":"2024-03-12T03:48:58.445711+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:47:13.591375+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.591375+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1011,"num_read_kb":1006,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":628,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.627013+0000","last_change":"2024-03-12T03:48:58.464609+0000","last_active":"2024-03-12T03:50:20.627013+0000","last_peered":"2024-03-12T03:50:20.627013+0000","last_clean":"2024-03-12T03:43:13.332262+0000","last_became_active":"2024-03-12T03:48:58.464609+0000","last_became_peered":"2024-03-12T03:48:58.464609+0000","last_unstale":"2024-03-12T03:50:20.627013+0000","last_undegraded":"2024-03-12T03:50:20.627013+0000","last_fullsized":"2024-03-12T03:48:58.443632+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:58.050985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.050985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949481+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":757,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617673+0000","last_change":"2024-03-12T03:49:13.598817+0000","last_active":"2024-03-12T03:50:20.617673+0000","last_peered":"2024-03-12T03:50:20.617673+0000","last_clean":"2024-03-12T03:50:20.617673+0000","last_became_active":"2024-03-12T03:46:25.302256+0000","last_became_peered":"2024-03-12T03:46:25.302256+0000","last_unstale":"2024-03-12T03:50:20.617673+0000","last_undegraded":"2024-03-12T03:50:20.617673+0000","last_fullsized":"2024-03-12T03:50:20.617673+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:13.598692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:13.598692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:34.615813+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":874,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.618417+0000","last_change":"2024-03-12T03:49:50.468068+0000","last_active":"2024-03-12T03:50:20.618417+0000","last_peered":"2024-03-12T03:50:20.618417+0000","last_clean":"2024-03-12T03:50:20.618417+0000","last_became_active":"2024-03-12T03:44:19.366060+0000","last_became_peered":"2024-03-12T03:44:19.366060+0000","last_unstale":"2024-03-12T03:50:20.618417+0000","last_undegraded":"2024-03-12T03:50:20.618417+0000","last_fullsized":"2024-03-12T03:50:20.618417+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:50.467976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.467976+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:15.492614+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"406'12167","reported_seq":12260,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:22.756122+0000","last_change":"2024-03-12T03:48:58.444686+0000","last_active":"2024-03-12T03:50:22.756122+0000","last_peered":"2024-03-12T03:50:22.756122+0000","last_clean":"2024-03-12T03:48:57.546566+0000","last_became_active":"2024-03-12T03:48:58.444686+0000","last_became_peered":"2024-03-12T03:48:58.444686+0000","last_unstale":"2024-03-12T03:50:22.756122+0000","last_undegraded":"2024-03-12T03:48:58.438881+0000","last_fullsized":"2024-03-12T03:48:58.438716+0000","mapping_epoch":384,"log_start":"391'7300","ondisk_log_start":"391'7300","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322633728,"num_objects":4923,"num_object_clones":0,"num_object_copies":14769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12167,"num_write_kb":546880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4923}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":748,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.618634+0000","last_change":"2024-03-12T03:49:07.521537+0000","last_active":"2024-03-12T03:50:20.618634+0000","last_peered":"2024-03-12T03:50:20.618634+0000","last_clean":"2024-03-12T03:50:20.618634+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:50:20.618634+0000","last_undegraded":"2024-03-12T03:50:20.618634+0000","last_fullsized":"2024-03-12T03:50:20.618634+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:07.521281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:07.521281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:22.447642+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"315'29","reported_seq":880,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.628313+0000","last_change":"2024-03-12T03:48:08.875139+0000","last_active":"2024-03-12T03:50:20.628313+0000","last_peered":"2024-03-12T03:50:20.628313+0000","last_clean":"2024-03-12T03:50:20.628313+0000","last_became_active":"2024-03-12T03:45:18.109048+0000","last_became_peered":"2024-03-12T03:45:18.109048+0000","last_unstale":"2024-03-12T03:50:20.628313+0000","last_undegraded":"2024-03-12T03:50:20.628313+0000","last_fullsized":"2024-03-12T03:50:20.628313+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'29","last_scrub_stamp":"2024-03-12T03:48:08.874995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:08.874995+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":511,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.660972+0000","last_change":"2024-03-12T03:49:08.478632+0000","last_active":"2024-03-12T03:50:07.660972+0000","last_peered":"2024-03-12T03:50:07.660972+0000","last_clean":"2024-03-12T03:50:07.660972+0000","last_became_active":"2024-03-12T03:46:25.292535+0000","last_became_peered":"2024-03-12T03:46:25.292535+0000","last_unstale":"2024-03-12T03:50:07.660972+0000","last_undegraded":"2024-03-12T03:50:07.660972+0000","last_fullsized":"2024-03-12T03:50:07.660972+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:08.478517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:49:08.478517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:33.314572+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1020,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.619439+0000","last_change":"2024-03-12T03:48:58.452024+0000","last_active":"2024-03-12T03:50:20.619439+0000","last_peered":"2024-03-12T03:50:20.619439+0000","last_clean":"2024-03-12T03:48:53.448868+0000","last_became_active":"2024-03-12T03:48:58.452024+0000","last_became_peered":"2024-03-12T03:48:58.452024+0000","last_unstale":"2024-03-12T03:50:20.619439+0000","last_undegraded":"2024-03-12T03:50:20.619439+0000","last_fullsized":"2024-03-12T03:48:58.444796+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_clean_scrub_stamp":"2024-03-12T03:48:11.435684+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":739,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626708+0000","last_change":"2024-03-12T03:49:50.444531+0000","last_active":"2024-03-12T03:50:20.626708+0000","last_peered":"2024-03-12T03:50:20.626708+0000","last_clean":"2024-03-12T03:50:20.626708+0000","last_became_active":"2024-03-12T03:46:25.295668+0000","last_became_peered":"2024-03-12T03:46:25.295668+0000","last_unstale":"2024-03-12T03:50:20.626708+0000","last_undegraded":"2024-03-12T03:50:20.626708+0000","last_fullsized":"2024-03-12T03:50:20.626708+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:50.444395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.444395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:53.830638+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"406'12270","reported_seq":12446,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:21.640585+0000","last_change":"2024-03-12T03:49:56.704066+0000","last_active":"2024-03-12T03:50:21.640585+0000","last_peered":"2024-03-12T03:50:21.640585+0000","last_clean":"2024-03-12T03:50:21.640585+0000","last_became_active":"2024-03-12T03:47:57.693619+0000","last_became_peered":"2024-03-12T03:47:57.693619+0000","last_unstale":"2024-03-12T03:50:21.640585+0000","last_undegraded":"2024-03-12T03:50:21.640585+0000","last_fullsized":"2024-03-12T03:50:21.640585+0000","mapping_epoch":366,"log_start":"392'7500","ondisk_log_start":"392'7500","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'9930","last_scrub_stamp":"2024-03-12T03:49:56.703697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:49:56.703697+0000","objects_scrubbed":7586,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:23.082230+0000","scrub_duration":2410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336199680,"num_objects":5130,"num_object_clones":0,"num_object_copies":15390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12270,"num_write_kb":556800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":478,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.622796+0000","last_change":"2024-03-12T03:49:04.011166+0000","last_active":"2024-03-12T03:50:20.622796+0000","last_peered":"2024-03-12T03:50:20.622796+0000","last_clean":"2024-03-12T03:50:20.622796+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:50:20.622796+0000","last_undegraded":"2024-03-12T03:50:20.622796+0000","last_fullsized":"2024-03-12T03:50:20.622796+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:04.010993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:49:04.010993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"269'23","reported_seq":786,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:20.618314+0000","last_change":"2024-03-12T03:48:58.451116+0000","last_active":"2024-03-12T03:50:20.618314+0000","last_peered":"2024-03-12T03:50:20.618314+0000","last_clean":"2024-03-12T03:48:53.448806+0000","last_became_active":"2024-03-12T03:48:58.451116+0000","last_became_peered":"2024-03-12T03:48:58.451116+0000","last_unstale":"2024-03-12T03:50:20.618314+0000","last_undegraded":"2024-03-12T03:48:58.440926+0000","last_fullsized":"2024-03-12T03:48:58.440811+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'23","last_scrub_stamp":"2024-03-12T03:45:59.998395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.998395+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":476,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.623821+0000","last_change":"2024-03-12T03:48:58.469230+0000","last_active":"2024-03-12T03:50:20.623821+0000","last_peered":"2024-03-12T03:50:20.623821+0000","last_clean":"2024-03-12T03:48:53.446422+0000","last_became_active":"2024-03-12T03:48:58.469230+0000","last_became_peered":"2024-03-12T03:48:58.469230+0000","last_unstale":"2024-03-12T03:50:20.623821+0000","last_undegraded":"2024-03-12T03:50:20.623821+0000","last_fullsized":"2024-03-12T03:48:58.446209+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:06.903506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.903506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":763,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617286+0000","last_change":"2024-03-12T03:49:05.549180+0000","last_active":"2024-03-12T03:50:20.617286+0000","last_peered":"2024-03-12T03:50:20.617286+0000","last_clean":"2024-03-12T03:50:20.617286+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:50:20.617286+0000","last_undegraded":"2024-03-12T03:50:20.617286+0000","last_fullsized":"2024-03-12T03:50:20.617286+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:05.549009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:49:05.549009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":752,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.618383+0000","last_change":"2024-03-12T03:49:18.522193+0000","last_active":"2024-03-12T03:50:20.618383+0000","last_peered":"2024-03-12T03:50:20.618383+0000","last_clean":"2024-03-12T03:50:20.618383+0000","last_became_active":"2024-03-12T03:46:25.284168+0000","last_became_peered":"2024-03-12T03:46:25.284168+0000","last_unstale":"2024-03-12T03:50:20.618383+0000","last_undegraded":"2024-03-12T03:50:20.618383+0000","last_fullsized":"2024-03-12T03:50:20.618383+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:49:18.522103+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:49:18.522103+0000","last_clean_scrub_stamp":"2024-03-12T03:49:18.522103+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:22.287849+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1035,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.617998+0000","last_change":"2024-03-12T03:48:58.455161+0000","last_active":"2024-03-12T03:50:20.617998+0000","last_peered":"2024-03-12T03:50:20.617998+0000","last_clean":"2024-03-12T03:48:53.449188+0000","last_became_active":"2024-03-12T03:48:58.455161+0000","last_became_peered":"2024-03-12T03:48:58.455161+0000","last_unstale":"2024-03-12T03:50:20.617998+0000","last_undegraded":"2024-03-12T03:50:20.617998+0000","last_fullsized":"2024-03-12T03:48:58.444402+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:01.456424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:01.456424+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"406'12086","reported_seq":12178,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:21.643464+0000","last_change":"2024-03-12T03:48:58.457912+0000","last_active":"2024-03-12T03:50:21.643464+0000","last_peered":"2024-03-12T03:50:21.643464+0000","last_clean":"2024-03-12T03:48:57.500736+0000","last_became_active":"2024-03-12T03:48:58.457912+0000","last_became_peered":"2024-03-12T03:48:58.457912+0000","last_unstale":"2024-03-12T03:50:21.643464+0000","last_undegraded":"2024-03-12T03:48:58.444591+0000","last_fullsized":"2024-03-12T03:48:58.442369+0000","mapping_epoch":384,"log_start":"391'7300","ondisk_log_start":"391'7300","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4786,"log_dups_size":0,"ondisk_log_size":4786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:26.245750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341180416,"num_objects":5206,"num_object_clones":0,"num_object_copies":15618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":553344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":5206}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":51,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.618601+0000","last_change":"2024-03-12T03:48:58.458916+0000","last_active":"2024-03-12T03:50:20.618601+0000","last_peered":"2024-03-12T03:50:20.618601+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.458916+0000","last_became_peered":"2024-03-12T03:48:58.458916+0000","last_unstale":"2024-03-12T03:50:20.618601+0000","last_undegraded":"2024-03-12T03:50:20.618601+0000","last_fullsized":"2024-03-12T03:48:58.440354+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:06.864708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.864708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"382'54","reported_seq":112,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617624+0000","last_change":"2024-03-12T03:49:15.597576+0000","last_active":"2024-03-12T03:50:20.617624+0000","last_peered":"2024-03-12T03:50:20.617624+0000","last_clean":"2024-03-12T03:50:20.617624+0000","last_became_active":"2024-03-12T03:48:48.462111+0000","last_became_peered":"2024-03-12T03:48:48.462111+0000","last_unstale":"2024-03-12T03:50:20.617624+0000","last_undegraded":"2024-03-12T03:50:20.617624+0000","last_fullsized":"2024-03-12T03:50:20.617624+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'54","last_scrub_stamp":"2024-03-12T03:49:15.597431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:15.597431+0000","objects_scrubbed":11,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:34.051010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":58,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":747,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.626107+0000","last_change":"2024-03-12T03:48:58.453277+0000","last_active":"2024-03-12T03:50:20.626107+0000","last_peered":"2024-03-12T03:50:20.626107+0000","last_clean":"2024-03-12T03:48:53.447092+0000","last_became_active":"2024-03-12T03:48:58.453277+0000","last_became_peered":"2024-03-12T03:48:58.453277+0000","last_unstale":"2024-03-12T03:50:20.626107+0000","last_undegraded":"2024-03-12T03:50:20.626107+0000","last_fullsized":"2024-03-12T03:48:58.443592+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.648824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.648824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949375+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":753,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627780+0000","last_change":"2024-03-12T03:49:08.211828+0000","last_active":"2024-03-12T03:50:20.627780+0000","last_peered":"2024-03-12T03:50:20.627780+0000","last_clean":"2024-03-12T03:50:20.627780+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:50:20.627780+0000","last_undegraded":"2024-03-12T03:50:20.627780+0000","last_fullsized":"2024-03-12T03:50:20.627780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:49:08.211712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:49:08.211712+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:35.785906+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":551,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.672071+0000","last_change":"2024-03-12T03:49:03.575443+0000","last_active":"2024-03-12T03:50:07.672071+0000","last_peered":"2024-03-12T03:50:07.672071+0000","last_clean":"2024-03-12T03:50:07.672071+0000","last_became_active":"2024-03-12T03:46:25.284416+0000","last_became_peered":"2024-03-12T03:46:25.284416+0000","last_unstale":"2024-03-12T03:50:07.672071+0000","last_undegraded":"2024-03-12T03:50:07.672071+0000","last_fullsized":"2024-03-12T03:50:07.672071+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:03.575207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:03.575207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:12.563614+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":877,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.627514+0000","last_change":"2024-03-12T03:48:58.460115+0000","last_active":"2024-03-12T03:50:20.627514+0000","last_peered":"2024-03-12T03:50:20.627514+0000","last_clean":"2024-03-12T03:48:53.471868+0000","last_became_active":"2024-03-12T03:48:58.460115+0000","last_became_peered":"2024-03-12T03:48:58.460115+0000","last_unstale":"2024-03-12T03:50:20.627514+0000","last_undegraded":"2024-03-12T03:50:20.627514+0000","last_fullsized":"2024-03-12T03:48:58.445534+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:06.690873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.690873+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"406'12249","reported_seq":12341,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:21.776587+0000","last_change":"2024-03-12T03:48:58.491732+0000","last_active":"2024-03-12T03:50:21.776587+0000","last_peered":"2024-03-12T03:50:21.776587+0000","last_clean":"2024-03-12T03:48:57.560710+0000","last_became_active":"2024-03-12T03:48:58.473224+0000","last_became_peered":"2024-03-12T03:48:58.473224+0000","last_unstale":"2024-03-12T03:50:21.776587+0000","last_undegraded":"2024-03-12T03:48:58.454399+0000","last_fullsized":"2024-03-12T03:48:58.446729+0000","mapping_epoch":384,"log_start":"386'6100","ondisk_log_start":"386'6100","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":6149,"log_dups_size":0,"ondisk_log_size":6149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346095616,"num_objects":5281,"num_object_clones":0,"num_object_copies":15843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12249,"num_write_kb":560960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":5281}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":551,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.628614+0000","last_change":"2024-03-12T03:48:11.924861+0000","last_active":"2024-03-12T03:50:20.628614+0000","last_peered":"2024-03-12T03:50:20.628614+0000","last_clean":"2024-03-12T03:50:20.628614+0000","last_became_active":"2024-03-12T03:46:26.281472+0000","last_became_peered":"2024-03-12T03:46:26.281472+0000","last_unstale":"2024-03-12T03:50:20.628614+0000","last_undegraded":"2024-03-12T03:50:20.628614+0000","last_fullsized":"2024-03-12T03:50:20.628614+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:11.924742+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.924742+0000","last_clean_scrub_stamp":"2024-03-12T03:48:11.924742+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"269'26","reported_seq":530,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:20.618499+0000","last_change":"2024-03-12T03:48:58.462328+0000","last_active":"2024-03-12T03:50:20.618499+0000","last_peered":"2024-03-12T03:50:20.618499+0000","last_clean":"2024-03-12T03:43:28.842466+0000","last_became_active":"2024-03-12T03:48:58.462328+0000","last_became_peered":"2024-03-12T03:48:58.462328+0000","last_unstale":"2024-03-12T03:50:20.618499+0000","last_undegraded":"2024-03-12T03:48:58.442501+0000","last_fullsized":"2024-03-12T03:48:58.442398+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:15.774076+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.774076+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":751,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627136+0000","last_change":"2024-03-12T03:49:45.255407+0000","last_active":"2024-03-12T03:50:20.627136+0000","last_peered":"2024-03-12T03:50:20.627136+0000","last_clean":"2024-03-12T03:50:20.627136+0000","last_became_active":"2024-03-12T03:46:25.303784+0000","last_became_peered":"2024-03-12T03:46:25.303784+0000","last_unstale":"2024-03-12T03:50:20.627136+0000","last_undegraded":"2024-03-12T03:50:20.627136+0000","last_fullsized":"2024-03-12T03:50:20.627136+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:45.255294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:49:45.255294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:51.355334+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":480,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.622736+0000","last_change":"2024-03-12T03:49:19.918452+0000","last_active":"2024-03-12T03:50:20.622736+0000","last_peered":"2024-03-12T03:50:20.622736+0000","last_clean":"2024-03-12T03:50:20.622736+0000","last_became_active":"2024-03-12T03:46:25.284556+0000","last_became_peered":"2024-03-12T03:46:25.284556+0000","last_unstale":"2024-03-12T03:50:20.622736+0000","last_undegraded":"2024-03-12T03:50:20.622736+0000","last_fullsized":"2024-03-12T03:50:20.622736+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:19.918313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:19.918313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:39.095294+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":333,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.618931+0000","last_change":"2024-03-12T03:48:58.464502+0000","last_active":"2024-03-12T03:50:20.618931+0000","last_peered":"2024-03-12T03:50:20.618931+0000","last_clean":"2024-03-12T03:39:28.535672+0000","last_became_active":"2024-03-12T03:48:58.464502+0000","last_became_peered":"2024-03-12T03:48:58.464502+0000","last_unstale":"2024-03-12T03:50:20.618931+0000","last_undegraded":"2024-03-12T03:50:20.618931+0000","last_fullsized":"2024-03-12T03:48:58.446034+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:55.192996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:55.192996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"406'12020","reported_seq":12112,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:21.252910+0000","last_change":"2024-03-12T03:48:58.465249+0000","last_active":"2024-03-12T03:50:21.252910+0000","last_peered":"2024-03-12T03:50:21.252910+0000","last_clean":"2024-03-12T03:48:57.534492+0000","last_became_active":"2024-03-12T03:48:58.465249+0000","last_became_peered":"2024-03-12T03:48:58.465249+0000","last_unstale":"2024-03-12T03:50:21.252910+0000","last_undegraded":"2024-03-12T03:48:58.444047+0000","last_fullsized":"2024-03-12T03:48:58.443929+0000","mapping_epoch":384,"log_start":"395'8009","ondisk_log_start":"395'8009","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4011,"log_dups_size":0,"ondisk_log_size":4011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330432512,"num_objects":5042,"num_object_clones":0,"num_object_copies":15126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5042,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12020,"num_write_kb":545984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":5042}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":847,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626562+0000","last_change":"2024-03-12T03:49:03.235534+0000","last_active":"2024-03-12T03:50:20.626562+0000","last_peered":"2024-03-12T03:50:20.626562+0000","last_clean":"2024-03-12T03:50:20.626562+0000","last_became_active":"2024-03-12T03:45:18.108107+0000","last_became_peered":"2024-03-12T03:45:18.108107+0000","last_unstale":"2024-03-12T03:50:20.626562+0000","last_undegraded":"2024-03-12T03:50:20.626562+0000","last_fullsized":"2024-03-12T03:50:20.626562+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:03.235270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:03.235270+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'51","reported_seq":742,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617511+0000","last_change":"2024-03-12T03:49:53.484809+0000","last_active":"2024-03-12T03:50:20.617511+0000","last_peered":"2024-03-12T03:50:20.617511+0000","last_clean":"2024-03-12T03:50:20.617511+0000","last_became_active":"2024-03-12T03:45:18.106503+0000","last_became_peered":"2024-03-12T03:45:18.106503+0000","last_unstale":"2024-03-12T03:50:20.617511+0000","last_undegraded":"2024-03-12T03:50:20.617511+0000","last_fullsized":"2024-03-12T03:50:20.617511+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'51","last_scrub_stamp":"2024-03-12T03:49:53.484646+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:49:53.484646+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:08.271186+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":751,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.618062+0000","last_change":"2024-03-12T03:50:03.440676+0000","last_active":"2024-03-12T03:50:20.618062+0000","last_peered":"2024-03-12T03:50:20.618062+0000","last_clean":"2024-03-12T03:50:20.618062+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:50:20.618062+0000","last_undegraded":"2024-03-12T03:50:20.618062+0000","last_fullsized":"2024-03-12T03:50:20.618062+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:03.440539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:03.440539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:11.407734+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":465,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.622158+0000","last_change":"2024-03-12T03:48:58.459230+0000","last_active":"2024-03-12T03:50:20.622158+0000","last_peered":"2024-03-12T03:50:20.622158+0000","last_clean":"2024-03-12T03:48:53.446740+0000","last_became_active":"2024-03-12T03:48:58.459230+0000","last_became_peered":"2024-03-12T03:48:58.459230+0000","last_unstale":"2024-03-12T03:50:20.622158+0000","last_undegraded":"2024-03-12T03:50:20.622158+0000","last_fullsized":"2024-03-12T03:48:58.447475+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.954731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.954731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":761,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.619061+0000","last_change":"2024-03-12T03:49:10.437704+0000","last_active":"2024-03-12T03:50:20.619061+0000","last_peered":"2024-03-12T03:50:20.619061+0000","last_clean":"2024-03-12T03:50:20.619061+0000","last_became_active":"2024-03-12T03:46:25.292122+0000","last_became_peered":"2024-03-12T03:46:25.292122+0000","last_unstale":"2024-03-12T03:50:20.619061+0000","last_undegraded":"2024-03-12T03:50:20.619061+0000","last_fullsized":"2024-03-12T03:50:20.619061+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:10.437601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:10.437601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"405'12003","reported_seq":12169,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:18.534632+0000","last_change":"2024-03-12T03:50:05.652788+0000","last_active":"2024-03-12T03:50:18.534632+0000","last_peered":"2024-03-12T03:50:18.534632+0000","last_clean":"2024-03-12T03:50:18.534632+0000","last_became_active":"2024-03-12T03:47:58.027372+0000","last_became_peered":"2024-03-12T03:47:58.027372+0000","last_unstale":"2024-03-12T03:50:18.534632+0000","last_undegraded":"2024-03-12T03:50:18.534632+0000","last_fullsized":"2024-03-12T03:50:18.534632+0000","mapping_epoch":366,"log_start":"393'7600","ondisk_log_start":"393'7600","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'10749","last_scrub_stamp":"2024-03-12T03:50:05.652456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:50:05.652456+0000","objects_scrubbed":6904,"log_size":4403,"log_dups_size":0,"ondisk_log_size":4403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:06.690702+0000","scrub_duration":2133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364707840,"num_objects":5565,"num_object_clones":0,"num_object_copies":16695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12003,"num_write_kb":562176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":676,"reported_epoch":405,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:07.661016+0000","last_change":"2024-03-12T03:48:58.449413+0000","last_active":"2024-03-12T03:50:07.661016+0000","last_peered":"2024-03-12T03:50:07.661016+0000","last_clean":"2024-03-12T03:48:53.447854+0000","last_became_active":"2024-03-12T03:48:58.449413+0000","last_became_peered":"2024-03-12T03:48:58.449413+0000","last_unstale":"2024-03-12T03:50:07.661016+0000","last_undegraded":"2024-03-12T03:48:58.440707+0000","last_fullsized":"2024-03-12T03:48:58.440595+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:51.818256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:51.818256+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:11.523353+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":481,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.622703+0000","last_change":"2024-03-12T03:48:58.461069+0000","last_active":"2024-03-12T03:50:20.622703+0000","last_peered":"2024-03-12T03:50:20.622703+0000","last_clean":"2024-03-12T03:48:53.445898+0000","last_became_active":"2024-03-12T03:48:58.461069+0000","last_became_peered":"2024-03-12T03:48:58.461069+0000","last_unstale":"2024-03-12T03:50:20.622703+0000","last_undegraded":"2024-03-12T03:50:20.622703+0000","last_fullsized":"2024-03-12T03:48:58.444677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.981014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.981014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"269'30","reported_seq":822,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.628111+0000","last_change":"2024-03-12T03:49:16.863752+0000","last_active":"2024-03-12T03:50:20.628111+0000","last_peered":"2024-03-12T03:50:20.628111+0000","last_clean":"2024-03-12T03:50:20.628111+0000","last_became_active":"2024-03-12T03:46:25.287885+0000","last_became_peered":"2024-03-12T03:46:25.287885+0000","last_unstale":"2024-03-12T03:50:20.628111+0000","last_undegraded":"2024-03-12T03:50:20.628111+0000","last_fullsized":"2024-03-12T03:50:20.628111+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'30","last_scrub_stamp":"2024-03-12T03:49:16.863622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:16.863622+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.298379+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":751,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617567+0000","last_change":"2024-03-12T03:49:08.492723+0000","last_active":"2024-03-12T03:50:20.617567+0000","last_peered":"2024-03-12T03:50:20.617567+0000","last_clean":"2024-03-12T03:50:20.617567+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:50:20.617567+0000","last_undegraded":"2024-03-12T03:50:20.617567+0000","last_fullsized":"2024-03-12T03:50:20.617567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:08.492606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:49:08.492606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":474,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.669826+0000","last_change":"2024-03-12T03:50:01.616545+0000","last_active":"2024-03-12T03:50:07.669826+0000","last_peered":"2024-03-12T03:50:07.669826+0000","last_clean":"2024-03-12T03:50:07.669826+0000","last_became_active":"2024-03-12T03:46:25.280784+0000","last_became_peered":"2024-03-12T03:46:25.280784+0000","last_unstale":"2024-03-12T03:50:07.669826+0000","last_undegraded":"2024-03-12T03:50:07.669826+0000","last_fullsized":"2024-03-12T03:50:07.669826+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:01.616355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:01.616355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:23.896666+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"406'12273","reported_seq":12366,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:21.625845+0000","last_change":"2024-03-12T03:48:58.457494+0000","last_active":"2024-03-12T03:50:21.625845+0000","last_peered":"2024-03-12T03:50:21.625845+0000","last_clean":"2024-03-12T03:48:57.506664+0000","last_became_active":"2024-03-12T03:48:58.457494+0000","last_became_peered":"2024-03-12T03:48:58.457494+0000","last_unstale":"2024-03-12T03:50:21.625845+0000","last_undegraded":"2024-03-12T03:48:58.442176+0000","last_fullsized":"2024-03-12T03:48:58.442057+0000","mapping_epoch":384,"log_start":"392'7500","ondisk_log_start":"392'7500","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:26.245657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343605248,"num_objects":5243,"num_object_clones":0,"num_object_copies":15729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12273,"num_write_kb":560512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5243}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":671,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617122+0000","last_change":"2024-03-12T03:50:01.383042+0000","last_active":"2024-03-12T03:50:20.617122+0000","last_peered":"2024-03-12T03:50:20.617122+0000","last_clean":"2024-03-12T03:50:20.617122+0000","last_became_active":"2024-03-12T03:44:24.373190+0000","last_became_peered":"2024-03-12T03:44:24.373190+0000","last_unstale":"2024-03-12T03:50:20.617122+0000","last_undegraded":"2024-03-12T03:50:20.617122+0000","last_fullsized":"2024-03-12T03:50:20.617122+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:01.382914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:01.382914+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:20.832601+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":948,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.619471+0000","last_change":"2024-03-12T03:48:18.441645+0000","last_active":"2024-03-12T03:50:20.619471+0000","last_peered":"2024-03-12T03:50:20.619471+0000","last_clean":"2024-03-12T03:50:20.619471+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:50:20.619471+0000","last_undegraded":"2024-03-12T03:50:20.619471+0000","last_fullsized":"2024-03-12T03:50:20.619471+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:48:18.441562+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:48:18.441562+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"269'48","reported_seq":813,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626174+0000","last_change":"2024-03-12T03:49:01.279886+0000","last_active":"2024-03-12T03:50:20.626174+0000","last_peered":"2024-03-12T03:50:20.626174+0000","last_clean":"2024-03-12T03:50:20.626174+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:50:20.626174+0000","last_undegraded":"2024-03-12T03:50:20.626174+0000","last_fullsized":"2024-03-12T03:50:20.626174+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'48","last_scrub_stamp":"2024-03-12T03:49:01.279743+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:49:01.279743+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:29.124804+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":750,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:20.617520+0000","last_change":"2024-03-12T03:48:58.456357+0000","last_active":"2024-03-12T03:50:20.617520+0000","last_peered":"2024-03-12T03:50:20.617520+0000","last_clean":"2024-03-12T03:48:53.450296+0000","last_became_active":"2024-03-12T03:48:58.456357+0000","last_became_peered":"2024-03-12T03:48:58.456357+0000","last_unstale":"2024-03-12T03:50:20.617520+0000","last_undegraded":"2024-03-12T03:48:58.445880+0000","last_fullsized":"2024-03-12T03:48:58.445769+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:47:05.588582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:05.588582+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":724,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.627595+0000","last_change":"2024-03-12T03:48:58.452652+0000","last_active":"2024-03-12T03:50:20.627595+0000","last_peered":"2024-03-12T03:50:20.627595+0000","last_clean":"2024-03-12T03:48:53.472055+0000","last_became_active":"2024-03-12T03:48:58.452652+0000","last_became_peered":"2024-03-12T03:48:58.452652+0000","last_unstale":"2024-03-12T03:50:20.627595+0000","last_undegraded":"2024-03-12T03:50:20.627595+0000","last_fullsized":"2024-03-12T03:48:58.441674+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:59.567890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.567890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":554,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626554+0000","last_change":"2024-03-12T03:50:06.385123+0000","last_active":"2024-03-12T03:50:20.626554+0000","last_peered":"2024-03-12T03:50:20.626554+0000","last_clean":"2024-03-12T03:50:20.626554+0000","last_became_active":"2024-03-12T03:44:24.365642+0000","last_became_peered":"2024-03-12T03:44:24.365642+0000","last_unstale":"2024-03-12T03:50:20.626554+0000","last_undegraded":"2024-03-12T03:50:20.626554+0000","last_fullsized":"2024-03-12T03:50:20.626554+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:06.384808+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:50:06.384808+0000","last_clean_scrub_stamp":"2024-03-12T03:50:06.384808+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:14.012985+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"269'18","reported_seq":537,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.672905+0000","last_change":"2024-03-12T03:49:16.403355+0000","last_active":"2024-03-12T03:50:07.672905+0000","last_peered":"2024-03-12T03:50:07.672905+0000","last_clean":"2024-03-12T03:50:07.672905+0000","last_became_active":"2024-03-12T03:46:25.288573+0000","last_became_peered":"2024-03-12T03:46:25.288573+0000","last_unstale":"2024-03-12T03:50:07.672905+0000","last_undegraded":"2024-03-12T03:50:07.672905+0000","last_fullsized":"2024-03-12T03:50:07.672905+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-03-12T03:49:16.403227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:16.403227+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:33.599737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":53,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.626245+0000","last_change":"2024-03-12T03:48:58.452359+0000","last_active":"2024-03-12T03:50:20.626245+0000","last_peered":"2024-03-12T03:50:20.626245+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:48:58.452359+0000","last_became_peered":"2024-03-12T03:48:58.452359+0000","last_unstale":"2024-03-12T03:50:20.626245+0000","last_undegraded":"2024-03-12T03:50:20.626245+0000","last_fullsized":"2024-03-12T03:48:58.441273+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:10.400135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.400135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":743,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626915+0000","last_change":"2024-03-12T03:49:20.206032+0000","last_active":"2024-03-12T03:50:20.626915+0000","last_peered":"2024-03-12T03:50:20.626915+0000","last_clean":"2024-03-12T03:50:20.626915+0000","last_became_active":"2024-03-12T03:46:25.293521+0000","last_became_peered":"2024-03-12T03:46:25.293521+0000","last_unstale":"2024-03-12T03:50:20.626915+0000","last_undegraded":"2024-03-12T03:50:20.626915+0000","last_fullsized":"2024-03-12T03:50:20.626915+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:20.205902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:20.205902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:26.823147+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":739,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617696+0000","last_change":"2024-03-12T03:49:58.457008+0000","last_active":"2024-03-12T03:50:20.617696+0000","last_peered":"2024-03-12T03:50:20.617696+0000","last_clean":"2024-03-12T03:50:20.617696+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:50:20.617696+0000","last_undegraded":"2024-03-12T03:50:20.617696+0000","last_fullsized":"2024-03-12T03:50:20.617696+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:58.456901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:49:58.456901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:58.456901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:13.229872+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":886,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617972+0000","last_change":"2024-03-12T03:49:59.481031+0000","last_active":"2024-03-12T03:50:20.617972+0000","last_peered":"2024-03-12T03:50:20.617972+0000","last_clean":"2024-03-12T03:50:20.617972+0000","last_became_active":"2024-03-12T03:46:28.316406+0000","last_became_peered":"2024-03-12T03:46:28.316406+0000","last_unstale":"2024-03-12T03:50:20.617972+0000","last_undegraded":"2024-03-12T03:50:20.617972+0000","last_fullsized":"2024-03-12T03:50:20.617972+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:59.480806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:59.480806+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:14.990823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"269'27","reported_seq":671,"reported_epoch":405,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:07.672369+0000","last_change":"2024-03-12T03:48:58.462615+0000","last_active":"2024-03-12T03:50:07.672369+0000","last_peered":"2024-03-12T03:50:07.672369+0000","last_clean":"2024-03-12T03:43:13.333279+0000","last_became_active":"2024-03-12T03:48:58.462615+0000","last_became_peered":"2024-03-12T03:48:58.462615+0000","last_unstale":"2024-03-12T03:50:07.672369+0000","last_undegraded":"2024-03-12T03:48:58.443892+0000","last_fullsized":"2024-03-12T03:48:58.443761+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:48:06.104297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.104297+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:12.563536+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":743,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.619292+0000","last_change":"2024-03-12T03:48:04.420937+0000","last_active":"2024-03-12T03:50:20.619292+0000","last_peered":"2024-03-12T03:50:20.619292+0000","last_clean":"2024-03-12T03:50:20.619292+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:50:20.619292+0000","last_undegraded":"2024-03-12T03:50:20.619292+0000","last_fullsized":"2024-03-12T03:50:20.619292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:04.420853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:04.420853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":500,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.673014+0000","last_change":"2024-03-12T03:49:17.403558+0000","last_active":"2024-03-12T03:50:07.673014+0000","last_peered":"2024-03-12T03:50:07.673014+0000","last_clean":"2024-03-12T03:50:07.673014+0000","last_became_active":"2024-03-12T03:46:25.286866+0000","last_became_peered":"2024-03-12T03:46:25.286866+0000","last_unstale":"2024-03-12T03:50:07.673014+0000","last_undegraded":"2024-03-12T03:50:07.673014+0000","last_fullsized":"2024-03-12T03:50:07.673014+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:17.403434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:49:17.403434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:22.588623+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":470,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.622525+0000","last_change":"2024-03-12T03:49:02.083280+0000","last_active":"2024-03-12T03:50:20.622525+0000","last_peered":"2024-03-12T03:50:20.622525+0000","last_clean":"2024-03-12T03:50:20.622525+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:50:20.622525+0000","last_undegraded":"2024-03-12T03:50:20.622525+0000","last_fullsized":"2024-03-12T03:50:20.622525+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:02.083177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:02.083177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":546,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626070+0000","last_change":"2024-03-12T03:47:58.612030+0000","last_active":"2024-03-12T03:50:20.626070+0000","last_peered":"2024-03-12T03:50:20.626070+0000","last_clean":"2024-03-12T03:50:20.626070+0000","last_became_active":"2024-03-12T03:44:24.376141+0000","last_became_peered":"2024-03-12T03:44:24.376141+0000","last_unstale":"2024-03-12T03:50:20.626070+0000","last_undegraded":"2024-03-12T03:50:20.626070+0000","last_fullsized":"2024-03-12T03:50:20.626070+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:58.611912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.611912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"269'21","reported_seq":801,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627119+0000","last_change":"2024-03-12T03:49:19.221180+0000","last_active":"2024-03-12T03:50:20.627119+0000","last_peered":"2024-03-12T03:50:20.627119+0000","last_clean":"2024-03-12T03:50:20.627119+0000","last_became_active":"2024-03-12T03:46:25.298007+0000","last_became_peered":"2024-03-12T03:46:25.298007+0000","last_unstale":"2024-03-12T03:50:20.627119+0000","last_undegraded":"2024-03-12T03:50:20.627119+0000","last_fullsized":"2024-03-12T03:50:20.627119+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'21","last_scrub_stamp":"2024-03-12T03:49:19.221045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:19.221045+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:42.537709+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":627,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.617663+0000","last_change":"2024-03-12T03:48:58.462149+0000","last_active":"2024-03-12T03:50:20.617663+0000","last_peered":"2024-03-12T03:50:20.617663+0000","last_clean":"2024-03-12T03:43:13.331412+0000","last_became_active":"2024-03-12T03:48:58.462149+0000","last_became_peered":"2024-03-12T03:48:58.462149+0000","last_unstale":"2024-03-12T03:50:20.617663+0000","last_undegraded":"2024-03-12T03:50:20.617663+0000","last_fullsized":"2024-03-12T03:48:58.443909+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"332'14","reported_seq":734,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:11.829206+0000","last_change":"2024-03-12T03:49:10.474096+0000","last_active":"2024-03-12T03:50:11.829206+0000","last_peered":"2024-03-12T03:50:11.829206+0000","last_clean":"2024-03-12T03:50:11.829206+0000","last_became_active":"2024-03-12T03:46:25.300463+0000","last_became_peered":"2024-03-12T03:46:25.300463+0000","last_unstale":"2024-03-12T03:50:11.829206+0000","last_undegraded":"2024-03-12T03:50:11.829206+0000","last_fullsized":"2024-03-12T03:50:11.829206+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'14","last_scrub_stamp":"2024-03-12T03:49:10.473983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:10.473983+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.746305+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":751,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.629150+0000","last_change":"2024-03-12T03:50:04.042481+0000","last_active":"2024-03-12T03:50:20.629150+0000","last_peered":"2024-03-12T03:50:20.629150+0000","last_clean":"2024-03-12T03:50:20.629150+0000","last_became_active":"2024-03-12T03:39:33.265499+0000","last_became_peered":"2024-03-12T03:39:33.265499+0000","last_unstale":"2024-03-12T03:50:20.629150+0000","last_undegraded":"2024-03-12T03:50:20.629150+0000","last_fullsized":"2024-03-12T03:50:20.629150+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:04.042226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:04.042226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:08.428366+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":676,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.627080+0000","last_change":"2024-03-12T03:48:58.458244+0000","last_active":"2024-03-12T03:50:20.627080+0000","last_peered":"2024-03-12T03:50:20.627080+0000","last_clean":"2024-03-12T03:44:19.377035+0000","last_became_active":"2024-03-12T03:48:58.458244+0000","last_became_peered":"2024-03-12T03:48:58.458244+0000","last_unstale":"2024-03-12T03:50:20.627080+0000","last_undegraded":"2024-03-12T03:50:20.627080+0000","last_fullsized":"2024-03-12T03:48:58.445185+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:16.223801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:16.223801+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:21.218239+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"403'39","reported_seq":607,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.670055+0000","last_change":"2024-03-12T03:50:06.535902+0000","last_active":"2024-03-12T03:50:07.670055+0000","last_peered":"2024-03-12T03:50:07.670055+0000","last_clean":"2024-03-12T03:50:07.670055+0000","last_became_active":"2024-03-12T03:46:25.299798+0000","last_became_peered":"2024-03-12T03:46:25.299798+0000","last_unstale":"2024-03-12T03:50:07.670055+0000","last_undegraded":"2024-03-12T03:50:07.670055+0000","last_fullsized":"2024-03-12T03:50:07.670055+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'39","last_scrub_stamp":"2024-03-12T03:50:06.535772+0000","last_deep_scrub":"403'39","last_deep_scrub_stamp":"2024-03-12T03:50:06.535772+0000","last_clean_scrub_stamp":"2024-03-12T03:50:06.535772+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:13.213921+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":778,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617290+0000","last_change":"2024-03-12T03:50:00.385862+0000","last_active":"2024-03-12T03:50:20.617290+0000","last_peered":"2024-03-12T03:50:20.617290+0000","last_clean":"2024-03-12T03:50:20.617290+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:50:20.617290+0000","last_undegraded":"2024-03-12T03:50:20.617290+0000","last_fullsized":"2024-03-12T03:50:20.617290+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:50:00.385750+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:50:00.385750+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:11.005858+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"385'9","reported_seq":941,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:20.626864+0000","last_change":"2024-03-12T03:48:58.448464+0000","last_active":"2024-03-12T03:50:20.626864+0000","last_peered":"2024-03-12T03:50:20.626864+0000","last_clean":"2024-03-12T03:48:56.813639+0000","last_became_active":"2024-03-12T03:48:58.448464+0000","last_became_peered":"2024-03-12T03:48:58.448464+0000","last_unstale":"2024-03-12T03:50:20.626864+0000","last_undegraded":"2024-03-12T03:48:58.442644+0000","last_fullsized":"2024-03-12T03:48:58.442315+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:53.458170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.458170+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:26.245374+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":615,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.617289+0000","last_change":"2024-03-12T03:48:58.463189+0000","last_active":"2024-03-12T03:50:20.617289+0000","last_peered":"2024-03-12T03:50:20.617289+0000","last_clean":"2024-03-12T03:43:13.332853+0000","last_became_active":"2024-03-12T03:48:58.463189+0000","last_became_peered":"2024-03-12T03:48:58.463189+0000","last_unstale":"2024-03-12T03:50:20.617289+0000","last_undegraded":"2024-03-12T03:50:20.617289+0000","last_fullsized":"2024-03-12T03:48:58.445060+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.079569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.079569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"269'19","reported_seq":778,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626681+0000","last_change":"2024-03-12T03:49:08.759600+0000","last_active":"2024-03-12T03:50:20.626681+0000","last_peered":"2024-03-12T03:50:20.626681+0000","last_clean":"2024-03-12T03:50:20.626681+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:50:20.626681+0000","last_undegraded":"2024-03-12T03:50:20.626681+0000","last_fullsized":"2024-03-12T03:50:20.626681+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'19","last_scrub_stamp":"2024-03-12T03:49:08.759497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:08.759497+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949868+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":751,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627192+0000","last_change":"2024-03-12T03:49:50.830712+0000","last_active":"2024-03-12T03:50:20.627192+0000","last_peered":"2024-03-12T03:50:20.627192+0000","last_clean":"2024-03-12T03:50:20.627192+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:50:20.627192+0000","last_undegraded":"2024-03-12T03:50:20.627192+0000","last_fullsized":"2024-03-12T03:50:20.627192+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:50.830611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.830611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:52.147471+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":752,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617877+0000","last_change":"2024-03-12T03:49:08.586759+0000","last_active":"2024-03-12T03:50:20.617877+0000","last_peered":"2024-03-12T03:50:20.617877+0000","last_clean":"2024-03-12T03:50:20.617877+0000","last_became_active":"2024-03-12T03:46:25.288050+0000","last_became_peered":"2024-03-12T03:46:25.288050+0000","last_unstale":"2024-03-12T03:50:20.617877+0000","last_undegraded":"2024-03-12T03:50:20.617877+0000","last_fullsized":"2024-03-12T03:50:20.617877+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:08.586525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:08.586525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:20.900165+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":519,"reported_epoch":405,"state":"active+undersized","last_fresh":"2024-03-12T03:50:07.669149+0000","last_change":"2024-03-12T03:48:58.456788+0000","last_active":"2024-03-12T03:50:07.669149+0000","last_peered":"2024-03-12T03:50:07.669149+0000","last_clean":"2024-03-12T03:48:53.447742+0000","last_became_active":"2024-03-12T03:48:58.456788+0000","last_became_peered":"2024-03-12T03:48:58.456788+0000","last_unstale":"2024-03-12T03:50:07.669149+0000","last_undegraded":"2024-03-12T03:50:07.669149+0000","last_fullsized":"2024-03-12T03:48:58.442213+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.433184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.433184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:11.523403+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"269'15","reported_seq":342,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627363+0000","last_change":"2024-03-12T03:49:56.329520+0000","last_active":"2024-03-12T03:50:20.627363+0000","last_peered":"2024-03-12T03:50:20.627363+0000","last_clean":"2024-03-12T03:50:20.627363+0000","last_became_active":"2024-03-12T03:43:01.037348+0000","last_became_peered":"2024-03-12T03:43:01.037348+0000","last_unstale":"2024-03-12T03:50:20.627363+0000","last_undegraded":"2024-03-12T03:50:20.627363+0000","last_fullsized":"2024-03-12T03:50:20.627363+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'15","last_scrub_stamp":"2024-03-12T03:49:56.329393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:56.329393+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:21.835051+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":761,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617742+0000","last_change":"2024-03-12T03:49:12.446186+0000","last_active":"2024-03-12T03:50:20.617742+0000","last_peered":"2024-03-12T03:50:20.617742+0000","last_clean":"2024-03-12T03:50:20.617742+0000","last_became_active":"2024-03-12T03:46:25.283602+0000","last_became_peered":"2024-03-12T03:46:25.283602+0000","last_unstale":"2024-03-12T03:50:20.617742+0000","last_undegraded":"2024-03-12T03:50:20.617742+0000","last_fullsized":"2024-03-12T03:50:20.617742+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:12.446075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:49:12.446075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":531,"reported_epoch":405,"state":"active+undersized","last_fresh":"2024-03-12T03:50:07.669479+0000","last_change":"2024-03-12T03:48:58.460970+0000","last_active":"2024-03-12T03:50:07.669479+0000","last_peered":"2024-03-12T03:50:07.669479+0000","last_clean":"2024-03-12T03:48:53.448497+0000","last_became_active":"2024-03-12T03:48:58.460970+0000","last_became_peered":"2024-03-12T03:48:58.460970+0000","last_unstale":"2024-03-12T03:50:07.669479+0000","last_undegraded":"2024-03-12T03:50:07.669479+0000","last_fullsized":"2024-03-12T03:48:58.444135+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_clean_scrub_stamp":"2024-03-12T03:48:10.625582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:12.563585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":51,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.626270+0000","last_change":"2024-03-12T03:48:58.453783+0000","last_active":"2024-03-12T03:50:20.626270+0000","last_peered":"2024-03-12T03:50:20.626270+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.453783+0000","last_became_peered":"2024-03-12T03:48:58.453783+0000","last_unstale":"2024-03-12T03:50:20.626270+0000","last_undegraded":"2024-03-12T03:50:20.626270+0000","last_fullsized":"2024-03-12T03:48:58.441749+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:13.228117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:48:13.228117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":668,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.621985+0000","last_change":"2024-03-12T03:49:09.041208+0000","last_active":"2024-03-12T03:50:20.621985+0000","last_peered":"2024-03-12T03:50:20.621985+0000","last_clean":"2024-03-12T03:50:20.621985+0000","last_became_active":"2024-03-12T03:44:24.387247+0000","last_became_peered":"2024-03-12T03:44:24.387247+0000","last_unstale":"2024-03-12T03:50:20.621985+0000","last_undegraded":"2024-03-12T03:50:20.621985+0000","last_fullsized":"2024-03-12T03:50:20.621985+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:09.041066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:09.041066+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"269'26","reported_seq":547,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.669476+0000","last_change":"2024-03-12T03:49:47.515912+0000","last_active":"2024-03-12T03:50:07.669476+0000","last_peered":"2024-03-12T03:50:07.669476+0000","last_clean":"2024-03-12T03:50:07.669476+0000","last_became_active":"2024-03-12T03:46:25.287095+0000","last_became_peered":"2024-03-12T03:46:25.287095+0000","last_unstale":"2024-03-12T03:50:07.669476+0000","last_undegraded":"2024-03-12T03:50:07.669476+0000","last_fullsized":"2024-03-12T03:50:07.669476+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:49:47.515799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:47.515799+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:59.734985+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":749,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626970+0000","last_change":"2024-03-12T03:50:00.379727+0000","last_active":"2024-03-12T03:50:20.626970+0000","last_peered":"2024-03-12T03:50:20.626970+0000","last_clean":"2024-03-12T03:50:20.626970+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:50:20.626970+0000","last_undegraded":"2024-03-12T03:50:20.626970+0000","last_fullsized":"2024-03-12T03:50:20.626970+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:00.379623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:50:00.379623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:07.116347+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":735,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.619756+0000","last_change":"2024-03-12T03:48:09.377483+0000","last_active":"2024-03-12T03:50:20.619756+0000","last_peered":"2024-03-12T03:50:20.619756+0000","last_clean":"2024-03-12T03:50:20.619756+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:50:20.619756+0000","last_undegraded":"2024-03-12T03:50:20.619756+0000","last_fullsized":"2024-03-12T03:50:20.619756+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:09.377400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:09.377400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":546,"reported_epoch":405,"state":"active+undersized","last_fresh":"2024-03-12T03:50:07.669863+0000","last_change":"2024-03-12T03:48:58.449798+0000","last_active":"2024-03-12T03:50:07.669863+0000","last_peered":"2024-03-12T03:50:07.669863+0000","last_clean":"2024-03-12T03:48:53.448767+0000","last_became_active":"2024-03-12T03:48:58.449798+0000","last_became_peered":"2024-03-12T03:48:58.449798+0000","last_unstale":"2024-03-12T03:50:07.669863+0000","last_undegraded":"2024-03-12T03:50:07.669863+0000","last_fullsized":"2024-03-12T03:48:58.443857+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:11.523322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"269'10","reported_seq":518,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.660684+0000","last_change":"2024-03-12T03:48:06.646761+0000","last_active":"2024-03-12T03:50:07.660684+0000","last_peered":"2024-03-12T03:50:07.660684+0000","last_clean":"2024-03-12T03:50:07.660684+0000","last_became_active":"2024-03-12T03:46:25.286315+0000","last_became_peered":"2024-03-12T03:46:25.286315+0000","last_unstale":"2024-03-12T03:50:07.660684+0000","last_undegraded":"2024-03-12T03:50:07.660684+0000","last_fullsized":"2024-03-12T03:50:07.660684+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'10","last_scrub_stamp":"2024-03-12T03:48:06.646668+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.646668+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:12.563570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":627,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.627026+0000","last_change":"2024-03-12T03:48:58.463096+0000","last_active":"2024-03-12T03:50:20.627026+0000","last_peered":"2024-03-12T03:50:20.627026+0000","last_clean":"2024-03-12T03:43:13.332505+0000","last_became_active":"2024-03-12T03:48:58.463096+0000","last_became_peered":"2024-03-12T03:48:58.463096+0000","last_unstale":"2024-03-12T03:50:20.627026+0000","last_undegraded":"2024-03-12T03:50:20.627026+0000","last_fullsized":"2024-03-12T03:48:58.443705+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.851271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.851271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":206,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626655+0000","last_change":"2024-03-12T03:49:00.707749+0000","last_active":"2024-03-12T03:50:20.626655+0000","last_peered":"2024-03-12T03:50:20.626655+0000","last_clean":"2024-03-12T03:50:20.626655+0000","last_became_active":"2024-03-12T03:45:18.109983+0000","last_became_peered":"2024-03-12T03:45:18.109983+0000","last_unstale":"2024-03-12T03:50:20.626655+0000","last_undegraded":"2024-03-12T03:50:20.626655+0000","last_fullsized":"2024-03-12T03:50:20.626655+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:00.707518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:00.707518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949839+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":743,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.618198+0000","last_change":"2024-03-12T03:49:04.449644+0000","last_active":"2024-03-12T03:50:20.618198+0000","last_peered":"2024-03-12T03:50:20.618198+0000","last_clean":"2024-03-12T03:50:20.618198+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:50:20.618198+0000","last_undegraded":"2024-03-12T03:50:20.618198+0000","last_fullsized":"2024-03-12T03:50:20.618198+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:04.449485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:04.449485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"269'31","reported_seq":714,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627174+0000","last_change":"2024-03-12T03:49:53.291968+0000","last_active":"2024-03-12T03:50:20.627174+0000","last_peered":"2024-03-12T03:50:20.627174+0000","last_clean":"2024-03-12T03:50:20.627174+0000","last_became_active":"2024-03-12T03:45:18.107047+0000","last_became_peered":"2024-03-12T03:45:18.107047+0000","last_unstale":"2024-03-12T03:50:20.627174+0000","last_undegraded":"2024-03-12T03:50:20.627174+0000","last_fullsized":"2024-03-12T03:50:20.627174+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:49:53.291824+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:49:53.291824+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:13.883282+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":547,"reported_epoch":405,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:07.660757+0000","last_change":"2024-03-12T03:48:58.454467+0000","last_active":"2024-03-12T03:50:07.660757+0000","last_peered":"2024-03-12T03:50:07.660757+0000","last_clean":"2024-03-12T03:48:53.477110+0000","last_became_active":"2024-03-12T03:48:58.454467+0000","last_became_peered":"2024-03-12T03:48:58.454467+0000","last_unstale":"2024-03-12T03:50:07.660757+0000","last_undegraded":"2024-03-12T03:48:58.439427+0000","last_fullsized":"2024-03-12T03:48:58.439320+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:48:00.458460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.458460+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:12.563521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":338,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.617650+0000","last_change":"2024-03-12T03:48:58.461994+0000","last_active":"2024-03-12T03:50:20.617650+0000","last_peered":"2024-03-12T03:50:20.617650+0000","last_clean":"2024-03-12T03:39:31.213817+0000","last_became_active":"2024-03-12T03:48:58.461994+0000","last_became_peered":"2024-03-12T03:48:58.461994+0000","last_unstale":"2024-03-12T03:50:20.617650+0000","last_undegraded":"2024-03-12T03:50:20.617650+0000","last_fullsized":"2024-03-12T03:48:58.446456+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:09.160277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:09.160277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":737,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.618804+0000","last_change":"2024-03-12T03:49:07.757495+0000","last_active":"2024-03-12T03:50:20.618804+0000","last_peered":"2024-03-12T03:50:20.618804+0000","last_clean":"2024-03-12T03:50:20.618804+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:50:20.618804+0000","last_undegraded":"2024-03-12T03:50:20.618804+0000","last_fullsized":"2024-03-12T03:50:20.618804+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:07.757361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:07.757361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:20.907704+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"269'27","reported_seq":544,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:20.617892+0000","last_change":"2024-03-12T03:48:58.462533+0000","last_active":"2024-03-12T03:50:20.617892+0000","last_peered":"2024-03-12T03:50:20.617892+0000","last_clean":"2024-03-12T03:43:13.333148+0000","last_became_active":"2024-03-12T03:48:58.462533+0000","last_became_peered":"2024-03-12T03:48:58.462533+0000","last_unstale":"2024-03-12T03:50:20.617892+0000","last_undegraded":"2024-03-12T03:48:58.446989+0000","last_fullsized":"2024-03-12T03:48:58.446871+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:15.795515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.795515+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"351'2","reported_seq":762,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617296+0000","last_change":"2024-03-12T03:49:14.611362+0000","last_active":"2024-03-12T03:50:20.617296+0000","last_peered":"2024-03-12T03:50:20.617296+0000","last_clean":"2024-03-12T03:50:20.617296+0000","last_became_active":"2024-03-12T03:46:25.283344+0000","last_became_peered":"2024-03-12T03:46:25.283344+0000","last_unstale":"2024-03-12T03:50:20.617296+0000","last_undegraded":"2024-03-12T03:50:20.617296+0000","last_fullsized":"2024-03-12T03:50:20.617296+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'2","last_scrub_stamp":"2024-03-12T03:49:14.611213+0000","last_deep_scrub":"351'2","last_deep_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_clean_scrub_stamp":"2024-03-12T03:49:14.611213+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.636078+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'9","reported_seq":294,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.629222+0000","last_change":"2024-03-12T03:48:02.854911+0000","last_active":"2024-03-12T03:50:20.629222+0000","last_peered":"2024-03-12T03:50:20.629222+0000","last_clean":"2024-03-12T03:50:20.629222+0000","last_became_active":"2024-03-12T03:45:18.108935+0000","last_became_peered":"2024-03-12T03:45:18.108935+0000","last_unstale":"2024-03-12T03:50:20.629222+0000","last_undegraded":"2024-03-12T03:50:20.629222+0000","last_fullsized":"2024-03-12T03:50:20.629222+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'9","last_scrub_stamp":"2024-03-12T03:48:02.854820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:02.854820+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":759,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.618545+0000","last_change":"2024-03-12T03:49:48.376454+0000","last_active":"2024-03-12T03:50:20.618545+0000","last_peered":"2024-03-12T03:50:20.618545+0000","last_clean":"2024-03-12T03:50:20.618545+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:50:20.618545+0000","last_undegraded":"2024-03-12T03:50:20.618545+0000","last_fullsized":"2024-03-12T03:50:20.618545+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:49:48.375999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:48.375999+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:49.566799+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"269'24","reported_seq":584,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.670145+0000","last_change":"2024-03-12T03:50:05.585005+0000","last_active":"2024-03-12T03:50:07.670145+0000","last_peered":"2024-03-12T03:50:07.670145+0000","last_clean":"2024-03-12T03:50:07.670145+0000","last_became_active":"2024-03-12T03:46:25.283193+0000","last_became_peered":"2024-03-12T03:46:25.283193+0000","last_unstale":"2024-03-12T03:50:07.670145+0000","last_undegraded":"2024-03-12T03:50:07.670145+0000","last_fullsized":"2024-03-12T03:50:07.670145+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:50:05.584740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:05.584740+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.476261+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":762,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.629407+0000","last_change":"2024-03-12T03:48:01.902683+0000","last_active":"2024-03-12T03:50:20.629407+0000","last_peered":"2024-03-12T03:50:20.629407+0000","last_clean":"2024-03-12T03:50:20.629407+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:50:20.629407+0000","last_undegraded":"2024-03-12T03:50:20.629407+0000","last_fullsized":"2024-03-12T03:50:20.629407+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:01.902570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:01.902570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":764,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627449+0000","last_change":"2024-03-12T03:49:19.894833+0000","last_active":"2024-03-12T03:50:20.627449+0000","last_peered":"2024-03-12T03:50:20.627449+0000","last_clean":"2024-03-12T03:50:20.627449+0000","last_became_active":"2024-03-12T03:46:25.301691+0000","last_became_peered":"2024-03-12T03:46:25.301691+0000","last_unstale":"2024-03-12T03:50:20.627449+0000","last_undegraded":"2024-03-12T03:50:20.627449+0000","last_fullsized":"2024-03-12T03:50:20.627449+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:19.894587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:19.894587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:25.339221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":553,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.673191+0000","last_change":"2024-03-12T03:49:54.607508+0000","last_active":"2024-03-12T03:50:07.673191+0000","last_peered":"2024-03-12T03:50:07.673191+0000","last_clean":"2024-03-12T03:50:07.673191+0000","last_became_active":"2024-03-12T03:46:25.296764+0000","last_became_peered":"2024-03-12T03:46:25.296764+0000","last_unstale":"2024-03-12T03:50:07.673191+0000","last_undegraded":"2024-03-12T03:50:07.673191+0000","last_fullsized":"2024-03-12T03:50:07.673191+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:54.607394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.607394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:55.661971+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"269'27","reported_seq":529,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.622894+0000","last_change":"2024-03-12T03:49:57.717659+0000","last_active":"2024-03-12T03:50:20.622894+0000","last_peered":"2024-03-12T03:50:20.622894+0000","last_clean":"2024-03-12T03:50:20.622894+0000","last_became_active":"2024-03-12T03:46:25.301700+0000","last_became_peered":"2024-03-12T03:46:25.301700+0000","last_unstale":"2024-03-12T03:50:20.622894+0000","last_undegraded":"2024-03-12T03:50:20.622894+0000","last_fullsized":"2024-03-12T03:50:20.622894+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:49:57.717507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:57.717507+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:00.235756+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":51,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.626328+0000","last_change":"2024-03-12T03:48:58.455613+0000","last_active":"2024-03-12T03:50:20.626328+0000","last_peered":"2024-03-12T03:50:20.626328+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:48:58.455613+0000","last_became_peered":"2024-03-12T03:48:58.455613+0000","last_unstale":"2024-03-12T03:50:20.626328+0000","last_undegraded":"2024-03-12T03:50:20.626328+0000","last_fullsized":"2024-03-12T03:48:58.445944+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.797296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.797296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":735,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.618124+0000","last_change":"2024-03-12T03:48:58.458438+0000","last_active":"2024-03-12T03:50:20.618124+0000","last_peered":"2024-03-12T03:50:20.618124+0000","last_clean":"2024-03-12T03:48:53.448127+0000","last_became_active":"2024-03-12T03:48:58.458438+0000","last_became_peered":"2024-03-12T03:48:58.458438+0000","last_unstale":"2024-03-12T03:50:20.618124+0000","last_undegraded":"2024-03-12T03:50:20.618124+0000","last_fullsized":"2024-03-12T03:48:58.440067+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.016030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.016030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":732,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.628749+0000","last_change":"2024-03-12T03:48:00.918264+0000","last_active":"2024-03-12T03:50:20.628749+0000","last_peered":"2024-03-12T03:50:20.628749+0000","last_clean":"2024-03-12T03:50:20.628749+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:50:20.628749+0000","last_undegraded":"2024-03-12T03:50:20.628749+0000","last_fullsized":"2024-03-12T03:50:20.628749+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:00.918131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.918131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949718+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"269'16","reported_seq":825,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626512+0000","last_change":"2024-03-12T03:49:51.806758+0000","last_active":"2024-03-12T03:50:20.626512+0000","last_peered":"2024-03-12T03:50:20.626512+0000","last_clean":"2024-03-12T03:50:20.626512+0000","last_became_active":"2024-03-12T03:46:25.283025+0000","last_became_peered":"2024-03-12T03:46:25.283025+0000","last_unstale":"2024-03-12T03:50:20.626512+0000","last_undegraded":"2024-03-12T03:50:20.626512+0000","last_fullsized":"2024-03-12T03:50:20.626512+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'16","last_scrub_stamp":"2024-03-12T03:49:51.806651+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:49:51.806651+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:56.931992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":740,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627341+0000","last_change":"2024-03-12T03:50:02.319955+0000","last_active":"2024-03-12T03:50:20.627341+0000","last_peered":"2024-03-12T03:50:20.627341+0000","last_clean":"2024-03-12T03:50:20.627341+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:50:20.627341+0000","last_undegraded":"2024-03-12T03:50:20.627341+0000","last_fullsized":"2024-03-12T03:50:20.627341+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:02.319837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:02.319837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:25.368545+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":734,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617911+0000","last_change":"2024-03-12T03:49:02.468991+0000","last_active":"2024-03-12T03:50:20.617911+0000","last_peered":"2024-03-12T03:50:20.617911+0000","last_clean":"2024-03-12T03:50:20.617911+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:50:20.617911+0000","last_undegraded":"2024-03-12T03:50:20.617911+0000","last_fullsized":"2024-03-12T03:50:20.617911+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:02.468865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:49:02.468865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:31.413640+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":738,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.627750+0000","last_change":"2024-03-12T03:49:46.378515+0000","last_active":"2024-03-12T03:50:20.627750+0000","last_peered":"2024-03-12T03:50:20.627750+0000","last_clean":"2024-03-12T03:50:20.627750+0000","last_became_active":"2024-03-12T03:46:25.293777+0000","last_became_peered":"2024-03-12T03:46:25.293777+0000","last_unstale":"2024-03-12T03:50:20.627750+0000","last_undegraded":"2024-03-12T03:50:20.627750+0000","last_fullsized":"2024-03-12T03:50:20.627750+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:46.378413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:46.378413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:12.689217+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"345'33","reported_seq":815,"reported_epoch":406,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:20.629181+0000","last_change":"2024-03-12T03:48:58.457301+0000","last_active":"2024-03-12T03:50:20.629181+0000","last_peered":"2024-03-12T03:50:20.629181+0000","last_clean":"2024-03-12T03:48:53.446883+0000","last_became_active":"2024-03-12T03:48:58.457301+0000","last_became_peered":"2024-03-12T03:48:58.457301+0000","last_unstale":"2024-03-12T03:50:20.629181+0000","last_undegraded":"2024-03-12T03:48:58.444868+0000","last_fullsized":"2024-03-12T03:48:58.444750+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'33","last_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_deep_scrub":"345'33","last_deep_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_clean_scrub_stamp":"2024-03-12T03:47:02.110517+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949389+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":740,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626651+0000","last_change":"2024-03-12T03:49:04.233108+0000","last_active":"2024-03-12T03:50:20.626651+0000","last_peered":"2024-03-12T03:50:20.626651+0000","last_clean":"2024-03-12T03:50:20.626651+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:50:20.626651+0000","last_undegraded":"2024-03-12T03:50:20.626651+0000","last_fullsized":"2024-03-12T03:50:20.626651+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:04.232977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:49:04.232977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":727,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.628501+0000","last_change":"2024-03-12T03:48:58.453634+0000","last_active":"2024-03-12T03:50:20.628501+0000","last_peered":"2024-03-12T03:50:20.628501+0000","last_clean":"2024-03-12T03:48:53.472261+0000","last_became_active":"2024-03-12T03:48:58.453634+0000","last_became_peered":"2024-03-12T03:48:58.453634+0000","last_unstale":"2024-03-12T03:50:20.628501+0000","last_undegraded":"2024-03-12T03:50:20.628501+0000","last_fullsized":"2024-03-12T03:48:58.444137+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:02.618599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_clean_scrub_stamp":"2024-03-12T03:46:02.618599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.616850+0000","last_change":"2024-03-12T03:48:58.463158+0000","last_active":"2024-03-12T03:50:20.616850+0000","last_peered":"2024-03-12T03:50:20.616850+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.463158+0000","last_became_peered":"2024-03-12T03:48:58.463158+0000","last_unstale":"2024-03-12T03:50:20.616850+0000","last_undegraded":"2024-03-12T03:50:20.616850+0000","last_fullsized":"2024-03-12T03:48:58.439250+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_clean_scrub_stamp":"2024-03-12T03:45:58.095196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"269'62","reported_seq":698,"reported_epoch":405,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:07.660866+0000","last_change":"2024-03-12T03:48:58.457134+0000","last_active":"2024-03-12T03:50:07.660866+0000","last_peered":"2024-03-12T03:50:07.660866+0000","last_clean":"2024-03-12T03:48:53.477142+0000","last_became_active":"2024-03-12T03:48:58.457134+0000","last_became_peered":"2024-03-12T03:48:58.457134+0000","last_unstale":"2024-03-12T03:50:07.660866+0000","last_undegraded":"2024-03-12T03:48:58.441299+0000","last_fullsized":"2024-03-12T03:48:58.441188+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:11.523339+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":570,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.672283+0000","last_change":"2024-03-12T03:48:12.573604+0000","last_active":"2024-03-12T03:50:07.672283+0000","last_peered":"2024-03-12T03:50:07.672283+0000","last_clean":"2024-03-12T03:50:07.672283+0000","last_became_active":"2024-03-12T03:46:25.279131+0000","last_became_peered":"2024-03-12T03:46:25.279131+0000","last_unstale":"2024-03-12T03:50:07.672283+0000","last_undegraded":"2024-03-12T03:50:07.672283+0000","last_fullsized":"2024-03-12T03:50:07.672283+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:12.573481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:48:12.573481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:12.563502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":734,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626194+0000","last_change":"2024-03-12T03:47:55.924374+0000","last_active":"2024-03-12T03:50:20.626194+0000","last_peered":"2024-03-12T03:50:20.626194+0000","last_clean":"2024-03-12T03:50:20.626194+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:50:20.626194+0000","last_undegraded":"2024-03-12T03:50:20.626194+0000","last_fullsized":"2024-03-12T03:50:20.626194+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:55.924292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:47:55.924292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":529,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-12T03:50:07.669729+0000","last_change":"2024-03-12T03:49:55.557456+0000","last_active":"2024-03-12T03:50:07.669729+0000","last_peered":"2024-03-12T03:50:07.669729+0000","last_clean":"2024-03-12T03:50:07.669729+0000","last_became_active":"2024-03-12T03:46:25.281081+0000","last_became_peered":"2024-03-12T03:46:25.281081+0000","last_unstale":"2024-03-12T03:50:07.669729+0000","last_undegraded":"2024-03-12T03:50:07.669729+0000","last_fullsized":"2024-03-12T03:50:07.669729+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:55.557335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:55.557335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.968328+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"269'26","reported_seq":780,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.626096+0000","last_change":"2024-03-12T03:49:01.728877+0000","last_active":"2024-03-12T03:50:20.626096+0000","last_peered":"2024-03-12T03:50:20.626096+0000","last_clean":"2024-03-12T03:50:20.626096+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:50:20.626096+0000","last_undegraded":"2024-03-12T03:50:20.626096+0000","last_fullsized":"2024-03-12T03:50:20.626096+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:49:01.728616+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:49:01.728616+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:23.949814+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":345,"reported_epoch":406,"state":"active+undersized","last_fresh":"2024-03-12T03:50:20.626853+0000","last_change":"2024-03-12T03:48:58.462711+0000","last_active":"2024-03-12T03:50:20.626853+0000","last_peered":"2024-03-12T03:50:20.626853+0000","last_clean":"2024-03-12T03:39:28.536135+0000","last_became_active":"2024-03-12T03:48:58.462711+0000","last_became_peered":"2024-03-12T03:48:58.462711+0000","last_unstale":"2024-03-12T03:50:20.626853+0000","last_undegraded":"2024-03-12T03:50:20.626853+0000","last_fullsized":"2024-03-12T03:48:58.443314+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:12.186146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:48:12.186146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":749,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.617349+0000","last_change":"2024-03-12T03:50:04.535064+0000","last_active":"2024-03-12T03:50:20.617349+0000","last_peered":"2024-03-12T03:50:20.617349+0000","last_clean":"2024-03-12T03:50:20.617349+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:50:20.617349+0000","last_undegraded":"2024-03-12T03:50:20.617349+0000","last_fullsized":"2024-03-12T03:50:20.617349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_clean_scrub_stamp":"2024-03-12T03:50:04.534937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:34.126590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":738,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-12T03:50:20.628888+0000","last_change":"2024-03-12T03:49:03.833199+0000","last_active":"2024-03-12T03:50:20.628888+0000","last_peered":"2024-03-12T03:50:20.628888+0000","last_clean":"2024-03-12T03:50:20.628888+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:50:20.628888+0000","last_undegraded":"2024-03-12T03:50:20.628888+0000","last_fullsized":"2024-03-12T03:50:20.628888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:03.832786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_clean_scrub_stamp":"2024-03-12T03:49:03.832786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:24.984726+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5468913688,"num_objects":83450,"num_object_clones":0,"num_object_copies":250350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":51455,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83450,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194536,"num_write_kb":8895489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16263421952,"data_stored":16263413808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77177,"ondisk_log_size":77177,"up":38,"acting":38,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1011,"num_read_kb":1006,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":605880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":56,"acting":56,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1397,"num_read_kb":1122,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":2642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":83,"acting":83,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":87,"acting":87,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070673,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2090800,"kb_used_data":1329380,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":91654352,"statfs":{"total":95995035648,"available":93854056448,"internally_reserved":0,"allocated":1361285120,"data_stored":1360090136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":779680202},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332310,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3004700,"kb_used_data":2351440,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":90740452,"statfs":{"total":95995035648,"available":92918222848,"internally_reserved":0,"allocated":2407874560,"data_stored":2406698117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":668793286},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575087,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1733972,"kb_used_data":962824,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92011180,"statfs":{"total":95995035648,"available":94219448320,"internally_reserved":0,"allocated":985931776,"data_stored":984700063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":789641669},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738595,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3725208,"kb_used_data":3102608,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":90019944,"statfs":{"total":95995035648,"available":92180422656,"internally_reserved":0,"allocated":3177070592,"data_stored":3175997927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411561,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2732056,"kb_used_data":2024780,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":91013096,"statfs":{"total":95995035648,"available":93197410304,"internally_reserved":0,"allocated":2073374720,"data_stored":2072203906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":330,"seq":1417339207728,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2062644,"kb_used_data":1811500,"kb_used_omap":13,"kb_used_meta":251122,"kb_avail":91682508,"statfs":{"total":95995035648,"available":93882888192,"internally_reserved":0,"allocated":1854976000,"data_stored":1853765872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13485,"internal_metadata":257149779},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953697,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3273784,"kb_used_data":2674604,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":90471368,"statfs":{"total":95995035648,"available":92642680832,"internally_reserved":0,"allocated":2738794496,"data_stored":2737590884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659355,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2467676,"kb_used_data":1692372,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":91277476,"statfs":{"total":95995035648,"available":93468135424,"internally_reserved":0,"allocated":1732988928,"data_stored":1731809332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3205627904,"data_stored":3205627904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":978649088,"data_stored":978649088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2063794176,"data_stored":2063794176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1819344896,"data_stored":1819344896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2721058816,"data_stored":2721054744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1715863552,"data_stored":1715863552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2400784384,"data_stored":2400780312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1358299136,"data_stored":1358299136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:50:25.209 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-12T03:50:25.239 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:25.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:25.341 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:25.370 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:25.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:25.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:25.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:25.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:25.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:25.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:25.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:25.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:25.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:25.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:25.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:25.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:25.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:25.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:25 smithi169 ceph-mon[29344]: pgmap v819: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.97k op/s; 51529/248001 objects degraded (20.778%) 2024-03-12T03:50:25.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:25 smithi169 ceph-mon[39949]: pgmap v819: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.97k op/s; 51529/248001 objects degraded (20.778%) 2024-03-12T03:50:25.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:25.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:25.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:25 smithi173 ceph-mon[33544]: pgmap v819: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.97k op/s; 51529/248001 objects degraded (20.778%) 2024-03-12T03:50:26.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:26.037 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:26.038 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:26.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:26.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:26.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:26.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:26.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:26.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:26.327 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:26.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:26.428 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:26.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:26.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:26.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:26.601 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:26.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:26.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:26.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:26.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:26.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:26.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:26.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:26.976 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:27.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:27.010 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:27.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:27.111 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:27.146 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:27.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:27.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:27.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:27.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:27.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:27.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:27.417 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:27.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:27.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:27.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:27.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:27.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:27.658 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:27.689 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:27.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:27.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:27.791 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:27.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:27 smithi169 ceph-mon[29344]: pgmap v820: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 51529/248001 objects degraded (20.778%) 2024-03-12T03:50:27.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:27 smithi169 ceph-mon[39949]: pgmap v820: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 51529/248001 objects degraded (20.778%) 2024-03-12T03:50:27.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:27.966 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:27.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:27 smithi173 ceph-mon[33544]: pgmap v820: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 51529/248001 objects degraded (20.778%) 2024-03-12T03:50:28.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:28.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:28.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:28.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:28.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:28.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:28.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:28.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:28.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:28.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:28.521 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:28.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:28.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:28.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:28.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:28.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:28.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:28.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:28.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:28 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 51529/248001 objects degraded (20.778%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:28.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:28 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 51529/248001 objects degraded (20.778%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:28.929 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:28.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:28.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:28.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:28 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 51529/248001 objects degraded (20.778%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:29.031 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:29.068 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:29.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:29.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:29.205 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:29.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:29.307 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:29.350 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:29.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:29.398 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:50:29.433 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:50:29.451 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:29.468 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:50:29.491 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:29.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:29.507 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:50:29.542 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:50:29.592 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:50:29.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:29.624 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:29.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:29.629 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:50:29.629 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:50:29.629 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:50:29.629 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:50:29.629 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:50:29.629 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:50:29.630 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:50:29.630 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:50:29.630 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:50:29.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:29.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:29.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:29.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:29.898 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:29 smithi173 ceph-mon[33544]: pgmap v821: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.84k op/s; 46741/227580 objects degraded (20.538%) 2024-03-12T03:50:29.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:29 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2171909617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:50:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:29.900 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:29.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:29.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:29 smithi169 ceph-mon[39949]: pgmap v821: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.84k op/s; 46741/227580 objects degraded (20.538%) 2024-03-12T03:50:29.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:29 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2171909617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:50:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:29 smithi169 ceph-mon[29344]: pgmap v821: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.84k op/s; 46741/227580 objects degraded (20.538%) 2024-03-12T03:50:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:29 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2171909617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:50:30.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:30.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:30.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:30.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:30.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:30.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:30.210 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:50:30.210 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:50:30.210 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-12T03:50:30.211 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-12T03:50:30.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:50:30.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:30.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:30.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:30.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:30.442 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:30.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:30.544 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:30.580 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:30.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:30.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:30.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:30.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:30.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:30.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:30.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:30.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:30.960 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:30.992 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:30.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:30.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:31.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:31.126 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:31.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:31.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:31.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:31.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:31.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:31.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:31.398 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:31.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:31.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:31.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:31.531 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:31.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:31.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:31.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:31.655 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:50:31.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:31.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:31.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:31.770 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:31.798 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:31.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:31.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:31.900 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:31.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:31 smithi169 ceph-mon[29344]: pgmap v822: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.65k op/s; 46741/224559 objects degraded (20.815%) 2024-03-12T03:50:31.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:31 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1764341527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:50:31.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:31 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:50:31.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:31.924 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:50:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:50:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:50:31.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:31 smithi169 ceph-mon[39949]: pgmap v822: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.65k op/s; 46741/224559 objects degraded (20.815%) 2024-03-12T03:50:31.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:31 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1764341527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:50:31.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:31 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:50:31.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:31.933 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:31.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:31 smithi173 ceph-mon[33544]: pgmap v822: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.65k op/s; 46741/224559 objects degraded (20.815%) 2024-03-12T03:50:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:31 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1764341527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:50:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:31 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:50:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:32.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:32.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:32.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:32.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:32.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:32.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:32.310 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:32.338 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:32.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:32.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:32.476 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:32.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:32.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:32.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:32.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:32.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:32.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:50:32.673 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:50:32.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:32.746 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:32.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:32.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:32.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:32.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:32.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:32.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:32.891 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:32 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:50:32.891 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:32 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:50:32.891 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:32 smithi169 ceph-mon[29344]: osdmap e408: 8 total, 7 up, 8 in 2024-03-12T03:50:32.892 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:32 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:50:32.892 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:32 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:50:32.892 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:32 smithi169 ceph-mon[39949]: osdmap e408: 8 total, 7 up, 8 in 2024-03-12T03:50:32.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:32.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:32 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:50:32.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:32 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:50:32.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:32 smithi173 ceph-mon[33544]: osdmap e408: 8 total, 7 up, 8 in 2024-03-12T03:50:33.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:33.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:33.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:33.160 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:33.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:33.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:33.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:33.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:33.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:33.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:33.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:33.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:33.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:33.573 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:33.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:33.669 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:50:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:50:33.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:33.709 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:33.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:33.756 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:50:33.756 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:50:33.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.756 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:212733 10.b 10:d88e9bb6:::benchmark_data_smithi169_162694_object73738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-03-12T03:50:33.756 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:50:33.387930+0000", 2024-03-12T03:50:33.756 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.011334759, 2024-03-12T03:50:33.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011360336, 2024-03-12T03:50:33.756 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:50:33.756 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:50:33.756 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 212733 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.387930+0000", 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.387930+0000", 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:50:33.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.387931+0000", 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2470000000000001e-06 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.387936+0000", 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4020000000000001e-06 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.387951+0000", 2024-03-12T03:50:33.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4610000000000001e-05 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.387959+0000", 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1580000000000005e-06 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.388018+0000", 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8553e-05 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:50:33.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.388140+0000", 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012296799999999999 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6", 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.388383+0000", 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000242243 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.394223+0000", 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058405690000000003 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:212734 10.b 10:d93eeb22:::benchmark_data_smithi169_162694_object73739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:50:33.388286+0000", 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.010978023, 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011073972, 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 212734 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.761 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.388286+0000", 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.388286+0000", 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.388292+0000", 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7359999999999998e-06 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.388302+0000", 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8490000000000001e-06 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.388312+0000", 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.024e-05 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.388319+0000", 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3429999999999998e-06 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.391361+0000", 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0030416610000000002 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.391490+0000", 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000129393 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6", 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.391614+0000", 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012389899999999999 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.394403+0000", 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027886439999999998 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:212740 10.0 10:0175e77c:::benchmark_data_smithi169_162694_object73745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:50:33.393282+0000", 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0059824559999999997, 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061709340000000003, 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:50:33.765 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 212740 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.393282+0000", 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.393282+0000", 2024-03-12T03:50:33.766 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.393283+0000", 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.482e-06 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.393289+0000", 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1410000000000002e-06 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:50:33.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.393305+0000", 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5536000000000002e-05 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.393313+0000", 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6660000000000008e-06 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.393366+0000", 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2848e-05 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:50:33.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.393541+0000", 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000175718 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4,6", 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.393657+0000", 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011560100000000001 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.396228+0000", 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025713469999999999 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:50:33.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.399461+0000", 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003232698 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:212745 10.b 10:d56f9f6e:::benchmark_data_smithi169_162694_object73750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:50:33.396581+0000", 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0026832700000000002, 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029404959999999999, 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:50:33.770 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 212745 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.396581+0000", 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.396581+0000", 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.396582+0000", 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.361e-06 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.396588+0000", 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2630000000000003e-06 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.396600+0000", 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2815e-05 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.396609+0000", 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4100000000000008e-06 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.396670+0000", 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0665999999999998e-05 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.396798+0000", 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000128052 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6", 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.396933+0000", 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013495799999999999 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:50:33.399309+0000", 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023767630000000001 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 4 2024-03-12T03:50:33.774 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:50:33.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:33.846 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:33.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:33.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:33.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[29344]: pgmap v824: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.61k op/s; 42952/209067 objects degraded (20.545%) 2024-03-12T03:50:33.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[29344]: 5.0 scrub starts 2024-03-12T03:50:33.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[29344]: 5.0 scrub ok 2024-03-12T03:50:33.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[29344]: 10.e scrub starts 2024-03-12T03:50:33.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[29344]: 1.10 scrub starts 2024-03-12T03:50:33.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[29344]: 1.10 scrub ok 2024-03-12T03:50:33.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 42952/209067 objects degraded (20.545%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:33.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[29344]: osdmap e409: 8 total, 7 up, 8 in 2024-03-12T03:50:33.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[39949]: pgmap v824: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.61k op/s; 42952/209067 objects degraded (20.545%) 2024-03-12T03:50:33.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[39949]: 5.0 scrub starts 2024-03-12T03:50:33.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[39949]: 5.0 scrub ok 2024-03-12T03:50:33.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[39949]: 10.e scrub starts 2024-03-12T03:50:33.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[39949]: 1.10 scrub starts 2024-03-12T03:50:33.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[39949]: 1.10 scrub ok 2024-03-12T03:50:33.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 42952/209067 objects degraded (20.545%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:33.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:33 smithi169 ceph-mon[39949]: osdmap e409: 8 total, 7 up, 8 in 2024-03-12T03:50:33.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:33.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:33.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:33 smithi173 ceph-mon[33544]: pgmap v824: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.61k op/s; 42952/209067 objects degraded (20.545%) 2024-03-12T03:50:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:33 smithi173 ceph-mon[33544]: 5.0 scrub starts 2024-03-12T03:50:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:33 smithi173 ceph-mon[33544]: 5.0 scrub ok 2024-03-12T03:50:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:33 smithi173 ceph-mon[33544]: 10.e scrub starts 2024-03-12T03:50:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:33 smithi173 ceph-mon[33544]: 1.10 scrub starts 2024-03-12T03:50:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:33 smithi173 ceph-mon[33544]: 1.10 scrub ok 2024-03-12T03:50:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:33 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 42952/209067 objects degraded (20.545%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:33 smithi173 ceph-mon[33544]: osdmap e409: 8 total, 7 up, 8 in 2024-03-12T03:50:34.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:34.117 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:34.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:34.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:34.219 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:34.246 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:34.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:34.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:34.348 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:34.378 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:34.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:34.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:34.529 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:34.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-12T03:50:34.542 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:34.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:34.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:34.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:34.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:50:34.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:34.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:34.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:34.834 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:34.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:34.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:34.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[29344]: 3.2 scrub starts 2024-03-12T03:50:34.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[29344]: 3.2 scrub ok 2024-03-12T03:50:34.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:50:34.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:50:34.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[29344]: 1.a scrub starts 2024-03-12T03:50:34.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[29344]: 1.a scrub ok 2024-03-12T03:50:34.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:34.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[39949]: 3.2 scrub starts 2024-03-12T03:50:34.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[39949]: 3.2 scrub ok 2024-03-12T03:50:34.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:50:34.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:50:34.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[39949]: 1.a scrub starts 2024-03-12T03:50:34.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:34 smithi169 ceph-mon[39949]: 1.a scrub ok 2024-03-12T03:50:34.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:34.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:34.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:34.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:34.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:34.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:34 smithi173 ceph-mon[33544]: 3.2 scrub starts 2024-03-12T03:50:34.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:34 smithi173 ceph-mon[33544]: 3.2 scrub ok 2024-03-12T03:50:34.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:34 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:50:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:34 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:50:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:34 smithi173 ceph-mon[33544]: 1.a scrub starts 2024-03-12T03:50:34.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:34 smithi173 ceph-mon[33544]: 1.a scrub ok 2024-03-12T03:50:35.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:35.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:35.110 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:35.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:35.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:35.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:35.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:35.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:35.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:35.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:35.388 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:35.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:35.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:35.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:35.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:35.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:35.651 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:35.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:35.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:35.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:50:35.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:35.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:35.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:35.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:35.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:35.920 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:35.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:35.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:35 smithi173 ceph-mon[33544]: pgmap v826: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.90k op/s; 42109/202458 objects degraded (20.799%) 2024-03-12T03:50:35.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:35 smithi173 ceph-mon[33544]: 10.e scrub ok 2024-03-12T03:50:35.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:35 smithi173 ceph-mon[33544]: osdmap e410: 8 total, 7 up, 8 in 2024-03-12T03:50:36.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:36.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:36.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:36.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:36.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:35 smithi169 ceph-mon[29344]: pgmap v826: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.90k op/s; 42109/202458 objects degraded (20.799%) 2024-03-12T03:50:36.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:35 smithi169 ceph-mon[29344]: 10.e scrub ok 2024-03-12T03:50:36.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:35 smithi169 ceph-mon[29344]: osdmap e410: 8 total, 7 up, 8 in 2024-03-12T03:50:36.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:35 smithi169 ceph-mon[39949]: pgmap v826: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.90k op/s; 42109/202458 objects degraded (20.799%) 2024-03-12T03:50:36.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:35 smithi169 ceph-mon[39949]: 10.e scrub ok 2024-03-12T03:50:36.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:35 smithi169 ceph-mon[39949]: osdmap e410: 8 total, 7 up, 8 in 2024-03-12T03:50:36.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:36.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:36.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:36.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:36.331 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:36.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:36.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:36.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:36.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:36.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:36.598 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:36.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:36.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:36.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:36.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:36.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:36.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:36.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:36.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:36.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:36.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:36 smithi173 ceph-mon[33544]: 3.11 scrub starts 2024-03-12T03:50:36.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:36 smithi173 ceph-mon[33544]: 3.11 scrub ok 2024-03-12T03:50:36.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:36 smithi173 ceph-mon[33544]: 4.2 scrub starts 2024-03-12T03:50:36.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:36 smithi173 ceph-mon[33544]: 4.2 scrub ok 2024-03-12T03:50:36.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:36 smithi173 ceph-mon[33544]: 4.1 scrub starts 2024-03-12T03:50:36.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:36 smithi173 ceph-mon[33544]: 4.1 scrub ok 2024-03-12T03:50:37.002 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:37.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:37.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:37.137 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:37.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:37.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:36 smithi169 ceph-mon[29344]: 3.11 scrub starts 2024-03-12T03:50:37.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:36 smithi169 ceph-mon[29344]: 3.11 scrub ok 2024-03-12T03:50:37.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:36 smithi169 ceph-mon[29344]: 4.2 scrub starts 2024-03-12T03:50:37.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:36 smithi169 ceph-mon[29344]: 4.2 scrub ok 2024-03-12T03:50:37.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:36 smithi169 ceph-mon[29344]: 4.1 scrub starts 2024-03-12T03:50:37.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:36 smithi169 ceph-mon[29344]: 4.1 scrub ok 2024-03-12T03:50:37.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:36 smithi169 ceph-mon[39949]: 3.11 scrub starts 2024-03-12T03:50:37.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:36 smithi169 ceph-mon[39949]: 3.11 scrub ok 2024-03-12T03:50:37.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:36 smithi169 ceph-mon[39949]: 4.2 scrub starts 2024-03-12T03:50:37.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:36 smithi169 ceph-mon[39949]: 4.2 scrub ok 2024-03-12T03:50:37.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:36 smithi169 ceph-mon[39949]: 4.1 scrub starts 2024-03-12T03:50:37.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:36 smithi169 ceph-mon[39949]: 4.1 scrub ok 2024-03-12T03:50:37.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:37.267 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:37.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:37.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:37.369 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:37.431 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:37.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:37.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:37.560 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:37.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:37.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:37.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:37.693 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:37.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:37.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:37.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:37.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:37 smithi169 ceph-mon[29344]: pgmap v828: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.23k op/s; 42109/202458 objects degraded (20.799%) 2024-03-12T03:50:37.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:37 smithi169 ceph-mon[29344]: 5.10 scrub starts 2024-03-12T03:50:37.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:37 smithi169 ceph-mon[29344]: 5.10 scrub ok 2024-03-12T03:50:37.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:37 smithi169 ceph-mon[29344]: 1.f scrub starts 2024-03-12T03:50:37.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:37 smithi169 ceph-mon[29344]: 1.f scrub ok 2024-03-12T03:50:37.823 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:37 smithi169 ceph-mon[39949]: pgmap v828: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.23k op/s; 42109/202458 objects degraded (20.799%) 2024-03-12T03:50:37.823 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:37 smithi169 ceph-mon[39949]: 5.10 scrub starts 2024-03-12T03:50:37.823 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:37 smithi169 ceph-mon[39949]: 5.10 scrub ok 2024-03-12T03:50:37.823 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:37 smithi169 ceph-mon[39949]: 1.f scrub starts 2024-03-12T03:50:37.823 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:37 smithi169 ceph-mon[39949]: 1.f scrub ok 2024-03-12T03:50:37.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:37.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:37.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:37.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:37.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:37.986 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:37.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:37 smithi173 ceph-mon[33544]: pgmap v828: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.23k op/s; 42109/202458 objects degraded (20.799%) 2024-03-12T03:50:37.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:37 smithi173 ceph-mon[33544]: 5.10 scrub starts 2024-03-12T03:50:37.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:37 smithi173 ceph-mon[33544]: 5.10 scrub ok 2024-03-12T03:50:37.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:37 smithi173 ceph-mon[33544]: 1.f scrub starts 2024-03-12T03:50:37.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:37 smithi173 ceph-mon[33544]: 1.f scrub ok 2024-03-12T03:50:37.997 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:38.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:38.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:38.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:38.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:38.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:38.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:38.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:38.315 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:50:38.318 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:50:38.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:38.394 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:38.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:38.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:38.495 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:38.526 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:38.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:38.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:38.659 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:38.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:38.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:38.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:38.795 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:38.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:38.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:38.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:38.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:38.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:38.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:38 smithi173 ceph-mon[33544]: 5.9 scrub starts 2024-03-12T03:50:38.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:38 smithi173 ceph-mon[33544]: 5.9 scrub ok 2024-03-12T03:50:38.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:38 smithi173 ceph-mon[33544]: 3.1e scrub starts 2024-03-12T03:50:38.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:38 smithi173 ceph-mon[33544]: 3.1e scrub ok 2024-03-12T03:50:38.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:38 smithi173 ceph-mon[33544]: 6.14 scrub starts 2024-03-12T03:50:38.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:38 smithi173 ceph-mon[33544]: 6.14 scrub ok 2024-03-12T03:50:38.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:38 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 42109/202458 objects degraded (20.799%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:38.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:38 smithi173 ceph-mon[33544]: osdmap e411: 8 total, 7 up, 8 in 2024-03-12T03:50:39.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:39.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:39.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:39.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:39.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[29344]: 5.9 scrub starts 2024-03-12T03:50:39.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[29344]: 5.9 scrub ok 2024-03-12T03:50:39.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[29344]: 3.1e scrub starts 2024-03-12T03:50:39.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[29344]: 3.1e scrub ok 2024-03-12T03:50:39.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[29344]: 6.14 scrub starts 2024-03-12T03:50:39.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[29344]: 6.14 scrub ok 2024-03-12T03:50:39.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 42109/202458 objects degraded (20.799%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:39.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[29344]: osdmap e411: 8 total, 7 up, 8 in 2024-03-12T03:50:39.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[39949]: 5.9 scrub starts 2024-03-12T03:50:39.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[39949]: 5.9 scrub ok 2024-03-12T03:50:39.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[39949]: 3.1e scrub starts 2024-03-12T03:50:39.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[39949]: 3.1e scrub ok 2024-03-12T03:50:39.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[39949]: 6.14 scrub starts 2024-03-12T03:50:39.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[39949]: 6.14 scrub ok 2024-03-12T03:50:39.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 42109/202458 objects degraded (20.799%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:39.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:38 smithi169 ceph-mon[39949]: osdmap e411: 8 total, 7 up, 8 in 2024-03-12T03:50:39.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:39.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:39.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:39.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:39.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:39.349 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:39.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:39.451 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:39.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:39.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:39.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:39.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:39.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:39.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:39.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:39.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:39.774 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:39 smithi169 ceph-mon[29344]: pgmap v829: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 785 B/s rd, 0 B/s wr, 1.75k op/s; 37525/183222 objects degraded (20.481%) 2024-03-12T03:50:39.774 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:39 smithi169 ceph-mon[29344]: from='client.25471 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:50:39.774 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:39 smithi169 ceph-mon[29344]: 6.12 scrub starts 2024-03-12T03:50:39.775 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:39 smithi169 ceph-mon[29344]: 6.12 scrub ok 2024-03-12T03:50:39.775 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:39 smithi169 ceph-mon[39949]: pgmap v829: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 785 B/s rd, 0 B/s wr, 1.75k op/s; 37525/183222 objects degraded (20.481%) 2024-03-12T03:50:39.775 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:39 smithi169 ceph-mon[39949]: from='client.25471 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:50:39.775 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:39 smithi169 ceph-mon[39949]: 6.12 scrub starts 2024-03-12T03:50:39.775 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:39 smithi169 ceph-mon[39949]: 6.12 scrub ok 2024-03-12T03:50:39.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:39.896 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:39.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:39 smithi173 ceph-mon[33544]: pgmap v829: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 785 B/s rd, 0 B/s wr, 1.75k op/s; 37525/183222 objects degraded (20.481%) 2024-03-12T03:50:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:39 smithi173 ceph-mon[33544]: from='client.25471 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:50:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:39 smithi173 ceph-mon[33544]: 6.12 scrub starts 2024-03-12T03:50:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:39 smithi173 ceph-mon[33544]: 6.12 scrub ok 2024-03-12T03:50:39.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:40.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:40.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:40.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:40.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:40.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:40.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:40.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:40.313 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:40.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:40.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:40.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:40.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:40.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:40.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:40.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:40.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:40.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:40.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:40.688 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:40.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:40.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:40.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:40.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:40.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:40.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:40.965 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:40.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:40 smithi173 ceph-mon[33544]: 3.e scrub starts 2024-03-12T03:50:40.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:40 smithi173 ceph-mon[33544]: 3.e scrub ok 2024-03-12T03:50:40.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:40 smithi173 ceph-mon[33544]: 4.10 scrub starts 2024-03-12T03:50:40.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:40 smithi173 ceph-mon[33544]: 4.10 scrub ok 2024-03-12T03:50:40.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:40 smithi173 ceph-mon[33544]: 5.18 scrub starts 2024-03-12T03:50:40.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:40 smithi173 ceph-mon[33544]: 5.18 scrub ok 2024-03-12T03:50:40.999 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:41.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:41.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:41.129 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:41.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:41.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:41.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:40 smithi169 ceph-mon[29344]: 3.e scrub starts 2024-03-12T03:50:41.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:40 smithi169 ceph-mon[29344]: 3.e scrub ok 2024-03-12T03:50:41.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:40 smithi169 ceph-mon[29344]: 4.10 scrub starts 2024-03-12T03:50:41.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:40 smithi169 ceph-mon[29344]: 4.10 scrub ok 2024-03-12T03:50:41.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:40 smithi169 ceph-mon[29344]: 5.18 scrub starts 2024-03-12T03:50:41.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:40 smithi169 ceph-mon[29344]: 5.18 scrub ok 2024-03-12T03:50:41.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:40 smithi169 ceph-mon[39949]: 3.e scrub starts 2024-03-12T03:50:41.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:40 smithi169 ceph-mon[39949]: 3.e scrub ok 2024-03-12T03:50:41.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:40 smithi169 ceph-mon[39949]: 4.10 scrub starts 2024-03-12T03:50:41.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:40 smithi169 ceph-mon[39949]: 4.10 scrub ok 2024-03-12T03:50:41.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:40 smithi169 ceph-mon[39949]: 5.18 scrub starts 2024-03-12T03:50:41.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:40 smithi169 ceph-mon[39949]: 5.18 scrub ok 2024-03-12T03:50:41.231 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:41.258 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:41.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:41.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:41.360 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:41.399 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:41.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:41.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:41.501 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:41.534 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:41.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:41.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:41.635 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:41.666 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:41.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:41.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:41.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:41.802 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:41.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:41.903 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:41.929 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":829,"stamp":"2024-03-12T03:50:38.166743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3988030099,"num_objects":61074,"num_object_clones":0,"num_object_copies":183222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61074,"num_whiteouts":0,"num_read":2450,"num_read_kb":2174,"num_write":218064,"num_write_kb":8895644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":676620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81038,"ondisk_log_size":81038,"up":441,"acting":441,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17761948,"kb_used_data":12549896,"kb_used_omap":24,"kb_used_meta":5211751,"kb_avail":732199268,"statfs":{"total":767960285184,"available":749772050432,"internally_reserved":0,"allocated":12851093504,"data_stored":12841507293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24644,"internal_metadata":5336833980},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,12,5,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-747438080,"num_objects":-11405,"num_object_clones":0,"num_object_copies":-34215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-7303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11405,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11405,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.520737"},"pg_stats":[{"pgid":"10.2","version":"410'13592","reported_seq":13693,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:37.758718+0000","last_change":"2024-03-12T03:48:58.458222+0000","last_active":"2024-03-12T03:50:37.758718+0000","last_peered":"2024-03-12T03:50:37.758718+0000","last_clean":"2024-03-12T03:48:57.540498+0000","last_became_active":"2024-03-12T03:48:58.458222+0000","last_became_peered":"2024-03-12T03:48:58.458222+0000","last_unstale":"2024-03-12T03:50:37.758718+0000","last_undegraded":"2024-03-12T03:48:58.441694+0000","last_fullsized":"2024-03-12T03:48:58.441567+0000","mapping_epoch":384,"log_start":"397'8700","ondisk_log_start":"397'8700","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4892,"log_dups_size":0,"ondisk_log_size":4892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:40.592683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230948864,"num_objects":3524,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13592,"num_write_kb":547712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":3524}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1063,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.697780+0000","last_change":"2024-03-12T03:49:15.453480+0000","last_active":"2024-03-12T03:50:34.697780+0000","last_peered":"2024-03-12T03:50:34.697780+0000","last_clean":"2024-03-12T03:50:34.697780+0000","last_became_active":"2024-03-12T03:46:25.289005+0000","last_became_peered":"2024-03-12T03:46:25.289005+0000","last_unstale":"2024-03-12T03:50:34.697780+0000","last_undegraded":"2024-03-12T03:50:34.697780+0000","last_fullsized":"2024-03-12T03:50:34.697780+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:15.453179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:15.453179+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"344'25","reported_seq":813,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.689410+0000","last_change":"2024-03-12T03:49:17.840270+0000","last_active":"2024-03-12T03:50:34.689410+0000","last_peered":"2024-03-12T03:50:34.689410+0000","last_clean":"2024-03-12T03:50:34.689410+0000","last_became_active":"2024-03-12T03:46:25.301377+0000","last_became_peered":"2024-03-12T03:46:25.301377+0000","last_unstale":"2024-03-12T03:50:34.689410+0000","last_undegraded":"2024-03-12T03:50:34.689410+0000","last_fullsized":"2024-03-12T03:50:34.689410+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'25","last_scrub_stamp":"2024-03-12T03:49:17.839910+0000","last_deep_scrub":"344'25","last_deep_scrub_stamp":"2024-03-12T03:49:17.839910+0000","last_clean_scrub_stamp":"2024-03-12T03:49:17.839910+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:34.771372+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":519,"reported_epoch":409,"state":"active+undersized","last_fresh":"2024-03-12T03:50:33.484953+0000","last_change":"2024-03-12T03:48:58.454102+0000","last_active":"2024-03-12T03:50:33.484953+0000","last_peered":"2024-03-12T03:50:33.484953+0000","last_clean":"2024-03-12T03:48:53.448575+0000","last_became_active":"2024-03-12T03:48:58.454102+0000","last_became_peered":"2024-03-12T03:48:58.454102+0000","last_unstale":"2024-03-12T03:50:33.484953+0000","last_undegraded":"2024-03-12T03:50:33.484953+0000","last_fullsized":"2024-03-12T03:48:58.443116+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:07.363439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:07.363439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":622,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.693417+0000","last_change":"2024-03-12T03:49:48.662261+0000","last_active":"2024-03-12T03:50:34.693417+0000","last_peered":"2024-03-12T03:50:34.693417+0000","last_clean":"2024-03-12T03:50:34.693417+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:50:34.693417+0000","last_undegraded":"2024-03-12T03:50:34.693417+0000","last_fullsized":"2024-03-12T03:50:34.693417+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:49:48.662179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:48.662179+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:02.935042+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":759,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.688740+0000","last_change":"2024-03-12T03:49:54.872090+0000","last_active":"2024-03-12T03:50:34.688740+0000","last_peered":"2024-03-12T03:50:34.688740+0000","last_clean":"2024-03-12T03:50:34.688740+0000","last_became_active":"2024-03-12T03:39:33.265665+0000","last_became_peered":"2024-03-12T03:39:33.265665+0000","last_unstale":"2024-03-12T03:50:34.688740+0000","last_undegraded":"2024-03-12T03:50:34.688740+0000","last_fullsized":"2024-03-12T03:50:34.688740+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:54.871895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.871895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:21.677087+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"410'15444","reported_seq":15731,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:36.761281+0000","last_change":"2024-03-12T03:49:54.072615+0000","last_active":"2024-03-12T03:50:36.761281+0000","last_peered":"2024-03-12T03:50:36.761281+0000","last_clean":"2024-03-12T03:50:36.761281+0000","last_became_active":"2024-03-12T03:47:57.695210+0000","last_became_peered":"2024-03-12T03:47:57.695210+0000","last_unstale":"2024-03-12T03:50:36.761281+0000","last_undegraded":"2024-03-12T03:50:36.761281+0000","last_fullsized":"2024-03-12T03:50:36.761281+0000","mapping_epoch":366,"log_start":"393'9301","ondisk_log_start":"393'9301","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'11483","last_scrub_stamp":"2024-03-12T03:49:54.072402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.072402+0000","objects_scrubbed":7964,"log_size":6143,"log_dups_size":0,"ondisk_log_size":6143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:04.899343+0000","scrub_duration":2423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255000576,"num_objects":3891,"num_object_clones":0,"num_object_copies":11673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13893,"num_write_kb":569088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"269'31","reported_seq":852,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.688307+0000","last_change":"2024-03-12T03:50:01.871630+0000","last_active":"2024-03-12T03:50:34.688307+0000","last_peered":"2024-03-12T03:50:34.688307+0000","last_clean":"2024-03-12T03:50:34.688307+0000","last_became_active":"2024-03-12T03:39:33.268652+0000","last_became_peered":"2024-03-12T03:39:33.268652+0000","last_unstale":"2024-03-12T03:50:34.688307+0000","last_undegraded":"2024-03-12T03:50:34.688307+0000","last_fullsized":"2024-03-12T03:50:34.688307+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:50:01.871412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:01.871412+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:05.340467+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":214,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.690467+0000","last_change":"2024-03-12T03:48:18.820400+0000","last_active":"2024-03-12T03:50:34.690467+0000","last_peered":"2024-03-12T03:50:34.690467+0000","last_clean":"2024-03-12T03:50:34.690467+0000","last_became_active":"2024-03-12T03:45:18.108377+0000","last_became_peered":"2024-03-12T03:45:18.108377+0000","last_unstale":"2024-03-12T03:50:34.690467+0000","last_undegraded":"2024-03-12T03:50:34.690467+0000","last_fullsized":"2024-03-12T03:50:34.690467+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:18.820292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:48:18.820292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":745,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.694592+0000","last_change":"2024-03-12T03:48:58.457284+0000","last_active":"2024-03-12T03:50:34.694592+0000","last_peered":"2024-03-12T03:50:34.694592+0000","last_clean":"2024-03-12T03:48:53.449180+0000","last_became_active":"2024-03-12T03:48:58.457284+0000","last_became_peered":"2024-03-12T03:48:58.457284+0000","last_unstale":"2024-03-12T03:50:34.694592+0000","last_undegraded":"2024-03-12T03:50:34.694592+0000","last_fullsized":"2024-03-12T03:48:58.441935+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.054936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.054936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":496,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.693282+0000","last_change":"2024-03-12T03:49:17.061923+0000","last_active":"2024-03-12T03:50:34.693282+0000","last_peered":"2024-03-12T03:50:34.693282+0000","last_clean":"2024-03-12T03:50:34.693282+0000","last_became_active":"2024-03-12T03:46:25.293293+0000","last_became_peered":"2024-03-12T03:46:25.293293+0000","last_unstale":"2024-03-12T03:50:34.693282+0000","last_undegraded":"2024-03-12T03:50:34.693282+0000","last_fullsized":"2024-03-12T03:50:34.693282+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:17.061809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:17.061809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:45.913368+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"388'129","reported_seq":904,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.682815+0000","last_change":"2024-03-12T03:49:17.168749+0000","last_active":"2024-03-12T03:50:34.682815+0000","last_peered":"2024-03-12T03:50:34.682815+0000","last_clean":"2024-03-12T03:50:34.682815+0000","last_became_active":"2024-03-12T03:46:25.297913+0000","last_became_peered":"2024-03-12T03:46:25.297913+0000","last_unstale":"2024-03-12T03:50:34.682815+0000","last_undegraded":"2024-03-12T03:50:34.682815+0000","last_fullsized":"2024-03-12T03:50:34.682815+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"388'129","last_scrub_stamp":"2024-03-12T03:49:17.168603+0000","last_deep_scrub":"388'129","last_deep_scrub_stamp":"2024-03-12T03:49:17.168603+0000","last_clean_scrub_stamp":"2024-03-12T03:49:17.168603+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:46.165998+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"410'13791","reported_seq":13969,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:37.792154+0000","last_change":"2024-03-12T03:49:20.727415+0000","last_active":"2024-03-12T03:50:37.792154+0000","last_peered":"2024-03-12T03:50:37.792154+0000","last_clean":"2024-03-12T03:50:37.792154+0000","last_became_active":"2024-03-12T03:47:58.027417+0000","last_became_peered":"2024-03-12T03:47:58.027417+0000","last_unstale":"2024-03-12T03:50:37.792154+0000","last_undegraded":"2024-03-12T03:50:37.792154+0000","last_fullsized":"2024-03-12T03:50:37.792154+0000","mapping_epoch":366,"log_start":"398'8900","ondisk_log_start":"398'8900","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'7006","last_scrub_stamp":"2024-03-12T03:49:20.727145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:49:20.727145+0000","objects_scrubbed":6935,"log_size":4891,"log_dups_size":0,"ondisk_log_size":4891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237043712,"num_objects":3617,"num_object_clones":0,"num_object_copies":10851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13791,"num_write_kb":557056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":884,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.694164+0000","last_change":"2024-03-12T03:48:58.456582+0000","last_active":"2024-03-12T03:50:34.694164+0000","last_peered":"2024-03-12T03:50:34.694164+0000","last_clean":"2024-03-12T03:48:53.447572+0000","last_became_active":"2024-03-12T03:48:58.456582+0000","last_became_peered":"2024-03-12T03:48:58.456582+0000","last_unstale":"2024-03-12T03:50:34.694164+0000","last_undegraded":"2024-03-12T03:50:34.694164+0000","last_fullsized":"2024-03-12T03:48:58.441708+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:08.710391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:08.710391+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"269'31","reported_seq":804,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.687788+0000","last_change":"2024-03-12T03:49:09.755599+0000","last_active":"2024-03-12T03:50:34.687788+0000","last_peered":"2024-03-12T03:50:34.687788+0000","last_clean":"2024-03-12T03:50:34.687788+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:50:34.687788+0000","last_undegraded":"2024-03-12T03:50:34.687788+0000","last_fullsized":"2024-03-12T03:50:34.687788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:49:09.755469+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:49:09.755469+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:35.253531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":763,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.687585+0000","last_change":"2024-03-12T03:47:57.920336+0000","last_active":"2024-03-12T03:50:34.687585+0000","last_peered":"2024-03-12T03:50:34.687585+0000","last_clean":"2024-03-12T03:50:34.687585+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:50:34.687585+0000","last_undegraded":"2024-03-12T03:50:34.687585+0000","last_fullsized":"2024-03-12T03:50:34.687585+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:57.920257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_clean_scrub_stamp":"2024-03-12T03:47:57.920257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":741,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.686647+0000","last_change":"2024-03-12T03:49:06.194181+0000","last_active":"2024-03-12T03:50:34.686647+0000","last_peered":"2024-03-12T03:50:34.686647+0000","last_clean":"2024-03-12T03:50:34.686647+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:50:34.686647+0000","last_undegraded":"2024-03-12T03:50:34.686647+0000","last_fullsized":"2024-03-12T03:50:34.686647+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:06.192749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:06.192749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:34.863444+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":534,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.478580+0000","last_change":"2024-03-12T03:49:48.536574+0000","last_active":"2024-03-12T03:50:33.478580+0000","last_peered":"2024-03-12T03:50:33.478580+0000","last_clean":"2024-03-12T03:50:33.478580+0000","last_became_active":"2024-03-12T03:46:25.300814+0000","last_became_peered":"2024-03-12T03:46:25.300814+0000","last_unstale":"2024-03-12T03:50:33.478580+0000","last_undegraded":"2024-03-12T03:50:33.478580+0000","last_fullsized":"2024-03-12T03:50:33.478580+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:48.536435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:48.536435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:04.740323+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"410'13313","reported_seq":13415,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:36.268474+0000","last_change":"2024-03-12T03:48:58.455008+0000","last_active":"2024-03-12T03:50:36.268474+0000","last_peered":"2024-03-12T03:50:36.268474+0000","last_clean":"2024-03-12T03:48:57.553732+0000","last_became_active":"2024-03-12T03:48:58.455008+0000","last_became_peered":"2024-03-12T03:48:58.455008+0000","last_unstale":"2024-03-12T03:50:36.268474+0000","last_undegraded":"2024-03-12T03:48:58.446599+0000","last_fullsized":"2024-03-12T03:48:58.446493+0000","mapping_epoch":384,"log_start":"399'9300","ondisk_log_start":"399'9300","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247660544,"num_objects":3779,"num_object_clones":0,"num_object_copies":11337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13313,"num_write_kb":546944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":3779}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":885,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.682988+0000","last_change":"2024-03-12T03:50:34.205833+0000","last_active":"2024-03-12T03:50:34.682988+0000","last_peered":"2024-03-12T03:50:34.682988+0000","last_clean":"2024-03-12T03:50:34.682988+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:50:34.682988+0000","last_undegraded":"2024-03-12T03:50:34.682988+0000","last_fullsized":"2024-03-12T03:50:34.682988+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:34.205711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:34.205711+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:50.625397+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":607,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.696120+0000","last_change":"2024-03-12T03:49:12.640242+0000","last_active":"2024-03-12T03:50:34.696120+0000","last_peered":"2024-03-12T03:50:34.696120+0000","last_clean":"2024-03-12T03:50:34.696120+0000","last_became_active":"2024-03-12T03:46:25.287373+0000","last_became_peered":"2024-03-12T03:46:25.287373+0000","last_unstale":"2024-03-12T03:50:34.696120+0000","last_undegraded":"2024-03-12T03:50:34.696120+0000","last_fullsized":"2024-03-12T03:50:34.696120+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:49:12.640132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:12.640132+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":244,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.687310+0000","last_change":"2024-03-12T03:48:58.456372+0000","last_active":"2024-03-12T03:50:34.687310+0000","last_peered":"2024-03-12T03:50:34.687310+0000","last_clean":"2024-03-12T03:48:53.446320+0000","last_became_active":"2024-03-12T03:48:58.456372+0000","last_became_peered":"2024-03-12T03:48:58.456372+0000","last_unstale":"2024-03-12T03:50:34.687310+0000","last_undegraded":"2024-03-12T03:50:34.687310+0000","last_fullsized":"2024-03-12T03:48:58.444143+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:16.876204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:16.876204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"385'9","reported_seq":962,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:34.688543+0000","last_change":"2024-03-12T03:48:58.455645+0000","last_active":"2024-03-12T03:50:34.688543+0000","last_peered":"2024-03-12T03:50:34.688543+0000","last_clean":"2024-03-12T03:48:56.818397+0000","last_became_active":"2024-03-12T03:48:58.455645+0000","last_became_peered":"2024-03-12T03:48:58.455645+0000","last_unstale":"2024-03-12T03:50:34.688543+0000","last_undegraded":"2024-03-12T03:48:58.448928+0000","last_fullsized":"2024-03-12T03:48:58.448807+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:47:53.964216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:53.964216+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":732,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.683781+0000","last_change":"2024-03-12T03:48:58.448113+0000","last_active":"2024-03-12T03:50:34.683781+0000","last_peered":"2024-03-12T03:50:34.683781+0000","last_clean":"2024-03-12T03:48:53.472090+0000","last_became_active":"2024-03-12T03:48:58.448113+0000","last_became_peered":"2024-03-12T03:48:58.448113+0000","last_unstale":"2024-03-12T03:50:34.683781+0000","last_undegraded":"2024-03-12T03:50:34.683781+0000","last_fullsized":"2024-03-12T03:48:58.441990+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:04.770746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:04.770746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.231417+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"410'13515","reported_seq":13663,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:36.257553+0000","last_change":"2024-03-12T03:50:34.323820+0000","last_active":"2024-03-12T03:50:36.257553+0000","last_peered":"2024-03-12T03:50:36.257553+0000","last_clean":"2024-03-12T03:50:36.257553+0000","last_became_active":"2024-03-12T03:47:58.027802+0000","last_became_peered":"2024-03-12T03:47:58.027802+0000","last_unstale":"2024-03-12T03:50:36.257553+0000","last_undegraded":"2024-03-12T03:50:36.257553+0000","last_fullsized":"2024-03-12T03:50:36.257553+0000","mapping_epoch":366,"log_start":"400'9500","ondisk_log_start":"400'9500","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'13357","last_scrub_stamp":"2024-03-12T03:50:34.323591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:50:34.323591+0000","objects_scrubbed":4095,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:00.580249+0000","scrub_duration":1476,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253558784,"num_objects":3869,"num_object_clones":0,"num_object_copies":11607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13515,"num_write_kb":556288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":754,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.687248+0000","last_change":"2024-03-12T03:49:11.818292+0000","last_active":"2024-03-12T03:50:34.687248+0000","last_peered":"2024-03-12T03:50:34.687248+0000","last_clean":"2024-03-12T03:50:34.687248+0000","last_became_active":"2024-03-12T03:46:25.286985+0000","last_became_peered":"2024-03-12T03:46:25.286985+0000","last_unstale":"2024-03-12T03:50:34.687248+0000","last_undegraded":"2024-03-12T03:50:34.687248+0000","last_fullsized":"2024-03-12T03:50:34.687248+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:11.818116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:11.818116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"269'42","reported_seq":608,"reported_epoch":409,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:33.479728+0000","last_change":"2024-03-12T03:48:58.463377+0000","last_active":"2024-03-12T03:50:33.479728+0000","last_peered":"2024-03-12T03:50:33.479728+0000","last_clean":"2024-03-12T03:48:53.447338+0000","last_became_active":"2024-03-12T03:48:58.463377+0000","last_became_peered":"2024-03-12T03:48:58.463377+0000","last_unstale":"2024-03-12T03:50:33.479728+0000","last_undegraded":"2024-03-12T03:48:58.440183+0000","last_fullsized":"2024-03-12T03:48:58.440082+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:47:13.347128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.347128+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358227+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":523,"reported_epoch":409,"state":"active+undersized","last_fresh":"2024-03-12T03:50:33.478392+0000","last_change":"2024-03-12T03:48:58.464970+0000","last_active":"2024-03-12T03:50:33.478392+0000","last_peered":"2024-03-12T03:50:33.478392+0000","last_clean":"2024-03-12T03:48:53.448054+0000","last_became_active":"2024-03-12T03:48:58.464970+0000","last_became_peered":"2024-03-12T03:48:58.464970+0000","last_unstale":"2024-03-12T03:50:33.478392+0000","last_undegraded":"2024-03-12T03:50:33.478392+0000","last_fullsized":"2024-03-12T03:48:58.441302+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:05.639950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:05.639950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":770,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.690388+0000","last_change":"2024-03-12T03:49:14.836471+0000","last_active":"2024-03-12T03:50:34.690388+0000","last_peered":"2024-03-12T03:50:34.690388+0000","last_clean":"2024-03-12T03:50:34.690388+0000","last_became_active":"2024-03-12T03:46:25.280860+0000","last_became_peered":"2024-03-12T03:46:25.280860+0000","last_unstale":"2024-03-12T03:50:34.690388+0000","last_undegraded":"2024-03-12T03:50:34.690388+0000","last_fullsized":"2024-03-12T03:50:34.690388+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:14.836179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_clean_scrub_stamp":"2024-03-12T03:49:14.836179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:41.107430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":893,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.687317+0000","last_change":"2024-03-12T03:48:15.895744+0000","last_active":"2024-03-12T03:50:34.687317+0000","last_peered":"2024-03-12T03:50:34.687317+0000","last_clean":"2024-03-12T03:50:34.687317+0000","last_became_active":"2024-03-12T03:44:19.373715+0000","last_became_peered":"2024-03-12T03:44:19.373715+0000","last_unstale":"2024-03-12T03:50:34.687317+0000","last_undegraded":"2024-03-12T03:50:34.687317+0000","last_fullsized":"2024-03-12T03:50:34.687317+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:15.895638+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:15.895638+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.895638+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"409'13457","reported_seq":13636,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.541778+0000","last_change":"2024-03-12T03:49:58.948258+0000","last_active":"2024-03-12T03:50:33.541778+0000","last_peered":"2024-03-12T03:50:33.541778+0000","last_clean":"2024-03-12T03:50:33.541778+0000","last_became_active":"2024-03-12T03:47:58.027347+0000","last_became_peered":"2024-03-12T03:47:58.027347+0000","last_unstale":"2024-03-12T03:50:33.541778+0000","last_undegraded":"2024-03-12T03:50:33.541778+0000","last_fullsized":"2024-03-12T03:50:33.541778+0000","mapping_epoch":366,"log_start":"398'9100","ondisk_log_start":"398'9100","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'10207","last_scrub_stamp":"2024-03-12T03:49:58.947894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:49:58.947894+0000","objects_scrubbed":7488,"log_size":4357,"log_dups_size":0,"ondisk_log_size":4357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:01.246152+0000","scrub_duration":2515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270729216,"num_objects":4131,"num_object_clones":0,"num_object_copies":12393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13457,"num_write_kb":562816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"332'12","reported_seq":736,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.479565+0000","last_change":"2024-03-12T03:50:32.360621+0000","last_active":"2024-03-12T03:50:33.479565+0000","last_peered":"2024-03-12T03:50:33.479565+0000","last_clean":"2024-03-12T03:50:33.479565+0000","last_became_active":"2024-03-12T03:46:25.288674+0000","last_became_peered":"2024-03-12T03:46:25.288674+0000","last_unstale":"2024-03-12T03:50:33.479565+0000","last_undegraded":"2024-03-12T03:50:33.479565+0000","last_fullsized":"2024-03-12T03:50:33.479565+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'12","last_scrub_stamp":"2024-03-12T03:50:32.360502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:32.360502+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:39.022017+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"269'42","reported_seq":818,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:35.839929+0000","last_change":"2024-03-12T03:50:35.839987+0000","last_active":"2024-03-12T03:50:35.839929+0000","last_peered":"2024-03-12T03:50:35.839929+0000","last_clean":"2024-03-12T03:50:35.839929+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:50:35.839929+0000","last_undegraded":"2024-03-12T03:50:35.839929+0000","last_fullsized":"2024-03-12T03:50:35.839929+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:50:35.839859+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.839859+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:04.798992+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"320'2","reported_seq":616,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:34.695098+0000","last_change":"2024-03-12T03:48:58.457502+0000","last_active":"2024-03-12T03:50:34.695098+0000","last_peered":"2024-03-12T03:50:34.695098+0000","last_clean":"2024-03-12T03:45:56.084290+0000","last_became_active":"2024-03-12T03:48:58.457502+0000","last_became_peered":"2024-03-12T03:48:58.457502+0000","last_unstale":"2024-03-12T03:50:34.695098+0000","last_undegraded":"2024-03-12T03:48:58.439950+0000","last_fullsized":"2024-03-12T03:48:58.439792+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'2","last_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_deep_scrub":"320'2","last_deep_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_clean_scrub_stamp":"2024-03-12T03:45:56.089247+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":760,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.697026+0000","last_change":"2024-03-12T03:49:54.471443+0000","last_active":"2024-03-12T03:50:34.697026+0000","last_peered":"2024-03-12T03:50:34.697026+0000","last_clean":"2024-03-12T03:50:34.697026+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:50:34.697026+0000","last_undegraded":"2024-03-12T03:50:34.697026+0000","last_fullsized":"2024-03-12T03:50:34.697026+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:54.471078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:49:54.471078+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.471078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:55.755964+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":700,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.478284+0000","last_change":"2024-03-12T03:49:53.442784+0000","last_active":"2024-03-12T03:50:33.478284+0000","last_peered":"2024-03-12T03:50:33.478284+0000","last_clean":"2024-03-12T03:50:33.478284+0000","last_became_active":"2024-03-12T03:46:25.299964+0000","last_became_peered":"2024-03-12T03:46:25.299964+0000","last_unstale":"2024-03-12T03:50:33.478284+0000","last_undegraded":"2024-03-12T03:50:33.478284+0000","last_fullsized":"2024-03-12T03:50:33.478284+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:53.442638+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_clean_scrub_stamp":"2024-03-12T03:49:53.442638+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:06.708718+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"410'13450","reported_seq":13552,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:36.431320+0000","last_change":"2024-03-12T03:48:58.456284+0000","last_active":"2024-03-12T03:50:36.431320+0000","last_peered":"2024-03-12T03:50:36.431320+0000","last_clean":"2024-03-12T03:48:57.535679+0000","last_became_active":"2024-03-12T03:48:58.456284+0000","last_became_peered":"2024-03-12T03:48:58.456284+0000","last_unstale":"2024-03-12T03:50:36.431320+0000","last_undegraded":"2024-03-12T03:48:58.446353+0000","last_fullsized":"2024-03-12T03:48:58.446227+0000","mapping_epoch":384,"log_start":"397'8700","ondisk_log_start":"397'8700","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4750,"log_dups_size":0,"ondisk_log_size":4750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:40.378935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244056064,"num_objects":3724,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13450,"num_write_kb":549568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":3724}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"385'9","reported_seq":103,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:34.683395+0000","last_change":"2024-03-12T03:48:58.451360+0000","last_active":"2024-03-12T03:50:34.683395+0000","last_peered":"2024-03-12T03:50:34.683395+0000","last_clean":"2024-03-12T03:33:28.360558+0000","last_became_active":"2024-03-12T03:48:58.451360+0000","last_became_peered":"2024-03-12T03:48:58.451360+0000","last_unstale":"2024-03-12T03:50:34.683395+0000","last_undegraded":"2024-03-12T03:48:58.442329+0000","last_fullsized":"2024-03-12T03:48:58.441949+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:48:14.138048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:14.138048+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.231405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"269'26","reported_seq":805,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:35.597424+0000","last_change":"2024-03-12T03:50:35.597476+0000","last_active":"2024-03-12T03:50:35.597424+0000","last_peered":"2024-03-12T03:50:35.597424+0000","last_clean":"2024-03-12T03:50:35.597424+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:50:35.597424+0000","last_undegraded":"2024-03-12T03:50:35.597424+0000","last_fullsized":"2024-03-12T03:50:35.597424+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:50:35.597225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.597225+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:53.172520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":772,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.683985+0000","last_change":"2024-03-12T03:49:09.155688+0000","last_active":"2024-03-12T03:50:34.683985+0000","last_peered":"2024-03-12T03:50:34.683985+0000","last_clean":"2024-03-12T03:50:34.683985+0000","last_became_active":"2024-03-12T03:46:25.294348+0000","last_became_peered":"2024-03-12T03:46:25.294348+0000","last_unstale":"2024-03-12T03:50:34.683985+0000","last_undegraded":"2024-03-12T03:50:34.683985+0000","last_fullsized":"2024-03-12T03:50:34.683985+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:09.155572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:49:09.155572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":634,"reported_epoch":409,"state":"active+undersized","last_fresh":"2024-03-12T03:50:33.479558+0000","last_change":"2024-03-12T03:48:58.455443+0000","last_active":"2024-03-12T03:50:33.479558+0000","last_peered":"2024-03-12T03:50:33.479558+0000","last_clean":"2024-03-12T03:43:13.330724+0000","last_became_active":"2024-03-12T03:48:58.455443+0000","last_became_peered":"2024-03-12T03:48:58.455443+0000","last_unstale":"2024-03-12T03:50:33.479558+0000","last_undegraded":"2024-03-12T03:50:33.479558+0000","last_fullsized":"2024-03-12T03:48:58.439578+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:13.892760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.892760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358219+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":487,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.479674+0000","last_change":"2024-03-12T03:49:50.458935+0000","last_active":"2024-03-12T03:50:33.479674+0000","last_peered":"2024-03-12T03:50:33.479674+0000","last_clean":"2024-03-12T03:50:33.479674+0000","last_became_active":"2024-03-12T03:46:25.287360+0000","last_became_peered":"2024-03-12T03:46:25.287360+0000","last_unstale":"2024-03-12T03:50:33.479674+0000","last_undegraded":"2024-03-12T03:50:33.479674+0000","last_fullsized":"2024-03-12T03:50:33.479674+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:50.458811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.458811+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:59.838393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"410'13595","reported_seq":13696,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:36.272605+0000","last_change":"2024-03-12T03:48:58.455501+0000","last_active":"2024-03-12T03:50:36.272605+0000","last_peered":"2024-03-12T03:50:36.272605+0000","last_clean":"2024-03-12T03:48:57.549912+0000","last_became_active":"2024-03-12T03:48:58.455501+0000","last_became_peered":"2024-03-12T03:48:58.455501+0000","last_unstale":"2024-03-12T03:50:36.272605+0000","last_undegraded":"2024-03-12T03:48:58.444776+0000","last_fullsized":"2024-03-12T03:48:58.444662+0000","mapping_epoch":384,"log_start":"399'9500","ondisk_log_start":"399'9500","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4095,"log_dups_size":0,"ondisk_log_size":4095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253755416,"num_objects":3873,"num_object_clones":0,"num_object_copies":11619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13595,"num_write_kb":558913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":3873}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":61,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.685994+0000","last_change":"2024-03-12T03:48:58.449166+0000","last_active":"2024-03-12T03:50:34.685994+0000","last_peered":"2024-03-12T03:50:34.685994+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.449166+0000","last_became_peered":"2024-03-12T03:48:58.449166+0000","last_unstale":"2024-03-12T03:50:34.685994+0000","last_undegraded":"2024-03-12T03:50:34.685994+0000","last_fullsized":"2024-03-12T03:48:58.441301+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.179416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.179416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"269'35","reported_seq":559,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:34.683667+0000","last_change":"2024-03-12T03:48:58.454600+0000","last_active":"2024-03-12T03:50:34.683667+0000","last_peered":"2024-03-12T03:50:34.683667+0000","last_clean":"2024-03-12T03:43:28.848207+0000","last_became_active":"2024-03-12T03:48:58.454600+0000","last_became_peered":"2024-03-12T03:48:58.454600+0000","last_unstale":"2024-03-12T03:50:34.683667+0000","last_undegraded":"2024-03-12T03:48:58.443785+0000","last_fullsized":"2024-03-12T03:48:58.443677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-03-12T03:45:59.055318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.055318+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.225865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":47,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":779,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.694497+0000","last_change":"2024-03-12T03:49:51.465759+0000","last_active":"2024-03-12T03:50:34.694497+0000","last_peered":"2024-03-12T03:50:34.694497+0000","last_clean":"2024-03-12T03:50:34.694497+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:50:34.694497+0000","last_undegraded":"2024-03-12T03:50:34.694497+0000","last_fullsized":"2024-03-12T03:50:34.694497+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:51.465633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:49:51.465633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:55.149499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":759,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.692085+0000","last_change":"2024-03-12T03:49:02.614970+0000","last_active":"2024-03-12T03:50:34.692085+0000","last_peered":"2024-03-12T03:50:34.692085+0000","last_clean":"2024-03-12T03:50:34.692085+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:50:34.692085+0000","last_undegraded":"2024-03-12T03:50:34.692085+0000","last_fullsized":"2024-03-12T03:50:34.692085+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:02.614847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:02.614847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":357,"reported_epoch":409,"state":"active+undersized","last_fresh":"2024-03-12T03:50:33.478820+0000","last_change":"2024-03-12T03:48:58.454186+0000","last_active":"2024-03-12T03:50:33.478820+0000","last_peered":"2024-03-12T03:50:33.478820+0000","last_clean":"2024-03-12T03:34:01.860694+0000","last_became_active":"2024-03-12T03:48:58.454186+0000","last_became_peered":"2024-03-12T03:48:58.454186+0000","last_unstale":"2024-03-12T03:50:33.478820+0000","last_undegraded":"2024-03-12T03:50:33.478820+0000","last_fullsized":"2024-03-12T03:48:58.443407+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:04.857986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:04.857986+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.400254+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"410'13716","reported_seq":13814,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:36.775950+0000","last_change":"2024-03-12T03:48:58.462488+0000","last_active":"2024-03-12T03:50:36.775950+0000","last_peered":"2024-03-12T03:50:36.775950+0000","last_clean":"2024-03-12T03:48:57.504096+0000","last_became_active":"2024-03-12T03:48:58.462488+0000","last_became_peered":"2024-03-12T03:48:58.462488+0000","last_unstale":"2024-03-12T03:50:36.775950+0000","last_undegraded":"2024-03-12T03:48:58.445111+0000","last_fullsized":"2024-03-12T03:48:58.444978+0000","mapping_epoch":384,"log_start":"392'7500","ondisk_log_start":"392'7500","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248905728,"num_objects":3798,"num_object_clones":0,"num_object_copies":11394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3798,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13716,"num_write_kb":560448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":3798}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":736,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.686421+0000","last_change":"2024-03-12T03:48:58.467253+0000","last_active":"2024-03-12T03:50:34.686421+0000","last_peered":"2024-03-12T03:50:34.686421+0000","last_clean":"2024-03-12T03:48:53.472859+0000","last_became_active":"2024-03-12T03:48:58.467253+0000","last_became_peered":"2024-03-12T03:48:58.467253+0000","last_unstale":"2024-03-12T03:50:34.686421+0000","last_undegraded":"2024-03-12T03:50:34.686421+0000","last_fullsized":"2024-03-12T03:48:58.444374+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:00.577693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:46:00.577693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.225888+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"269'24","reported_seq":580,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:34.695543+0000","last_change":"2024-03-12T03:48:58.449415+0000","last_active":"2024-03-12T03:50:34.695543+0000","last_peered":"2024-03-12T03:50:34.695543+0000","last_clean":"2024-03-12T03:43:28.788554+0000","last_became_active":"2024-03-12T03:48:58.449415+0000","last_became_peered":"2024-03-12T03:48:58.449415+0000","last_unstale":"2024-03-12T03:50:34.695543+0000","last_undegraded":"2024-03-12T03:48:58.441112+0000","last_fullsized":"2024-03-12T03:48:58.441001+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:48:17.191968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:17.191968+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":1795,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:34.720093+0000","last_change":"2024-03-12T03:48:58.462253+0000","last_active":"2024-03-12T03:50:34.720093+0000","last_peered":"2024-03-12T03:50:34.720093+0000","last_clean":"2024-03-12T03:48:56.580904+0000","last_became_active":"2024-03-12T03:48:58.462253+0000","last_became_peered":"2024-03-12T03:48:58.462253+0000","last_unstale":"2024-03-12T03:50:34.720093+0000","last_undegraded":"2024-03-12T03:48:58.445825+0000","last_fullsized":"2024-03-12T03:48:58.445711+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:47:13.591375+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.591375+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:40.378906+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1026,"num_read_kb":1021,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":636,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.687680+0000","last_change":"2024-03-12T03:48:58.464609+0000","last_active":"2024-03-12T03:50:34.687680+0000","last_peered":"2024-03-12T03:50:34.687680+0000","last_clean":"2024-03-12T03:43:13.332262+0000","last_became_active":"2024-03-12T03:48:58.464609+0000","last_became_peered":"2024-03-12T03:48:58.464609+0000","last_unstale":"2024-03-12T03:50:34.687680+0000","last_undegraded":"2024-03-12T03:50:34.687680+0000","last_fullsized":"2024-03-12T03:48:58.443632+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:58.050985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.050985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":765,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.694292+0000","last_change":"2024-03-12T03:49:13.598817+0000","last_active":"2024-03-12T03:50:34.694292+0000","last_peered":"2024-03-12T03:50:34.694292+0000","last_clean":"2024-03-12T03:50:34.694292+0000","last_became_active":"2024-03-12T03:46:25.302256+0000","last_became_peered":"2024-03-12T03:46:25.302256+0000","last_unstale":"2024-03-12T03:50:34.694292+0000","last_undegraded":"2024-03-12T03:50:34.694292+0000","last_fullsized":"2024-03-12T03:50:34.694292+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:13.598692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:13.598692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":882,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.694600+0000","last_change":"2024-03-12T03:49:50.468068+0000","last_active":"2024-03-12T03:50:34.694600+0000","last_peered":"2024-03-12T03:50:34.694600+0000","last_clean":"2024-03-12T03:50:34.694600+0000","last_became_active":"2024-03-12T03:44:19.366060+0000","last_became_peered":"2024-03-12T03:44:19.366060+0000","last_unstale":"2024-03-12T03:50:34.694600+0000","last_undegraded":"2024-03-12T03:50:34.694600+0000","last_fullsized":"2024-03-12T03:50:34.694600+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:50.467976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.467976+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:15.492614+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"410'13499","reported_seq":13600,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:37.745005+0000","last_change":"2024-03-12T03:48:58.444686+0000","last_active":"2024-03-12T03:50:37.745005+0000","last_peered":"2024-03-12T03:50:37.745005+0000","last_clean":"2024-03-12T03:48:57.546566+0000","last_became_active":"2024-03-12T03:48:58.444686+0000","last_became_peered":"2024-03-12T03:48:58.444686+0000","last_unstale":"2024-03-12T03:50:37.745005+0000","last_undegraded":"2024-03-12T03:48:58.438881+0000","last_fullsized":"2024-03-12T03:48:58.438716+0000","mapping_epoch":384,"log_start":"397'8600","ondisk_log_start":"397'8600","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4899,"log_dups_size":0,"ondisk_log_size":4899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:40.592057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235339776,"num_objects":3591,"num_object_clones":0,"num_object_copies":10773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3591,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13499,"num_write_kb":546880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":3591}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":756,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.698048+0000","last_change":"2024-03-12T03:49:07.521537+0000","last_active":"2024-03-12T03:50:34.698048+0000","last_peered":"2024-03-12T03:50:34.698048+0000","last_clean":"2024-03-12T03:50:34.698048+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:50:34.698048+0000","last_undegraded":"2024-03-12T03:50:34.698048+0000","last_fullsized":"2024-03-12T03:50:34.698048+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:07.521281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:07.521281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"315'29","reported_seq":891,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.687751+0000","last_change":"2024-03-12T03:48:08.875139+0000","last_active":"2024-03-12T03:50:34.687751+0000","last_peered":"2024-03-12T03:50:34.687751+0000","last_clean":"2024-03-12T03:50:34.687751+0000","last_became_active":"2024-03-12T03:45:18.109048+0000","last_became_peered":"2024-03-12T03:45:18.109048+0000","last_unstale":"2024-03-12T03:50:34.687751+0000","last_undegraded":"2024-03-12T03:50:34.687751+0000","last_fullsized":"2024-03-12T03:50:34.687751+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'29","last_scrub_stamp":"2024-03-12T03:48:08.874995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:08.874995+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":527,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.478438+0000","last_change":"2024-03-12T03:50:33.413216+0000","last_active":"2024-03-12T03:50:33.478438+0000","last_peered":"2024-03-12T03:50:33.478438+0000","last_clean":"2024-03-12T03:50:33.478438+0000","last_became_active":"2024-03-12T03:46:25.292535+0000","last_became_peered":"2024-03-12T03:46:25.292535+0000","last_unstale":"2024-03-12T03:50:33.478438+0000","last_undegraded":"2024-03-12T03:50:33.478438+0000","last_fullsized":"2024-03-12T03:50:33.478438+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:33.413098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:33.413098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:56.872026+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1028,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.700323+0000","last_change":"2024-03-12T03:48:58.452024+0000","last_active":"2024-03-12T03:50:34.700323+0000","last_peered":"2024-03-12T03:50:34.700323+0000","last_clean":"2024-03-12T03:48:53.448868+0000","last_became_active":"2024-03-12T03:48:58.452024+0000","last_became_peered":"2024-03-12T03:48:58.452024+0000","last_unstale":"2024-03-12T03:50:34.700323+0000","last_undegraded":"2024-03-12T03:50:34.700323+0000","last_fullsized":"2024-03-12T03:48:58.444796+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_clean_scrub_stamp":"2024-03-12T03:48:11.435684+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":747,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.685235+0000","last_change":"2024-03-12T03:49:50.444531+0000","last_active":"2024-03-12T03:50:34.685235+0000","last_peered":"2024-03-12T03:50:34.685235+0000","last_clean":"2024-03-12T03:50:34.685235+0000","last_became_active":"2024-03-12T03:46:25.295668+0000","last_became_peered":"2024-03-12T03:46:25.295668+0000","last_unstale":"2024-03-12T03:50:34.685235+0000","last_undegraded":"2024-03-12T03:50:34.685235+0000","last_fullsized":"2024-03-12T03:50:34.685235+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:50.444395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.444395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:53.830638+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"410'13675","reported_seq":13859,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:36.651308+0000","last_change":"2024-03-12T03:49:56.704066+0000","last_active":"2024-03-12T03:50:36.651308+0000","last_peered":"2024-03-12T03:50:36.651308+0000","last_clean":"2024-03-12T03:50:36.651308+0000","last_became_active":"2024-03-12T03:47:57.693619+0000","last_became_peered":"2024-03-12T03:47:57.693619+0000","last_unstale":"2024-03-12T03:50:36.651308+0000","last_undegraded":"2024-03-12T03:50:36.651308+0000","last_fullsized":"2024-03-12T03:50:36.651308+0000","mapping_epoch":366,"log_start":"398'8900","ondisk_log_start":"398'8900","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'9930","last_scrub_stamp":"2024-03-12T03:49:56.703697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:49:56.703697+0000","objects_scrubbed":7586,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:23.082230+0000","scrub_duration":2410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244121600,"num_objects":3725,"num_object_clones":0,"num_object_copies":11175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13675,"num_write_kb":556800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":487,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.692641+0000","last_change":"2024-03-12T03:49:04.011166+0000","last_active":"2024-03-12T03:50:34.692641+0000","last_peered":"2024-03-12T03:50:34.692641+0000","last_clean":"2024-03-12T03:50:34.692641+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:50:34.692641+0000","last_undegraded":"2024-03-12T03:50:34.692641+0000","last_fullsized":"2024-03-12T03:50:34.692641+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:04.010993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:49:04.010993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"269'23","reported_seq":794,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:34.695898+0000","last_change":"2024-03-12T03:48:58.451116+0000","last_active":"2024-03-12T03:50:34.695898+0000","last_peered":"2024-03-12T03:50:34.695898+0000","last_clean":"2024-03-12T03:48:53.448806+0000","last_became_active":"2024-03-12T03:48:58.451116+0000","last_became_peered":"2024-03-12T03:48:58.451116+0000","last_unstale":"2024-03-12T03:50:34.695898+0000","last_undegraded":"2024-03-12T03:48:58.440926+0000","last_fullsized":"2024-03-12T03:48:58.440811+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'23","last_scrub_stamp":"2024-03-12T03:45:59.998395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.998395+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":484,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.692641+0000","last_change":"2024-03-12T03:48:58.469230+0000","last_active":"2024-03-12T03:50:34.692641+0000","last_peered":"2024-03-12T03:50:34.692641+0000","last_clean":"2024-03-12T03:48:53.446422+0000","last_became_active":"2024-03-12T03:48:58.469230+0000","last_became_peered":"2024-03-12T03:48:58.469230+0000","last_unstale":"2024-03-12T03:50:34.692641+0000","last_undegraded":"2024-03-12T03:50:34.692641+0000","last_fullsized":"2024-03-12T03:48:58.446209+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:06.903506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.903506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":771,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.692187+0000","last_change":"2024-03-12T03:49:05.549180+0000","last_active":"2024-03-12T03:50:34.692187+0000","last_peered":"2024-03-12T03:50:34.692187+0000","last_clean":"2024-03-12T03:50:34.692187+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:50:34.692187+0000","last_undegraded":"2024-03-12T03:50:34.692187+0000","last_fullsized":"2024-03-12T03:50:34.692187+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:05.549009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:49:05.549009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":760,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.698231+0000","last_change":"2024-03-12T03:49:18.522193+0000","last_active":"2024-03-12T03:50:34.698231+0000","last_peered":"2024-03-12T03:50:34.698231+0000","last_clean":"2024-03-12T03:50:34.698231+0000","last_became_active":"2024-03-12T03:46:25.284168+0000","last_became_peered":"2024-03-12T03:46:25.284168+0000","last_unstale":"2024-03-12T03:50:34.698231+0000","last_undegraded":"2024-03-12T03:50:34.698231+0000","last_fullsized":"2024-03-12T03:50:34.698231+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:49:18.522103+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:49:18.522103+0000","last_clean_scrub_stamp":"2024-03-12T03:49:18.522103+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1044,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.696795+0000","last_change":"2024-03-12T03:48:58.455161+0000","last_active":"2024-03-12T03:50:34.696795+0000","last_peered":"2024-03-12T03:50:34.696795+0000","last_clean":"2024-03-12T03:48:53.449188+0000","last_became_active":"2024-03-12T03:48:58.455161+0000","last_became_peered":"2024-03-12T03:48:58.455161+0000","last_unstale":"2024-03-12T03:50:34.696795+0000","last_undegraded":"2024-03-12T03:50:34.696795+0000","last_fullsized":"2024-03-12T03:48:58.444402+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:01.456424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:01.456424+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:40.378981+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"410'13492","reported_seq":13592,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:36.618675+0000","last_change":"2024-03-12T03:48:58.457912+0000","last_active":"2024-03-12T03:50:36.618675+0000","last_peered":"2024-03-12T03:50:36.618675+0000","last_clean":"2024-03-12T03:48:57.500736+0000","last_became_active":"2024-03-12T03:48:58.457912+0000","last_became_peered":"2024-03-12T03:48:58.457912+0000","last_unstale":"2024-03-12T03:50:36.618675+0000","last_undegraded":"2024-03-12T03:48:58.444591+0000","last_fullsized":"2024-03-12T03:48:58.442369+0000","mapping_epoch":384,"log_start":"397'8700","ondisk_log_start":"397'8700","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.231431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249036800,"num_objects":3800,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13492,"num_write_kb":553344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":3800}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":59,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.694805+0000","last_change":"2024-03-12T03:48:58.458916+0000","last_active":"2024-03-12T03:50:34.694805+0000","last_peered":"2024-03-12T03:50:34.694805+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.458916+0000","last_became_peered":"2024-03-12T03:48:58.458916+0000","last_unstale":"2024-03-12T03:50:34.694805+0000","last_undegraded":"2024-03-12T03:50:34.694805+0000","last_fullsized":"2024-03-12T03:48:58.440354+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:06.864708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.864708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"382'54","reported_seq":123,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.694868+0000","last_change":"2024-03-12T03:49:15.597576+0000","last_active":"2024-03-12T03:50:34.694868+0000","last_peered":"2024-03-12T03:50:34.694868+0000","last_clean":"2024-03-12T03:50:34.694868+0000","last_became_active":"2024-03-12T03:48:48.462111+0000","last_became_peered":"2024-03-12T03:48:48.462111+0000","last_unstale":"2024-03-12T03:50:34.694868+0000","last_undegraded":"2024-03-12T03:50:34.694868+0000","last_fullsized":"2024-03-12T03:50:34.694868+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'54","last_scrub_stamp":"2024-03-12T03:49:15.597431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:15.597431+0000","objects_scrubbed":11,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":58,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":755,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.687371+0000","last_change":"2024-03-12T03:48:58.453277+0000","last_active":"2024-03-12T03:50:34.687371+0000","last_peered":"2024-03-12T03:50:34.687371+0000","last_clean":"2024-03-12T03:48:53.447092+0000","last_became_active":"2024-03-12T03:48:58.453277+0000","last_became_peered":"2024-03-12T03:48:58.453277+0000","last_unstale":"2024-03-12T03:50:34.687371+0000","last_undegraded":"2024-03-12T03:50:34.687371+0000","last_fullsized":"2024-03-12T03:48:58.443592+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.648824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.648824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:37.845461+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":761,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.685882+0000","last_change":"2024-03-12T03:49:08.211828+0000","last_active":"2024-03-12T03:50:34.685882+0000","last_peered":"2024-03-12T03:50:34.685882+0000","last_clean":"2024-03-12T03:50:34.685882+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:50:34.685882+0000","last_undegraded":"2024-03-12T03:50:34.685882+0000","last_fullsized":"2024-03-12T03:50:34.685882+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:49:08.211712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:49:08.211712+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:35.785906+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":559,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.479604+0000","last_change":"2024-03-12T03:49:03.575443+0000","last_active":"2024-03-12T03:50:33.479604+0000","last_peered":"2024-03-12T03:50:33.479604+0000","last_clean":"2024-03-12T03:50:33.479604+0000","last_became_active":"2024-03-12T03:46:25.284416+0000","last_became_peered":"2024-03-12T03:46:25.284416+0000","last_unstale":"2024-03-12T03:50:33.479604+0000","last_undegraded":"2024-03-12T03:50:33.479604+0000","last_fullsized":"2024-03-12T03:50:33.479604+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:03.575207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:03.575207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358339+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":886,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.689201+0000","last_change":"2024-03-12T03:48:58.460115+0000","last_active":"2024-03-12T03:50:34.689201+0000","last_peered":"2024-03-12T03:50:34.689201+0000","last_clean":"2024-03-12T03:48:53.471868+0000","last_became_active":"2024-03-12T03:48:58.460115+0000","last_became_peered":"2024-03-12T03:48:58.460115+0000","last_unstale":"2024-03-12T03:50:34.689201+0000","last_undegraded":"2024-03-12T03:50:34.689201+0000","last_fullsized":"2024-03-12T03:48:58.445534+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:06.690873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.690873+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.225912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"410'13701","reported_seq":13805,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:36.767734+0000","last_change":"2024-03-12T03:48:58.491732+0000","last_active":"2024-03-12T03:50:36.767734+0000","last_peered":"2024-03-12T03:50:36.767734+0000","last_clean":"2024-03-12T03:48:57.560710+0000","last_became_active":"2024-03-12T03:48:58.473224+0000","last_became_peered":"2024-03-12T03:48:58.473224+0000","last_unstale":"2024-03-12T03:50:36.767734+0000","last_undegraded":"2024-03-12T03:48:58.454399+0000","last_fullsized":"2024-03-12T03:48:58.446729+0000","mapping_epoch":384,"log_start":"392'7500","ondisk_log_start":"392'7500","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":6201,"log_dups_size":0,"ondisk_log_size":6201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250937344,"num_objects":3829,"num_object_clones":0,"num_object_copies":11487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13701,"num_write_kb":560960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3829}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":559,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.687423+0000","last_change":"2024-03-12T03:48:11.924861+0000","last_active":"2024-03-12T03:50:34.687423+0000","last_peered":"2024-03-12T03:50:34.687423+0000","last_clean":"2024-03-12T03:50:34.687423+0000","last_became_active":"2024-03-12T03:46:26.281472+0000","last_became_peered":"2024-03-12T03:46:26.281472+0000","last_unstale":"2024-03-12T03:50:34.687423+0000","last_undegraded":"2024-03-12T03:50:34.687423+0000","last_fullsized":"2024-03-12T03:50:34.687423+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:11.924742+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.924742+0000","last_clean_scrub_stamp":"2024-03-12T03:48:11.924742+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"269'26","reported_seq":538,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:34.694725+0000","last_change":"2024-03-12T03:48:58.462328+0000","last_active":"2024-03-12T03:50:34.694725+0000","last_peered":"2024-03-12T03:50:34.694725+0000","last_clean":"2024-03-12T03:43:28.842466+0000","last_became_active":"2024-03-12T03:48:58.462328+0000","last_became_peered":"2024-03-12T03:48:58.462328+0000","last_unstale":"2024-03-12T03:50:34.694725+0000","last_undegraded":"2024-03-12T03:48:58.442501+0000","last_fullsized":"2024-03-12T03:48:58.442398+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:15.774076+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.774076+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":759,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.683461+0000","last_change":"2024-03-12T03:49:45.255407+0000","last_active":"2024-03-12T03:50:34.683461+0000","last_peered":"2024-03-12T03:50:34.683461+0000","last_clean":"2024-03-12T03:50:34.683461+0000","last_became_active":"2024-03-12T03:46:25.303784+0000","last_became_peered":"2024-03-12T03:46:25.303784+0000","last_unstale":"2024-03-12T03:50:34.683461+0000","last_undegraded":"2024-03-12T03:50:34.683461+0000","last_fullsized":"2024-03-12T03:50:34.683461+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:45.255294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:49:45.255294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:51.355334+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":489,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.692524+0000","last_change":"2024-03-12T03:49:19.918452+0000","last_active":"2024-03-12T03:50:34.692524+0000","last_peered":"2024-03-12T03:50:34.692524+0000","last_clean":"2024-03-12T03:50:34.692524+0000","last_became_active":"2024-03-12T03:46:25.284556+0000","last_became_peered":"2024-03-12T03:46:25.284556+0000","last_unstale":"2024-03-12T03:50:34.692524+0000","last_undegraded":"2024-03-12T03:50:34.692524+0000","last_fullsized":"2024-03-12T03:50:34.692524+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:19.918313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:19.918313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:39.095294+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":341,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.697886+0000","last_change":"2024-03-12T03:48:58.464502+0000","last_active":"2024-03-12T03:50:34.697886+0000","last_peered":"2024-03-12T03:50:34.697886+0000","last_clean":"2024-03-12T03:39:28.535672+0000","last_became_active":"2024-03-12T03:48:58.464502+0000","last_became_peered":"2024-03-12T03:48:58.464502+0000","last_unstale":"2024-03-12T03:50:34.697886+0000","last_undegraded":"2024-03-12T03:50:34.697886+0000","last_fullsized":"2024-03-12T03:48:58.446034+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:55.192996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:55.192996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"410'13371","reported_seq":13471,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:36.268224+0000","last_change":"2024-03-12T03:48:58.465249+0000","last_active":"2024-03-12T03:50:36.268224+0000","last_peered":"2024-03-12T03:50:36.268224+0000","last_clean":"2024-03-12T03:48:57.534492+0000","last_became_active":"2024-03-12T03:48:58.465249+0000","last_became_peered":"2024-03-12T03:48:58.465249+0000","last_unstale":"2024-03-12T03:50:36.268224+0000","last_undegraded":"2024-03-12T03:48:58.444047+0000","last_fullsized":"2024-03-12T03:48:58.443929+0000","mapping_epoch":384,"log_start":"399'9309","ondisk_log_start":"399'9309","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4062,"log_dups_size":0,"ondisk_log_size":4062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:37.845492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241893376,"num_objects":3691,"num_object_clones":0,"num_object_copies":11073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13371,"num_write_kb":545984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":3691}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":855,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.685345+0000","last_change":"2024-03-12T03:49:03.235534+0000","last_active":"2024-03-12T03:50:34.685345+0000","last_peered":"2024-03-12T03:50:34.685345+0000","last_clean":"2024-03-12T03:50:34.685345+0000","last_became_active":"2024-03-12T03:45:18.108107+0000","last_became_peered":"2024-03-12T03:45:18.108107+0000","last_unstale":"2024-03-12T03:50:34.685345+0000","last_undegraded":"2024-03-12T03:50:34.685345+0000","last_fullsized":"2024-03-12T03:50:34.685345+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:03.235270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:03.235270+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'51","reported_seq":750,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.692369+0000","last_change":"2024-03-12T03:49:53.484809+0000","last_active":"2024-03-12T03:50:34.692369+0000","last_peered":"2024-03-12T03:50:34.692369+0000","last_clean":"2024-03-12T03:50:34.692369+0000","last_became_active":"2024-03-12T03:45:18.106503+0000","last_became_peered":"2024-03-12T03:45:18.106503+0000","last_unstale":"2024-03-12T03:50:34.692369+0000","last_undegraded":"2024-03-12T03:50:34.692369+0000","last_fullsized":"2024-03-12T03:50:34.692369+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'51","last_scrub_stamp":"2024-03-12T03:49:53.484646+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:49:53.484646+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:08.271186+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":759,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.698092+0000","last_change":"2024-03-12T03:50:03.440676+0000","last_active":"2024-03-12T03:50:34.698092+0000","last_peered":"2024-03-12T03:50:34.698092+0000","last_clean":"2024-03-12T03:50:34.698092+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:50:34.698092+0000","last_undegraded":"2024-03-12T03:50:34.698092+0000","last_fullsized":"2024-03-12T03:50:34.698092+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:03.440539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:03.440539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:11.407734+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":473,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.693319+0000","last_change":"2024-03-12T03:48:58.459230+0000","last_active":"2024-03-12T03:50:34.693319+0000","last_peered":"2024-03-12T03:50:34.693319+0000","last_clean":"2024-03-12T03:48:53.446740+0000","last_became_active":"2024-03-12T03:48:58.459230+0000","last_became_peered":"2024-03-12T03:48:58.459230+0000","last_unstale":"2024-03-12T03:50:34.693319+0000","last_undegraded":"2024-03-12T03:50:34.693319+0000","last_fullsized":"2024-03-12T03:48:58.447475+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.954731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.954731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":769,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.698595+0000","last_change":"2024-03-12T03:49:10.437704+0000","last_active":"2024-03-12T03:50:34.698595+0000","last_peered":"2024-03-12T03:50:34.698595+0000","last_clean":"2024-03-12T03:50:34.698595+0000","last_became_active":"2024-03-12T03:46:25.292122+0000","last_became_peered":"2024-03-12T03:46:25.292122+0000","last_unstale":"2024-03-12T03:50:34.698595+0000","last_undegraded":"2024-03-12T03:50:34.698595+0000","last_fullsized":"2024-03-12T03:50:34.698595+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:10.437601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:10.437601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"409'13408","reported_seq":13584,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.541761+0000","last_change":"2024-03-12T03:50:05.652788+0000","last_active":"2024-03-12T03:50:33.541761+0000","last_peered":"2024-03-12T03:50:33.541761+0000","last_clean":"2024-03-12T03:50:33.541761+0000","last_became_active":"2024-03-12T03:47:58.027372+0000","last_became_peered":"2024-03-12T03:47:58.027372+0000","last_unstale":"2024-03-12T03:50:33.541761+0000","last_undegraded":"2024-03-12T03:50:33.541761+0000","last_fullsized":"2024-03-12T03:50:33.541761+0000","mapping_epoch":366,"log_start":"398'9000","ondisk_log_start":"398'9000","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'10749","last_scrub_stamp":"2024-03-12T03:50:05.652456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:50:05.652456+0000","objects_scrubbed":6904,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:06.690702+0000","scrub_duration":2133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":4160,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13408,"num_write_kb":562176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":684,"reported_epoch":409,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:33.478083+0000","last_change":"2024-03-12T03:48:58.449413+0000","last_active":"2024-03-12T03:50:33.478083+0000","last_peered":"2024-03-12T03:50:33.478083+0000","last_clean":"2024-03-12T03:48:53.447854+0000","last_became_active":"2024-03-12T03:48:58.449413+0000","last_became_peered":"2024-03-12T03:48:58.449413+0000","last_unstale":"2024-03-12T03:50:33.478083+0000","last_undegraded":"2024-03-12T03:48:58.440707+0000","last_fullsized":"2024-03-12T03:48:58.440595+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:51.818256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:51.818256+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358200+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":490,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.693320+0000","last_change":"2024-03-12T03:48:58.461069+0000","last_active":"2024-03-12T03:50:34.693320+0000","last_peered":"2024-03-12T03:50:34.693320+0000","last_clean":"2024-03-12T03:48:53.445898+0000","last_became_active":"2024-03-12T03:48:58.461069+0000","last_became_peered":"2024-03-12T03:48:58.461069+0000","last_unstale":"2024-03-12T03:50:34.693320+0000","last_undegraded":"2024-03-12T03:50:34.693320+0000","last_fullsized":"2024-03-12T03:48:58.444677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.981014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.981014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"269'30","reported_seq":830,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.689720+0000","last_change":"2024-03-12T03:49:16.863752+0000","last_active":"2024-03-12T03:50:34.689720+0000","last_peered":"2024-03-12T03:50:34.689720+0000","last_clean":"2024-03-12T03:50:34.689720+0000","last_became_active":"2024-03-12T03:46:25.287885+0000","last_became_peered":"2024-03-12T03:46:25.287885+0000","last_unstale":"2024-03-12T03:50:34.689720+0000","last_undegraded":"2024-03-12T03:50:34.689720+0000","last_fullsized":"2024-03-12T03:50:34.689720+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'30","last_scrub_stamp":"2024-03-12T03:49:16.863622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:16.863622+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.298379+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":759,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.699064+0000","last_change":"2024-03-12T03:49:08.492723+0000","last_active":"2024-03-12T03:50:34.699064+0000","last_peered":"2024-03-12T03:50:34.699064+0000","last_clean":"2024-03-12T03:50:34.699064+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:50:34.699064+0000","last_undegraded":"2024-03-12T03:50:34.699064+0000","last_fullsized":"2024-03-12T03:50:34.699064+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:08.492606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:49:08.492606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":482,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.485239+0000","last_change":"2024-03-12T03:50:01.616545+0000","last_active":"2024-03-12T03:50:33.485239+0000","last_peered":"2024-03-12T03:50:33.485239+0000","last_clean":"2024-03-12T03:50:33.485239+0000","last_became_active":"2024-03-12T03:46:25.280784+0000","last_became_peered":"2024-03-12T03:46:25.280784+0000","last_unstale":"2024-03-12T03:50:33.485239+0000","last_undegraded":"2024-03-12T03:50:33.485239+0000","last_fullsized":"2024-03-12T03:50:33.485239+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:01.616355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:01.616355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:23.896666+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"410'13674","reported_seq":13775,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:36.625742+0000","last_change":"2024-03-12T03:48:58.457494+0000","last_active":"2024-03-12T03:50:36.625742+0000","last_peered":"2024-03-12T03:50:36.625742+0000","last_clean":"2024-03-12T03:48:57.506664+0000","last_became_active":"2024-03-12T03:48:58.457494+0000","last_became_peered":"2024-03-12T03:48:58.457494+0000","last_unstale":"2024-03-12T03:50:36.625742+0000","last_undegraded":"2024-03-12T03:48:58.442176+0000","last_fullsized":"2024-03-12T03:48:58.442057+0000","mapping_epoch":384,"log_start":"397'8900","ondisk_log_start":"397'8900","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4774,"log_dups_size":0,"ondisk_log_size":4774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.231352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251789312,"num_objects":3842,"num_object_clones":0,"num_object_copies":11526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13674,"num_write_kb":560512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":3842}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":679,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.697669+0000","last_change":"2024-03-12T03:50:01.383042+0000","last_active":"2024-03-12T03:50:34.697669+0000","last_peered":"2024-03-12T03:50:34.697669+0000","last_clean":"2024-03-12T03:50:34.697669+0000","last_became_active":"2024-03-12T03:44:24.373190+0000","last_became_peered":"2024-03-12T03:44:24.373190+0000","last_unstale":"2024-03-12T03:50:34.697669+0000","last_undegraded":"2024-03-12T03:50:34.697669+0000","last_fullsized":"2024-03-12T03:50:34.697669+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:01.382914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:01.382914+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:20.832601+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":959,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.698568+0000","last_change":"2024-03-12T03:48:18.441645+0000","last_active":"2024-03-12T03:50:34.698568+0000","last_peered":"2024-03-12T03:50:34.698568+0000","last_clean":"2024-03-12T03:50:34.698568+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:50:34.698568+0000","last_undegraded":"2024-03-12T03:50:34.698568+0000","last_fullsized":"2024-03-12T03:50:34.698568+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:48:18.441562+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:48:18.441562+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"269'48","reported_seq":821,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.682395+0000","last_change":"2024-03-12T03:49:01.279886+0000","last_active":"2024-03-12T03:50:34.682395+0000","last_peered":"2024-03-12T03:50:34.682395+0000","last_clean":"2024-03-12T03:50:34.682395+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:50:34.682395+0000","last_undegraded":"2024-03-12T03:50:34.682395+0000","last_fullsized":"2024-03-12T03:50:34.682395+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'48","last_scrub_stamp":"2024-03-12T03:49:01.279743+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:49:01.279743+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":758,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:34.698730+0000","last_change":"2024-03-12T03:48:58.456357+0000","last_active":"2024-03-12T03:50:34.698730+0000","last_peered":"2024-03-12T03:50:34.698730+0000","last_clean":"2024-03-12T03:48:53.450296+0000","last_became_active":"2024-03-12T03:48:58.456357+0000","last_became_peered":"2024-03-12T03:48:58.456357+0000","last_unstale":"2024-03-12T03:50:34.698730+0000","last_undegraded":"2024-03-12T03:48:58.445880+0000","last_fullsized":"2024-03-12T03:48:58.445769+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:47:05.588582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:05.588582+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":732,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.683160+0000","last_change":"2024-03-12T03:48:58.452652+0000","last_active":"2024-03-12T03:50:34.683160+0000","last_peered":"2024-03-12T03:50:34.683160+0000","last_clean":"2024-03-12T03:48:53.472055+0000","last_became_active":"2024-03-12T03:48:58.452652+0000","last_became_peered":"2024-03-12T03:48:58.452652+0000","last_unstale":"2024-03-12T03:50:34.683160+0000","last_undegraded":"2024-03-12T03:50:34.683160+0000","last_fullsized":"2024-03-12T03:48:58.441674+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:59.567890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.567890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.225872+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":562,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.682236+0000","last_change":"2024-03-12T03:50:06.385123+0000","last_active":"2024-03-12T03:50:34.682236+0000","last_peered":"2024-03-12T03:50:34.682236+0000","last_clean":"2024-03-12T03:50:34.682236+0000","last_became_active":"2024-03-12T03:44:24.365642+0000","last_became_peered":"2024-03-12T03:44:24.365642+0000","last_unstale":"2024-03-12T03:50:34.682236+0000","last_undegraded":"2024-03-12T03:50:34.682236+0000","last_fullsized":"2024-03-12T03:50:34.682236+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:06.384808+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:50:06.384808+0000","last_clean_scrub_stamp":"2024-03-12T03:50:06.384808+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:14.012985+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"269'18","reported_seq":545,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.478643+0000","last_change":"2024-03-12T03:49:16.403355+0000","last_active":"2024-03-12T03:50:33.478643+0000","last_peered":"2024-03-12T03:50:33.478643+0000","last_clean":"2024-03-12T03:50:33.478643+0000","last_became_active":"2024-03-12T03:46:25.288573+0000","last_became_peered":"2024-03-12T03:46:25.288573+0000","last_unstale":"2024-03-12T03:50:33.478643+0000","last_undegraded":"2024-03-12T03:50:33.478643+0000","last_fullsized":"2024-03-12T03:50:33.478643+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-03-12T03:49:16.403227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:16.403227+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:33.599737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":61,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.682481+0000","last_change":"2024-03-12T03:48:58.452359+0000","last_active":"2024-03-12T03:50:34.682481+0000","last_peered":"2024-03-12T03:50:34.682481+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:48:58.452359+0000","last_became_peered":"2024-03-12T03:48:58.452359+0000","last_unstale":"2024-03-12T03:50:34.682481+0000","last_undegraded":"2024-03-12T03:50:34.682481+0000","last_fullsized":"2024-03-12T03:48:58.441273+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:10.400135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.400135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.225854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":751,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.684612+0000","last_change":"2024-03-12T03:49:20.206032+0000","last_active":"2024-03-12T03:50:34.684612+0000","last_peered":"2024-03-12T03:50:34.684612+0000","last_clean":"2024-03-12T03:50:34.684612+0000","last_became_active":"2024-03-12T03:46:25.293521+0000","last_became_peered":"2024-03-12T03:46:25.293521+0000","last_unstale":"2024-03-12T03:50:34.684612+0000","last_undegraded":"2024-03-12T03:50:34.684612+0000","last_fullsized":"2024-03-12T03:50:34.684612+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:20.205902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:20.205902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":747,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.697771+0000","last_change":"2024-03-12T03:49:58.457008+0000","last_active":"2024-03-12T03:50:34.697771+0000","last_peered":"2024-03-12T03:50:34.697771+0000","last_clean":"2024-03-12T03:50:34.697771+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:50:34.697771+0000","last_undegraded":"2024-03-12T03:50:34.697771+0000","last_fullsized":"2024-03-12T03:50:34.697771+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:58.456901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:49:58.456901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:58.456901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:13.229872+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":894,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.694047+0000","last_change":"2024-03-12T03:49:59.481031+0000","last_active":"2024-03-12T03:50:34.694047+0000","last_peered":"2024-03-12T03:50:34.694047+0000","last_clean":"2024-03-12T03:50:34.694047+0000","last_became_active":"2024-03-12T03:46:28.316406+0000","last_became_peered":"2024-03-12T03:46:28.316406+0000","last_unstale":"2024-03-12T03:50:34.694047+0000","last_undegraded":"2024-03-12T03:50:34.694047+0000","last_fullsized":"2024-03-12T03:50:34.694047+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:59.480806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:59.480806+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:14.990823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"269'27","reported_seq":679,"reported_epoch":409,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:33.480014+0000","last_change":"2024-03-12T03:48:58.462615+0000","last_active":"2024-03-12T03:50:33.480014+0000","last_peered":"2024-03-12T03:50:33.480014+0000","last_clean":"2024-03-12T03:43:13.333279+0000","last_became_active":"2024-03-12T03:48:58.462615+0000","last_became_peered":"2024-03-12T03:48:58.462615+0000","last_unstale":"2024-03-12T03:50:33.480014+0000","last_undegraded":"2024-03-12T03:48:58.443892+0000","last_fullsized":"2024-03-12T03:48:58.443761+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:48:06.104297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.104297+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358300+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":759,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:35.416860+0000","last_change":"2024-03-12T03:50:35.416894+0000","last_active":"2024-03-12T03:50:35.416860+0000","last_peered":"2024-03-12T03:50:35.416860+0000","last_clean":"2024-03-12T03:50:35.416860+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:50:35.416860+0000","last_undegraded":"2024-03-12T03:50:35.416860+0000","last_fullsized":"2024-03-12T03:50:35.416860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:35.416780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.416780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.856705+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":508,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.478794+0000","last_change":"2024-03-12T03:49:17.403558+0000","last_active":"2024-03-12T03:50:33.478794+0000","last_peered":"2024-03-12T03:50:33.478794+0000","last_clean":"2024-03-12T03:50:33.478794+0000","last_became_active":"2024-03-12T03:46:25.286866+0000","last_became_peered":"2024-03-12T03:46:25.286866+0000","last_unstale":"2024-03-12T03:50:33.478794+0000","last_undegraded":"2024-03-12T03:50:33.478794+0000","last_fullsized":"2024-03-12T03:50:33.478794+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:17.403434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:49:17.403434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358353+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":478,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.693221+0000","last_change":"2024-03-12T03:49:02.083280+0000","last_active":"2024-03-12T03:50:34.693221+0000","last_peered":"2024-03-12T03:50:34.693221+0000","last_clean":"2024-03-12T03:50:34.693221+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:50:34.693221+0000","last_undegraded":"2024-03-12T03:50:34.693221+0000","last_fullsized":"2024-03-12T03:50:34.693221+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:02.083177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:02.083177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":562,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.682296+0000","last_change":"2024-03-12T03:50:33.243653+0000","last_active":"2024-03-12T03:50:34.682296+0000","last_peered":"2024-03-12T03:50:34.682296+0000","last_clean":"2024-03-12T03:50:34.682296+0000","last_became_active":"2024-03-12T03:44:24.376141+0000","last_became_peered":"2024-03-12T03:44:24.376141+0000","last_unstale":"2024-03-12T03:50:34.682296+0000","last_undegraded":"2024-03-12T03:50:34.682296+0000","last_fullsized":"2024-03-12T03:50:34.682296+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:33.243526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:33.243526+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:36.732091+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"269'21","reported_seq":809,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.684204+0000","last_change":"2024-03-12T03:49:19.221180+0000","last_active":"2024-03-12T03:50:34.684204+0000","last_peered":"2024-03-12T03:50:34.684204+0000","last_clean":"2024-03-12T03:50:34.684204+0000","last_became_active":"2024-03-12T03:46:25.298007+0000","last_became_peered":"2024-03-12T03:46:25.298007+0000","last_unstale":"2024-03-12T03:50:34.684204+0000","last_undegraded":"2024-03-12T03:50:34.684204+0000","last_fullsized":"2024-03-12T03:50:34.684204+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'21","last_scrub_stamp":"2024-03-12T03:49:19.221045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:19.221045+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:42.537709+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":635,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.696389+0000","last_change":"2024-03-12T03:48:58.462149+0000","last_active":"2024-03-12T03:50:34.696389+0000","last_peered":"2024-03-12T03:50:34.696389+0000","last_clean":"2024-03-12T03:43:13.331412+0000","last_became_active":"2024-03-12T03:48:58.462149+0000","last_became_peered":"2024-03-12T03:48:58.462149+0000","last_unstale":"2024-03-12T03:50:34.696389+0000","last_undegraded":"2024-03-12T03:50:34.696389+0000","last_fullsized":"2024-03-12T03:48:58.443909+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"332'14","reported_seq":745,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.478843+0000","last_change":"2024-03-12T03:49:10.474096+0000","last_active":"2024-03-12T03:50:33.478843+0000","last_peered":"2024-03-12T03:50:33.478843+0000","last_clean":"2024-03-12T03:50:33.478843+0000","last_became_active":"2024-03-12T03:46:25.300463+0000","last_became_peered":"2024-03-12T03:46:25.300463+0000","last_unstale":"2024-03-12T03:50:33.478843+0000","last_undegraded":"2024-03-12T03:50:33.478843+0000","last_fullsized":"2024-03-12T03:50:33.478843+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'14","last_scrub_stamp":"2024-03-12T03:49:10.473983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:10.473983+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.746305+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":759,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.687632+0000","last_change":"2024-03-12T03:50:04.042481+0000","last_active":"2024-03-12T03:50:34.687632+0000","last_peered":"2024-03-12T03:50:34.687632+0000","last_clean":"2024-03-12T03:50:34.687632+0000","last_became_active":"2024-03-12T03:39:33.265499+0000","last_became_peered":"2024-03-12T03:39:33.265499+0000","last_unstale":"2024-03-12T03:50:34.687632+0000","last_undegraded":"2024-03-12T03:50:34.687632+0000","last_fullsized":"2024-03-12T03:50:34.687632+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:04.042226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:04.042226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:08.428366+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":684,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.685727+0000","last_change":"2024-03-12T03:48:58.458244+0000","last_active":"2024-03-12T03:50:34.685727+0000","last_peered":"2024-03-12T03:50:34.685727+0000","last_clean":"2024-03-12T03:44:19.377035+0000","last_became_active":"2024-03-12T03:48:58.458244+0000","last_became_peered":"2024-03-12T03:48:58.458244+0000","last_unstale":"2024-03-12T03:50:34.685727+0000","last_undegraded":"2024-03-12T03:50:34.685727+0000","last_fullsized":"2024-03-12T03:48:58.445185+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:16.223801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:16.223801+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:37.213736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"403'39","reported_seq":615,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.478480+0000","last_change":"2024-03-12T03:50:06.535902+0000","last_active":"2024-03-12T03:50:33.478480+0000","last_peered":"2024-03-12T03:50:33.478480+0000","last_clean":"2024-03-12T03:50:33.478480+0000","last_became_active":"2024-03-12T03:46:25.299798+0000","last_became_peered":"2024-03-12T03:46:25.299798+0000","last_unstale":"2024-03-12T03:50:33.478480+0000","last_undegraded":"2024-03-12T03:50:33.478480+0000","last_fullsized":"2024-03-12T03:50:33.478480+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'39","last_scrub_stamp":"2024-03-12T03:50:06.535772+0000","last_deep_scrub":"403'39","last_deep_scrub_stamp":"2024-03-12T03:50:06.535772+0000","last_clean_scrub_stamp":"2024-03-12T03:50:06.535772+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:13.213921+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":786,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.697114+0000","last_change":"2024-03-12T03:50:00.385862+0000","last_active":"2024-03-12T03:50:34.697114+0000","last_peered":"2024-03-12T03:50:34.697114+0000","last_clean":"2024-03-12T03:50:34.697114+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:50:34.697114+0000","last_undegraded":"2024-03-12T03:50:34.697114+0000","last_fullsized":"2024-03-12T03:50:34.697114+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:50:00.385750+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:50:00.385750+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:11.005858+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"385'9","reported_seq":952,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:34.684073+0000","last_change":"2024-03-12T03:48:58.448464+0000","last_active":"2024-03-12T03:50:34.684073+0000","last_peered":"2024-03-12T03:50:34.684073+0000","last_clean":"2024-03-12T03:48:56.813639+0000","last_became_active":"2024-03-12T03:48:58.448464+0000","last_became_peered":"2024-03-12T03:48:58.448464+0000","last_unstale":"2024-03-12T03:50:34.684073+0000","last_undegraded":"2024-03-12T03:48:58.442644+0000","last_fullsized":"2024-03-12T03:48:58.442315+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:53.458170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.458170+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.225880+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":623,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.698390+0000","last_change":"2024-03-12T03:48:58.463189+0000","last_active":"2024-03-12T03:50:34.698390+0000","last_peered":"2024-03-12T03:50:34.698390+0000","last_clean":"2024-03-12T03:43:13.332853+0000","last_became_active":"2024-03-12T03:48:58.463189+0000","last_became_peered":"2024-03-12T03:48:58.463189+0000","last_unstale":"2024-03-12T03:50:34.698390+0000","last_undegraded":"2024-03-12T03:50:34.698390+0000","last_fullsized":"2024-03-12T03:48:58.445060+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.079569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.079569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"269'19","reported_seq":786,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.688837+0000","last_change":"2024-03-12T03:49:08.759600+0000","last_active":"2024-03-12T03:50:34.688837+0000","last_peered":"2024-03-12T03:50:34.688837+0000","last_clean":"2024-03-12T03:50:34.688837+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:50:34.688837+0000","last_undegraded":"2024-03-12T03:50:34.688837+0000","last_fullsized":"2024-03-12T03:50:34.688837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'19","last_scrub_stamp":"2024-03-12T03:49:08.759497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:08.759497+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":759,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.689287+0000","last_change":"2024-03-12T03:49:50.830712+0000","last_active":"2024-03-12T03:50:34.689287+0000","last_peered":"2024-03-12T03:50:34.689287+0000","last_clean":"2024-03-12T03:50:34.689287+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:50:34.689287+0000","last_undegraded":"2024-03-12T03:50:34.689287+0000","last_fullsized":"2024-03-12T03:50:34.689287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:50.830611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.830611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:52.147471+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":760,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.694016+0000","last_change":"2024-03-12T03:49:08.586759+0000","last_active":"2024-03-12T03:50:34.694016+0000","last_peered":"2024-03-12T03:50:34.694016+0000","last_clean":"2024-03-12T03:50:34.694016+0000","last_became_active":"2024-03-12T03:46:25.288050+0000","last_became_peered":"2024-03-12T03:46:25.288050+0000","last_unstale":"2024-03-12T03:50:34.694016+0000","last_undegraded":"2024-03-12T03:50:34.694016+0000","last_fullsized":"2024-03-12T03:50:34.694016+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:08.586525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:08.586525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":527,"reported_epoch":409,"state":"active+undersized","last_fresh":"2024-03-12T03:50:33.477972+0000","last_change":"2024-03-12T03:48:58.456788+0000","last_active":"2024-03-12T03:50:33.477972+0000","last_peered":"2024-03-12T03:50:33.477972+0000","last_clean":"2024-03-12T03:48:53.447742+0000","last_became_active":"2024-03-12T03:48:58.456788+0000","last_became_peered":"2024-03-12T03:48:58.456788+0000","last_unstale":"2024-03-12T03:50:33.477972+0000","last_undegraded":"2024-03-12T03:50:33.477972+0000","last_fullsized":"2024-03-12T03:48:58.442213+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.433184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.433184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358238+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"269'15","reported_seq":350,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.682358+0000","last_change":"2024-03-12T03:49:56.329520+0000","last_active":"2024-03-12T03:50:34.682358+0000","last_peered":"2024-03-12T03:50:34.682358+0000","last_clean":"2024-03-12T03:50:34.682358+0000","last_became_active":"2024-03-12T03:43:01.037348+0000","last_became_peered":"2024-03-12T03:43:01.037348+0000","last_unstale":"2024-03-12T03:50:34.682358+0000","last_undegraded":"2024-03-12T03:50:34.682358+0000","last_fullsized":"2024-03-12T03:50:34.682358+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'15","last_scrub_stamp":"2024-03-12T03:49:56.329393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:56.329393+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:21.835051+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":769,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.697871+0000","last_change":"2024-03-12T03:49:12.446186+0000","last_active":"2024-03-12T03:50:34.697871+0000","last_peered":"2024-03-12T03:50:34.697871+0000","last_clean":"2024-03-12T03:50:34.697871+0000","last_became_active":"2024-03-12T03:46:25.283602+0000","last_became_peered":"2024-03-12T03:46:25.283602+0000","last_unstale":"2024-03-12T03:50:34.697871+0000","last_undegraded":"2024-03-12T03:50:34.697871+0000","last_fullsized":"2024-03-12T03:50:34.697871+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:12.446075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:49:12.446075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":539,"reported_epoch":409,"state":"active+undersized","last_fresh":"2024-03-12T03:50:33.484710+0000","last_change":"2024-03-12T03:48:58.460970+0000","last_active":"2024-03-12T03:50:33.484710+0000","last_peered":"2024-03-12T03:50:33.484710+0000","last_clean":"2024-03-12T03:48:53.448497+0000","last_became_active":"2024-03-12T03:48:58.460970+0000","last_became_peered":"2024-03-12T03:48:58.460970+0000","last_unstale":"2024-03-12T03:50:33.484710+0000","last_undegraded":"2024-03-12T03:50:33.484710+0000","last_fullsized":"2024-03-12T03:48:58.444135+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_clean_scrub_stamp":"2024-03-12T03:48:10.625582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":59,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.682644+0000","last_change":"2024-03-12T03:48:58.453783+0000","last_active":"2024-03-12T03:50:34.682644+0000","last_peered":"2024-03-12T03:50:34.682644+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.453783+0000","last_became_peered":"2024-03-12T03:48:58.453783+0000","last_unstale":"2024-03-12T03:50:34.682644+0000","last_undegraded":"2024-03-12T03:50:34.682644+0000","last_fullsized":"2024-03-12T03:48:58.441749+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:13.228117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:48:13.228117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.231390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":676,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.692497+0000","last_change":"2024-03-12T03:49:09.041208+0000","last_active":"2024-03-12T03:50:34.692497+0000","last_peered":"2024-03-12T03:50:34.692497+0000","last_clean":"2024-03-12T03:50:34.692497+0000","last_became_active":"2024-03-12T03:44:24.387247+0000","last_became_peered":"2024-03-12T03:44:24.387247+0000","last_unstale":"2024-03-12T03:50:34.692497+0000","last_undegraded":"2024-03-12T03:50:34.692497+0000","last_fullsized":"2024-03-12T03:50:34.692497+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:09.041066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:09.041066+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"269'26","reported_seq":555,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.478238+0000","last_change":"2024-03-12T03:49:47.515912+0000","last_active":"2024-03-12T03:50:33.478238+0000","last_peered":"2024-03-12T03:50:33.478238+0000","last_clean":"2024-03-12T03:50:33.478238+0000","last_became_active":"2024-03-12T03:46:25.287095+0000","last_became_peered":"2024-03-12T03:46:25.287095+0000","last_unstale":"2024-03-12T03:50:33.478238+0000","last_undegraded":"2024-03-12T03:50:33.478238+0000","last_fullsized":"2024-03-12T03:50:33.478238+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:49:47.515799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:47.515799+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:59.734985+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":757,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.684692+0000","last_change":"2024-03-12T03:50:00.379727+0000","last_active":"2024-03-12T03:50:34.684692+0000","last_peered":"2024-03-12T03:50:34.684692+0000","last_clean":"2024-03-12T03:50:34.684692+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:50:34.684692+0000","last_undegraded":"2024-03-12T03:50:34.684692+0000","last_fullsized":"2024-03-12T03:50:34.684692+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:00.379623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:50:00.379623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:07.116347+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":751,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:36.405387+0000","last_change":"2024-03-12T03:50:36.405438+0000","last_active":"2024-03-12T03:50:36.405387+0000","last_peered":"2024-03-12T03:50:36.405387+0000","last_clean":"2024-03-12T03:50:36.405387+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:50:36.405387+0000","last_undegraded":"2024-03-12T03:50:36.405387+0000","last_fullsized":"2024-03-12T03:50:36.405387+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:36.405298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:36.405298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:39.013601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":554,"reported_epoch":409,"state":"active+undersized","last_fresh":"2024-03-12T03:50:33.485336+0000","last_change":"2024-03-12T03:48:58.449798+0000","last_active":"2024-03-12T03:50:33.485336+0000","last_peered":"2024-03-12T03:50:33.485336+0000","last_clean":"2024-03-12T03:48:53.448767+0000","last_became_active":"2024-03-12T03:48:58.449798+0000","last_became_peered":"2024-03-12T03:48:58.449798+0000","last_unstale":"2024-03-12T03:50:33.485336+0000","last_undegraded":"2024-03-12T03:50:33.485336+0000","last_fullsized":"2024-03-12T03:48:58.443857+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"269'10","reported_seq":526,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.478950+0000","last_change":"2024-03-12T03:48:06.646761+0000","last_active":"2024-03-12T03:50:33.478950+0000","last_peered":"2024-03-12T03:50:33.478950+0000","last_clean":"2024-03-12T03:50:33.478950+0000","last_became_active":"2024-03-12T03:46:25.286315+0000","last_became_peered":"2024-03-12T03:46:25.286315+0000","last_unstale":"2024-03-12T03:50:33.478950+0000","last_undegraded":"2024-03-12T03:50:33.478950+0000","last_fullsized":"2024-03-12T03:50:33.478950+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'10","last_scrub_stamp":"2024-03-12T03:48:06.646668+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.646668+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":635,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.685679+0000","last_change":"2024-03-12T03:48:58.463096+0000","last_active":"2024-03-12T03:50:34.685679+0000","last_peered":"2024-03-12T03:50:34.685679+0000","last_clean":"2024-03-12T03:43:13.332505+0000","last_became_active":"2024-03-12T03:48:58.463096+0000","last_became_peered":"2024-03-12T03:48:58.463096+0000","last_unstale":"2024-03-12T03:50:34.685679+0000","last_undegraded":"2024-03-12T03:50:34.685679+0000","last_fullsized":"2024-03-12T03:48:58.443705+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.851271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.851271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.225903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":214,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.687145+0000","last_change":"2024-03-12T03:49:00.707749+0000","last_active":"2024-03-12T03:50:34.687145+0000","last_peered":"2024-03-12T03:50:34.687145+0000","last_clean":"2024-03-12T03:50:34.687145+0000","last_became_active":"2024-03-12T03:45:18.109983+0000","last_became_peered":"2024-03-12T03:45:18.109983+0000","last_unstale":"2024-03-12T03:50:34.687145+0000","last_undegraded":"2024-03-12T03:50:34.687145+0000","last_fullsized":"2024-03-12T03:50:34.687145+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:00.707518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:00.707518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":751,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.696880+0000","last_change":"2024-03-12T03:49:04.449644+0000","last_active":"2024-03-12T03:50:34.696880+0000","last_peered":"2024-03-12T03:50:34.696880+0000","last_clean":"2024-03-12T03:50:34.696880+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:50:34.696880+0000","last_undegraded":"2024-03-12T03:50:34.696880+0000","last_fullsized":"2024-03-12T03:50:34.696880+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:04.449485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:04.449485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"269'31","reported_seq":722,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.686201+0000","last_change":"2024-03-12T03:49:53.291968+0000","last_active":"2024-03-12T03:50:34.686201+0000","last_peered":"2024-03-12T03:50:34.686201+0000","last_clean":"2024-03-12T03:50:34.686201+0000","last_became_active":"2024-03-12T03:45:18.107047+0000","last_became_peered":"2024-03-12T03:45:18.107047+0000","last_unstale":"2024-03-12T03:50:34.686201+0000","last_undegraded":"2024-03-12T03:50:34.686201+0000","last_fullsized":"2024-03-12T03:50:34.686201+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:49:53.291824+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:49:53.291824+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:13.883282+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":555,"reported_epoch":409,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:33.479358+0000","last_change":"2024-03-12T03:48:58.454467+0000","last_active":"2024-03-12T03:50:33.479358+0000","last_peered":"2024-03-12T03:50:33.479358+0000","last_clean":"2024-03-12T03:48:53.477110+0000","last_became_active":"2024-03-12T03:48:58.454467+0000","last_became_peered":"2024-03-12T03:48:58.454467+0000","last_unstale":"2024-03-12T03:50:33.479358+0000","last_undegraded":"2024-03-12T03:48:58.439427+0000","last_fullsized":"2024-03-12T03:48:58.439320+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:48:00.458460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.458460+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":346,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.699143+0000","last_change":"2024-03-12T03:48:58.461994+0000","last_active":"2024-03-12T03:50:34.699143+0000","last_peered":"2024-03-12T03:50:34.699143+0000","last_clean":"2024-03-12T03:39:31.213817+0000","last_became_active":"2024-03-12T03:48:58.461994+0000","last_became_peered":"2024-03-12T03:48:58.461994+0000","last_unstale":"2024-03-12T03:50:34.699143+0000","last_undegraded":"2024-03-12T03:50:34.699143+0000","last_fullsized":"2024-03-12T03:48:58.446456+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:09.160277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:09.160277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":745,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.694889+0000","last_change":"2024-03-12T03:49:07.757495+0000","last_active":"2024-03-12T03:50:34.694889+0000","last_peered":"2024-03-12T03:50:34.694889+0000","last_clean":"2024-03-12T03:50:34.694889+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:50:34.694889+0000","last_undegraded":"2024-03-12T03:50:34.694889+0000","last_fullsized":"2024-03-12T03:50:34.694889+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:07.757361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:07.757361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"269'27","reported_seq":552,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:34.699360+0000","last_change":"2024-03-12T03:48:58.462533+0000","last_active":"2024-03-12T03:50:34.699360+0000","last_peered":"2024-03-12T03:50:34.699360+0000","last_clean":"2024-03-12T03:43:13.333148+0000","last_became_active":"2024-03-12T03:48:58.462533+0000","last_became_peered":"2024-03-12T03:48:58.462533+0000","last_unstale":"2024-03-12T03:50:34.699360+0000","last_undegraded":"2024-03-12T03:48:58.446989+0000","last_fullsized":"2024-03-12T03:48:58.446871+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:15.795515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.795515+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"351'2","reported_seq":770,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.691893+0000","last_change":"2024-03-12T03:49:14.611362+0000","last_active":"2024-03-12T03:50:34.691893+0000","last_peered":"2024-03-12T03:50:34.691893+0000","last_clean":"2024-03-12T03:50:34.691893+0000","last_became_active":"2024-03-12T03:46:25.283344+0000","last_became_peered":"2024-03-12T03:46:25.283344+0000","last_unstale":"2024-03-12T03:50:34.691893+0000","last_undegraded":"2024-03-12T03:50:34.691893+0000","last_fullsized":"2024-03-12T03:50:34.691893+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'2","last_scrub_stamp":"2024-03-12T03:49:14.611213+0000","last_deep_scrub":"351'2","last_deep_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_clean_scrub_stamp":"2024-03-12T03:49:14.611213+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.636078+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'9","reported_seq":305,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.687810+0000","last_change":"2024-03-12T03:48:02.854911+0000","last_active":"2024-03-12T03:50:34.687810+0000","last_peered":"2024-03-12T03:50:34.687810+0000","last_clean":"2024-03-12T03:50:34.687810+0000","last_became_active":"2024-03-12T03:45:18.108935+0000","last_became_peered":"2024-03-12T03:45:18.108935+0000","last_unstale":"2024-03-12T03:50:34.687810+0000","last_undegraded":"2024-03-12T03:50:34.687810+0000","last_fullsized":"2024-03-12T03:50:34.687810+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'9","last_scrub_stamp":"2024-03-12T03:48:02.854820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:02.854820+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":767,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.697936+0000","last_change":"2024-03-12T03:49:48.376454+0000","last_active":"2024-03-12T03:50:34.697936+0000","last_peered":"2024-03-12T03:50:34.697936+0000","last_clean":"2024-03-12T03:50:34.697936+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:50:34.697936+0000","last_undegraded":"2024-03-12T03:50:34.697936+0000","last_fullsized":"2024-03-12T03:50:34.697936+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:49:48.375999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:48.375999+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:49.566799+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"269'24","reported_seq":592,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.477392+0000","last_change":"2024-03-12T03:50:05.585005+0000","last_active":"2024-03-12T03:50:33.477392+0000","last_peered":"2024-03-12T03:50:33.477392+0000","last_clean":"2024-03-12T03:50:33.477392+0000","last_became_active":"2024-03-12T03:46:25.283193+0000","last_became_peered":"2024-03-12T03:46:25.283193+0000","last_unstale":"2024-03-12T03:50:33.477392+0000","last_undegraded":"2024-03-12T03:50:33.477392+0000","last_fullsized":"2024-03-12T03:50:33.477392+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:50:05.584740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:05.584740+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.476261+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":770,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.693584+0000","last_change":"2024-03-12T03:48:01.902683+0000","last_active":"2024-03-12T03:50:34.693584+0000","last_peered":"2024-03-12T03:50:34.693584+0000","last_clean":"2024-03-12T03:50:34.693584+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:50:34.693584+0000","last_undegraded":"2024-03-12T03:50:34.693584+0000","last_fullsized":"2024-03-12T03:50:34.693584+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:01.902570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:01.902570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":772,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.689495+0000","last_change":"2024-03-12T03:49:19.894833+0000","last_active":"2024-03-12T03:50:34.689495+0000","last_peered":"2024-03-12T03:50:34.689495+0000","last_clean":"2024-03-12T03:50:34.689495+0000","last_became_active":"2024-03-12T03:46:25.301691+0000","last_became_peered":"2024-03-12T03:46:25.301691+0000","last_unstale":"2024-03-12T03:50:34.689495+0000","last_undegraded":"2024-03-12T03:50:34.689495+0000","last_fullsized":"2024-03-12T03:50:34.689495+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:19.894587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:19.894587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":561,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.479565+0000","last_change":"2024-03-12T03:49:54.607508+0000","last_active":"2024-03-12T03:50:33.479565+0000","last_peered":"2024-03-12T03:50:33.479565+0000","last_clean":"2024-03-12T03:50:33.479565+0000","last_became_active":"2024-03-12T03:46:25.296764+0000","last_became_peered":"2024-03-12T03:46:25.296764+0000","last_unstale":"2024-03-12T03:50:33.479565+0000","last_undegraded":"2024-03-12T03:50:33.479565+0000","last_fullsized":"2024-03-12T03:50:33.479565+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:54.607394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.607394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:55.661971+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"269'27","reported_seq":537,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.693038+0000","last_change":"2024-03-12T03:49:57.717659+0000","last_active":"2024-03-12T03:50:34.693038+0000","last_peered":"2024-03-12T03:50:34.693038+0000","last_clean":"2024-03-12T03:50:34.693038+0000","last_became_active":"2024-03-12T03:46:25.301700+0000","last_became_peered":"2024-03-12T03:46:25.301700+0000","last_unstale":"2024-03-12T03:50:34.693038+0000","last_undegraded":"2024-03-12T03:50:34.693038+0000","last_fullsized":"2024-03-12T03:50:34.693038+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:49:57.717507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:57.717507+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:00.235756+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":59,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.688183+0000","last_change":"2024-03-12T03:48:58.455613+0000","last_active":"2024-03-12T03:50:34.688183+0000","last_peered":"2024-03-12T03:50:34.688183+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:48:58.455613+0000","last_became_peered":"2024-03-12T03:48:58.455613+0000","last_unstale":"2024-03-12T03:50:34.688183+0000","last_undegraded":"2024-03-12T03:50:34.688183+0000","last_fullsized":"2024-03-12T03:48:58.445944+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.797296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.797296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:37.845431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":743,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.694609+0000","last_change":"2024-03-12T03:48:58.458438+0000","last_active":"2024-03-12T03:50:34.694609+0000","last_peered":"2024-03-12T03:50:34.694609+0000","last_clean":"2024-03-12T03:48:53.448127+0000","last_became_active":"2024-03-12T03:48:58.458438+0000","last_became_peered":"2024-03-12T03:48:58.458438+0000","last_unstale":"2024-03-12T03:50:34.694609+0000","last_undegraded":"2024-03-12T03:50:34.694609+0000","last_fullsized":"2024-03-12T03:48:58.440067+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.016030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.016030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":740,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.687581+0000","last_change":"2024-03-12T03:48:00.918264+0000","last_active":"2024-03-12T03:50:34.687581+0000","last_peered":"2024-03-12T03:50:34.687581+0000","last_clean":"2024-03-12T03:50:34.687581+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:50:34.687581+0000","last_undegraded":"2024-03-12T03:50:34.687581+0000","last_fullsized":"2024-03-12T03:50:34.687581+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:00.918131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.918131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"269'16","reported_seq":833,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.688661+0000","last_change":"2024-03-12T03:49:51.806758+0000","last_active":"2024-03-12T03:50:34.688661+0000","last_peered":"2024-03-12T03:50:34.688661+0000","last_clean":"2024-03-12T03:50:34.688661+0000","last_became_active":"2024-03-12T03:46:25.283025+0000","last_became_peered":"2024-03-12T03:46:25.283025+0000","last_unstale":"2024-03-12T03:50:34.688661+0000","last_undegraded":"2024-03-12T03:50:34.688661+0000","last_fullsized":"2024-03-12T03:50:34.688661+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'16","last_scrub_stamp":"2024-03-12T03:49:51.806651+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:49:51.806651+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:56.931992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":748,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.683633+0000","last_change":"2024-03-12T03:50:02.319955+0000","last_active":"2024-03-12T03:50:34.683633+0000","last_peered":"2024-03-12T03:50:34.683633+0000","last_clean":"2024-03-12T03:50:34.683633+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:50:34.683633+0000","last_undegraded":"2024-03-12T03:50:34.683633+0000","last_fullsized":"2024-03-12T03:50:34.683633+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:02.319837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:02.319837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:25.368545+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":742,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.696620+0000","last_change":"2024-03-12T03:49:02.468991+0000","last_active":"2024-03-12T03:50:34.696620+0000","last_peered":"2024-03-12T03:50:34.696620+0000","last_clean":"2024-03-12T03:50:34.696620+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:50:34.696620+0000","last_undegraded":"2024-03-12T03:50:34.696620+0000","last_fullsized":"2024-03-12T03:50:34.696620+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:02.468865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:49:02.468865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":747,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.689374+0000","last_change":"2024-03-12T03:49:46.378515+0000","last_active":"2024-03-12T03:50:34.689374+0000","last_peered":"2024-03-12T03:50:34.689374+0000","last_clean":"2024-03-12T03:50:34.689374+0000","last_became_active":"2024-03-12T03:46:25.293777+0000","last_became_peered":"2024-03-12T03:46:25.293777+0000","last_unstale":"2024-03-12T03:50:34.689374+0000","last_undegraded":"2024-03-12T03:50:34.689374+0000","last_fullsized":"2024-03-12T03:50:34.689374+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:46.378413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:46.378413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:12.689217+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"345'33","reported_seq":823,"reported_epoch":410,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:34.690546+0000","last_change":"2024-03-12T03:48:58.457301+0000","last_active":"2024-03-12T03:50:34.690546+0000","last_peered":"2024-03-12T03:50:34.690546+0000","last_clean":"2024-03-12T03:48:53.446883+0000","last_became_active":"2024-03-12T03:48:58.457301+0000","last_became_peered":"2024-03-12T03:48:58.457301+0000","last_unstale":"2024-03-12T03:50:34.690546+0000","last_undegraded":"2024-03-12T03:48:58.444868+0000","last_fullsized":"2024-03-12T03:48:58.444750+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'33","last_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_deep_scrub":"345'33","last_deep_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_clean_scrub_stamp":"2024-03-12T03:47:02.110517+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:37.845476+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":748,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.684067+0000","last_change":"2024-03-12T03:49:04.233108+0000","last_active":"2024-03-12T03:50:34.684067+0000","last_peered":"2024-03-12T03:50:34.684067+0000","last_clean":"2024-03-12T03:50:34.684067+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:50:34.684067+0000","last_undegraded":"2024-03-12T03:50:34.684067+0000","last_fullsized":"2024-03-12T03:50:34.684067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:04.232977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:49:04.232977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":735,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.683972+0000","last_change":"2024-03-12T03:48:58.453634+0000","last_active":"2024-03-12T03:50:34.683972+0000","last_peered":"2024-03-12T03:50:34.683972+0000","last_clean":"2024-03-12T03:48:53.472261+0000","last_became_active":"2024-03-12T03:48:58.453634+0000","last_became_peered":"2024-03-12T03:48:58.453634+0000","last_unstale":"2024-03-12T03:50:34.683972+0000","last_undegraded":"2024-03-12T03:50:34.683972+0000","last_fullsized":"2024-03-12T03:48:58.444137+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:02.618599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_clean_scrub_stamp":"2024-03-12T03:46:02.618599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.225896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":59,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.691703+0000","last_change":"2024-03-12T03:48:58.463158+0000","last_active":"2024-03-12T03:50:34.691703+0000","last_peered":"2024-03-12T03:50:34.691703+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.463158+0000","last_became_peered":"2024-03-12T03:48:58.463158+0000","last_unstale":"2024-03-12T03:50:34.691703+0000","last_undegraded":"2024-03-12T03:50:34.691703+0000","last_fullsized":"2024-03-12T03:48:58.439250+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_clean_scrub_stamp":"2024-03-12T03:45:58.095196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"269'62","reported_seq":706,"reported_epoch":409,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:50:33.477583+0000","last_change":"2024-03-12T03:48:58.457134+0000","last_active":"2024-03-12T03:50:33.477583+0000","last_peered":"2024-03-12T03:50:33.477583+0000","last_clean":"2024-03-12T03:48:53.477142+0000","last_became_active":"2024-03-12T03:48:58.457134+0000","last_became_peered":"2024-03-12T03:48:58.457134+0000","last_unstale":"2024-03-12T03:50:33.477583+0000","last_undegraded":"2024-03-12T03:48:58.441299+0000","last_fullsized":"2024-03-12T03:48:58.441188+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358183+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":579,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.479914+0000","last_change":"2024-03-12T03:48:12.573604+0000","last_active":"2024-03-12T03:50:33.479914+0000","last_peered":"2024-03-12T03:50:33.479914+0000","last_clean":"2024-03-12T03:50:33.479914+0000","last_became_active":"2024-03-12T03:46:25.279131+0000","last_became_peered":"2024-03-12T03:46:25.279131+0000","last_unstale":"2024-03-12T03:50:33.479914+0000","last_undegraded":"2024-03-12T03:50:33.479914+0000","last_fullsized":"2024-03-12T03:50:33.479914+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:12.573481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:48:12.573481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:36.358283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":743,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.689014+0000","last_change":"2024-03-12T03:47:55.924374+0000","last_active":"2024-03-12T03:50:34.689014+0000","last_peered":"2024-03-12T03:50:34.689014+0000","last_clean":"2024-03-12T03:50:34.689014+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:50:34.689014+0000","last_undegraded":"2024-03-12T03:50:34.689014+0000","last_fullsized":"2024-03-12T03:50:34.689014+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:55.924292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:47:55.924292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":537,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-12T03:50:33.485142+0000","last_change":"2024-03-12T03:49:55.557456+0000","last_active":"2024-03-12T03:50:33.485142+0000","last_peered":"2024-03-12T03:50:33.485142+0000","last_clean":"2024-03-12T03:50:33.485142+0000","last_became_active":"2024-03-12T03:46:25.281081+0000","last_became_peered":"2024-03-12T03:46:25.281081+0000","last_unstale":"2024-03-12T03:50:33.485142+0000","last_undegraded":"2024-03-12T03:50:33.485142+0000","last_fullsized":"2024-03-12T03:50:33.485142+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:55.557335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:55.557335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.968328+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"269'26","reported_seq":789,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.688915+0000","last_change":"2024-03-12T03:49:01.728877+0000","last_active":"2024-03-12T03:50:34.688915+0000","last_peered":"2024-03-12T03:50:34.688915+0000","last_clean":"2024-03-12T03:50:34.688915+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:50:34.688915+0000","last_undegraded":"2024-03-12T03:50:34.688915+0000","last_fullsized":"2024-03-12T03:50:34.688915+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:49:01.728616+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:49:01.728616+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":353,"reported_epoch":410,"state":"active+undersized","last_fresh":"2024-03-12T03:50:34.685518+0000","last_change":"2024-03-12T03:48:58.462711+0000","last_active":"2024-03-12T03:50:34.685518+0000","last_peered":"2024-03-12T03:50:34.685518+0000","last_clean":"2024-03-12T03:39:28.536135+0000","last_became_active":"2024-03-12T03:48:58.462711+0000","last_became_peered":"2024-03-12T03:48:58.462711+0000","last_unstale":"2024-03-12T03:50:34.685518+0000","last_undegraded":"2024-03-12T03:50:34.685518+0000","last_fullsized":"2024-03-12T03:48:58.443314+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:12.186146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:48:12.186146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:50:38.231377+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":758,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.691997+0000","last_change":"2024-03-12T03:50:04.535064+0000","last_active":"2024-03-12T03:50:34.691997+0000","last_peered":"2024-03-12T03:50:34.691997+0000","last_clean":"2024-03-12T03:50:34.691997+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:50:34.691997+0000","last_undegraded":"2024-03-12T03:50:34.691997+0000","last_fullsized":"2024-03-12T03:50:34.691997+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_clean_scrub_stamp":"2024-03-12T03:50:04.534937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:34.126590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":746,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-12T03:50:34.690284+0000","last_change":"2024-03-12T03:49:03.833199+0000","last_active":"2024-03-12T03:50:34.690284+0000","last_peered":"2024-03-12T03:50:34.690284+0000","last_clean":"2024-03-12T03:50:34.690284+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:50:34.690284+0000","last_undegraded":"2024-03-12T03:50:34.690284+0000","last_fullsized":"2024-03-12T03:50:34.690284+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:03.832786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_clean_scrub_stamp":"2024-03-12T03:49:03.832786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3987406872,"num_objects":60844,"num_object_clones":0,"num_object_copies":182532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60844,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217142,"num_write_kb":8895489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12732473344,"data_stored":12732465200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77283,"ondisk_log_size":77283,"up":38,"acting":38,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1026,"num_read_kb":1021,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":605880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":56,"acting":56,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1397,"num_read_kb":1122,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":2642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":83,"acting":83,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":87,"acting":87,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070676,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1735068,"kb_used_data":968084,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92010084,"statfs":{"total":95995035648,"available":94218326016,"internally_reserved":0,"allocated":991318016,"data_stored":990101370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":785381834},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332313,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2406800,"kb_used_data":1753664,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":91338352,"statfs":{"total":95995035648,"available":93530472448,"internally_reserved":0,"allocated":1795751936,"data_stored":1794553831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":668793286},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575090,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1468096,"kb_used_data":692792,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92277056,"statfs":{"total":95995035648,"available":94491705344,"internally_reserved":0,"allocated":709419008,"data_stored":708165633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738595,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3725208,"kb_used_data":3102608,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":90019944,"statfs":{"total":95995035648,"available":92180422656,"internally_reserved":0,"allocated":3177070592,"data_stored":3175997927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411564,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2210120,"kb_used_data":1507004,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":91535032,"statfs":{"total":95995035648,"available":93731872768,"internally_reserved":0,"allocated":1543172096,"data_stored":1541979620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":330,"seq":1417339207731,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1607448,"kb_used_data":1324368,"kb_used_omap":13,"kb_used_meta":283058,"kb_avail":92137704,"statfs":{"total":95995035648,"available":94349008896,"internally_reserved":0,"allocated":1356152832,"data_stored":1354926420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13485,"internal_metadata":289852243},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953700,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2564908,"kb_used_data":1960156,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91180244,"statfs":{"total":95995035648,"available":93368569856,"internally_reserved":0,"allocated":2007199744,"data_stored":2005974470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659358,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2044300,"kb_used_data":1241220,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":91700852,"statfs":{"total":95995035648,"available":93901672448,"internally_reserved":0,"allocated":1271009280,"data_stored":1269808022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":822344133},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3205627904,"data_stored":3205627904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":703266816,"data_stored":703266816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1501364224,"data_stored":1501364224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1348403200,"data_stored":1348403200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1982337024,"data_stored":1982332952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1250426880,"data_stored":1250426880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1753616384,"data_stored":1753612312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":987430912,"data_stored":987430912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:50:41.936 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-03-12T03:50:41.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-12T03:50:41.950 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:41.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:41.985 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:41 smithi169 ceph-mon[29344]: pgmap v831: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 764 B/s rd, 0 B/s wr, 1.23k op/s; 37525/180357 objects degraded (20.806%) 2024-03-12T03:50:41.985 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:41 smithi169 ceph-mon[29344]: 6.6 scrub starts 2024-03-12T03:50:41.985 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:41 smithi169 ceph-mon[29344]: 6.6 scrub ok 2024-03-12T03:50:41.985 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:41 smithi169 ceph-mon[29344]: 3.1d scrub starts 2024-03-12T03:50:41.985 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:41 smithi169 ceph-mon[29344]: 3.1d scrub ok 2024-03-12T03:50:41.986 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:50:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:50:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:50:41.986 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:41 smithi169 ceph-mon[39949]: pgmap v831: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 764 B/s rd, 0 B/s wr, 1.23k op/s; 37525/180357 objects degraded (20.806%) 2024-03-12T03:50:41.986 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:41 smithi169 ceph-mon[39949]: 6.6 scrub starts 2024-03-12T03:50:41.986 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:41 smithi169 ceph-mon[39949]: 6.6 scrub ok 2024-03-12T03:50:41.986 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:41 smithi169 ceph-mon[39949]: 3.1d scrub starts 2024-03-12T03:50:41.986 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:41 smithi169 ceph-mon[39949]: 3.1d scrub ok 2024-03-12T03:50:41.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:41 smithi173 ceph-mon[33544]: pgmap v831: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 764 B/s rd, 0 B/s wr, 1.23k op/s; 37525/180357 objects degraded (20.806%) 2024-03-12T03:50:41.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:41 smithi173 ceph-mon[33544]: 6.6 scrub starts 2024-03-12T03:50:41.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:41 smithi173 ceph-mon[33544]: 6.6 scrub ok 2024-03-12T03:50:41.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:41 smithi173 ceph-mon[33544]: 3.1d scrub starts 2024-03-12T03:50:41.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:41 smithi173 ceph-mon[33544]: 3.1d scrub ok 2024-03-12T03:50:42.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:42.093 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:42.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:42.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:42.229 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:42.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:42.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:42.331 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:42.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:42.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:42.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:42.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:42.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:42.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:42.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:42.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:42.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:42.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:42.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:42.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:42.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:42.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:42.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:42.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:42 smithi173 ceph-mon[33544]: 3.0 scrub starts 2024-03-12T03:50:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:42 smithi173 ceph-mon[33544]: 3.0 scrub ok 2024-03-12T03:50:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:42 smithi173 ceph-mon[33544]: 5.16 scrub starts 2024-03-12T03:50:42.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:42 smithi173 ceph-mon[33544]: 5.16 scrub ok 2024-03-12T03:50:43.003 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:42 smithi169 ceph-mon[29344]: 3.0 scrub starts 2024-03-12T03:50:43.003 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:42 smithi169 ceph-mon[29344]: 3.0 scrub ok 2024-03-12T03:50:43.003 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:42 smithi169 ceph-mon[29344]: 5.16 scrub starts 2024-03-12T03:50:43.003 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:42 smithi169 ceph-mon[29344]: 5.16 scrub ok 2024-03-12T03:50:43.004 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:42 smithi169 ceph-mon[39949]: 3.0 scrub starts 2024-03-12T03:50:43.004 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:42 smithi169 ceph-mon[39949]: 3.0 scrub ok 2024-03-12T03:50:43.004 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:42 smithi169 ceph-mon[39949]: 5.16 scrub starts 2024-03-12T03:50:43.004 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:42 smithi169 ceph-mon[39949]: 5.16 scrub ok 2024-03-12T03:50:43.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:43.048 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:43.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:43.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:43.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:43.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:43.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:43.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:43.318 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:43.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:43.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:43.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:43.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:43.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:43.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:43.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:43.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:43.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:43.733 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:50:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:50:43.741 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:43.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:43.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:43.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:43.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:43 smithi169 ceph-mon[29344]: pgmap v832: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 33691/164448 objects degraded (20.487%) 2024-03-12T03:50:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:43 smithi169 ceph-mon[29344]: 6.8 scrub starts 2024-03-12T03:50:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:43 smithi169 ceph-mon[29344]: 6.8 scrub ok 2024-03-12T03:50:43.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:43 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 33691/164448 objects degraded (20.487%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:43.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:43 smithi169 ceph-mon[39949]: pgmap v832: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 33691/164448 objects degraded (20.487%) 2024-03-12T03:50:43.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:43 smithi169 ceph-mon[39949]: 6.8 scrub starts 2024-03-12T03:50:43.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:43 smithi169 ceph-mon[39949]: 6.8 scrub ok 2024-03-12T03:50:43.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:43 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 33691/164448 objects degraded (20.487%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:43.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:43 smithi173 ceph-mon[33544]: pgmap v832: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 33691/164448 objects degraded (20.487%) 2024-03-12T03:50:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:43 smithi173 ceph-mon[33544]: 6.8 scrub starts 2024-03-12T03:50:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:43 smithi173 ceph-mon[33544]: 6.8 scrub ok 2024-03-12T03:50:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:43 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 33691/164448 objects degraded (20.487%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:44.031 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:44.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:44.105 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:50:44.106 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:50:44.106 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-03-12T03:50:44.106 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-03-12T03:50:44.106 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:50:44.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:44.166 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:44.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:44.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:44.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:44.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:44.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:44.449 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:44.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:44.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:44.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:44.578 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:44.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:44.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:44.601 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:50:44.633 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:50:44.666 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:50:44.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:44.703 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:50:44.728 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:44.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:44.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:44.733 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:50:44.750 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:50:44.770 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:50:44.808 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:50:44.808 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:50:44.808 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:50:44.808 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:50:44.808 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:50:44.808 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:50:44.808 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:50:44.808 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:50:44.809 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:50:44.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:44.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:44.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:44.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:44 smithi169 ceph-mon[29344]: 3.15 scrub starts 2024-03-12T03:50:44.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:44 smithi169 ceph-mon[29344]: 3.15 scrub ok 2024-03-12T03:50:44.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:44.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:44 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/318846633' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:50:44.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:44 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:50:44.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:44 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3846588216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:50:44.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:44 smithi169 ceph-mon[39949]: 3.15 scrub starts 2024-03-12T03:50:44.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:44 smithi169 ceph-mon[39949]: 3.15 scrub ok 2024-03-12T03:50:44.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:44.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:44 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/318846633' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:50:44.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:44 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:50:44.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:44 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3846588216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:50:44.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:44.997 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:44.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:45.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:45.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:45.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:45.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:44 smithi173 ceph-mon[33544]: 3.15 scrub starts 2024-03-12T03:50:45.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:44 smithi173 ceph-mon[33544]: 3.15 scrub ok 2024-03-12T03:50:45.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:45.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:44 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/318846633' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:50:45.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:44 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:50:45.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:44 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3846588216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:50:45.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:45.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:45.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:45.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:45.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:45.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:45.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:45.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:45.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:45.534 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:45.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:45.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:45.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:45.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:45.775 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:45.801 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:45.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:45.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:45.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:45.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:45.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:46.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:46.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:46.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:46.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:45 smithi169 ceph-mon[29344]: pgmap v833: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.60k op/s; 32741/156957 objects degraded (20.860%) 2024-03-12T03:50:46.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:45 smithi169 ceph-mon[29344]: 6.5 scrub starts 2024-03-12T03:50:46.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:45 smithi169 ceph-mon[29344]: 6.5 scrub ok 2024-03-12T03:50:46.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:45 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:50:46.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:45 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:50:46.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:45 smithi169 ceph-mon[29344]: osdmap e412: 8 total, 7 up, 8 in 2024-03-12T03:50:46.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:45 smithi169 ceph-mon[39949]: pgmap v833: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.60k op/s; 32741/156957 objects degraded (20.860%) 2024-03-12T03:50:46.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:45 smithi169 ceph-mon[39949]: 6.5 scrub starts 2024-03-12T03:50:46.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:45 smithi169 ceph-mon[39949]: 6.5 scrub ok 2024-03-12T03:50:46.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:45 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:50:46.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:45 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:50:46.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:45 smithi169 ceph-mon[39949]: osdmap e412: 8 total, 7 up, 8 in 2024-03-12T03:50:46.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:46.210 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:46.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:46.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:45 smithi173 ceph-mon[33544]: pgmap v833: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.60k op/s; 32741/156957 objects degraded (20.860%) 2024-03-12T03:50:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:45 smithi173 ceph-mon[33544]: 6.5 scrub starts 2024-03-12T03:50:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:45 smithi173 ceph-mon[33544]: 6.5 scrub ok 2024-03-12T03:50:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:45 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:50:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:45 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:50:46.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:45 smithi173 ceph-mon[33544]: osdmap e412: 8 total, 7 up, 8 in 2024-03-12T03:50:46.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:46.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:46.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:46.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:46.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:46.484 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:46.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:46.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:46.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:46.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:46.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:46.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:46.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:46.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:46.897 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:46.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:46.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:46.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:46 smithi169 ceph-mon[29344]: 4.5 scrub starts 2024-03-12T03:50:46.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:46 smithi169 ceph-mon[29344]: 4.5 scrub ok 2024-03-12T03:50:46.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:46 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:50:46.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:46 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:46.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:46 smithi169 ceph-mon[39949]: 4.5 scrub starts 2024-03-12T03:50:46.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:46 smithi169 ceph-mon[39949]: 4.5 scrub ok 2024-03-12T03:50:46.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:46 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:50:46.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:46 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:46.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:47.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:47.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:47.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:47.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:47.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:47.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:46 smithi173 ceph-mon[33544]: 4.5 scrub starts 2024-03-12T03:50:47.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:46 smithi173 ceph-mon[33544]: 4.5 scrub ok 2024-03-12T03:50:47.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:46 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:50:47.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:46 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:47.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:47.303 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:47.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:47.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:47.438 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:47.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:47.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:47.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:47.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:47.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:47.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:47.708 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:47.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:47.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:47.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:47.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[29344]: pgmap v835: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.09k op/s; 32741/156957 objects degraded (20.860%) 2024-03-12T03:50:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[29344]: 4.f scrub starts 2024-03-12T03:50:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[29344]: 4.f scrub ok 2024-03-12T03:50:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[29344]: 5.1f scrub starts 2024-03-12T03:50:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[29344]: 5.1f scrub ok 2024-03-12T03:50:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[29344]: 4.8 scrub starts 2024-03-12T03:50:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[29344]: 4.8 scrub ok 2024-03-12T03:50:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[39949]: pgmap v835: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.09k op/s; 32741/156957 objects degraded (20.860%) 2024-03-12T03:50:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[39949]: 4.f scrub starts 2024-03-12T03:50:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[39949]: 4.f scrub ok 2024-03-12T03:50:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[39949]: 5.1f scrub starts 2024-03-12T03:50:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[39949]: 5.1f scrub ok 2024-03-12T03:50:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[39949]: 4.8 scrub starts 2024-03-12T03:50:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:47 smithi169 ceph-mon[39949]: 4.8 scrub ok 2024-03-12T03:50:47.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:47.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:47.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:48.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:48.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:48.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:48.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:48.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:47 smithi173 ceph-mon[33544]: pgmap v835: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 3.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.09k op/s; 32741/156957 objects degraded (20.860%) 2024-03-12T03:50:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:47 smithi173 ceph-mon[33544]: 4.f scrub starts 2024-03-12T03:50:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:47 smithi173 ceph-mon[33544]: 4.f scrub ok 2024-03-12T03:50:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:47 smithi173 ceph-mon[33544]: 5.1f scrub starts 2024-03-12T03:50:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:47 smithi173 ceph-mon[33544]: 5.1f scrub ok 2024-03-12T03:50:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:47 smithi173 ceph-mon[33544]: 4.8 scrub starts 2024-03-12T03:50:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:47 smithi173 ceph-mon[33544]: 4.8 scrub ok 2024-03-12T03:50:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:48.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:48.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:48.368 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:48.402 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:48.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:48.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:48.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:48.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:48.549 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-12T03:50:48.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:48.676 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:48.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:48.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:48.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:48.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:48.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:48.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:48.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:48.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:49.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:49.084 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:49.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:49.099 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[29344]: 5.17 scrub starts 2024-03-12T03:50:49.099 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[29344]: 5.17 scrub ok 2024-03-12T03:50:49.099 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[29344]: 3.9 scrub starts 2024-03-12T03:50:49.099 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[29344]: 3.9 scrub ok 2024-03-12T03:50:49.099 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[29344]: 3.5 scrub starts 2024-03-12T03:50:49.099 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[29344]: 3.5 scrub ok 2024-03-12T03:50:49.099 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 32741/156957 objects degraded (20.860%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:49.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[29344]: osdmap e413: 8 total, 7 up, 8 in 2024-03-12T03:50:49.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[39949]: 5.17 scrub starts 2024-03-12T03:50:49.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[39949]: 5.17 scrub ok 2024-03-12T03:50:49.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[39949]: 3.9 scrub starts 2024-03-12T03:50:49.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[39949]: 3.9 scrub ok 2024-03-12T03:50:49.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[39949]: 3.5 scrub starts 2024-03-12T03:50:49.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[39949]: 3.5 scrub ok 2024-03-12T03:50:49.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 32741/156957 objects degraded (20.860%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:49.100 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:48 smithi169 ceph-mon[39949]: osdmap e413: 8 total, 7 up, 8 in 2024-03-12T03:50:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:50:49.140 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:50:49.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:49.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:49.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:49.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:48 smithi173 ceph-mon[33544]: 5.17 scrub starts 2024-03-12T03:50:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:48 smithi173 ceph-mon[33544]: 5.17 scrub ok 2024-03-12T03:50:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:48 smithi173 ceph-mon[33544]: 3.9 scrub starts 2024-03-12T03:50:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:48 smithi173 ceph-mon[33544]: 3.9 scrub ok 2024-03-12T03:50:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:48 smithi173 ceph-mon[33544]: 3.5 scrub starts 2024-03-12T03:50:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:48 smithi173 ceph-mon[33544]: 3.5 scrub ok 2024-03-12T03:50:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:48 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 32741/156957 objects degraded (20.860%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:48 smithi173 ceph-mon[33544]: osdmap e413: 8 total, 7 up, 8 in 2024-03-12T03:50:49.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:49.353 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:49.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:49.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:49.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:49.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:49.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:49.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:49.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:49.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:49.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:49.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:49.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:49.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:49.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:49.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[39949]: pgmap v836: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.56k op/s; 28180/137832 objects degraded (20.445%) 2024-03-12T03:50:49.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[39949]: 5.4 scrub starts 2024-03-12T03:50:49.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[39949]: 5.4 scrub ok 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[39949]: 4.17 scrub starts 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[39949]: 4.17 scrub ok 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[39949]: 5.12 scrub starts 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[39949]: 5.12 scrub ok 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[39949]: 1.5 scrub starts 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[39949]: 1.5 scrub ok 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[29344]: pgmap v836: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.56k op/s; 28180/137832 objects degraded (20.445%) 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[29344]: 5.4 scrub starts 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[29344]: 5.4 scrub ok 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[29344]: 4.17 scrub starts 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[29344]: 4.17 scrub ok 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[29344]: 5.12 scrub starts 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[29344]: 5.12 scrub ok 2024-03-12T03:50:49.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[29344]: 1.5 scrub starts 2024-03-12T03:50:49.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:49 smithi169 ceph-mon[29344]: 1.5 scrub ok 2024-03-12T03:50:50.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:50.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:50.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:50.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:50.184 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:50.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:50.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:50.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:50.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:49 smithi173 ceph-mon[33544]: pgmap v836: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.56k op/s; 28180/137832 objects degraded (20.445%) 2024-03-12T03:50:50.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:49 smithi173 ceph-mon[33544]: 5.4 scrub starts 2024-03-12T03:50:50.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:49 smithi173 ceph-mon[33544]: 5.4 scrub ok 2024-03-12T03:50:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:49 smithi173 ceph-mon[33544]: 4.17 scrub starts 2024-03-12T03:50:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:49 smithi173 ceph-mon[33544]: 4.17 scrub ok 2024-03-12T03:50:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:49 smithi173 ceph-mon[33544]: 5.12 scrub starts 2024-03-12T03:50:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:49 smithi173 ceph-mon[33544]: 5.12 scrub ok 2024-03-12T03:50:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:49 smithi173 ceph-mon[33544]: 1.5 scrub starts 2024-03-12T03:50:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:49 smithi173 ceph-mon[33544]: 1.5 scrub ok 2024-03-12T03:50:50.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:50.322 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:50.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:50.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:50.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:50.453 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:50.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:50.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:50.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:50.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:50.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:50.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:50.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:50.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:50.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:50.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:50.870 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:50.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:50.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:50.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:50 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:50:50.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:50 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:50:50.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:50 smithi169 ceph-mon[29344]: 3.1a scrub starts 2024-03-12T03:50:50.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:50 smithi169 ceph-mon[29344]: 3.1a scrub ok 2024-03-12T03:50:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:50 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:50:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:50 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:50:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:50 smithi169 ceph-mon[39949]: 3.1a scrub starts 2024-03-12T03:50:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:50 smithi169 ceph-mon[39949]: 3.1a scrub ok 2024-03-12T03:50:50.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:51.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:51.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:51.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:51.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:50:51.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:51.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:51.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:50 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:50:51.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:50 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:50:51.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:50 smithi173 ceph-mon[33544]: 3.1a scrub starts 2024-03-12T03:50:51.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:50 smithi173 ceph-mon[33544]: 3.1a scrub ok 2024-03-12T03:50:51.250 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:51.290 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:51.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:51.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:51.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:51.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:51.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:51.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:51.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:51.637 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:50:51.661 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:51.722 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:51.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:51.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:51.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:51.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:51.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:51.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:51.982 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:51 smithi169 ceph-mon[29344]: pgmap v838: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.22k op/s; 28180/135231 objects degraded (20.838%) 2024-03-12T03:50:51.982 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:51 smithi169 ceph-mon[29344]: osdmap e414: 8 total, 7 up, 8 in 2024-03-12T03:50:51.982 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:51 smithi169 ceph-mon[29344]: 5.f scrub starts 2024-03-12T03:50:51.982 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:51 smithi169 ceph-mon[29344]: 5.f scrub ok 2024-03-12T03:50:51.982 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:51 smithi169 ceph-mon[39949]: pgmap v838: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.22k op/s; 28180/135231 objects degraded (20.838%) 2024-03-12T03:50:51.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:51 smithi169 ceph-mon[39949]: osdmap e414: 8 total, 7 up, 8 in 2024-03-12T03:50:51.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:51 smithi169 ceph-mon[39949]: 5.f scrub starts 2024-03-12T03:50:51.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:51 smithi169 ceph-mon[39949]: 5.f scrub ok 2024-03-12T03:50:51.983 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:50:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:50:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:50:51.990 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:51.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:52.079 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:52.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:52.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:52.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:52.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:52.147 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:50:52.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:51 smithi173 ceph-mon[33544]: pgmap v838: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.22k op/s; 28180/135231 objects degraded (20.838%) 2024-03-12T03:50:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:51 smithi173 ceph-mon[33544]: osdmap e414: 8 total, 7 up, 8 in 2024-03-12T03:50:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:51 smithi173 ceph-mon[33544]: 5.f scrub starts 2024-03-12T03:50:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:51 smithi173 ceph-mon[33544]: 5.f scrub ok 2024-03-12T03:50:52.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:52.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:52.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:52.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:52.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:52.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:52.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:52.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:52.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:52.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:52.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:52.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:52.800 INFO:teuthology.orchestra.run.smithi169.stderr:set pool 1 pg_num to 11 2024-03-12T03:50:52.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:52.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:52.895 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:50:52.895 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:50:52.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:52.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:52.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:52.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:53.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:53.087 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:53.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:53.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:52 smithi169 ceph-mon[29344]: 1.14 scrub starts 2024-03-12T03:50:53.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:52 smithi169 ceph-mon[29344]: 1.14 scrub ok 2024-03-12T03:50:53.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:52 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3621996862' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-12T03:50:53.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:52 smithi169 ceph-mon[29344]: 6.1e deep-scrub starts 2024-03-12T03:50:53.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:52 smithi169 ceph-mon[29344]: 6.1e deep-scrub ok 2024-03-12T03:50:53.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:52 smithi169 ceph-mon[39949]: 1.14 scrub starts 2024-03-12T03:50:53.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:52 smithi169 ceph-mon[39949]: 1.14 scrub ok 2024-03-12T03:50:53.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:52 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3621996862' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-12T03:50:53.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:52 smithi169 ceph-mon[39949]: 6.1e deep-scrub starts 2024-03-12T03:50:53.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:52 smithi169 ceph-mon[39949]: 6.1e deep-scrub ok 2024-03-12T03:50:53.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:53.223 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:53.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:53.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:52 smithi173 ceph-mon[33544]: 1.14 scrub starts 2024-03-12T03:50:53.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:52 smithi173 ceph-mon[33544]: 1.14 scrub ok 2024-03-12T03:50:53.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:52 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3621996862' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-12T03:50:53.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:52 smithi173 ceph-mon[33544]: 6.1e deep-scrub starts 2024-03-12T03:50:53.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:52 smithi173 ceph-mon[33544]: 6.1e deep-scrub ok 2024-03-12T03:50:53.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:53.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:53.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:53.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:53.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:53.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:53.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:53.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:53.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:53.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:50:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:50:53.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:53.777 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:53.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:53.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:53.879 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:53.907 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:53.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:53.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:54.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:54.037 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:54.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:54.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:54.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:53 smithi169 ceph-mon[39949]: pgmap v840: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.70k op/s; 24303/119025 objects degraded (20.418%) 2024-03-12T03:50:54.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:53 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3621996862' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-12T03:50:54.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:53 smithi169 ceph-mon[39949]: osdmap e415: 8 total, 7 up, 8 in 2024-03-12T03:50:54.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:53 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 24303/119025 objects degraded (20.418%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:54.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:53 smithi169 ceph-mon[29344]: pgmap v840: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.70k op/s; 24303/119025 objects degraded (20.418%) 2024-03-12T03:50:54.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:53 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3621996862' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-12T03:50:54.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:53 smithi169 ceph-mon[29344]: osdmap e415: 8 total, 7 up, 8 in 2024-03-12T03:50:54.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:53 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 24303/119025 objects degraded (20.418%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:54.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:54.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:54.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:53 smithi173 ceph-mon[33544]: pgmap v840: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.70k op/s; 24303/119025 objects degraded (20.418%) 2024-03-12T03:50:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:53 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3621996862' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-12T03:50:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:53 smithi173 ceph-mon[33544]: osdmap e415: 8 total, 7 up, 8 in 2024-03-12T03:50:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:53 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 24303/119025 objects degraded (20.418%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:54.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:54.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:54.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:54.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:54.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:54.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:54.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:54.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:54.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:54.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:54.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:54.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:54.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:54.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:54.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:54.867 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:54.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:54.968 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:55.002 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:55.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:55.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:55.135 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:55.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:55.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[29344]: 1.9 scrub starts 2024-03-12T03:50:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[29344]: 1.9 scrub ok 2024-03-12T03:50:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[29344]: 5.14 scrub starts 2024-03-12T03:50:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[29344]: 5.14 scrub ok 2024-03-12T03:50:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[29344]: 6.1d scrub starts 2024-03-12T03:50:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[29344]: 6.1d scrub ok 2024-03-12T03:50:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-12T03:50:55.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-12T03:50:55.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:55.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[39949]: 1.9 scrub starts 2024-03-12T03:50:55.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[39949]: 1.9 scrub ok 2024-03-12T03:50:55.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[39949]: 5.14 scrub starts 2024-03-12T03:50:55.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[39949]: 5.14 scrub ok 2024-03-12T03:50:55.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[39949]: 6.1d scrub starts 2024-03-12T03:50:55.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[39949]: 6.1d scrub ok 2024-03-12T03:50:55.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-12T03:50:55.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:54 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-12T03:50:55.236 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:54 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:55.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:54 smithi173 ceph-mon[33544]: 1.9 scrub starts 2024-03-12T03:50:55.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:54 smithi173 ceph-mon[33544]: 1.9 scrub ok 2024-03-12T03:50:55.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:54 smithi173 ceph-mon[33544]: 5.14 scrub starts 2024-03-12T03:50:55.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:54 smithi173 ceph-mon[33544]: 5.14 scrub ok 2024-03-12T03:50:55.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:54 smithi173 ceph-mon[33544]: 6.1d scrub starts 2024-03-12T03:50:55.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:54 smithi173 ceph-mon[33544]: 6.1d scrub ok 2024-03-12T03:50:55.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:54 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-12T03:50:55.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:54 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-12T03:50:55.267 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:55.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:55.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:55.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:55.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:55.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:55.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:55.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:55.543 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:55.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:55.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:55.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:55.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:55.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:55.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:55.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:55.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:55.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:55.957 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:55.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:56.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:56.066 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:55 smithi169 ceph-mon[29344]: pgmap v842: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.44k op/s; 23386/111918 objects degraded (20.896%) 2024-03-12T03:50:56.067 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:55 smithi169 ceph-mon[29344]: 1.7 scrub starts 2024-03-12T03:50:56.067 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:55 smithi169 ceph-mon[29344]: 1.7 scrub ok 2024-03-12T03:50:56.067 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:55 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-12T03:50:56.067 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:55 smithi169 ceph-mon[29344]: osdmap e416: 8 total, 7 up, 8 in 2024-03-12T03:50:56.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:55 smithi169 ceph-mon[39949]: pgmap v842: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.44k op/s; 23386/111918 objects degraded (20.896%) 2024-03-12T03:50:56.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:55 smithi169 ceph-mon[39949]: 1.7 scrub starts 2024-03-12T03:50:56.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:55 smithi169 ceph-mon[39949]: 1.7 scrub ok 2024-03-12T03:50:56.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:55 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-12T03:50:56.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:55 smithi169 ceph-mon[39949]: osdmap e416: 8 total, 7 up, 8 in 2024-03-12T03:50:56.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:56.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:56.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:56.235 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:56.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:55 smithi173 ceph-mon[33544]: pgmap v842: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.44k op/s; 23386/111918 objects degraded (20.896%) 2024-03-12T03:50:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:55 smithi173 ceph-mon[33544]: 1.7 scrub starts 2024-03-12T03:50:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:55 smithi173 ceph-mon[33544]: 1.7 scrub ok 2024-03-12T03:50:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:55 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-12T03:50:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:55 smithi173 ceph-mon[33544]: osdmap e416: 8 total, 7 up, 8 in 2024-03-12T03:50:56.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:56.375 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:56.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:56.476 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:56.506 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:56.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:56.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:56.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:56.636 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:56.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:56.738 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:56.771 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:56.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:56.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:56.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:56.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:56.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:57.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:57.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:57.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:57.147 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:57.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[29344]: osdmap e417: 8 total, 7 up, 8 in 2024-03-12T03:50:57.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:50:57.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:50:57.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[29344]: 3.a scrub starts 2024-03-12T03:50:57.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[29344]: 3.a scrub ok 2024-03-12T03:50:57.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[29344]: 1.8 scrub starts 2024-03-12T03:50:57.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[29344]: 1.8 scrub ok 2024-03-12T03:50:57.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[39949]: osdmap e417: 8 total, 7 up, 8 in 2024-03-12T03:50:57.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:50:57.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:50:57.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[39949]: 3.a scrub starts 2024-03-12T03:50:57.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[39949]: 3.a scrub ok 2024-03-12T03:50:57.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[39949]: 1.8 scrub starts 2024-03-12T03:50:57.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:56 smithi169 ceph-mon[39949]: 1.8 scrub ok 2024-03-12T03:50:57.178 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:57.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:56 smithi173 ceph-mon[33544]: osdmap e417: 8 total, 7 up, 8 in 2024-03-12T03:50:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:56 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:50:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:56 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:50:57.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:56 smithi173 ceph-mon[33544]: 3.a scrub starts 2024-03-12T03:50:57.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:56 smithi173 ceph-mon[33544]: 3.a scrub ok 2024-03-12T03:50:57.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:56 smithi173 ceph-mon[33544]: 1.8 scrub starts 2024-03-12T03:50:57.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:56 smithi173 ceph-mon[33544]: 1.8 scrub ok 2024-03-12T03:50:57.280 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:57.318 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:57.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:57.419 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:50:57.455 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:57.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:50:57.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:57.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:57.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:57.693 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:57.722 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:57.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:57.824 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:57.859 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:57.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:57.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:57.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:57.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:58.097 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:57 smithi169 ceph-mon[29344]: pgmap v845: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 947 B/s rd, 0 B/s wr, 1.44k op/s; 23386/111918 objects degraded (20.896%) 2024-03-12T03:50:58.097 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:57 smithi169 ceph-mon[29344]: osdmap e418: 8 total, 7 up, 8 in 2024-03-12T03:50:58.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:57 smithi169 ceph-mon[39949]: pgmap v845: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 947 B/s rd, 0 B/s wr, 1.44k op/s; 23386/111918 objects degraded (20.896%) 2024-03-12T03:50:58.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:57 smithi169 ceph-mon[39949]: osdmap e418: 8 total, 7 up, 8 in 2024-03-12T03:50:58.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:58.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:58.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:58.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-12T03:50:58.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:57 smithi173 ceph-mon[33544]: pgmap v845: 168 pgs: 27 active+undersized+degraded, 36 active+undersized, 105 active+clean; 2.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 947 B/s rd, 0 B/s wr, 1.44k op/s; 23386/111918 objects degraded (20.896%) 2024-03-12T03:50:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:57 smithi173 ceph-mon[33544]: osdmap e418: 8 total, 7 up, 8 in 2024-03-12T03:50:58.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:58.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:58.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:58.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:58.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:58.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:58.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:58.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:58.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:58.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:58.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:58.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:58.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:58.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:58.817 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:58.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:58.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:50:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:58.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:50:59.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:50:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:59.085 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:59.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:50:59.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:58 smithi173 ceph-mon[33544]: 1.10 scrub starts 2024-03-12T03:50:59.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:58 smithi173 ceph-mon[33544]: 1.10 scrub ok 2024-03-12T03:50:59.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:58 smithi173 ceph-mon[33544]: osdmap e419: 8 total, 7 up, 8 in 2024-03-12T03:50:59.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:58 smithi173 ceph-mon[33544]: 4.18 scrub starts 2024-03-12T03:50:59.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:58 smithi173 ceph-mon[33544]: 4.18 scrub ok 2024-03-12T03:50:59.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:58 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:50:59.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:58 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:50:59.139 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:58 smithi173 ceph-mon[33544]: 2.2 scrub starts 2024-03-12T03:50:59.139 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:58 smithi173 ceph-mon[33544]: 2.2 scrub ok 2024-03-12T03:50:59.139 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:58 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 23386/111918 objects degraded (20.896%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:59.139 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:58 smithi173 ceph-mon[33544]: osdmap e420: 8 total, 7 up, 8 in 2024-03-12T03:50:59.163 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[39949]: 1.10 scrub starts 2024-03-12T03:50:59.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[39949]: 1.10 scrub ok 2024-03-12T03:50:59.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[39949]: osdmap e419: 8 total, 7 up, 8 in 2024-03-12T03:50:59.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[39949]: 4.18 scrub starts 2024-03-12T03:50:59.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[39949]: 4.18 scrub ok 2024-03-12T03:50:59.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:50:59.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:50:59.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[39949]: 2.2 scrub starts 2024-03-12T03:50:59.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[39949]: 2.2 scrub ok 2024-03-12T03:50:59.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 23386/111918 objects degraded (20.896%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:59.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[39949]: osdmap e420: 8 total, 7 up, 8 in 2024-03-12T03:50:59.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[29344]: 1.10 scrub starts 2024-03-12T03:50:59.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[29344]: 1.10 scrub ok 2024-03-12T03:50:59.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[29344]: osdmap e419: 8 total, 7 up, 8 in 2024-03-12T03:50:59.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[29344]: 4.18 scrub starts 2024-03-12T03:50:59.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[29344]: 4.18 scrub ok 2024-03-12T03:50:59.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:50:59.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:50:59.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[29344]: 2.2 scrub starts 2024-03-12T03:50:59.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[29344]: 2.2 scrub ok 2024-03-12T03:50:59.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 23386/111918 objects degraded (20.896%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-12T03:50:59.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:58 smithi169 ceph-mon[29344]: osdmap e420: 8 total, 7 up, 8 in 2024-03-12T03:50:59.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:59.222 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:59.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:59.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:50:59.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:50:59.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:59.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:50:59.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:59.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:59.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:59.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:50:59.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:59.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:59.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:50:59.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:50:59.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:59.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:50:59.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:50:59.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:50:59.912 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:50:59.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:59.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:00.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:00.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:00.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:00.097 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[29344]: pgmap v848: 168 pgs: 4 remapped+peering, 4 active+undersized+remapped, 27 active+undersized+degraded, 34 active+undersized, 99 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.66k op/s; 18688/92067 objects degraded (20.298%); 1/92067 objects misplaced (0.001%); 20 KiB/s, 0 objects/s recovering 2024-03-12T03:51:00.097 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[29344]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-12T03:51:00.097 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[29344]: 1.d scrub starts 2024-03-12T03:51:00.097 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[29344]: 1.d scrub ok 2024-03-12T03:51:00.097 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[29344]: 1.11 scrub starts 2024-03-12T03:51:00.097 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[29344]: 1.11 scrub ok 2024-03-12T03:51:00.098 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[29344]: 1.f scrub starts 2024-03-12T03:51:00.098 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[29344]: 1.f scrub ok 2024-03-12T03:51:00.098 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[39949]: pgmap v848: 168 pgs: 4 remapped+peering, 4 active+undersized+remapped, 27 active+undersized+degraded, 34 active+undersized, 99 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.66k op/s; 18688/92067 objects degraded (20.298%); 1/92067 objects misplaced (0.001%); 20 KiB/s, 0 objects/s recovering 2024-03-12T03:51:00.098 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[39949]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-12T03:51:00.098 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[39949]: 1.d scrub starts 2024-03-12T03:51:00.098 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[39949]: 1.d scrub ok 2024-03-12T03:51:00.098 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[39949]: 1.11 scrub starts 2024-03-12T03:51:00.098 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[39949]: 1.11 scrub ok 2024-03-12T03:51:00.098 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[39949]: 1.f scrub starts 2024-03-12T03:51:00.098 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:50:59 smithi169 ceph-mon[39949]: 1.f scrub ok 2024-03-12T03:51:00.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:00.181 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:00.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:00.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:59 smithi173 ceph-mon[33544]: pgmap v848: 168 pgs: 4 remapped+peering, 4 active+undersized+remapped, 27 active+undersized+degraded, 34 active+undersized, 99 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.66k op/s; 18688/92067 objects degraded (20.298%); 1/92067 objects misplaced (0.001%); 20 KiB/s, 0 objects/s recovering 2024-03-12T03:51:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:59 smithi173 ceph-mon[33544]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-12T03:51:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:59 smithi173 ceph-mon[33544]: 1.d scrub starts 2024-03-12T03:51:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:59 smithi173 ceph-mon[33544]: 1.d scrub ok 2024-03-12T03:51:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:59 smithi173 ceph-mon[33544]: 1.11 scrub starts 2024-03-12T03:51:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:59 smithi173 ceph-mon[33544]: 1.11 scrub ok 2024-03-12T03:51:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:59 smithi173 ceph-mon[33544]: 1.f scrub starts 2024-03-12T03:51:00.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:50:59 smithi173 ceph-mon[33544]: 1.f scrub ok 2024-03-12T03:51:00.283 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:00.313 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:00.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:00.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:00.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:00.449 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:00.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:00.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:00.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:00.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:00.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:00.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:00.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:00.723 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:00.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:00.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:00.743 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:51:00.781 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:51:00.813 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:51:00.821 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:51:00.822 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:51:00.822 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-12T03:51:00.822 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-12T03:51:00.822 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-12T03:51:00.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-03-12T03:51:00.824 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:00.851 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:51:00.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:00.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:00.887 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:51:00.888 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[29344]: 1.12 scrub starts 2024-03-12T03:51:00.888 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[29344]: 1.12 scrub ok 2024-03-12T03:51:00.888 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[29344]: 5.6 scrub starts 2024-03-12T03:51:00.888 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[29344]: 5.6 scrub ok 2024-03-12T03:51:00.888 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[29344]: 1.e scrub starts 2024-03-12T03:51:00.888 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[29344]: 1.e scrub ok 2024-03-12T03:51:00.888 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/454304692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:00.888 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[39949]: 1.12 scrub starts 2024-03-12T03:51:00.889 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[39949]: 1.12 scrub ok 2024-03-12T03:51:00.889 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[39949]: 5.6 scrub starts 2024-03-12T03:51:00.889 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[39949]: 5.6 scrub ok 2024-03-12T03:51:00.889 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[39949]: 1.e scrub starts 2024-03-12T03:51:00.889 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[39949]: 1.e scrub ok 2024-03-12T03:51:00.889 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:00 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/454304692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:00.926 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:51:00.957 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:51:00.957 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:51:00.957 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:51:00.957 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:51:00.957 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:51:00.957 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:51:00.957 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:51:00.957 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:51:00.957 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:51:00.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:01.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:01.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:01.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:01.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:01.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:01.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:01.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:00 smithi173 ceph-mon[33544]: 1.12 scrub starts 2024-03-12T03:51:01.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:00 smithi173 ceph-mon[33544]: 1.12 scrub ok 2024-03-12T03:51:01.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:00 smithi173 ceph-mon[33544]: 5.6 scrub starts 2024-03-12T03:51:01.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:00 smithi173 ceph-mon[33544]: 5.6 scrub ok 2024-03-12T03:51:01.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:00 smithi173 ceph-mon[33544]: 1.e scrub starts 2024-03-12T03:51:01.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:00 smithi173 ceph-mon[33544]: 1.e scrub ok 2024-03-12T03:51:01.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:00 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/454304692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:01.260 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:01.290 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:01.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:01.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:01.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:01.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:01.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:01.492 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:01.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:01.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:01.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:01.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:01.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:51:01.715 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:51:01.728 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:01.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:01.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:01.866 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:01.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:01.869 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:51:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[29344]: pgmap v850: 168 pgs: 3 peering, 4 remapped+peering, 3 active+undersized+remapped, 26 active+undersized+degraded, 34 active+undersized, 98 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 7.5 KiB/s rd, 50 KiB/s wr, 1.74k op/s; 18687/89145 objects degraded (20.962%); 1/89145 objects misplaced (0.001%); 109 KiB/s, 0 objects/s recovering 2024-03-12T03:51:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[29344]: 5.11 scrub starts 2024-03-12T03:51:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[29344]: 5.11 scrub ok 2024-03-12T03:51:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1822400257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:51:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:51:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[29344]: 4.15 scrub starts 2024-03-12T03:51:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[29344]: 4.15 scrub ok 2024-03-12T03:51:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:01.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[39949]: pgmap v850: 168 pgs: 3 peering, 4 remapped+peering, 3 active+undersized+remapped, 26 active+undersized+degraded, 34 active+undersized, 98 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 7.5 KiB/s rd, 50 KiB/s wr, 1.74k op/s; 18687/89145 objects degraded (20.962%); 1/89145 objects misplaced (0.001%); 109 KiB/s, 0 objects/s recovering 2024-03-12T03:51:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[39949]: 5.11 scrub starts 2024-03-12T03:51:01.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[39949]: 5.11 scrub ok 2024-03-12T03:51:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1822400257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:51:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:51:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[39949]: 4.15 scrub starts 2024-03-12T03:51:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[39949]: 4.15 scrub ok 2024-03-12T03:51:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:01.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:01.925 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:51:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:51:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:51:01.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:01 smithi173 ceph-mon[33544]: pgmap v850: 168 pgs: 3 peering, 4 remapped+peering, 3 active+undersized+remapped, 26 active+undersized+degraded, 34 active+undersized, 98 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 7.5 KiB/s rd, 50 KiB/s wr, 1.74k op/s; 18687/89145 objects degraded (20.962%); 1/89145 objects misplaced (0.001%); 109 KiB/s, 0 objects/s recovering 2024-03-12T03:51:01.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:01 smithi173 ceph-mon[33544]: 5.11 scrub starts 2024-03-12T03:51:01.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:01 smithi173 ceph-mon[33544]: 5.11 scrub ok 2024-03-12T03:51:01.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:01 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1822400257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:51:01.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:01 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:51:01.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:01 smithi173 ceph-mon[33544]: 4.15 scrub starts 2024-03-12T03:51:01.967 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:01 smithi173 ceph-mon[33544]: 4.15 scrub ok 2024-03-12T03:51:01.967 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:01 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:01.967 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:01 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:01.968 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:01.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:02.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:02.132 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:02.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:02.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:02.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:02.263 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:02.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:02.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:02.399 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:02.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:02.501 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:02.532 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:02.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:02.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:02.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:02.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:02.774 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:02.804 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:02.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:02.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:02.906 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:02.937 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:02.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:02 smithi173 ceph-mon[33544]: 5.1a scrub starts 2024-03-12T03:51:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:02 smithi173 ceph-mon[33544]: 5.1a scrub ok 2024-03-12T03:51:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:02 smithi173 ceph-mon[33544]: 4.13 scrub starts 2024-03-12T03:51:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:02 smithi173 ceph-mon[33544]: 4.13 scrub ok 2024-03-12T03:51:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:02 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:02 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:51:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:02 smithi173 ceph-mon[33544]: osdmap e421: 8 total, 7 up, 8 in 2024-03-12T03:51:03.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:03.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:03.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:03.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[29344]: 5.1a scrub starts 2024-03-12T03:51:03.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[29344]: 5.1a scrub ok 2024-03-12T03:51:03.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[29344]: 4.13 scrub starts 2024-03-12T03:51:03.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[29344]: 4.13 scrub ok 2024-03-12T03:51:03.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:03.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:51:03.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[29344]: osdmap e421: 8 total, 7 up, 8 in 2024-03-12T03:51:03.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[39949]: 5.1a scrub starts 2024-03-12T03:51:03.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[39949]: 5.1a scrub ok 2024-03-12T03:51:03.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[39949]: 4.13 scrub starts 2024-03-12T03:51:03.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[39949]: 4.13 scrub ok 2024-03-12T03:51:03.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:03.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:51:03.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:02 smithi169 ceph-mon[39949]: osdmap e421: 8 total, 7 up, 8 in 2024-03-12T03:51:03.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:03.265 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:03.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:03.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:03.396 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:03.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:03.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:03.426 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:51:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:51:03.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:03.528 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:03.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:03.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:03.650 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:51:03.650 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "" 2024-03-12T03:51:03.650 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:51:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:03.670 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:03.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:03.716 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:51:03.771 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:03.811 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:03.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:03.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:03.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:03.965 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:03.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:03 smithi173 ceph-mon[33544]: pgmap v852: 168 pgs: 3 active+clean+remapped, 3 peering, 1 remapped+peering, 2 active+undersized+remapped, 26 active+undersized+degraded, 34 active+undersized, 99 active+clean; 1.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.5 KiB/s rd, 20 KiB/s wr, 1.71k op/s; 14817/73230 objects degraded (20.234%); 74 KiB/s, 0 objects/s recovering 2024-03-12T03:51:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:03 smithi173 ceph-mon[33544]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-12T03:51:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:03 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 14817/73230 objects degraded (20.234%), 26 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:04.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:04.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:04.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:04.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:03 smithi169 ceph-mon[29344]: pgmap v852: 168 pgs: 3 active+clean+remapped, 3 peering, 1 remapped+peering, 2 active+undersized+remapped, 26 active+undersized+degraded, 34 active+undersized, 99 active+clean; 1.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.5 KiB/s rd, 20 KiB/s wr, 1.71k op/s; 14817/73230 objects degraded (20.234%); 74 KiB/s, 0 objects/s recovering 2024-03-12T03:51:04.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:03 smithi169 ceph-mon[29344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-12T03:51:04.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:03 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 14817/73230 objects degraded (20.234%), 26 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:04.138 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:03 smithi169 ceph-mon[39949]: pgmap v852: 168 pgs: 3 active+clean+remapped, 3 peering, 1 remapped+peering, 2 active+undersized+remapped, 26 active+undersized+degraded, 34 active+undersized, 99 active+clean; 1.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 3.5 KiB/s rd, 20 KiB/s wr, 1.71k op/s; 14817/73230 objects degraded (20.234%); 74 KiB/s, 0 objects/s recovering 2024-03-12T03:51:04.138 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:03 smithi169 ceph-mon[39949]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-12T03:51:04.139 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:03 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 14817/73230 objects degraded (20.234%), 26 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:04.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:04.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:04.239 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:04.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:04.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:04.382 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:04.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:04.483 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:04.525 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:04.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:04.583 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:04.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:04.675 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:51:04.675 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:51:04.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.675 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:259875 10.8 10:1229d400:::benchmark_data_smithi169_162694_object120880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e421)", 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:51:04.666013+0000", 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0060989690000000001, 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061315140000000002, 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 259875 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:51:04.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.666013+0000", 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.666013+0000", 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.666014+0000", 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2589999999999999e-06 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:51:04.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.666020+0000", 2024-03-12T03:51:04.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.288e-06 2024-03-12T03:51:04.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.678 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:51:04.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.666033+0000", 2024-03-12T03:51:04.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3685e-05 2024-03-12T03:51:04.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.678 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:51:04.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.666041+0000", 2024-03-12T03:51:04.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8960000000000003e-06 2024-03-12T03:51:04.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.666125+0000", 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3416000000000002e-05 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.666247+0000", 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012267800000000001 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.666359+0000", 2024-03-12T03:51:04.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000111659 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:259876 10.8 10:15c0756c:::benchmark_data_smithi169_162694_object120881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e421)", 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:51:04.668324+0000", 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.003787753, 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003939634, 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:04.680 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 259876 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.668324+0000", 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.668324+0000", 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.668325+0000", 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1200000000000001e-06 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.668331+0000", 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4179999999999996e-06 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.668342+0000", 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0757000000000001e-05 2024-03-12T03:51:04.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.668350+0000", 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8849999999999999e-06 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.669573+0000", 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012237229999999999 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.669716+0000", 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014278199999999999 2024-03-12T03:51:04.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.669864+0000", 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014794400000000001 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:259880 10.8 10.6dfe19d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e421)", 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:51:04.671593+0000", 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.00051922999999999997, 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00076840600000000002, 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "queued for pg", 2024-03-12T03:51:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 259880 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.671593+0000", 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.671593+0000", 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:51:04.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.671594+0000", 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3e-06 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.671600+0000", 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.755e-06 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.671614+0000", 2024-03-12T03:51:04.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3879e-05 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.671622+0000", 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4959999999999996e-06 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:259882 10.3 10:c0008965:::benchmark_data_smithi169_162694_object120887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e421)", 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:51:04.672129+0000", 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4294967295.9999833, 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030462399999999998, 2024-03-12T03:51:04.687 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 259882 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.672129+0000", 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.672129+0000", 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.672130+0000", 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9500000000000001e-07 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.672135+0000", 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9030000000000003e-06 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.672156+0000", 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1812000000000001e-05 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.672165+0000", 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2309999999999996e-06 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.672215+0000", 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0348e-05 2024-03-12T03:51:04.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:51:04.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:51:04.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:51:04.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:51:04.672337+0000", 2024-03-12T03:51:04.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121809 2024-03-12T03:51:04.691 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:51:04.691 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:51:04.691 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:51:04.691 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:51:04.691 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:51:04.691 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 4 2024-03-12T03:51:04.691 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:51:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:04.692 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:04.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:04.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:04.822 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:04.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:04.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:04.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:04.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:04 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-12T03:51:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:04 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-12T03:51:04.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:04 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:51:05.069 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:05.070 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:04 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-12T03:51:05.070 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:04 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-12T03:51:05.070 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:04 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:51:05.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:05.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:04 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-12T03:51:05.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:04 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-12T03:51:05.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:04 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:51:05.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:05.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:05.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:05.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:05.248 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:05.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:05.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:05.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:05.389 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:05.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:05.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:51:05.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:05.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:05.530 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:05.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:05.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:05.661 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:05.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:05.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:05.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:05.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:05.793 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:05.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:05.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:05.964 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:05.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:05 smithi173 ceph-mon[33544]: pgmap v853: 168 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 104 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.36k op/s; 13884/66090 objects degraded (21.008%); 38 KiB/s, 0 objects/s recovering 2024-03-12T03:51:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-12T03:51:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:05 smithi173 ceph-mon[33544]: osdmap e422: 8 total, 7 up, 8 in 2024-03-12T03:51:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:05.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:05 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:06.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:06.092 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:06.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:06.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:05 smithi169 ceph-mon[29344]: pgmap v853: 168 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 104 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.36k op/s; 13884/66090 objects degraded (21.008%); 38 KiB/s, 0 objects/s recovering 2024-03-12T03:51:06.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-12T03:51:06.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:05 smithi169 ceph-mon[29344]: osdmap e422: 8 total, 7 up, 8 in 2024-03-12T03:51:06.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:06.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:05 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:05 smithi169 ceph-mon[39949]: pgmap v853: 168 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 104 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.36k op/s; 13884/66090 objects degraded (21.008%); 38 KiB/s, 0 objects/s recovering 2024-03-12T03:51:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-12T03:51:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:05 smithi169 ceph-mon[39949]: osdmap e422: 8 total, 7 up, 8 in 2024-03-12T03:51:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:05 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:06.194 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:06.233 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:06.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:06.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:06.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:06.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:06.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:06.379 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-12T03:51:06.472 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:06.507 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:06.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:06.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:06.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:06.644 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:06.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:06.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:06.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:06.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:06.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:06.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:06.903 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:06.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:06.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:06.936 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:07.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:07.042 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:06 smithi169 ceph-mon[29344]: osdmap e423: 8 total, 7 up, 8 in 2024-03-12T03:51:07.042 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:06 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:51:07.042 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:06 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:51:07.042 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:06 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:07.043 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:06 smithi169 ceph-mon[39949]: osdmap e423: 8 total, 7 up, 8 in 2024-03-12T03:51:07.043 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:06 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:51:07.043 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:06 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:51:07.043 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:06 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:07.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:07.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:07.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:07.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:07.194 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:07.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:07.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:06 smithi173 ceph-mon[33544]: osdmap e423: 8 total, 7 up, 8 in 2024-03-12T03:51:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:06 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:51:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:06 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:51:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:06 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:07.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:07.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:07.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:07.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:07.467 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:07.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:07.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:07.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:07.573 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:51:07.592 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:51:07.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:07.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:07.619 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:51:07.656 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:51:07.684 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:51:07.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:07.732 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:51:07.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:07.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:07.766 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:51:07.766 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:51:07.766 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:51:07.767 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:51:07.767 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:51:07.767 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:51:07.767 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:51:07.767 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:51:07.767 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:51:07.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:07.875 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:07.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:07.976 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:08.017 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:08.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:08.119 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:08.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:08.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:08.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:07 smithi169 ceph-mon[29344]: pgmap v856: 168 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 104 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.28k op/s; 13884/66090 objects degraded (21.008%) 2024-03-12T03:51:08.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:07 smithi169 ceph-mon[29344]: osdmap e424: 8 total, 7 up, 8 in 2024-03-12T03:51:08.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:07 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/662623803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:08.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:07 smithi169 ceph-mon[39949]: pgmap v856: 168 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 104 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.28k op/s; 13884/66090 objects degraded (21.008%) 2024-03-12T03:51:08.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:07 smithi169 ceph-mon[39949]: osdmap e424: 8 total, 7 up, 8 in 2024-03-12T03:51:08.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:07 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/662623803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:08.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:07 smithi173 ceph-mon[33544]: pgmap v856: 168 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 104 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.28k op/s; 13884/66090 objects degraded (21.008%) 2024-03-12T03:51:08.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:07 smithi173 ceph-mon[33544]: osdmap e424: 8 total, 7 up, 8 in 2024-03-12T03:51:08.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:07 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/662623803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:08.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:08.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:08.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:08.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:08.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:08.434 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:08.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:08.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:08.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:08.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:08.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:08.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:51:08.683 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:51:08.707 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:08.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:08.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:08.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:08.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:08.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:08.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:08.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:08.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:08.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:09.023 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:08 smithi169 ceph-mon[39949]: osdmap e425: 8 total, 7 up, 8 in 2024-03-12T03:51:09.023 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:08 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-12T03:51:09.023 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:08 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-12T03:51:09.023 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:08 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 13884/66090 objects degraded (21.008%), 26 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:09.023 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:08 smithi169 ceph-mon[29344]: osdmap e425: 8 total, 7 up, 8 in 2024-03-12T03:51:09.023 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:08 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-12T03:51:09.023 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:08 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-12T03:51:09.023 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:08 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 13884/66090 objects degraded (21.008%), 26 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:09.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:09.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:09.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:09.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:09.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:08 smithi173 ceph-mon[33544]: osdmap e425: 8 total, 7 up, 8 in 2024-03-12T03:51:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:08 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-12T03:51:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:08 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-12T03:51:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:08 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 13884/66090 objects degraded (21.008%), 26 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:09.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:09.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:09.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:09.362 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:09.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:09.395 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:09.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:09.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:09.526 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:09.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:09.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:09.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:09.653 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:09.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:09.720 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:51:09.720 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:51:09.720 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-12T03:51:09.720 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-12T03:51:09.720 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-12T03:51:09.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-12T03:51:09.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:09.802 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:09.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:09.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:09.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:09.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:09.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:10.036 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:09 smithi169 ceph-mon[29344]: pgmap v859: 167 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 103 active+clean; 995 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 9627/48402 objects degraded (19.890%) 2024-03-12T03:51:10.036 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-12T03:51:10.036 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:09 smithi169 ceph-mon[29344]: osdmap e426: 8 total, 7 up, 8 in 2024-03-12T03:51:10.037 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:09 smithi169 ceph-mon[39949]: pgmap v859: 167 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 103 active+clean; 995 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 9627/48402 objects degraded (19.890%) 2024-03-12T03:51:10.037 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-12T03:51:10.037 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:09 smithi169 ceph-mon[39949]: osdmap e426: 8 total, 7 up, 8 in 2024-03-12T03:51:10.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:10.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:10.090 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:10.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:10.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:10.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:10.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:10.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:10.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:09 smithi173 ceph-mon[33544]: pgmap v859: 167 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 103 active+clean; 995 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 9627/48402 objects degraded (19.890%) 2024-03-12T03:51:10.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-12T03:51:10.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:09 smithi173 ceph-mon[33544]: osdmap e426: 8 total, 7 up, 8 in 2024-03-12T03:51:10.336 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:10.368 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:10.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:10.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:10.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:10.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:10.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:10.599 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:10.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:10.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:10.644 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:10.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:10.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:51:10.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:10.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:10.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:10.880 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:10.933 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:10.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:11.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:11.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:11.071 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:11.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:11.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:11.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:10 smithi169 ceph-mon[29344]: osdmap e427: 8 total, 7 up, 8 in 2024-03-12T03:51:11.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:10 smithi169 ceph-mon[29344]: pgmap v862: 167 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 103 active+clean; 938 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s; 9627/45693 objects degraded (21.069%) 2024-03-12T03:51:11.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:10 smithi169 ceph-mon[39949]: osdmap e427: 8 total, 7 up, 8 in 2024-03-12T03:51:11.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:10 smithi169 ceph-mon[39949]: pgmap v862: 167 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 103 active+clean; 938 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s; 9627/45693 objects degraded (21.069%) 2024-03-12T03:51:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:11.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:11.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:11.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:10 smithi173 ceph-mon[33544]: osdmap e427: 8 total, 7 up, 8 in 2024-03-12T03:51:11.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:10 smithi173 ceph-mon[33544]: pgmap v862: 167 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 103 active+clean; 938 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s; 9627/45693 objects degraded (21.069%) 2024-03-12T03:51:11.301 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:51:11.301 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:51:11.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:11.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:11.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:11.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:11.474 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:11.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:11.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:11.576 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:11.606 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:11.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:11.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:11.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:11.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:11.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:11.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:11.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:11.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:11.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:11.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:11.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:51:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:51:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:51:11.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:12.018 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:12.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:12.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:12.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:12.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:12.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:12.203 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:11 smithi169 ceph-mon[39949]: osdmap e428: 8 total, 7 up, 8 in 2024-03-12T03:51:12.203 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:11 smithi169 ceph-mon[29344]: osdmap e428: 8 total, 7 up, 8 in 2024-03-12T03:51:12.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:11 smithi173 ceph-mon[33544]: osdmap e428: 8 total, 7 up, 8 in 2024-03-12T03:51:12.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:12.293 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:12.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:12.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:12.432 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:12.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:12.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:12.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:12.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:12.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:12.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:12.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-12T03:51:12.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:12.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:12.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:12.805 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:12.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:12.850 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:12.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:12.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:12.988 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:12.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:13.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:13.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:13.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:12 smithi169 ceph-mon[29344]: osdmap e429: 8 total, 7 up, 8 in 2024-03-12T03:51:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:12 smithi169 ceph-mon[29344]: pgmap v865: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 610 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 5839/29949 objects degraded (19.496%) 2024-03-12T03:51:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:12 smithi169 ceph-mon[39949]: osdmap e429: 8 total, 7 up, 8 in 2024-03-12T03:51:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:12 smithi169 ceph-mon[39949]: pgmap v865: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 610 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 5839/29949 objects degraded (19.496%) 2024-03-12T03:51:13.232 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:13.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:12 smithi173 ceph-mon[33544]: osdmap e429: 8 total, 7 up, 8 in 2024-03-12T03:51:13.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:12 smithi173 ceph-mon[33544]: pgmap v865: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 610 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 5839/29949 objects degraded (19.496%) 2024-03-12T03:51:13.268 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:13.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:13.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:13.377 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:13.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:13.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:13.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:13.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:13.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:13.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:13.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:13.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:13.676 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:13.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:13.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:13.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:51:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:51:13.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:13.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:13.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:13.920 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:13.946 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:13.955 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:13.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:13.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:13.998 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:51:14.038 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:51:14.038 INFO:teuthology.orchestra.run.smithi169.stdout: "success": "" 2024-03-12T03:51:14.038 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:51:14.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:14.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:14.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:14.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:14.105 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:13 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2165184860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:14.105 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:13 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:14.105 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:13 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:14.105 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:13 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 5839/29949 objects degraded (19.496%), 26 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:14.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:13 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2165184860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:14.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:13 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:14.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:13 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:14.106 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:13 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 5839/29949 objects degraded (19.496%), 26 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:14.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:14.232 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:14.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:14.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:13 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2165184860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:14.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:13 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:14.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:13 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:14.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:13 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 5839/29949 objects degraded (19.496%), 26 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:14.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:14.369 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:14.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:14.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:14.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:14.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:14.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:14.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:14.648 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:14.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:14.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:14.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:14.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:14.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:14.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:14.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:14.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:15.028 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:15.058 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:15.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:15.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:15.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:14 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:15.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:14 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:51:15.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:14 smithi169 ceph-mon[29344]: osdmap e430: 8 total, 7 up, 8 in 2024-03-12T03:51:15.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:14 smithi169 ceph-mon[29344]: pgmap v867: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 466 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.78k op/s; 4977/23055 objects degraded (21.588%) 2024-03-12T03:51:15.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:14 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:15.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:14 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:51:15.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:14 smithi169 ceph-mon[39949]: osdmap e430: 8 total, 7 up, 8 in 2024-03-12T03:51:15.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:14 smithi169 ceph-mon[39949]: pgmap v867: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 466 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.78k op/s; 4977/23055 objects degraded (21.588%) 2024-03-12T03:51:15.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:15.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:15.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:15.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:15.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:15.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:15.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:15.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:15.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:15.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:15.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:14 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:15.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:14 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:51:15.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:14 smithi173 ceph-mon[33544]: osdmap e430: 8 total, 7 up, 8 in 2024-03-12T03:51:15.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:14 smithi173 ceph-mon[33544]: pgmap v867: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 466 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.78k op/s; 4977/23055 objects degraded (21.588%) 2024-03-12T03:51:15.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:15.611 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:15.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:15.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:15.755 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:15.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:15.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:15.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:15.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:15.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:16.013 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:16.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:16.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:16.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:16.146 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:16.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:16.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:16.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:16.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:16.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:16.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:16.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:16.418 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:16.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:16.520 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:16.545 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:16.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:16.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:16.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:16.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:16.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:16.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:16.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:16.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:16.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:16.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:16.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:16.960 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:16.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:17.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:17.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:17.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:17.144 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Removed 138992 objects 2024-03-12T03:51:17.144 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Clean up completed and total clean up time :94.0725 2024-03-12T03:51:17.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:17.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:17.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:17.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:17.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:17.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:17.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:17.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:17.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:17.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:17.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:17.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:17.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:17.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:17.722 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:17.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:17.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:17.824 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:17.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:17 smithi169 ceph-mon[29344]: pgmap v868: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 466 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s; 4977/23055 objects degraded (21.588%) 2024-03-12T03:51:17.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:17 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:17.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:17 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:17.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:17 smithi169 ceph-mon[39949]: pgmap v868: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 466 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s; 4977/23055 objects degraded (21.588%) 2024-03-12T03:51:17.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:17 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:17.826 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:17 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:17.851 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:17.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:17.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:17.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:17.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:17.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:17 smithi173 ceph-mon[33544]: pgmap v868: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 466 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s; 4977/23055 objects degraded (21.588%) 2024-03-12T03:51:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:17 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:17 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:18.079 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:18.103 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:18.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:18.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:18.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:18.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:18.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:18.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:18.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:18.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:18.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:18.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:18.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:18.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:18.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:18.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:18.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:18.725 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:18.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:18.753 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:18.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:18.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:18.882 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:18.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:18.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:18.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:18 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:51:18.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:18 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:51:18.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:18 smithi169 ceph-mon[29344]: osdmap e431: 8 total, 7 up, 8 in 2024-03-12T03:51:18.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:18 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 4977/23055 objects degraded (21.588%), 26 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:18.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:18 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:51:18.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:18 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:51:18.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:18 smithi169 ceph-mon[39949]: osdmap e431: 8 total, 7 up, 8 in 2024-03-12T03:51:18.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:18 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 4977/23055 objects degraded (21.588%), 26 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:18.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:18 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:51:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:18 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:51:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:18 smithi173 ceph-mon[33544]: osdmap e431: 8 total, 7 up, 8 in 2024-03-12T03:51:18.988 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:18 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 4977/23055 objects degraded (21.588%), 26 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:19.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:19.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:19.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:19.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:51:19.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:19.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:19.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:19.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:19.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:19.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:19.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:19.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:19.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:19.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:19.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:19.528 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:19.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:19.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:19.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:19.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:19.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:19.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:19.784 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:19.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:19.826 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:19 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:51:19.827 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:19 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:51:19.827 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:19 smithi169 ceph-mon[29344]: pgmap v870: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 74 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 821 B/s rd, 0 B/s wr, 1.38k op/s; 459/4239 objects degraded (10.828%) 2024-03-12T03:51:19.827 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:19 smithi169 ceph-mon[29344]: osdmap e432: 8 total, 7 up, 8 in 2024-03-12T03:51:19.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:19 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:51:19.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:19 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:51:19.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:19 smithi169 ceph-mon[39949]: pgmap v870: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 74 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 821 B/s rd, 0 B/s wr, 1.38k op/s; 459/4239 objects degraded (10.828%) 2024-03-12T03:51:19.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:19 smithi169 ceph-mon[39949]: osdmap e432: 8 total, 7 up, 8 in 2024-03-12T03:51:19.873 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:51:19.873 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:51:19.873 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_3 2024-03-12T03:51:19.873 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_3 by 10 min 8 2024-03-12T03:51:19.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:51:19.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:19.919 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:19.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:19 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:51:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:19 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:51:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:19 smithi173 ceph-mon[33544]: pgmap v870: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 74 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 821 B/s rd, 0 B/s wr, 1.38k op/s; 459/4239 objects degraded (10.828%) 2024-03-12T03:51:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:19 smithi173 ceph-mon[33544]: osdmap e432: 8 total, 7 up, 8 in 2024-03-12T03:51:20.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:20.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:20.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:20.121 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:51:20.140 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:51:20.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:20.160 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:51:20.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:20.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:20.181 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:51:20.200 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:51:20.200 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:51:20.201 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:51:20.221 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:51:20.241 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:51:20.242 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:51:20.242 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:51:20.242 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:51:20.242 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:51:20.242 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:51:20.242 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:51:20.242 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:51:20.242 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:51:20.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:20.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:20.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:20.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:20.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:20.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:20.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:20.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:20.559 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:20.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:20.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:20.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:20.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:20.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:20.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-12T03:51:20.773 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:20 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1651544640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:20.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:20 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1651544640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:20.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:20.811 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:20.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:20.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:20.937 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:20.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:20.947 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:20.971 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:20 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1651544640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:21.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:21.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:21.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:21.164 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:21.188 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:21.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:21.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:21.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:21.315 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:21.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:21.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:51:21.438 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:51:21.441 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:21.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:21.524 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:21.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:21.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:21.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:21.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:21.694 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:21.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:21.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:21.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:21.820 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:21.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:21.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:21.945 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:21.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:21 smithi173 ceph-mon[33544]: pgmap v872: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 34 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 823 B/s rd, 0 B/s wr, 1.11k op/s; 459/2295 objects degraded (20.000%) 2024-03-12T03:51:22.047 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:22.072 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:22.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:22.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:21 smithi169 ceph-mon[29344]: pgmap v872: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 34 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 823 B/s rd, 0 B/s wr, 1.11k op/s; 459/2295 objects degraded (20.000%) 2024-03-12T03:51:22.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:51:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:51:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:51:22.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:21 smithi169 ceph-mon[39949]: pgmap v872: 166 pgs: 4 active+clean+remapped, 26 active+undersized+degraded, 34 active+undersized, 102 active+clean; 34 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 823 B/s rd, 0 B/s wr, 1.11k op/s; 459/2295 objects degraded (20.000%) 2024-03-12T03:51:22.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:22.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:22.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:22.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:22.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:22.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:22.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:22.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:22.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:22.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:22.549 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:22.576 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:22.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:22.652 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:51:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:51:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-03-12T03:51:22.653 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:51:22.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:22.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:22.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:22.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:22.803 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:22.830 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:22.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:22.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:22.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:22.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:22.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:23.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:23.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:23.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:23.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:23.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:23.211 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:23.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:23.312 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:23.341 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:23.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:23.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-12T03:51:23.439 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:51:23.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:23.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:23.466 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:23.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:23.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:51:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:51:23.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:23.592 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:23.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:23.605 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:23.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:23.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:23.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:23.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:23.793 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:23 smithi169 ceph-mon[29344]: pgmap v873: 166 pgs: 4 active+clean+remapped, 18 active+undersized+degraded, 42 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 932 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:23.793 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:23 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 73/690 objects degraded (10.580%), 18 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:23.793 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:23 smithi169 ceph-mon[29344]: osdmap e433: 8 total, 7 up, 8 in 2024-03-12T03:51:23.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:23 smithi169 ceph-mon[39949]: pgmap v873: 166 pgs: 4 active+clean+remapped, 18 active+undersized+degraded, 42 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 932 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:23.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:23 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 73/690 objects degraded (10.580%), 18 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:23.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:23 smithi169 ceph-mon[39949]: osdmap e433: 8 total, 7 up, 8 in 2024-03-12T03:51:23.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:23.844 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:23.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:23.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:23.971 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:23.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:23 smithi173 ceph-mon[33544]: pgmap v873: 166 pgs: 4 active+clean+remapped, 18 active+undersized+degraded, 42 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 932 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:23 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 73/690 objects degraded (10.580%), 18 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:23 smithi173 ceph-mon[33544]: osdmap e433: 8 total, 7 up, 8 in 2024-03-12T03:51:24.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:24.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:24.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:24.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:24.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:24.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:24.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:24.343 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:24.347 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:24.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:24.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:24.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:24.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:24.575 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:24.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:24.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:24.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:24.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:24.726 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:24.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:24.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:24.851 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:24.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:24.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:24.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:24.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:24.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:25.033 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:25.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:25.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:25.106 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:25.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:25.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:25.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:25.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:25.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:25.287 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:51:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:51:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:51:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:51:25.287 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:51:25.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:25.358 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:25.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:25.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:25.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:25.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:25.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:25.609 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:25.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:25.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:25.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:25.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:25.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:25.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:25.862 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:25.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:25.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-12T03:51:25.932 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:25 smithi169 ceph-mon[29344]: pgmap v875: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 785 B/s rd, 0 B/s wr, 359 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:25.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:25 smithi169 ceph-mon[39949]: pgmap v875: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 785 B/s rd, 0 B/s wr, 359 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:25.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:25 smithi173 ceph-mon[33544]: pgmap v875: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 785 B/s rd, 0 B/s wr, 359 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:25.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:25.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:25.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:26.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:26.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:26.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:26.128 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:26.214 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:26.240 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:26.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:26.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:26.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:26.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:26.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:26.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:26.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:26.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:26.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:26.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:26.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:26.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:26.713 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:51:26.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:26 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:51:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:26 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:51:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:26 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/482133931' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:51:26.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:26 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:51:26.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:26.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:26.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:26.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:26.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:26.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:26.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:26 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:51:26.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:26 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:51:26.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:26 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/482133931' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:51:26.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:26 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:51:26.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:26 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:51:26.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:26 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:51:26.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:26 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/482133931' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:51:26.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:26 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:51:26.970 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:26.995 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:26.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:27.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:27.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:27.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:27.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:27.245 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:27.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:27.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:27.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:27.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:27.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:27.472 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:27.496 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:27.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:27.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:27.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:27.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:27.709 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:27 smithi169 ceph-mon[29344]: pgmap v876: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 148 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:27.709 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:27 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:27.709 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:27 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:51:27.709 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:27 smithi169 ceph-mon[29344]: osdmap e434: 8 total, 7 up, 8 in 2024-03-12T03:51:27.710 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:27 smithi169 ceph-mon[39949]: pgmap v876: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 148 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:27.710 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:27 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:27.710 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:27 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:51:27.710 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:27 smithi169 ceph-mon[39949]: osdmap e434: 8 total, 7 up, 8 in 2024-03-12T03:51:27.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:27.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:27.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:27.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:27.876 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:27.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:27.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:27.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:27 smithi173 ceph-mon[33544]: pgmap v876: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 148 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:27.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:27 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:27.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:27 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:51:27.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:27 smithi173 ceph-mon[33544]: osdmap e434: 8 total, 7 up, 8 in 2024-03-12T03:51:27.981 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:51:27.981 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:29246 10.d 10:b0bae9f3:::benchmark_data_smithi169_162694_object29245:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:24.390047+0000", 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.586551136, 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.082813597000000003, 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29246 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.390047+0000", 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.390047+0000", 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.390049+0000", 2024-03-12T03:51:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5200000000000001e-06 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.390087+0000", 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8541999999999998e-05 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.390097+0000", 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9359999999999994e-06 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.390106+0000", 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0960000000000007e-06 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.390531+0000", 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00042460099999999998 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.390745+0000", 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021406999999999999 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,6", 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.390903+0000", 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015836299999999999 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.392339+0000", 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014352950000000001 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.394541+0000", 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00220221 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.472783+0000", 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078241643 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.472824+0000", 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1538000000000001e-05 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.472861+0000", 2024-03-12T03:51:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6782999999999997e-05 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:29250 10.d 10:b728cea9:::benchmark_data_smithi169_162694_object29249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:24.392412+0000", 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.58418633799999, 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.083460758999999995, 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29250 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.392412+0000", 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.392412+0000", 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.392413+0000", 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.108e-06 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.392443+0000", 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9927999999999999e-05 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.392453+0000", 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0156999999999999e-05 2024-03-12T03:51:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.392461+0000", 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2690000000000002e-06 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.392638+0000", 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000177703 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.392776+0000", 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013818500000000001 2024-03-12T03:51:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,6", 2024-03-12T03:51:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.392917+0000", 2024-03-12T03:51:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000141073 2024-03-12T03:51:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.394776+0000", 2024-03-12T03:51:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001858667 2024-03-12T03:51:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.394893+0000", 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117235 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.475757+0000", 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.080863396000000004 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.475825+0000", 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8549999999999999e-05 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.475873+0000", 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7487999999999997e-05 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:29256 10.1 10:830f94bf:::benchmark_data_smithi169_162694_object29255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:51:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:24.394304+0000", 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.58229413500001, 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.083358747999999996, 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29256 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.394304+0000", 2024-03-12T03:51:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.394304+0000", 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.394305+0000", 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.083e-06 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.394343+0000", 2024-03-12T03:51:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8099000000000002e-05 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.394358+0000", 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4773000000000001e-05 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.394366+0000", 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.661e-06 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.394416+0000", 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0287999999999997e-05 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.394537+0000", 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012070000000000001 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.394679+0000", 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000142383 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.397290+0000", 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026105 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.397805+0000", 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051486999999999995 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.477596+0000", 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.079791301999999995 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.477631+0000", 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4578e-05 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.477663+0000", 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2511000000000001e-05 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:29258 10.1 10:83b40c02:::benchmark_data_smithi169_162694_object29257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:24.395159+0000", 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.58143950600001, 2024-03-12T03:51:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.082590419999999998, 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29258 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.395159+0000", 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.395159+0000", 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.395160+0000", 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6400000000000001e-07 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.395209+0000", 2024-03-12T03:51:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9725999999999998e-05 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.395222+0000", 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2277e-05 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.395229+0000", 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.43e-06 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.395874+0000", 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00064523 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.396010+0000", 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013529699999999999 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.396150+0000", 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000140555 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.397379+0000", 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001229094 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.399027+0000", 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001648011 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.477694+0000", 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078667031999999998 2024-03-12T03:51:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.477725+0000", 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0196e-05 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.477749+0000", 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4808e-05 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:79290 10.d 10:b8ff49b1:::benchmark_data_smithi169_162694_object79289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.546097+0000", 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 150.43050179299999, 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91427323500000002, 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 79290 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.546097+0000", 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.546097+0000", 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.546097+0000", 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8100000000000002e-07 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.546153+0000", 2024-03-12T03:51:28.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5469e-05 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.546163+0000", 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0549e-05 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.546172+0000", 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1799999999999996e-06 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.546246+0000", 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.436e-05 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.546454+0000", 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020808900000000001 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,6", 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.547005+0000", 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00055060099999999996 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.550068+0000", 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0030637849999999999 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.551457+0000", 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001388387 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.564194+0000", 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012736997 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.564206+0000", 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2604e-05 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.460177+0000", 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89597082800000005 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.460370+0000", 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019260499999999999 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:79292 10.d 10:b5ef383d:::benchmark_data_smithi169_162694_object79291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.551498+0000", 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 150.42510020899999, 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90924486900000001, 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 79292 2024-03-12T03:51:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.551498+0000", 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.551498+0000", 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.551499+0000", 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2979999999999999e-06 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.551549+0000", 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9428999999999997e-05 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.551560+0000", 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1062e-05 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.551566+0000", 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3049999999999998e-06 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.551620+0000", 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3421000000000002e-05 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.551779+0000", 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000158806 2024-03-12T03:51:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,6", 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.551881+0000", 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102069 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.564258+0000", 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012377324 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.564267+0000", 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2720000000000003e-06 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.460682+0000", 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89641492899999997 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.460743+0000", 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0954e-05 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:79303 10.d 10:bd58d03c:::benchmark_data_smithi169_162694_object79302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.576185+0000", 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 150.40041304900001, 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89468187200000004, 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 79303 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.576185+0000", 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.576185+0000", 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.576187+0000", 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.897e-06 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.576216+0000", 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9016000000000001e-05 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.576225+0000", 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2180000000000005e-06 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.576235+0000", 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8090000000000004e-06 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.576280+0000", 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5037999999999998e-05 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.576290+0000", 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6569999999999992e-06 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.460771+0000", 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88448145899999997 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.460897+0000", 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012562599999999999 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.461057+0000", 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000159507 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.467108+0000", 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0060510980000000004 2024-03-12T03:51:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.470716+0000", 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003608048 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.470778+0000", 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2048e-05 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.470867+0000", 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9450999999999994e-05 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:79311 10.1 10:8bd419bb:::benchmark_data_smithi169_162694_object79310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.584817+0000", 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 150.3917817, 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88212886400000001, 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 79311 2024-03-12T03:51:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.584817+0000", 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.584817+0000", 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.584818+0000", 2024-03-12T03:51:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5799999999999998e-07 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.584879+0000", 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1397999999999997e-05 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.584890+0000", 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1284000000000001e-05 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.584898+0000", 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5070000000000003e-06 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.584950+0000", 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2471e-05 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.584968+0000", 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7946999999999999e-05 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.460112+0000", 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87514351400000001 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.460278+0000", 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000166124 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7", 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.460461+0000", 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018322799999999999 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.463276+0000", 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002815274 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.466783+0000", 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035065880000000002 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.466846+0000", 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3093000000000001e-05 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.466946+0000", 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9578000000000001e-05 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:79315 10.4 10:265c33b9:::benchmark_data_smithi169_162694_object79314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.591618+0000", 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 150.38498019400001, 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90104617300000001, 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 79315 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.591618+0000", 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.591618+0000", 2024-03-12T03:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.591620+0000", 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3009999999999999e-06 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.591667+0000", 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7824999999999997e-05 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.591682+0000", 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4769999999999999e-05 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.591690+0000", 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3399999999999998e-06 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.591736+0000", 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5488999999999998e-05 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.591742+0000", 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6849999999999998e-06 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.471079+0000", 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87933774899999995 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.472915+0000", 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001835628 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3", 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.473181+0000", 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026634599999999999 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.484635+0000", 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011454034 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.492515+0000", 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0078793809999999995 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.492571+0000", 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5939000000000001e-05 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.492664+0000", 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3685999999999997e-05 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:79318 10.4 10:251e1634:::benchmark_data_smithi169_162694_object79317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.596479+0000", 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 150.38011909400001, 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89640360200000002, 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 79318 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.596479+0000", 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.596479+0000", 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.596480+0000", 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4499999999999997e-07 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.596530+0000", 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0454999999999997e-05 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.596538+0000", 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1760000000000004e-06 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.596542+0000", 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7550000000000002e-06 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.596580+0000", 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7431999999999997e-05 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.596585+0000", 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.079e-06 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.480755+0000", 2024-03-12T03:51:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88417012800000006 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.480891+0000", 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013589099999999999 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3", 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.481078+0000", 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018701199999999999 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.484643+0000", 2024-03-12T03:51:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035653239999999999 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.492734+0000", 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0080910099999999992 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.492796+0000", 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1477999999999997e-05 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.492883+0000", 2024-03-12T03:51:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7217000000000002e-05 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:79320 10.d 10:b83ee52e:::benchmark_data_smithi169_162694_object79319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.600669+0000", 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 150.37592905700001, 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87036050499999995, 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 79320 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.600669+0000", 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.600669+0000", 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.600670+0000", 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4700000000000001e-07 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.600701+0000", 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1133999999999997e-05 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.600711+0000", 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9639999999999998e-06 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.600717+0000", 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4739999999999996e-06 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.600789+0000", 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2009999999999997e-05 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.600795+0000", 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8329999999999999e-06 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.463625+0000", 2024-03-12T03:51:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.86283062799999999 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.463767+0000", 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014169900000000001 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.463954+0000", 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018764000000000001 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.470903+0000", 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0069486469999999996 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.470912+0000", 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5529999999999993e-06 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.470946+0000", 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4397000000000003e-05 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.471030+0000", 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3778999999999999e-05 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24359.0:3628 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94067641428096 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:58.204095+0000", 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 149.772503642, 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26745803800000001, 2024-03-12T03:51:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24359", 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 3628 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204095+0000", 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204095+0000", 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204096+0000", 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4270000000000001e-06 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204105+0000", 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3079999999999997e-06 2024-03-12T03:51:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204118+0000", 2024-03-12T03:51:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2405999999999999e-05 2024-03-12T03:51:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204127+0000", 2024-03-12T03:51:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4080000000000004e-06 2024-03-12T03:51:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204193+0000", 2024-03-12T03:51:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5685999999999994e-05 2024-03-12T03:51:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204203+0000", 2024-03-12T03:51:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0363000000000001e-05 2024-03-12T03:51:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.456140+0000", 2024-03-12T03:51:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25193637499999999 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.456297+0000", 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000157293 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.471553+0000", 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.015255772000000001 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:111756 10.0 e389/366 10:0393f551:::benchmark_data_smithi169_162694_object111755:head v 389'7047, mlcod=389'7045)", 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:21.425708+0000", 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 126.55089029299999, 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.082895163999999993, 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425708+0000", 2024-03-12T03:51:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425708+0000", 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425709+0000", 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.37e-07 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425815+0000", 2024-03-12T03:51:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010610199999999999 2024-03-12T03:51:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425824+0000", 2024-03-12T03:51:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9590000000000001e-06 2024-03-12T03:51:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425830+0000", 2024-03-12T03:51:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3010000000000003e-06 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425878+0000", 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7777e-05 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425897+0000", 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9188999999999999e-05 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.508561+0000", 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.082663898999999999 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.508603+0000", 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2299999999999998e-05 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:215897 10.e 10:758b476d:::benchmark_data_smithi169_162694_object76902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:50:35.501024+0000", 2024-03-12T03:51:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 52.475574534000003, 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.084192144999999996, 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 215897 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.501024+0000", 2024-03-12T03:51:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.501024+0000", 2024-03-12T03:51:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:28.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:28.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.501025+0000", 2024-03-12T03:51:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5799999999999998e-07 2024-03-12T03:51:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.501028+0000", 2024-03-12T03:51:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5949999999999999e-06 2024-03-12T03:51:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.501036+0000", 2024-03-12T03:51:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2239999999999998e-06 2024-03-12T03:51:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.501041+0000", 2024-03-12T03:51:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7359999999999998e-06 2024-03-12T03:51:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.501222+0000", 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000180373 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.501785+0000", 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056356899999999998 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,6", 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.502748+0000", 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00096243000000000001 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.505608+0000", 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028604099999999999 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.506851+0000", 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012429089999999999 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.585114+0000", 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078262618000000006 2024-03-12T03:51:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.585172+0000", 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8779000000000001e-05 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:35.585216+0000", 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3544000000000002e-05 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:219701 10.1 10:88b2268e:::benchmark_data_smithi169_162694_object80706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:50:38.046696+0000", 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 49.929902167999998, 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.085030514000000001, 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 219701 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.046696+0000", 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.046696+0000", 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.046697+0000", 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0759999999999999e-06 2024-03-12T03:51:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.046702+0000", 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0830000000000003e-06 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.092418+0000", 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.045715801 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.092431+0000", 2024-03-12T03:51:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2371000000000001e-05 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.094943+0000", 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025123739999999999 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.095078+0000", 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013497499999999999 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7", 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.095203+0000", 2024-03-12T03:51:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000124869 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.101409+0000", 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0062063919999999998 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.131540+0000", 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.030131143999999999 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.131639+0000", 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8679999999999997e-05 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:50:38.131727+0000", 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7749000000000003e-05 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:257773 10.e 10:780eac3c:::benchmark_data_smithi169_162694_object118778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e421)", 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:51:03.091244+0000", 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 24.885354255999999, 2024-03-12T03:51:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.082944963999999996, 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 257773 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.091244+0000", 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.091244+0000", 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.091245+0000", 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.065e-06 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.091250+0000", 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7929999999999997e-06 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.091258+0000", 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5950000000000001e-06 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.091264+0000", 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.533e-06 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.091349+0000", 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5266999999999995e-05 2024-03-12T03:51:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.091457+0000", 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107431 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,6", 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.091561+0000", 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010377700000000001 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.094579+0000", 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0030186229999999998 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.095276+0000", 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069655400000000005 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.174073+0000", 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078797014999999998 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.174125+0000", 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2222000000000002e-05 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:03.174189+0000", 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4089000000000003e-05 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24359.0:3919 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94067641424640 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:51:21.841063+0000", 2024-03-12T03:51:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.1355357789999996, 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073224100000000001, 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24359", 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 3919 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841063+0000", 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841063+0000", 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841065+0000", 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4200000000000001e-06 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841076+0000", 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1229e-05 2024-03-12T03:51:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841112+0000", 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5599999999999998e-05 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841136+0000", 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3731999999999999e-05 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841606+0000", 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00046995799999999998 2024-03-12T03:51:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841683+0000", 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7553999999999996e-05 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841795+0000", 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111748 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24359.0:3920 5.e 5:7759931f:::notify.3:head [watch ping cookie 94067641428096 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:51:21.841219+0000", 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.135379726, 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051348400000000003, 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24359", 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 3920 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841219+0000", 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841219+0000", 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841220+0000", 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2100000000000001e-06 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841227+0000", 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.08e-06 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841246+0000", 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9483e-05 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841263+0000", 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6586e-05 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841520+0000", 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000256844 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841590+0000", 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9635000000000007e-05 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841732+0000", 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014264599999999999 2024-03-12T03:51:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24359.0:3921 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94067646758912 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:51:21.841299+0000", 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.135299131, 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038851899999999998, 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24359", 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:51:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 3921 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841299+0000", 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841299+0000", 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841300+0000", 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1799999999999999e-06 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841307+0000", 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9569999999999996e-06 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841323+0000", 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6039999999999999e-05 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841339+0000", 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5273000000000001e-05 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841447+0000", 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108117 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841565+0000", 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011845099999999999 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:21.841688+0000", 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012250100000000001 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24359.0:3929 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94067641424640 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:51:26.842124+0000", 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.1344742270000001, 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035145, 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24359", 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 3929 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:26.842124+0000", 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:26.842124+0000", 2024-03-12T03:51:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:26.842125+0000", 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0400000000000005e-07 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:26.842129+0000", 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4299999999999999e-06 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:26.842137+0000", 2024-03-12T03:51:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3370000000000002e-06 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:26.842145+0000", 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2830000000000001e-06 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:26.842269+0000", 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012415500000000001 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:26.842343+0000", 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3983e-05 2024-03-12T03:51:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:26.842476+0000", 2024-03-12T03:51:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013245800000000001 2024-03-12T03:51:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:28.098 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:51:28.166 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:28.190 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:28.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:28.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:28.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:28.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:28.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:28.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:28.441 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:28.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:28.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:28.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:28.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:28.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:28.668 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:28.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:28.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:28.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:28.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:28.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:28.853 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:51:28.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:28 smithi169 ceph-mon[29344]: 1.c scrub starts 2024-03-12T03:51:28.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:28 smithi169 ceph-mon[29344]: 1.c scrub ok 2024-03-12T03:51:28.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:28 smithi169 ceph-mon[29344]: 3.c scrub starts 2024-03-12T03:51:28.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:28 smithi169 ceph-mon[29344]: 3.c scrub ok 2024-03-12T03:51:28.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:28 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 73/690 objects degraded (10.580%), 16 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:28.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:28 smithi169 ceph-mon[29344]: osdmap e435: 8 total, 7 up, 8 in 2024-03-12T03:51:28.854 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:28 smithi169 ceph-mon[39949]: 1.c scrub starts 2024-03-12T03:51:28.854 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:28 smithi169 ceph-mon[39949]: 1.c scrub ok 2024-03-12T03:51:28.854 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:28 smithi169 ceph-mon[39949]: 3.c scrub starts 2024-03-12T03:51:28.854 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:28 smithi169 ceph-mon[39949]: 3.c scrub ok 2024-03-12T03:51:28.854 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:28 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 73/690 objects degraded (10.580%), 16 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:28.854 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:28 smithi169 ceph-mon[39949]: osdmap e435: 8 total, 7 up, 8 in 2024-03-12T03:51:28.871 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:51:28.891 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:51:28.911 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:51:28.920 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:28.926 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:28 smithi173 ceph-mon[33544]: 1.c scrub starts 2024-03-12T03:51:28.926 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:28 smithi173 ceph-mon[33544]: 1.c scrub ok 2024-03-12T03:51:28.926 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:28 smithi173 ceph-mon[33544]: 3.c scrub starts 2024-03-12T03:51:28.926 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:28 smithi173 ceph-mon[33544]: 3.c scrub ok 2024-03-12T03:51:28.926 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:28 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 73/690 objects degraded (10.580%), 16 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:28.926 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:28 smithi173 ceph-mon[33544]: osdmap e435: 8 total, 7 up, 8 in 2024-03-12T03:51:28.931 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:51:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:28.946 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:28.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:28.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-12T03:51:28.952 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:51:28.971 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:51:28.971 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:51:28.971 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:51:28.971 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:51:28.971 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:51:28.971 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:51:28.971 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:51:28.971 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:51:28.972 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:51:29.047 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:29.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:29.072 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:29.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:29.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:29.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:29.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:29.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:29.209 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:29.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:29.310 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:51:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:51:29.327 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:29.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:29.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:29.454 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:29.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:29.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:29.579 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:29.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:51:29.676 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:51:29.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:29.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:29.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:29.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:29.832 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:29.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:29.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:29 smithi169 ceph-mon[39949]: pgmap v878: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 68 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:29.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:29 smithi169 ceph-mon[39949]: 6.1c scrub starts 2024-03-12T03:51:29.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:29 smithi169 ceph-mon[39949]: 6.1c scrub ok 2024-03-12T03:51:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:29 smithi169 ceph-mon[39949]: 6.1f scrub starts 2024-03-12T03:51:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:29 smithi169 ceph-mon[39949]: 6.1f scrub ok 2024-03-12T03:51:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:29 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1218724049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:29 smithi169 ceph-mon[29344]: pgmap v878: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 68 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:29 smithi169 ceph-mon[29344]: 6.1c scrub starts 2024-03-12T03:51:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:29 smithi169 ceph-mon[29344]: 6.1c scrub ok 2024-03-12T03:51:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:29 smithi169 ceph-mon[29344]: 6.1f scrub starts 2024-03-12T03:51:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:29 smithi169 ceph-mon[29344]: 6.1f scrub ok 2024-03-12T03:51:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:29 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1218724049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:29.934 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:29.957 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:29.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:29.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:29 smithi173 ceph-mon[33544]: pgmap v878: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 68 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:29 smithi173 ceph-mon[33544]: 6.1c scrub starts 2024-03-12T03:51:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:29 smithi173 ceph-mon[33544]: 6.1c scrub ok 2024-03-12T03:51:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:29 smithi173 ceph-mon[33544]: 6.1f scrub starts 2024-03-12T03:51:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:29 smithi173 ceph-mon[33544]: 6.1f scrub ok 2024-03-12T03:51:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:29 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1218724049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:30.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:30.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:30.083 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:30.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:30.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:30.208 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:30.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:30.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:30.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:30.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:30.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:30.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:30.440 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":879,"stamp":"2024-03-12T03:51:28.484154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":623227,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":73,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":2500,"num_read_kb":2224,"num_write":278908,"num_write_kb":8895644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1004300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81308,"ondisk_log_size":81308,"up":434,"acting":438,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":501,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8507396,"kb_used_data":3124772,"kb_used_omap":24,"kb_used_meta":5382503,"kb_avail":741453820,"statfs":{"total":767960285184,"available":759248711680,"internally_reserved":0,"allocated":3199766528,"data_stored":3189344208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24644,"internal_metadata":5511684028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.294191"},"pg_stats":[{"pgid":"10.2","version":"430'17116","reported_seq":17265,"reported_epoch":434,"state":"active+undersized","last_fresh":"2024-03-12T03:51:26.706697+0000","last_change":"2024-03-12T03:51:18.215535+0000","last_active":"2024-03-12T03:51:26.706697+0000","last_peered":"2024-03-12T03:51:26.706697+0000","last_clean":"2024-03-12T03:48:57.540498+0000","last_became_active":"2024-03-12T03:48:58.458222+0000","last_became_peered":"2024-03-12T03:48:58.458222+0000","last_unstale":"2024-03-12T03:51:26.706697+0000","last_undegraded":"2024-03-12T03:51:26.706697+0000","last_fullsized":"2024-03-12T03:48:58.441567+0000","mapping_epoch":384,"log_start":"406'12100","ondisk_log_start":"406'12100","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.981968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17116,"num_write_kb":547712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1124,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.264913+0000","last_change":"2024-03-12T03:50:53.286652+0000","last_active":"2024-03-12T03:51:24.264913+0000","last_peered":"2024-03-12T03:51:24.264913+0000","last_clean":"2024-03-12T03:51:24.264913+0000","last_became_active":"2024-03-12T03:46:25.289005+0000","last_became_peered":"2024-03-12T03:46:25.289005+0000","last_unstale":"2024-03-12T03:51:24.264913+0000","last_undegraded":"2024-03-12T03:51:24.264913+0000","last_fullsized":"2024-03-12T03:51:24.264913+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:53.286515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.286515+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:21.938213+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"344'25","reported_seq":860,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418469+0000","last_change":"2024-03-12T03:49:17.840270+0000","last_active":"2024-03-12T03:51:24.418469+0000","last_peered":"2024-03-12T03:51:24.418469+0000","last_clean":"2024-03-12T03:51:24.418469+0000","last_became_active":"2024-03-12T03:46:25.301377+0000","last_became_peered":"2024-03-12T03:46:25.301377+0000","last_unstale":"2024-03-12T03:51:24.418469+0000","last_undegraded":"2024-03-12T03:51:24.418469+0000","last_fullsized":"2024-03-12T03:51:24.418469+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'25","last_scrub_stamp":"2024-03-12T03:49:17.839910+0000","last_deep_scrub":"344'25","last_deep_scrub_stamp":"2024-03-12T03:49:17.839910+0000","last_clean_scrub_stamp":"2024-03-12T03:49:17.839910+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731279+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":568,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.486025+0000","last_change":"2024-03-12T03:48:58.454102+0000","last_active":"2024-03-12T03:51:23.486025+0000","last_peered":"2024-03-12T03:51:23.486025+0000","last_clean":"2024-03-12T03:48:53.448575+0000","last_became_active":"2024-03-12T03:48:58.454102+0000","last_became_peered":"2024-03-12T03:48:58.454102+0000","last_unstale":"2024-03-12T03:51:23.486025+0000","last_undegraded":"2024-03-12T03:51:23.486025+0000","last_fullsized":"2024-03-12T03:48:58.443116+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:07.363439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:07.363439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":682,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.885811+0000","last_change":"2024-03-12T03:49:48.662261+0000","last_active":"2024-03-12T03:51:26.885811+0000","last_peered":"2024-03-12T03:51:26.885811+0000","last_clean":"2024-03-12T03:51:26.885811+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:51:26.885811+0000","last_undegraded":"2024-03-12T03:51:26.885811+0000","last_fullsized":"2024-03-12T03:51:26.885811+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:49:48.662179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:48.662179+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:31.696263+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":805,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418667+0000","last_change":"2024-03-12T03:49:54.872090+0000","last_active":"2024-03-12T03:51:24.418667+0000","last_peered":"2024-03-12T03:51:24.418667+0000","last_clean":"2024-03-12T03:51:24.418667+0000","last_became_active":"2024-03-12T03:39:33.265665+0000","last_became_peered":"2024-03-12T03:39:33.265665+0000","last_unstale":"2024-03-12T03:51:24.418667+0000","last_undegraded":"2024-03-12T03:51:24.418667+0000","last_fullsized":"2024-03-12T03:51:24.418667+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:54.871895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.871895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731408+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"430'19335","reported_seq":19670,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.885341+0000","last_change":"2024-03-12T03:49:54.072615+0000","last_active":"2024-03-12T03:51:26.885341+0000","last_peered":"2024-03-12T03:51:26.885341+0000","last_clean":"2024-03-12T03:51:26.885341+0000","last_became_active":"2024-03-12T03:47:57.695210+0000","last_became_peered":"2024-03-12T03:47:57.695210+0000","last_unstale":"2024-03-12T03:51:26.885341+0000","last_undegraded":"2024-03-12T03:51:26.885341+0000","last_fullsized":"2024-03-12T03:51:26.885341+0000","mapping_epoch":366,"log_start":"405'13270","ondisk_log_start":"405'13270","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'11483","last_scrub_stamp":"2024-03-12T03:49:54.072402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.072402+0000","objects_scrubbed":7964,"log_size":6065,"log_dups_size":0,"ondisk_log_size":6065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:31.696284+0000","scrub_duration":2423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17784,"num_write_kb":569088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"269'31","reported_seq":899,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418749+0000","last_change":"2024-03-12T03:50:01.871630+0000","last_active":"2024-03-12T03:51:24.418749+0000","last_peered":"2024-03-12T03:51:24.418749+0000","last_clean":"2024-03-12T03:51:24.418749+0000","last_became_active":"2024-03-12T03:39:33.268652+0000","last_became_peered":"2024-03-12T03:39:33.268652+0000","last_unstale":"2024-03-12T03:51:24.418749+0000","last_undegraded":"2024-03-12T03:51:24.418749+0000","last_fullsized":"2024-03-12T03:51:24.418749+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:50:01.871412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:01.871412+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731378+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":268,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.417786+0000","last_change":"2024-03-12T03:50:55.920975+0000","last_active":"2024-03-12T03:51:24.417786+0000","last_peered":"2024-03-12T03:51:24.417786+0000","last_clean":"2024-03-12T03:51:24.417786+0000","last_became_active":"2024-03-12T03:45:18.108377+0000","last_became_peered":"2024-03-12T03:45:18.108377+0000","last_unstale":"2024-03-12T03:51:24.417786+0000","last_undegraded":"2024-03-12T03:51:24.417786+0000","last_fullsized":"2024-03-12T03:51:24.417786+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:55.920704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:50:55.920704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:09.389962+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":793,"reported_epoch":434,"state":"active+undersized","last_fresh":"2024-03-12T03:51:26.706851+0000","last_change":"2024-03-12T03:48:58.457284+0000","last_active":"2024-03-12T03:51:26.706851+0000","last_peered":"2024-03-12T03:51:26.706851+0000","last_clean":"2024-03-12T03:48:53.449180+0000","last_became_active":"2024-03-12T03:48:58.457284+0000","last_became_peered":"2024-03-12T03:48:58.457284+0000","last_unstale":"2024-03-12T03:51:26.706851+0000","last_undegraded":"2024-03-12T03:51:26.706851+0000","last_fullsized":"2024-03-12T03:48:58.441935+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.054936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.054936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.981881+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":545,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.885802+0000","last_change":"2024-03-12T03:49:17.061923+0000","last_active":"2024-03-12T03:51:26.885802+0000","last_peered":"2024-03-12T03:51:26.885802+0000","last_clean":"2024-03-12T03:51:26.885802+0000","last_became_active":"2024-03-12T03:46:25.293293+0000","last_became_peered":"2024-03-12T03:46:25.293293+0000","last_unstale":"2024-03-12T03:51:26.885802+0000","last_undegraded":"2024-03-12T03:51:26.885802+0000","last_fullsized":"2024-03-12T03:51:26.885802+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:17.061809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:17.061809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:31.696227+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"388'129","reported_seq":965,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486062+0000","last_change":"2024-03-12T03:50:56.164839+0000","last_active":"2024-03-12T03:51:23.486062+0000","last_peered":"2024-03-12T03:51:23.486062+0000","last_clean":"2024-03-12T03:51:23.486062+0000","last_became_active":"2024-03-12T03:46:25.297913+0000","last_became_peered":"2024-03-12T03:46:25.297913+0000","last_unstale":"2024-03-12T03:51:23.486062+0000","last_undegraded":"2024-03-12T03:51:23.486062+0000","last_fullsized":"2024-03-12T03:51:23.486062+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"388'129","last_scrub_stamp":"2024-03-12T03:50:56.164705+0000","last_deep_scrub":"388'129","last_deep_scrub_stamp":"2024-03-12T03:49:17.168603+0000","last_clean_scrub_stamp":"2024-03-12T03:50:56.164705+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:06.277420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"430'17408","reported_seq":17634,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.705904+0000","last_change":"2024-03-12T03:49:20.727415+0000","last_active":"2024-03-12T03:51:26.705904+0000","last_peered":"2024-03-12T03:51:26.705904+0000","last_clean":"2024-03-12T03:51:26.705904+0000","last_became_active":"2024-03-12T03:47:58.027417+0000","last_became_peered":"2024-03-12T03:47:58.027417+0000","last_unstale":"2024-03-12T03:51:26.705904+0000","last_undegraded":"2024-03-12T03:51:26.705904+0000","last_fullsized":"2024-03-12T03:51:26.705904+0000","mapping_epoch":366,"log_start":"406'12400","ondisk_log_start":"406'12400","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'7006","last_scrub_stamp":"2024-03-12T03:49:20.727145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:49:20.727145+0000","objects_scrubbed":6935,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.981995+0000","scrub_duration":2177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17408,"num_write_kb":557056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":947,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.708075+0000","last_change":"2024-03-12T03:50:58.843012+0000","last_active":"2024-03-12T03:51:26.708075+0000","last_peered":"2024-03-12T03:51:26.708075+0000","last_clean":"2024-03-12T03:51:26.708075+0000","last_became_active":"2024-03-12T03:50:58.842635+0000","last_became_peered":"2024-03-12T03:50:58.842635+0000","last_unstale":"2024-03-12T03:51:26.708075+0000","last_undegraded":"2024-03-12T03:51:26.708075+0000","last_fullsized":"2024-03-12T03:51:26.708075+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:08.710391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:08.710391+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.981959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"269'31","reported_seq":850,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418538+0000","last_change":"2024-03-12T03:49:09.755599+0000","last_active":"2024-03-12T03:51:24.418538+0000","last_peered":"2024-03-12T03:51:24.418538+0000","last_clean":"2024-03-12T03:51:24.418538+0000","last_became_active":"2024-03-12T03:33:23.168645+0000","last_became_peered":"2024-03-12T03:33:23.168645+0000","last_unstale":"2024-03-12T03:51:24.418538+0000","last_undegraded":"2024-03-12T03:51:24.418538+0000","last_fullsized":"2024-03-12T03:51:24.418538+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:49:09.755469+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:49:09.755469+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731296+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":818,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.417797+0000","last_change":"2024-03-12T03:50:47.939438+0000","last_active":"2024-03-12T03:51:24.417797+0000","last_peered":"2024-03-12T03:51:24.417797+0000","last_clean":"2024-03-12T03:51:24.417797+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:51:24.417797+0000","last_undegraded":"2024-03-12T03:51:24.417797+0000","last_fullsized":"2024-03-12T03:51:24.417797+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:47.939203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_clean_scrub_stamp":"2024-03-12T03:50:47.939203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:07.451477+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":795,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.487175+0000","last_change":"2024-03-12T03:50:51.214639+0000","last_active":"2024-03-12T03:51:23.487175+0000","last_peered":"2024-03-12T03:51:23.487175+0000","last_clean":"2024-03-12T03:51:23.487175+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:51:23.487175+0000","last_undegraded":"2024-03-12T03:51:23.487175+0000","last_fullsized":"2024-03-12T03:51:23.487175+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:51.214503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:51.214503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:06.200621+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":583,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486272+0000","last_change":"2024-03-12T03:49:48.536574+0000","last_active":"2024-03-12T03:51:23.486272+0000","last_peered":"2024-03-12T03:51:23.486272+0000","last_clean":"2024-03-12T03:51:23.486272+0000","last_became_active":"2024-03-12T03:46:25.300814+0000","last_became_peered":"2024-03-12T03:46:25.300814+0000","last_unstale":"2024-03-12T03:51:23.486272+0000","last_undegraded":"2024-03-12T03:51:23.486272+0000","last_fullsized":"2024-03-12T03:51:23.486272+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:48.536435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:48.536435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"430'17092","reported_seq":17240,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.417474+0000","last_change":"2024-03-12T03:51:18.216082+0000","last_active":"2024-03-12T03:51:24.417474+0000","last_peered":"2024-03-12T03:51:24.417474+0000","last_clean":"2024-03-12T03:48:57.553732+0000","last_became_active":"2024-03-12T03:48:58.455008+0000","last_became_peered":"2024-03-12T03:48:58.455008+0000","last_unstale":"2024-03-12T03:51:24.417474+0000","last_undegraded":"2024-03-12T03:51:24.417474+0000","last_fullsized":"2024-03-12T03:48:58.446493+0000","mapping_epoch":384,"log_start":"408'13012","ondisk_log_start":"408'13012","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17092,"num_write_kb":546944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":938,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.485950+0000","last_change":"2024-03-12T03:50:34.205833+0000","last_active":"2024-03-12T03:51:23.485950+0000","last_peered":"2024-03-12T03:51:23.485950+0000","last_clean":"2024-03-12T03:51:23.485950+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:51:23.485950+0000","last_undegraded":"2024-03-12T03:51:23.485950+0000","last_fullsized":"2024-03-12T03:51:23.485950+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:34.205711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:34.205711+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:00.651194+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":663,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.705682+0000","last_change":"2024-03-12T03:50:46.613513+0000","last_active":"2024-03-12T03:51:26.705682+0000","last_peered":"2024-03-12T03:51:26.705682+0000","last_clean":"2024-03-12T03:51:26.705682+0000","last_became_active":"2024-03-12T03:46:25.287373+0000","last_became_peered":"2024-03-12T03:46:25.287373+0000","last_unstale":"2024-03-12T03:51:26.705682+0000","last_undegraded":"2024-03-12T03:51:26.705682+0000","last_fullsized":"2024-03-12T03:51:26.705682+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:50:46.613361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:46.613361+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:53.630234+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":290,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.417426+0000","last_change":"2024-03-12T03:48:58.456372+0000","last_active":"2024-03-12T03:51:24.417426+0000","last_peered":"2024-03-12T03:51:24.417426+0000","last_clean":"2024-03-12T03:48:53.446320+0000","last_became_active":"2024-03-12T03:48:58.456372+0000","last_became_peered":"2024-03-12T03:48:58.456372+0000","last_unstale":"2024-03-12T03:51:24.417426+0000","last_undegraded":"2024-03-12T03:51:24.417426+0000","last_fullsized":"2024-03-12T03:48:58.444143+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:16.876204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:16.876204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731210+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"385'9","reported_seq":1018,"reported_epoch":433,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:24.418626+0000","last_change":"2024-03-12T03:48:58.455645+0000","last_active":"2024-03-12T03:51:24.418626+0000","last_peered":"2024-03-12T03:51:24.418626+0000","last_clean":"2024-03-12T03:48:56.818397+0000","last_became_active":"2024-03-12T03:48:58.455645+0000","last_became_peered":"2024-03-12T03:48:58.455645+0000","last_unstale":"2024-03-12T03:51:24.418626+0000","last_undegraded":"2024-03-12T03:48:58.448928+0000","last_fullsized":"2024-03-12T03:48:58.448807+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:47:53.964216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:53.964216+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":779,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.486684+0000","last_change":"2024-03-12T03:48:58.448113+0000","last_active":"2024-03-12T03:51:23.486684+0000","last_peered":"2024-03-12T03:51:23.486684+0000","last_clean":"2024-03-12T03:48:53.472090+0000","last_became_active":"2024-03-12T03:48:58.448113+0000","last_became_peered":"2024-03-12T03:48:58.448113+0000","last_unstale":"2024-03-12T03:51:23.486684+0000","last_undegraded":"2024-03-12T03:51:23.486684+0000","last_fullsized":"2024-03-12T03:48:58.441990+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:04.770746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:04.770746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"430'17384","reported_seq":17578,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418172+0000","last_change":"2024-03-12T03:50:34.323820+0000","last_active":"2024-03-12T03:51:24.418172+0000","last_peered":"2024-03-12T03:51:24.418172+0000","last_clean":"2024-03-12T03:51:24.418172+0000","last_became_active":"2024-03-12T03:47:58.027802+0000","last_became_peered":"2024-03-12T03:47:58.027802+0000","last_unstale":"2024-03-12T03:51:24.418172+0000","last_undegraded":"2024-03-12T03:51:24.418172+0000","last_fullsized":"2024-03-12T03:51:24.418172+0000","mapping_epoch":366,"log_start":"409'13289","ondisk_log_start":"409'13289","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'13357","last_scrub_stamp":"2024-03-12T03:50:34.323591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:50:34.323591+0000","objects_scrubbed":4095,"log_size":4095,"log_dups_size":0,"ondisk_log_size":4095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:00.580249+0000","scrub_duration":1476,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17384,"num_write_kb":556288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":800,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.417668+0000","last_change":"2024-03-12T03:49:11.818292+0000","last_active":"2024-03-12T03:51:24.417668+0000","last_peered":"2024-03-12T03:51:24.417668+0000","last_clean":"2024-03-12T03:51:24.417668+0000","last_became_active":"2024-03-12T03:46:25.286985+0000","last_became_peered":"2024-03-12T03:46:25.286985+0000","last_unstale":"2024-03-12T03:51:24.417668+0000","last_undegraded":"2024-03-12T03:51:24.417668+0000","last_fullsized":"2024-03-12T03:51:24.417668+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:11.818116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:11.818116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731229+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"269'42","reported_seq":656,"reported_epoch":433,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:23.485783+0000","last_change":"2024-03-12T03:48:58.463377+0000","last_active":"2024-03-12T03:51:23.485783+0000","last_peered":"2024-03-12T03:51:23.485783+0000","last_clean":"2024-03-12T03:48:53.447338+0000","last_became_active":"2024-03-12T03:48:58.463377+0000","last_became_peered":"2024-03-12T03:48:58.463377+0000","last_unstale":"2024-03-12T03:51:23.485783+0000","last_undegraded":"2024-03-12T03:48:58.440183+0000","last_fullsized":"2024-03-12T03:48:58.440082+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:47:13.347128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.347128+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":572,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.487737+0000","last_change":"2024-03-12T03:48:58.464970+0000","last_active":"2024-03-12T03:51:23.487737+0000","last_peered":"2024-03-12T03:51:23.487737+0000","last_clean":"2024-03-12T03:48:53.448054+0000","last_became_active":"2024-03-12T03:48:58.464970+0000","last_became_peered":"2024-03-12T03:48:58.464970+0000","last_unstale":"2024-03-12T03:51:23.487737+0000","last_undegraded":"2024-03-12T03:51:23.487737+0000","last_fullsized":"2024-03-12T03:48:58.441302+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:05.639950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:05.639950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":816,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.417752+0000","last_change":"2024-03-12T03:49:14.836471+0000","last_active":"2024-03-12T03:51:24.417752+0000","last_peered":"2024-03-12T03:51:24.417752+0000","last_clean":"2024-03-12T03:51:24.417752+0000","last_became_active":"2024-03-12T03:46:25.280860+0000","last_became_peered":"2024-03-12T03:46:25.280860+0000","last_unstale":"2024-03-12T03:51:24.417752+0000","last_undegraded":"2024-03-12T03:51:24.417752+0000","last_fullsized":"2024-03-12T03:51:24.417752+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:14.836179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_clean_scrub_stamp":"2024-03-12T03:49:14.836179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":953,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418015+0000","last_change":"2024-03-12T03:50:48.924955+0000","last_active":"2024-03-12T03:51:24.418015+0000","last_peered":"2024-03-12T03:51:24.418015+0000","last_clean":"2024-03-12T03:51:24.418015+0000","last_became_active":"2024-03-12T03:44:19.373715+0000","last_became_peered":"2024-03-12T03:44:19.373715+0000","last_unstale":"2024-03-12T03:51:24.418015+0000","last_undegraded":"2024-03-12T03:51:24.418015+0000","last_fullsized":"2024-03-12T03:51:24.418015+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:48.924831+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:15.895638+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.924831+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:02.935590+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"430'17588","reported_seq":17815,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.487612+0000","last_change":"2024-03-12T03:49:58.948258+0000","last_active":"2024-03-12T03:51:23.487612+0000","last_peered":"2024-03-12T03:51:23.487612+0000","last_clean":"2024-03-12T03:51:23.487612+0000","last_became_active":"2024-03-12T03:47:58.027347+0000","last_became_peered":"2024-03-12T03:47:58.027347+0000","last_unstale":"2024-03-12T03:51:23.487612+0000","last_undegraded":"2024-03-12T03:51:23.487612+0000","last_fullsized":"2024-03-12T03:51:23.487612+0000","mapping_epoch":366,"log_start":"407'13218","ondisk_log_start":"407'13218","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'10207","last_scrub_stamp":"2024-03-12T03:49:58.947894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:49:58.947894+0000","objects_scrubbed":7488,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603286+0000","scrub_duration":2515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17588,"num_write_kb":562816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"332'12","reported_seq":794,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486153+0000","last_change":"2024-03-12T03:50:32.360621+0000","last_active":"2024-03-12T03:51:23.486153+0000","last_peered":"2024-03-12T03:51:23.486153+0000","last_clean":"2024-03-12T03:51:23.486153+0000","last_became_active":"2024-03-12T03:46:25.288674+0000","last_became_peered":"2024-03-12T03:46:25.288674+0000","last_unstale":"2024-03-12T03:51:23.486153+0000","last_undegraded":"2024-03-12T03:51:23.486153+0000","last_fullsized":"2024-03-12T03:51:23.486153+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'12","last_scrub_stamp":"2024-03-12T03:50:32.360502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:32.360502+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:39.022017+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"269'42","reported_seq":864,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.417975+0000","last_change":"2024-03-12T03:50:35.839987+0000","last_active":"2024-03-12T03:51:24.417975+0000","last_peered":"2024-03-12T03:51:24.417975+0000","last_clean":"2024-03-12T03:51:24.417975+0000","last_became_active":"2024-03-12T03:33:23.156556+0000","last_became_peered":"2024-03-12T03:33:23.156556+0000","last_unstale":"2024-03-12T03:51:24.417975+0000","last_undegraded":"2024-03-12T03:51:24.417975+0000","last_fullsized":"2024-03-12T03:51:24.417975+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:50:35.839859+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.839859+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:04.798992+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"320'2","reported_seq":665,"reported_epoch":434,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:26.707106+0000","last_change":"2024-03-12T03:48:58.457502+0000","last_active":"2024-03-12T03:51:26.707106+0000","last_peered":"2024-03-12T03:51:26.707106+0000","last_clean":"2024-03-12T03:45:56.084290+0000","last_became_active":"2024-03-12T03:48:58.457502+0000","last_became_peered":"2024-03-12T03:48:58.457502+0000","last_unstale":"2024-03-12T03:51:26.707106+0000","last_undegraded":"2024-03-12T03:48:58.439950+0000","last_fullsized":"2024-03-12T03:48:58.439792+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'2","last_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_deep_scrub":"320'2","last_deep_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_clean_scrub_stamp":"2024-03-12T03:45:56.089247+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.981925+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":806,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265092+0000","last_change":"2024-03-12T03:49:54.471443+0000","last_active":"2024-03-12T03:51:24.265092+0000","last_peered":"2024-03-12T03:51:24.265092+0000","last_clean":"2024-03-12T03:51:24.265092+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:51:24.265092+0000","last_undegraded":"2024-03-12T03:51:24.265092+0000","last_fullsized":"2024-03-12T03:51:24.265092+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:54.471078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:49:54.471078+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.471078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266268+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":765,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.485918+0000","last_change":"2024-03-12T03:51:07.938720+0000","last_active":"2024-03-12T03:51:23.485918+0000","last_peered":"2024-03-12T03:51:23.485918+0000","last_clean":"2024-03-12T03:51:23.485918+0000","last_became_active":"2024-03-12T03:51:07.938356+0000","last_became_peered":"2024-03-12T03:51:07.938356+0000","last_unstale":"2024-03-12T03:51:23.485918+0000","last_undegraded":"2024-03-12T03:51:23.485918+0000","last_fullsized":"2024-03-12T03:51:23.485918+0000","mapping_epoch":330,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:53.442638+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_clean_scrub_stamp":"2024-03-12T03:49:53.442638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603342+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"430'17174","reported_seq":17322,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.265579+0000","last_change":"2024-03-12T03:51:17.136158+0000","last_active":"2024-03-12T03:51:24.265579+0000","last_peered":"2024-03-12T03:51:24.265579+0000","last_clean":"2024-03-12T03:48:57.535679+0000","last_became_active":"2024-03-12T03:48:58.456284+0000","last_became_peered":"2024-03-12T03:48:58.456284+0000","last_unstale":"2024-03-12T03:51:24.265579+0000","last_undegraded":"2024-03-12T03:51:24.265579+0000","last_fullsized":"2024-03-12T03:48:58.446227+0000","mapping_epoch":384,"log_start":"407'12465","ondisk_log_start":"407'12465","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17174,"num_write_kb":549568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"385'9","reported_seq":160,"reported_epoch":433,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:23.485952+0000","last_change":"2024-03-12T03:48:58.451360+0000","last_active":"2024-03-12T03:51:23.485952+0000","last_peered":"2024-03-12T03:51:23.485952+0000","last_clean":"2024-03-12T03:33:28.360558+0000","last_became_active":"2024-03-12T03:48:58.451360+0000","last_became_peered":"2024-03-12T03:48:58.451360+0000","last_unstale":"2024-03-12T03:51:23.485952+0000","last_undegraded":"2024-03-12T03:48:58.442329+0000","last_fullsized":"2024-03-12T03:48:58.441949+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:48:14.138048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:14.138048+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"269'26","reported_seq":853,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.707087+0000","last_change":"2024-03-12T03:50:35.597476+0000","last_active":"2024-03-12T03:51:26.707087+0000","last_peered":"2024-03-12T03:51:26.707087+0000","last_clean":"2024-03-12T03:51:26.707087+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:51:26.707087+0000","last_undegraded":"2024-03-12T03:51:26.707087+0000","last_fullsized":"2024-03-12T03:51:26.707087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:50:35.597225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.597225+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:53.172520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":826,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486389+0000","last_change":"2024-03-12T03:50:48.222807+0000","last_active":"2024-03-12T03:51:23.486389+0000","last_peered":"2024-03-12T03:51:23.486389+0000","last_clean":"2024-03-12T03:51:23.486389+0000","last_became_active":"2024-03-12T03:46:25.294348+0000","last_became_peered":"2024-03-12T03:46:25.294348+0000","last_unstale":"2024-03-12T03:51:23.486389+0000","last_undegraded":"2024-03-12T03:51:23.486389+0000","last_fullsized":"2024-03-12T03:51:23.486389+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.222697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.222697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:14.336537+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":683,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.485515+0000","last_change":"2024-03-12T03:48:58.455443+0000","last_active":"2024-03-12T03:51:23.485515+0000","last_peered":"2024-03-12T03:51:23.485515+0000","last_clean":"2024-03-12T03:43:13.330724+0000","last_became_active":"2024-03-12T03:48:58.455443+0000","last_became_peered":"2024-03-12T03:48:58.455443+0000","last_unstale":"2024-03-12T03:51:23.485515+0000","last_undegraded":"2024-03-12T03:51:23.485515+0000","last_fullsized":"2024-03-12T03:48:58.439578+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:13.892760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.892760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603163+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":550,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.488028+0000","last_change":"2024-03-12T03:50:54.587747+0000","last_active":"2024-03-12T03:51:23.488028+0000","last_peered":"2024-03-12T03:51:23.488028+0000","last_clean":"2024-03-12T03:51:23.488028+0000","last_became_active":"2024-03-12T03:46:25.287360+0000","last_became_peered":"2024-03-12T03:46:25.287360+0000","last_unstale":"2024-03-12T03:51:23.488028+0000","last_undegraded":"2024-03-12T03:51:23.488028+0000","last_fullsized":"2024-03-12T03:51:23.488028+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:54.587604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:54.587604+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:20.215534+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"430'17468","reported_seq":17615,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.418201+0000","last_change":"2024-03-12T03:51:18.217080+0000","last_active":"2024-03-12T03:51:24.418201+0000","last_peered":"2024-03-12T03:51:24.418201+0000","last_clean":"2024-03-12T03:48:57.549912+0000","last_became_active":"2024-03-12T03:48:58.455501+0000","last_became_peered":"2024-03-12T03:48:58.455501+0000","last_unstale":"2024-03-12T03:51:24.418201+0000","last_undegraded":"2024-03-12T03:51:24.418201+0000","last_fullsized":"2024-03-12T03:48:58.444662+0000","mapping_epoch":384,"log_start":"409'13382","ondisk_log_start":"409'13382","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4086,"log_dups_size":0,"ondisk_log_size":4086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17468,"num_write_kb":558913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":108,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.486907+0000","last_change":"2024-03-12T03:48:58.449166+0000","last_active":"2024-03-12T03:51:23.486907+0000","last_peered":"2024-03-12T03:51:23.486907+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.449166+0000","last_became_peered":"2024-03-12T03:48:58.449166+0000","last_unstale":"2024-03-12T03:51:23.486907+0000","last_undegraded":"2024-03-12T03:51:23.486907+0000","last_fullsized":"2024-03-12T03:48:58.441301+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.179416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.179416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"269'35","reported_seq":606,"reported_epoch":433,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:23.486231+0000","last_change":"2024-03-12T03:48:58.454600+0000","last_active":"2024-03-12T03:51:23.486231+0000","last_peered":"2024-03-12T03:51:23.486231+0000","last_clean":"2024-03-12T03:43:28.848207+0000","last_became_active":"2024-03-12T03:48:58.454600+0000","last_became_peered":"2024-03-12T03:48:58.454600+0000","last_unstale":"2024-03-12T03:51:23.486231+0000","last_undegraded":"2024-03-12T03:48:58.443785+0000","last_fullsized":"2024-03-12T03:48:58.443677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-03-12T03:45:59.055318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.055318+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142510+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":47,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":827,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.706760+0000","last_change":"2024-03-12T03:49:51.465759+0000","last_active":"2024-03-12T03:51:26.706760+0000","last_peered":"2024-03-12T03:51:26.706760+0000","last_clean":"2024-03-12T03:51:26.706760+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:51:26.706760+0000","last_undegraded":"2024-03-12T03:51:26.706760+0000","last_fullsized":"2024-03-12T03:51:26.706760+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:51.465633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:49:51.465633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.982075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":808,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.706336+0000","last_change":"2024-03-12T03:49:02.614970+0000","last_active":"2024-03-12T03:51:26.706336+0000","last_peered":"2024-03-12T03:51:26.706336+0000","last_clean":"2024-03-12T03:51:26.706336+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:51:26.706336+0000","last_undegraded":"2024-03-12T03:51:26.706336+0000","last_fullsized":"2024-03-12T03:51:26.706336+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:02.614847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:02.614847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.982010+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":412,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.487133+0000","last_change":"2024-03-12T03:48:58.454186+0000","last_active":"2024-03-12T03:51:23.487133+0000","last_peered":"2024-03-12T03:51:23.487133+0000","last_clean":"2024-03-12T03:34:01.860694+0000","last_became_active":"2024-03-12T03:48:58.454186+0000","last_became_peered":"2024-03-12T03:48:58.454186+0000","last_unstale":"2024-03-12T03:51:23.487133+0000","last_undegraded":"2024-03-12T03:51:23.487133+0000","last_fullsized":"2024-03-12T03:48:58.443407+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:04.857986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:04.857986+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:27.613726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"430'17514","reported_seq":17663,"reported_epoch":434,"state":"active+undersized","last_fresh":"2024-03-12T03:51:26.885084+0000","last_change":"2024-03-12T03:51:17.143470+0000","last_active":"2024-03-12T03:51:26.885084+0000","last_peered":"2024-03-12T03:51:26.885084+0000","last_clean":"2024-03-12T03:48:57.504096+0000","last_became_active":"2024-03-12T03:48:58.462488+0000","last_became_peered":"2024-03-12T03:48:58.462488+0000","last_unstale":"2024-03-12T03:51:26.885084+0000","last_undegraded":"2024-03-12T03:51:26.885084+0000","last_fullsized":"2024-03-12T03:48:58.444978+0000","mapping_epoch":384,"log_start":"405'11513","ondisk_log_start":"405'11513","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":6001,"log_dups_size":0,"ondisk_log_size":6001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:31.696182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17514,"num_write_kb":560448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":783,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.487229+0000","last_change":"2024-03-12T03:48:58.467253+0000","last_active":"2024-03-12T03:51:23.487229+0000","last_peered":"2024-03-12T03:51:23.487229+0000","last_clean":"2024-03-12T03:48:53.472859+0000","last_became_active":"2024-03-12T03:48:58.467253+0000","last_became_peered":"2024-03-12T03:48:58.467253+0000","last_unstale":"2024-03-12T03:51:23.487229+0000","last_undegraded":"2024-03-12T03:51:23.487229+0000","last_fullsized":"2024-03-12T03:48:58.444374+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:00.577693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:46:00.577693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142538+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"269'24","reported_seq":629,"reported_epoch":434,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:26.707532+0000","last_change":"2024-03-12T03:48:58.449415+0000","last_active":"2024-03-12T03:51:26.707532+0000","last_peered":"2024-03-12T03:51:26.707532+0000","last_clean":"2024-03-12T03:43:28.788554+0000","last_became_active":"2024-03-12T03:48:58.449415+0000","last_became_peered":"2024-03-12T03:48:58.449415+0000","last_unstale":"2024-03-12T03:51:26.707532+0000","last_undegraded":"2024-03-12T03:48:58.441112+0000","last_fullsized":"2024-03-12T03:48:58.441001+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:48:17.191968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:17.191968+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.981976+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":1892,"reported_epoch":433,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:25.819921+0000","last_change":"2024-03-12T03:48:58.462253+0000","last_active":"2024-03-12T03:51:25.819921+0000","last_peered":"2024-03-12T03:51:25.819921+0000","last_clean":"2024-03-12T03:48:56.580904+0000","last_became_active":"2024-03-12T03:48:58.462253+0000","last_became_peered":"2024-03-12T03:48:58.462253+0000","last_unstale":"2024-03-12T03:51:25.819921+0000","last_undegraded":"2024-03-12T03:48:58.445825+0000","last_fullsized":"2024-03-12T03:48:58.445711+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:47:13.591375+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.591375+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266164+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1076,"num_read_kb":1071,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":683,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.417804+0000","last_change":"2024-03-12T03:48:58.464609+0000","last_active":"2024-03-12T03:51:24.417804+0000","last_peered":"2024-03-12T03:51:24.417804+0000","last_clean":"2024-03-12T03:43:13.332262+0000","last_became_active":"2024-03-12T03:48:58.464609+0000","last_became_peered":"2024-03-12T03:48:58.464609+0000","last_unstale":"2024-03-12T03:51:24.417804+0000","last_undegraded":"2024-03-12T03:51:24.417804+0000","last_fullsized":"2024-03-12T03:48:58.443632+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:58.050985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.050985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731175+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":813,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.706577+0000","last_change":"2024-03-12T03:49:13.598817+0000","last_active":"2024-03-12T03:51:26.706577+0000","last_peered":"2024-03-12T03:51:26.706577+0000","last_clean":"2024-03-12T03:51:26.706577+0000","last_became_active":"2024-03-12T03:46:25.302256+0000","last_became_peered":"2024-03-12T03:46:25.302256+0000","last_unstale":"2024-03-12T03:51:26.706577+0000","last_undegraded":"2024-03-12T03:51:26.706577+0000","last_fullsized":"2024-03-12T03:51:26.706577+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:13.598692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:13.598692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.982043+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":947,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.708250+0000","last_change":"2024-03-12T03:51:11.966626+0000","last_active":"2024-03-12T03:51:26.708250+0000","last_peered":"2024-03-12T03:51:26.708250+0000","last_clean":"2024-03-12T03:51:26.708250+0000","last_became_active":"2024-03-12T03:51:11.965655+0000","last_became_peered":"2024-03-12T03:51:11.965655+0000","last_unstale":"2024-03-12T03:51:26.708250+0000","last_undegraded":"2024-03-12T03:51:26.708250+0000","last_fullsized":"2024-03-12T03:51:26.708250+0000","mapping_epoch":154,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:50.467976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.467976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.982110+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"430'17090","reported_seq":17239,"reported_epoch":434,"state":"active+undersized","last_fresh":"2024-03-12T03:51:26.708016+0000","last_change":"2024-03-12T03:51:18.215500+0000","last_active":"2024-03-12T03:51:26.708016+0000","last_peered":"2024-03-12T03:51:26.708016+0000","last_clean":"2024-03-12T03:48:57.546566+0000","last_became_active":"2024-03-12T03:48:58.444686+0000","last_became_peered":"2024-03-12T03:48:58.444686+0000","last_unstale":"2024-03-12T03:51:26.708016+0000","last_undegraded":"2024-03-12T03:51:26.708016+0000","last_fullsized":"2024-03-12T03:48:58.438716+0000","mapping_epoch":384,"log_start":"406'12000","ondisk_log_start":"406'12000","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.981934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17090,"num_write_kb":546880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":810,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265872+0000","last_change":"2024-03-12T03:50:48.205459+0000","last_active":"2024-03-12T03:51:24.265872+0000","last_peered":"2024-03-12T03:51:24.265872+0000","last_clean":"2024-03-12T03:51:24.265872+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:51:24.265872+0000","last_undegraded":"2024-03-12T03:51:24.265872+0000","last_fullsized":"2024-03-12T03:51:24.265872+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.205352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.205352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:17.162359+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"315'29","reported_seq":955,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418622+0000","last_change":"2024-03-12T03:50:45.902898+0000","last_active":"2024-03-12T03:51:24.418622+0000","last_peered":"2024-03-12T03:51:24.418622+0000","last_clean":"2024-03-12T03:51:24.418622+0000","last_became_active":"2024-03-12T03:45:18.109048+0000","last_became_peered":"2024-03-12T03:45:18.109048+0000","last_unstale":"2024-03-12T03:51:24.418622+0000","last_undegraded":"2024-03-12T03:51:24.418622+0000","last_fullsized":"2024-03-12T03:51:24.418622+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'29","last_scrub_stamp":"2024-03-12T03:50:45.902742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:45.902742+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:53.657314+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":576,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486559+0000","last_change":"2024-03-12T03:50:33.413216+0000","last_active":"2024-03-12T03:51:23.486559+0000","last_peered":"2024-03-12T03:51:23.486559+0000","last_clean":"2024-03-12T03:51:23.486559+0000","last_became_active":"2024-03-12T03:46:25.292535+0000","last_became_peered":"2024-03-12T03:46:25.292535+0000","last_unstale":"2024-03-12T03:51:23.486559+0000","last_undegraded":"2024-03-12T03:51:23.486559+0000","last_fullsized":"2024-03-12T03:51:23.486559+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:33.413098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:33.413098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:56.872026+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1081,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.265775+0000","last_change":"2024-03-12T03:48:58.452024+0000","last_active":"2024-03-12T03:51:24.265775+0000","last_peered":"2024-03-12T03:51:24.265775+0000","last_clean":"2024-03-12T03:48:53.448868+0000","last_became_active":"2024-03-12T03:48:58.452024+0000","last_became_peered":"2024-03-12T03:48:58.452024+0000","last_unstale":"2024-03-12T03:51:24.265775+0000","last_undegraded":"2024-03-12T03:51:24.265775+0000","last_fullsized":"2024-03-12T03:48:58.444796+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_clean_scrub_stamp":"2024-03-12T03:48:11.435684+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266206+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":794,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486531+0000","last_change":"2024-03-12T03:49:50.444531+0000","last_active":"2024-03-12T03:51:23.486531+0000","last_peered":"2024-03-12T03:51:23.486531+0000","last_clean":"2024-03-12T03:51:23.486531+0000","last_became_active":"2024-03-12T03:46:25.295668+0000","last_became_peered":"2024-03-12T03:46:25.295668+0000","last_unstale":"2024-03-12T03:51:23.486531+0000","last_undegraded":"2024-03-12T03:51:23.486531+0000","last_fullsized":"2024-03-12T03:51:23.486531+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:50.444395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.444395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142695+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"430'17400","reported_seq":17630,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.485747+0000","last_change":"2024-03-12T03:49:56.704066+0000","last_active":"2024-03-12T03:51:23.485747+0000","last_peered":"2024-03-12T03:51:23.485747+0000","last_clean":"2024-03-12T03:51:23.485747+0000","last_became_active":"2024-03-12T03:47:57.693619+0000","last_became_peered":"2024-03-12T03:47:57.693619+0000","last_unstale":"2024-03-12T03:51:23.485747+0000","last_undegraded":"2024-03-12T03:51:23.485747+0000","last_fullsized":"2024-03-12T03:51:23.485747+0000","mapping_epoch":366,"log_start":"407'12472","ondisk_log_start":"407'12472","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'9930","last_scrub_stamp":"2024-03-12T03:49:56.703697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:49:56.703697+0000","objects_scrubbed":7586,"log_size":4928,"log_dups_size":0,"ondisk_log_size":4928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17400,"num_write_kb":556800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":543,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.885712+0000","last_change":"2024-03-12T03:50:59.748930+0000","last_active":"2024-03-12T03:51:26.885712+0000","last_peered":"2024-03-12T03:51:26.885712+0000","last_clean":"2024-03-12T03:51:26.885712+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:51:26.885712+0000","last_undegraded":"2024-03-12T03:51:26.885712+0000","last_fullsized":"2024-03-12T03:51:26.885712+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:59.748792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.748792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:08.499819+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"269'23","reported_seq":843,"reported_epoch":434,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:26.705563+0000","last_change":"2024-03-12T03:48:58.451116+0000","last_active":"2024-03-12T03:51:26.705563+0000","last_peered":"2024-03-12T03:51:26.705563+0000","last_clean":"2024-03-12T03:48:53.448806+0000","last_became_active":"2024-03-12T03:48:58.451116+0000","last_became_peered":"2024-03-12T03:48:58.451116+0000","last_unstale":"2024-03-12T03:51:26.705563+0000","last_undegraded":"2024-03-12T03:48:58.440926+0000","last_fullsized":"2024-03-12T03:48:58.440811+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'23","last_scrub_stamp":"2024-03-12T03:45:59.998395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.998395+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.981908+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":532,"reported_epoch":434,"state":"active+undersized","last_fresh":"2024-03-12T03:51:26.885699+0000","last_change":"2024-03-12T03:48:58.469230+0000","last_active":"2024-03-12T03:51:26.885699+0000","last_peered":"2024-03-12T03:51:26.885699+0000","last_clean":"2024-03-12T03:48:53.446422+0000","last_became_active":"2024-03-12T03:48:58.469230+0000","last_became_peered":"2024-03-12T03:48:58.469230+0000","last_unstale":"2024-03-12T03:51:26.885699+0000","last_undegraded":"2024-03-12T03:51:26.885699+0000","last_fullsized":"2024-03-12T03:48:58.446209+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:06.903506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.903506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:31.696173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":828,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.706096+0000","last_change":"2024-03-12T03:50:41.599042+0000","last_active":"2024-03-12T03:51:26.706096+0000","last_peered":"2024-03-12T03:51:26.706096+0000","last_clean":"2024-03-12T03:51:26.706096+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:51:26.706096+0000","last_undegraded":"2024-03-12T03:51:26.706096+0000","last_fullsized":"2024-03-12T03:51:26.706096+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:41.598934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:50:41.598934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:52.846019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":814,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265601+0000","last_change":"2024-03-12T03:50:44.293342+0000","last_active":"2024-03-12T03:51:24.265601+0000","last_peered":"2024-03-12T03:51:24.265601+0000","last_clean":"2024-03-12T03:51:24.265601+0000","last_became_active":"2024-03-12T03:46:25.284168+0000","last_became_peered":"2024-03-12T03:46:25.284168+0000","last_unstale":"2024-03-12T03:51:24.265601+0000","last_undegraded":"2024-03-12T03:51:24.265601+0000","last_fullsized":"2024-03-12T03:51:24.265601+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:50:44.293155+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:49:18.522103+0000","last_clean_scrub_stamp":"2024-03-12T03:50:44.293155+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:00.958822+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1096,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.265382+0000","last_change":"2024-03-12T03:48:58.455161+0000","last_active":"2024-03-12T03:51:24.265382+0000","last_peered":"2024-03-12T03:51:24.265382+0000","last_clean":"2024-03-12T03:48:53.449188+0000","last_became_active":"2024-03-12T03:48:58.455161+0000","last_became_peered":"2024-03-12T03:48:58.455161+0000","last_unstale":"2024-03-12T03:51:24.265382+0000","last_undegraded":"2024-03-12T03:51:24.265382+0000","last_fullsized":"2024-03-12T03:48:58.444402+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:01.456424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:01.456424+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"430'17292","reported_seq":17439,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.486316+0000","last_change":"2024-03-12T03:51:17.671152+0000","last_active":"2024-03-12T03:51:23.486316+0000","last_peered":"2024-03-12T03:51:23.486316+0000","last_clean":"2024-03-12T03:48:57.500736+0000","last_became_active":"2024-03-12T03:48:58.457912+0000","last_became_peered":"2024-03-12T03:48:58.457912+0000","last_unstale":"2024-03-12T03:51:23.486316+0000","last_undegraded":"2024-03-12T03:51:23.486316+0000","last_fullsized":"2024-03-12T03:48:58.442369+0000","mapping_epoch":384,"log_start":"407'12363","ondisk_log_start":"407'12363","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17292,"num_write_kb":553344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":108,"reported_epoch":434,"state":"active+undersized","last_fresh":"2024-03-12T03:51:26.708331+0000","last_change":"2024-03-12T03:48:58.458916+0000","last_active":"2024-03-12T03:51:26.708331+0000","last_peered":"2024-03-12T03:51:26.708331+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.458916+0000","last_became_peered":"2024-03-12T03:48:58.458916+0000","last_unstale":"2024-03-12T03:51:26.708331+0000","last_undegraded":"2024-03-12T03:51:26.708331+0000","last_fullsized":"2024-03-12T03:48:58.440354+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:06.864708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.864708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.981868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"382'54","reported_seq":182,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.706531+0000","last_change":"2024-03-12T03:49:15.597576+0000","last_active":"2024-03-12T03:51:26.706531+0000","last_peered":"2024-03-12T03:51:26.706531+0000","last_clean":"2024-03-12T03:51:26.706531+0000","last_became_active":"2024-03-12T03:48:48.462111+0000","last_became_peered":"2024-03-12T03:48:48.462111+0000","last_unstale":"2024-03-12T03:51:26.706531+0000","last_undegraded":"2024-03-12T03:51:26.706531+0000","last_fullsized":"2024-03-12T03:51:26.706531+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'54","last_scrub_stamp":"2024-03-12T03:49:15.597431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:15.597431+0000","objects_scrubbed":11,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.982028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":58,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":801,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.417802+0000","last_change":"2024-03-12T03:48:58.453277+0000","last_active":"2024-03-12T03:51:24.417802+0000","last_peered":"2024-03-12T03:51:24.417802+0000","last_clean":"2024-03-12T03:48:53.447092+0000","last_became_active":"2024-03-12T03:48:58.453277+0000","last_became_peered":"2024-03-12T03:48:58.453277+0000","last_unstale":"2024-03-12T03:51:24.417802+0000","last_undegraded":"2024-03-12T03:51:24.417802+0000","last_fullsized":"2024-03-12T03:48:58.443592+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.648824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.648824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731147+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":815,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486810+0000","last_change":"2024-03-12T03:50:58.107026+0000","last_active":"2024-03-12T03:51:23.486810+0000","last_peered":"2024-03-12T03:51:23.486810+0000","last_clean":"2024-03-12T03:51:23.486810+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:51:23.486810+0000","last_undegraded":"2024-03-12T03:51:23.486810+0000","last_fullsized":"2024-03-12T03:51:23.486810+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:50:58.106746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:50:58.106746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:24.247584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":616,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.487418+0000","last_change":"2024-03-12T03:50:40.511469+0000","last_active":"2024-03-12T03:51:23.487418+0000","last_peered":"2024-03-12T03:51:23.487418+0000","last_clean":"2024-03-12T03:51:23.487418+0000","last_became_active":"2024-03-12T03:46:25.284416+0000","last_became_peered":"2024-03-12T03:46:25.284416+0000","last_unstale":"2024-03-12T03:51:23.487418+0000","last_undegraded":"2024-03-12T03:51:23.487418+0000","last_fullsized":"2024-03-12T03:51:23.487418+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:40.511391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:40.511391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:06.001248+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":938,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.485905+0000","last_change":"2024-03-12T03:48:58.460115+0000","last_active":"2024-03-12T03:51:23.485905+0000","last_peered":"2024-03-12T03:51:23.485905+0000","last_clean":"2024-03-12T03:48:53.471868+0000","last_became_active":"2024-03-12T03:48:58.460115+0000","last_became_peered":"2024-03-12T03:48:58.460115+0000","last_unstale":"2024-03-12T03:51:23.485905+0000","last_undegraded":"2024-03-12T03:51:23.485905+0000","last_fullsized":"2024-03-12T03:48:58.445534+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:06.690873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.690873+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"430'17530","reported_seq":17682,"reported_epoch":434,"state":"active+undersized","last_fresh":"2024-03-12T03:51:26.885769+0000","last_change":"2024-03-12T03:51:17.771635+0000","last_active":"2024-03-12T03:51:26.885769+0000","last_peered":"2024-03-12T03:51:26.885769+0000","last_clean":"2024-03-12T03:48:57.560710+0000","last_became_active":"2024-03-12T03:48:58.473224+0000","last_became_peered":"2024-03-12T03:48:58.473224+0000","last_unstale":"2024-03-12T03:51:26.885769+0000","last_undegraded":"2024-03-12T03:51:26.885769+0000","last_fullsized":"2024-03-12T03:48:58.446729+0000","mapping_epoch":384,"log_start":"405'11434","ondisk_log_start":"405'11434","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":6096,"log_dups_size":0,"ondisk_log_size":6096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:31.696164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17530,"num_write_kb":560960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":359,"reported_epoch":433,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:51:23.487116+0000","last_change":"2024-03-12T03:51:00.633662+0000","last_active":"2024-03-12T03:51:23.487116+0000","last_peered":"2024-03-12T03:51:23.487116+0000","last_clean":"2024-03-12T03:51:23.487116+0000","last_became_active":"2024-03-12T03:50:58.843997+0000","last_became_peered":"2024-03-12T03:50:58.843997+0000","last_unstale":"2024-03-12T03:51:23.487116+0000","last_undegraded":"2024-03-12T03:51:23.487116+0000","last_fullsized":"2024-03-12T03:51:23.487116+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:00.633453+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.924742+0000","last_clean_scrub_stamp":"2024-03-12T03:51:00.633453+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:21.187556+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"269'26","reported_seq":587,"reported_epoch":434,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:26.708306+0000","last_change":"2024-03-12T03:48:58.462328+0000","last_active":"2024-03-12T03:51:26.708306+0000","last_peered":"2024-03-12T03:51:26.708306+0000","last_clean":"2024-03-12T03:43:28.842466+0000","last_became_active":"2024-03-12T03:48:58.462328+0000","last_became_peered":"2024-03-12T03:48:58.462328+0000","last_unstale":"2024-03-12T03:51:26.708306+0000","last_undegraded":"2024-03-12T03:48:58.442501+0000","last_fullsized":"2024-03-12T03:48:58.442398+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:15.774076+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.774076+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.981892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":806,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486280+0000","last_change":"2024-03-12T03:49:45.255407+0000","last_active":"2024-03-12T03:51:23.486280+0000","last_peered":"2024-03-12T03:51:23.486280+0000","last_clean":"2024-03-12T03:51:23.486280+0000","last_became_active":"2024-03-12T03:46:25.303784+0000","last_became_peered":"2024-03-12T03:46:25.303784+0000","last_unstale":"2024-03-12T03:51:23.486280+0000","last_undegraded":"2024-03-12T03:51:23.486280+0000","last_fullsized":"2024-03-12T03:51:23.486280+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:45.255294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:49:45.255294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142673+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":537,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.885499+0000","last_change":"2024-03-12T03:49:19.918452+0000","last_active":"2024-03-12T03:51:26.885499+0000","last_peered":"2024-03-12T03:51:26.885499+0000","last_clean":"2024-03-12T03:51:26.885499+0000","last_became_active":"2024-03-12T03:46:25.284556+0000","last_became_peered":"2024-03-12T03:46:25.284556+0000","last_unstale":"2024-03-12T03:51:26.885499+0000","last_undegraded":"2024-03-12T03:51:26.885499+0000","last_fullsized":"2024-03-12T03:51:26.885499+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:19.918313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:19.918313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:31.696211+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":388,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.264921+0000","last_change":"2024-03-12T03:48:58.464502+0000","last_active":"2024-03-12T03:51:24.264921+0000","last_peered":"2024-03-12T03:51:24.264921+0000","last_clean":"2024-03-12T03:39:28.535672+0000","last_became_active":"2024-03-12T03:48:58.464502+0000","last_became_peered":"2024-03-12T03:48:58.464502+0000","last_unstale":"2024-03-12T03:51:24.264921+0000","last_undegraded":"2024-03-12T03:51:24.264921+0000","last_fullsized":"2024-03-12T03:48:58.446034+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:55.192996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:55.192996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266133+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"430'17062","reported_seq":17209,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.417895+0000","last_change":"2024-03-12T03:51:18.216253+0000","last_active":"2024-03-12T03:51:24.417895+0000","last_peered":"2024-03-12T03:51:24.417895+0000","last_clean":"2024-03-12T03:48:57.534492+0000","last_became_active":"2024-03-12T03:48:58.465249+0000","last_became_peered":"2024-03-12T03:48:58.465249+0000","last_unstale":"2024-03-12T03:51:24.417895+0000","last_undegraded":"2024-03-12T03:51:24.417895+0000","last_fullsized":"2024-03-12T03:48:58.443929+0000","mapping_epoch":384,"log_start":"407'12916","ondisk_log_start":"407'12916","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":4146,"log_dups_size":0,"ondisk_log_size":4146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17062,"num_write_kb":545984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":688,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.487495+0000","last_change":"2024-03-12T03:50:59.586938+0000","last_active":"2024-03-12T03:51:23.487495+0000","last_peered":"2024-03-12T03:51:23.487495+0000","last_clean":"2024-03-12T03:51:23.487495+0000","last_became_active":"2024-03-12T03:50:58.840022+0000","last_became_peered":"2024-03-12T03:50:58.840022+0000","last_unstale":"2024-03-12T03:51:23.487495+0000","last_undegraded":"2024-03-12T03:51:23.487495+0000","last_fullsized":"2024-03-12T03:51:23.487495+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.586800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.586800+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:24.596398+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"269'51","reported_seq":798,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.706604+0000","last_change":"2024-03-12T03:49:53.484809+0000","last_active":"2024-03-12T03:51:26.706604+0000","last_peered":"2024-03-12T03:51:26.706604+0000","last_clean":"2024-03-12T03:51:26.706604+0000","last_became_active":"2024-03-12T03:45:18.106503+0000","last_became_peered":"2024-03-12T03:45:18.106503+0000","last_unstale":"2024-03-12T03:51:26.706604+0000","last_undegraded":"2024-03-12T03:51:26.706604+0000","last_fullsized":"2024-03-12T03:51:26.706604+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'51","last_scrub_stamp":"2024-03-12T03:49:53.484646+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:49:53.484646+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.982093+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":806,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265441+0000","last_change":"2024-03-12T03:50:03.440676+0000","last_active":"2024-03-12T03:51:24.265441+0000","last_peered":"2024-03-12T03:51:24.265441+0000","last_clean":"2024-03-12T03:51:24.265441+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:51:24.265441+0000","last_undegraded":"2024-03-12T03:51:24.265441+0000","last_fullsized":"2024-03-12T03:51:24.265441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:03.440539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:03.440539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266299+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":522,"reported_epoch":434,"state":"active+undersized","last_fresh":"2024-03-12T03:51:26.885438+0000","last_change":"2024-03-12T03:48:58.459230+0000","last_active":"2024-03-12T03:51:26.885438+0000","last_peered":"2024-03-12T03:51:26.885438+0000","last_clean":"2024-03-12T03:48:53.446740+0000","last_became_active":"2024-03-12T03:48:58.459230+0000","last_became_peered":"2024-03-12T03:48:58.459230+0000","last_unstale":"2024-03-12T03:51:26.885438+0000","last_undegraded":"2024-03-12T03:51:26.885438+0000","last_fullsized":"2024-03-12T03:48:58.447475+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.954731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.954731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:31.696151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":824,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.264971+0000","last_change":"2024-03-12T03:50:42.276895+0000","last_active":"2024-03-12T03:51:24.264971+0000","last_peered":"2024-03-12T03:51:24.264971+0000","last_clean":"2024-03-12T03:51:24.264971+0000","last_became_active":"2024-03-12T03:46:25.292122+0000","last_became_peered":"2024-03-12T03:46:25.292122+0000","last_unstale":"2024-03-12T03:51:24.264971+0000","last_undegraded":"2024-03-12T03:51:24.264971+0000","last_fullsized":"2024-03-12T03:51:24.264971+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:42.276562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:42.276562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:08.411799+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"430'17568","reported_seq":17792,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.487616+0000","last_change":"2024-03-12T03:50:05.652788+0000","last_active":"2024-03-12T03:51:23.487616+0000","last_peered":"2024-03-12T03:51:23.487616+0000","last_clean":"2024-03-12T03:51:23.487616+0000","last_became_active":"2024-03-12T03:47:58.027372+0000","last_became_peered":"2024-03-12T03:47:58.027372+0000","last_unstale":"2024-03-12T03:51:23.487616+0000","last_undegraded":"2024-03-12T03:51:23.487616+0000","last_fullsized":"2024-03-12T03:51:23.487616+0000","mapping_epoch":366,"log_start":"407'13127","ondisk_log_start":"407'13127","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'10749","last_scrub_stamp":"2024-03-12T03:50:05.652456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:50:05.652456+0000","objects_scrubbed":6904,"log_size":4441,"log_dups_size":0,"ondisk_log_size":4441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603324+0000","scrub_duration":2133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17568,"num_write_kb":562176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":752,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.485790+0000","last_change":"2024-03-12T03:50:58.836775+0000","last_active":"2024-03-12T03:51:23.485790+0000","last_peered":"2024-03-12T03:51:23.485790+0000","last_clean":"2024-03-12T03:51:23.485790+0000","last_became_active":"2024-03-12T03:50:58.836091+0000","last_became_peered":"2024-03-12T03:50:58.836091+0000","last_unstale":"2024-03-12T03:51:23.485790+0000","last_undegraded":"2024-03-12T03:51:23.485790+0000","last_fullsized":"2024-03-12T03:51:23.485790+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:44:51.818256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:44:51.818256+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603148+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":538,"reported_epoch":434,"state":"active+undersized","last_fresh":"2024-03-12T03:51:26.885430+0000","last_change":"2024-03-12T03:48:58.461069+0000","last_active":"2024-03-12T03:51:26.885430+0000","last_peered":"2024-03-12T03:51:26.885430+0000","last_clean":"2024-03-12T03:48:53.445898+0000","last_became_active":"2024-03-12T03:48:58.461069+0000","last_became_peered":"2024-03-12T03:48:58.461069+0000","last_unstale":"2024-03-12T03:51:26.885430+0000","last_undegraded":"2024-03-12T03:51:26.885430+0000","last_fullsized":"2024-03-12T03:48:58.444677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.981014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.981014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:31.696190+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"269'30","reported_seq":877,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418096+0000","last_change":"2024-03-12T03:49:16.863752+0000","last_active":"2024-03-12T03:51:24.418096+0000","last_peered":"2024-03-12T03:51:24.418096+0000","last_clean":"2024-03-12T03:51:24.418096+0000","last_became_active":"2024-03-12T03:46:25.287885+0000","last_became_peered":"2024-03-12T03:46:25.287885+0000","last_unstale":"2024-03-12T03:51:24.418096+0000","last_undegraded":"2024-03-12T03:51:24.418096+0000","last_fullsized":"2024-03-12T03:51:24.418096+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'30","last_scrub_stamp":"2024-03-12T03:49:16.863622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:16.863622+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731314+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":814,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265707+0000","last_change":"2024-03-12T03:50:39.308944+0000","last_active":"2024-03-12T03:51:24.265707+0000","last_peered":"2024-03-12T03:51:24.265707+0000","last_clean":"2024-03-12T03:51:24.265707+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:51:24.265707+0000","last_undegraded":"2024-03-12T03:51:24.265707+0000","last_fullsized":"2024-03-12T03:51:24.265707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:39.308832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:50:39.308832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.888490+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":531,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486383+0000","last_change":"2024-03-12T03:50:01.616545+0000","last_active":"2024-03-12T03:51:23.486383+0000","last_peered":"2024-03-12T03:51:23.486383+0000","last_clean":"2024-03-12T03:51:23.486383+0000","last_became_active":"2024-03-12T03:46:25.280784+0000","last_became_peered":"2024-03-12T03:46:25.280784+0000","last_unstale":"2024-03-12T03:51:23.486383+0000","last_undegraded":"2024-03-12T03:51:23.486383+0000","last_fullsized":"2024-03-12T03:51:23.486383+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:01.616355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:01.616355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:23.896666+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"430'17516","reported_seq":17663,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.486508+0000","last_change":"2024-03-12T03:51:17.671209+0000","last_active":"2024-03-12T03:51:23.486508+0000","last_peered":"2024-03-12T03:51:23.486508+0000","last_clean":"2024-03-12T03:48:57.506664+0000","last_became_active":"2024-03-12T03:48:58.457494+0000","last_became_peered":"2024-03-12T03:48:58.457494+0000","last_unstale":"2024-03-12T03:51:23.486508+0000","last_undegraded":"2024-03-12T03:51:23.486508+0000","last_fullsized":"2024-03-12T03:48:58.442057+0000","mapping_epoch":384,"log_start":"407'12511","ondisk_log_start":"407'12511","created":366,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:56.681156+0000","last_clean_scrub_stamp":"2024-03-12T03:47:56.681156+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17516,"num_write_kb":560512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":693,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.417913+0000","last_change":"2024-03-12T03:50:58.872833+0000","last_active":"2024-03-12T03:51:24.417913+0000","last_peered":"2024-03-12T03:51:24.417913+0000","last_clean":"2024-03-12T03:51:24.417913+0000","last_became_active":"2024-03-12T03:50:58.850740+0000","last_became_peered":"2024-03-12T03:50:58.850740+0000","last_unstale":"2024-03-12T03:51:24.417913+0000","last_undegraded":"2024-03-12T03:51:24.417913+0000","last_fullsized":"2024-03-12T03:51:24.417913+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:58.872716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:58.872716+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:05.807093+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":1024,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265599+0000","last_change":"2024-03-12T03:50:37.369892+0000","last_active":"2024-03-12T03:51:24.265599+0000","last_peered":"2024-03-12T03:51:24.265599+0000","last_clean":"2024-03-12T03:51:24.265599+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:51:24.265599+0000","last_undegraded":"2024-03-12T03:51:24.265599+0000","last_fullsized":"2024-03-12T03:51:24.265599+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:50:37.369758+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:50:37.369758+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:46.476805+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"269'48","reported_seq":876,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.485730+0000","last_change":"2024-03-12T03:50:47.265447+0000","last_active":"2024-03-12T03:51:23.485730+0000","last_peered":"2024-03-12T03:51:23.485730+0000","last_clean":"2024-03-12T03:51:23.485730+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:51:23.485730+0000","last_undegraded":"2024-03-12T03:51:23.485730+0000","last_fullsized":"2024-03-12T03:51:23.485730+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'48","last_scrub_stamp":"2024-03-12T03:50:47.265298+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:50:47.265298+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:02.448314+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":805,"reported_epoch":433,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:24.265905+0000","last_change":"2024-03-12T03:48:58.456357+0000","last_active":"2024-03-12T03:51:24.265905+0000","last_peered":"2024-03-12T03:51:24.265905+0000","last_clean":"2024-03-12T03:48:53.450296+0000","last_became_active":"2024-03-12T03:48:58.456357+0000","last_became_peered":"2024-03-12T03:48:58.456357+0000","last_unstale":"2024-03-12T03:51:24.265905+0000","last_undegraded":"2024-03-12T03:48:58.445880+0000","last_fullsized":"2024-03-12T03:48:58.445769+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:47:05.588582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:05.588582+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":779,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.486482+0000","last_change":"2024-03-12T03:48:58.452652+0000","last_active":"2024-03-12T03:51:23.486482+0000","last_peered":"2024-03-12T03:51:23.486482+0000","last_clean":"2024-03-12T03:48:53.472055+0000","last_became_active":"2024-03-12T03:48:58.452652+0000","last_became_peered":"2024-03-12T03:48:58.452652+0000","last_unstale":"2024-03-12T03:51:23.486482+0000","last_undegraded":"2024-03-12T03:51:23.486482+0000","last_fullsized":"2024-03-12T03:48:58.441674+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:59.567890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.567890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142519+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":352,"reported_epoch":433,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:51:24.265010+0000","last_change":"2024-03-12T03:50:59.305188+0000","last_active":"2024-03-12T03:51:24.265010+0000","last_peered":"2024-03-12T03:51:24.265010+0000","last_clean":"2024-03-12T03:51:24.265010+0000","last_became_active":"2024-03-12T03:50:58.836425+0000","last_became_peered":"2024-03-12T03:50:58.836425+0000","last_unstale":"2024-03-12T03:51:24.265010+0000","last_undegraded":"2024-03-12T03:51:24.265010+0000","last_fullsized":"2024-03-12T03:51:24.265010+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.304950+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:50:06.384808+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.304950+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:24.443886+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,1],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"269'18","reported_seq":602,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.487789+0000","last_change":"2024-03-12T03:50:48.504074+0000","last_active":"2024-03-12T03:51:23.487789+0000","last_peered":"2024-03-12T03:51:23.487789+0000","last_clean":"2024-03-12T03:51:23.487789+0000","last_became_active":"2024-03-12T03:46:25.288573+0000","last_became_peered":"2024-03-12T03:46:25.288573+0000","last_unstale":"2024-03-12T03:51:23.487789+0000","last_undegraded":"2024-03-12T03:51:23.487789+0000","last_fullsized":"2024-03-12T03:51:23.487789+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-03-12T03:50:48.503336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.503336+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:06.681325+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":108,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.485764+0000","last_change":"2024-03-12T03:48:58.452359+0000","last_active":"2024-03-12T03:51:23.485764+0000","last_peered":"2024-03-12T03:51:23.485764+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:48:58.452359+0000","last_became_peered":"2024-03-12T03:48:58.452359+0000","last_unstale":"2024-03-12T03:51:23.485764+0000","last_undegraded":"2024-03-12T03:51:23.485764+0000","last_fullsized":"2024-03-12T03:48:58.441273+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:10.400135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.400135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":805,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486517+0000","last_change":"2024-03-12T03:50:42.134792+0000","last_active":"2024-03-12T03:51:23.486517+0000","last_peered":"2024-03-12T03:51:23.486517+0000","last_clean":"2024-03-12T03:51:23.486517+0000","last_became_active":"2024-03-12T03:46:25.293521+0000","last_became_peered":"2024-03-12T03:46:25.293521+0000","last_unstale":"2024-03-12T03:51:23.486517+0000","last_undegraded":"2024-03-12T03:51:23.486517+0000","last_fullsized":"2024-03-12T03:51:23.486517+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:42.134482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:42.134482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:50.548221+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":793,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265277+0000","last_change":"2024-03-12T03:49:58.457008+0000","last_active":"2024-03-12T03:51:24.265277+0000","last_peered":"2024-03-12T03:51:24.265277+0000","last_clean":"2024-03-12T03:51:24.265277+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:51:24.265277+0000","last_undegraded":"2024-03-12T03:51:24.265277+0000","last_fullsized":"2024-03-12T03:51:24.265277+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:58.456901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:49:58.456901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:58.456901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266314+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"269'27","reported_seq":728,"reported_epoch":433,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:23.487682+0000","last_change":"2024-03-12T03:48:58.462615+0000","last_active":"2024-03-12T03:51:23.487682+0000","last_peered":"2024-03-12T03:51:23.487682+0000","last_clean":"2024-03-12T03:43:13.333279+0000","last_became_active":"2024-03-12T03:48:58.462615+0000","last_became_peered":"2024-03-12T03:48:58.462615+0000","last_unstale":"2024-03-12T03:51:23.487682+0000","last_undegraded":"2024-03-12T03:48:58.443892+0000","last_fullsized":"2024-03-12T03:48:58.443761+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:48:06.104297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.104297+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603226+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":805,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265787+0000","last_change":"2024-03-12T03:50:35.416894+0000","last_active":"2024-03-12T03:51:24.265787+0000","last_peered":"2024-03-12T03:51:24.265787+0000","last_clean":"2024-03-12T03:51:24.265787+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:51:24.265787+0000","last_undegraded":"2024-03-12T03:51:24.265787+0000","last_fullsized":"2024-03-12T03:51:24.265787+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:35.416780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.416780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.856705+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":565,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.487915+0000","last_change":"2024-03-12T03:50:47.530660+0000","last_active":"2024-03-12T03:51:23.487915+0000","last_peered":"2024-03-12T03:51:23.487915+0000","last_clean":"2024-03-12T03:51:23.487915+0000","last_became_active":"2024-03-12T03:46:25.286866+0000","last_became_peered":"2024-03-12T03:46:25.286866+0000","last_unstale":"2024-03-12T03:51:23.487915+0000","last_undegraded":"2024-03-12T03:51:23.487915+0000","last_fullsized":"2024-03-12T03:51:23.487915+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:47.530547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:50:47.530547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:12.449208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":535,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.885682+0000","last_change":"2024-03-12T03:50:37.807983+0000","last_active":"2024-03-12T03:51:26.885682+0000","last_peered":"2024-03-12T03:51:26.885682+0000","last_clean":"2024-03-12T03:51:26.885682+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:51:26.885682+0000","last_undegraded":"2024-03-12T03:51:26.885682+0000","last_fullsized":"2024-03-12T03:51:26.885682+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:37.807855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:37.807855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:44.902247+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":822,"reported_epoch":433,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:51:24.264840+0000","last_change":"2024-03-12T03:50:57.227680+0000","last_active":"2024-03-12T03:51:24.264840+0000","last_peered":"2024-03-12T03:51:24.264840+0000","last_clean":"2024-03-12T03:51:24.264840+0000","last_became_active":"2024-03-12T03:50:56.823948+0000","last_became_peered":"2024-03-12T03:50:56.823948+0000","last_unstale":"2024-03-12T03:51:24.264840+0000","last_undegraded":"2024-03-12T03:51:24.264840+0000","last_fullsized":"2024-03-12T03:51:24.264840+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:57.227429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:57.227429+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:27.157222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"269'21","reported_seq":863,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486820+0000","last_change":"2024-03-12T03:51:01.099499+0000","last_active":"2024-03-12T03:51:23.486820+0000","last_peered":"2024-03-12T03:51:23.486820+0000","last_clean":"2024-03-12T03:51:23.486820+0000","last_became_active":"2024-03-12T03:46:25.298007+0000","last_became_peered":"2024-03-12T03:46:25.298007+0000","last_unstale":"2024-03-12T03:51:23.486820+0000","last_undegraded":"2024-03-12T03:51:23.486820+0000","last_fullsized":"2024-03-12T03:51:23.486820+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'21","last_scrub_stamp":"2024-03-12T03:51:01.099350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:01.099350+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:04.580700+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":682,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.265075+0000","last_change":"2024-03-12T03:48:58.462149+0000","last_active":"2024-03-12T03:51:24.265075+0000","last_peered":"2024-03-12T03:51:24.265075+0000","last_clean":"2024-03-12T03:43:13.331412+0000","last_became_active":"2024-03-12T03:48:58.462149+0000","last_became_peered":"2024-03-12T03:48:58.462149+0000","last_unstale":"2024-03-12T03:51:24.265075+0000","last_undegraded":"2024-03-12T03:51:24.265075+0000","last_fullsized":"2024-03-12T03:48:58.443909+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266172+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"332'14","reported_seq":812,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486866+0000","last_change":"2024-03-12T03:50:53.568427+0000","last_active":"2024-03-12T03:51:23.486866+0000","last_peered":"2024-03-12T03:51:23.486866+0000","last_clean":"2024-03-12T03:51:23.486866+0000","last_became_active":"2024-03-12T03:46:25.300463+0000","last_became_peered":"2024-03-12T03:46:25.300463+0000","last_unstale":"2024-03-12T03:51:23.486866+0000","last_undegraded":"2024-03-12T03:51:23.486866+0000","last_fullsized":"2024-03-12T03:51:23.486866+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'14","last_scrub_stamp":"2024-03-12T03:50:53.568307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.568307+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:17.071409+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":805,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418564+0000","last_change":"2024-03-12T03:50:04.042481+0000","last_active":"2024-03-12T03:51:24.418564+0000","last_peered":"2024-03-12T03:51:24.418564+0000","last_clean":"2024-03-12T03:51:24.418564+0000","last_became_active":"2024-03-12T03:39:33.265499+0000","last_became_peered":"2024-03-12T03:39:33.265499+0000","last_unstale":"2024-03-12T03:51:24.418564+0000","last_undegraded":"2024-03-12T03:51:24.418564+0000","last_fullsized":"2024-03-12T03:51:24.418564+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:04.042226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:04.042226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731393+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":753,"reported_epoch":433,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:51:23.485759+0000","last_change":"2024-03-12T03:50:59.098952+0000","last_active":"2024-03-12T03:51:23.485759+0000","last_peered":"2024-03-12T03:51:23.485759+0000","last_clean":"2024-03-12T03:51:23.485759+0000","last_became_active":"2024-03-12T03:50:58.849455+0000","last_became_peered":"2024-03-12T03:50:58.849455+0000","last_unstale":"2024-03-12T03:51:23.485759+0000","last_undegraded":"2024-03-12T03:51:23.485759+0000","last_fullsized":"2024-03-12T03:51:23.485759+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.098619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.098619+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:22.927231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,3],"avail_no_missing":["5","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"403'39","reported_seq":664,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486159+0000","last_change":"2024-03-12T03:50:06.535902+0000","last_active":"2024-03-12T03:51:23.486159+0000","last_peered":"2024-03-12T03:51:23.486159+0000","last_clean":"2024-03-12T03:51:23.486159+0000","last_became_active":"2024-03-12T03:46:25.299798+0000","last_became_peered":"2024-03-12T03:46:25.299798+0000","last_unstale":"2024-03-12T03:51:23.486159+0000","last_undegraded":"2024-03-12T03:51:23.486159+0000","last_fullsized":"2024-03-12T03:51:23.486159+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'39","last_scrub_stamp":"2024-03-12T03:50:06.535772+0000","last_deep_scrub":"403'39","last_deep_scrub_stamp":"2024-03-12T03:50:06.535772+0000","last_clean_scrub_stamp":"2024-03-12T03:50:06.535772+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603363+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":832,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265183+0000","last_change":"2024-03-12T03:50:00.385862+0000","last_active":"2024-03-12T03:51:24.265183+0000","last_peered":"2024-03-12T03:51:24.265183+0000","last_clean":"2024-03-12T03:51:24.265183+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:51:24.265183+0000","last_undegraded":"2024-03-12T03:51:24.265183+0000","last_fullsized":"2024-03-12T03:51:24.265183+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:50:00.385750+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:50:00.385750+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266285+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"385'9","reported_seq":1008,"reported_epoch":433,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:23.486598+0000","last_change":"2024-03-12T03:48:58.448464+0000","last_active":"2024-03-12T03:51:23.486598+0000","last_peered":"2024-03-12T03:51:23.486598+0000","last_clean":"2024-03-12T03:48:56.813639+0000","last_became_active":"2024-03-12T03:48:58.448464+0000","last_became_peered":"2024-03-12T03:48:58.448464+0000","last_unstale":"2024-03-12T03:51:23.486598+0000","last_undegraded":"2024-03-12T03:48:58.442644+0000","last_fullsized":"2024-03-12T03:48:58.442315+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:53.458170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.458170+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142528+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":670,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.265655+0000","last_change":"2024-03-12T03:48:58.463189+0000","last_active":"2024-03-12T03:51:24.265655+0000","last_peered":"2024-03-12T03:51:24.265655+0000","last_clean":"2024-03-12T03:43:13.332853+0000","last_became_active":"2024-03-12T03:48:58.463189+0000","last_became_peered":"2024-03-12T03:48:58.463189+0000","last_unstale":"2024-03-12T03:51:24.265655+0000","last_undegraded":"2024-03-12T03:51:24.265655+0000","last_fullsized":"2024-03-12T03:48:58.445060+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.079569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.079569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266190+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"269'19","reported_seq":840,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418141+0000","last_change":"2024-03-12T03:51:01.797152+0000","last_active":"2024-03-12T03:51:24.418141+0000","last_peered":"2024-03-12T03:51:24.418141+0000","last_clean":"2024-03-12T03:51:24.418141+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:51:24.418141+0000","last_undegraded":"2024-03-12T03:51:24.418141+0000","last_fullsized":"2024-03-12T03:51:24.418141+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'19","last_scrub_stamp":"2024-03-12T03:51:01.797018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:01.797018+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:06.011630+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":805,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418055+0000","last_change":"2024-03-12T03:49:50.830712+0000","last_active":"2024-03-12T03:51:24.418055+0000","last_peered":"2024-03-12T03:51:24.418055+0000","last_clean":"2024-03-12T03:51:24.418055+0000","last_became_active":"2024-03-12T03:33:21.109119+0000","last_became_peered":"2024-03-12T03:33:21.109119+0000","last_unstale":"2024-03-12T03:51:24.418055+0000","last_undegraded":"2024-03-12T03:51:24.418055+0000","last_fullsized":"2024-03-12T03:51:24.418055+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:50.830611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.830611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731344+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":816,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.707002+0000","last_change":"2024-03-12T03:50:48.723164+0000","last_active":"2024-03-12T03:51:26.707002+0000","last_peered":"2024-03-12T03:51:26.707002+0000","last_clean":"2024-03-12T03:51:26.707002+0000","last_became_active":"2024-03-12T03:46:25.288050+0000","last_became_peered":"2024-03-12T03:46:25.288050+0000","last_unstale":"2024-03-12T03:51:26.707002+0000","last_undegraded":"2024-03-12T03:51:26.707002+0000","last_fullsized":"2024-03-12T03:51:26.707002+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.723073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.723073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.423812+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":576,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.486229+0000","last_change":"2024-03-12T03:48:58.456788+0000","last_active":"2024-03-12T03:51:23.486229+0000","last_peered":"2024-03-12T03:51:23.486229+0000","last_clean":"2024-03-12T03:48:53.447742+0000","last_became_active":"2024-03-12T03:48:58.456788+0000","last_became_peered":"2024-03-12T03:48:58.456788+0000","last_unstale":"2024-03-12T03:51:23.486229+0000","last_undegraded":"2024-03-12T03:51:23.486229+0000","last_fullsized":"2024-03-12T03:48:58.442213+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.433184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.433184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603178+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"269'15","reported_seq":396,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.485822+0000","last_change":"2024-03-12T03:49:56.329520+0000","last_active":"2024-03-12T03:51:23.485822+0000","last_peered":"2024-03-12T03:51:23.485822+0000","last_clean":"2024-03-12T03:51:23.485822+0000","last_became_active":"2024-03-12T03:43:01.037348+0000","last_became_peered":"2024-03-12T03:43:01.037348+0000","last_unstale":"2024-03-12T03:51:23.485822+0000","last_undegraded":"2024-03-12T03:51:23.485822+0000","last_fullsized":"2024-03-12T03:51:23.485822+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'15","last_scrub_stamp":"2024-03-12T03:49:56.329393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:56.329393+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:27.126136+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":823,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265275+0000","last_change":"2024-03-12T03:50:43.240774+0000","last_active":"2024-03-12T03:51:24.265275+0000","last_peered":"2024-03-12T03:51:24.265275+0000","last_clean":"2024-03-12T03:51:24.265275+0000","last_became_active":"2024-03-12T03:46:25.283602+0000","last_became_peered":"2024-03-12T03:46:25.283602+0000","last_unstale":"2024-03-12T03:51:24.265275+0000","last_undegraded":"2024-03-12T03:51:24.265275+0000","last_fullsized":"2024-03-12T03:51:24.265275+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:43.240654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:50:43.240654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:45.000091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":588,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.485811+0000","last_change":"2024-03-12T03:48:58.460970+0000","last_active":"2024-03-12T03:51:23.485811+0000","last_peered":"2024-03-12T03:51:23.485811+0000","last_clean":"2024-03-12T03:48:53.448497+0000","last_became_active":"2024-03-12T03:48:58.460970+0000","last_became_peered":"2024-03-12T03:48:58.460970+0000","last_unstale":"2024-03-12T03:51:23.485811+0000","last_undegraded":"2024-03-12T03:51:23.485811+0000","last_fullsized":"2024-03-12T03:48:58.444135+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_clean_scrub_stamp":"2024-03-12T03:48:10.625582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":106,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.485846+0000","last_change":"2024-03-12T03:48:58.453783+0000","last_active":"2024-03-12T03:51:23.485846+0000","last_peered":"2024-03-12T03:51:23.485846+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.453783+0000","last_became_peered":"2024-03-12T03:48:58.453783+0000","last_unstale":"2024-03-12T03:51:23.485846+0000","last_undegraded":"2024-03-12T03:51:23.485846+0000","last_fullsized":"2024-03-12T03:48:58.441749+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:13.228117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:48:13.228117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"269'26","reported_seq":604,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486628+0000","last_change":"2024-03-12T03:49:47.515912+0000","last_active":"2024-03-12T03:51:23.486628+0000","last_peered":"2024-03-12T03:51:23.486628+0000","last_clean":"2024-03-12T03:51:23.486628+0000","last_became_active":"2024-03-12T03:46:25.287095+0000","last_became_peered":"2024-03-12T03:46:25.287095+0000","last_unstale":"2024-03-12T03:51:23.486628+0000","last_undegraded":"2024-03-12T03:51:23.486628+0000","last_fullsized":"2024-03-12T03:51:23.486628+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:49:47.515799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:47.515799+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603268+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":803,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486616+0000","last_change":"2024-03-12T03:50:00.379727+0000","last_active":"2024-03-12T03:51:23.486616+0000","last_peered":"2024-03-12T03:51:23.486616+0000","last_clean":"2024-03-12T03:51:23.486616+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:51:23.486616+0000","last_undegraded":"2024-03-12T03:51:23.486616+0000","last_fullsized":"2024-03-12T03:51:23.486616+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:00.379623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:50:00.379623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142713+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":797,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.266295+0000","last_change":"2024-03-12T03:50:36.405438+0000","last_active":"2024-03-12T03:51:24.266295+0000","last_peered":"2024-03-12T03:51:24.266295+0000","last_clean":"2024-03-12T03:51:24.266295+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:51:24.266295+0000","last_undegraded":"2024-03-12T03:51:24.266295+0000","last_fullsized":"2024-03-12T03:51:24.266295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:36.405298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:36.405298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:39.013601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":603,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.486447+0000","last_change":"2024-03-12T03:48:58.449798+0000","last_active":"2024-03-12T03:51:23.486447+0000","last_peered":"2024-03-12T03:51:23.486447+0000","last_clean":"2024-03-12T03:48:53.448767+0000","last_became_active":"2024-03-12T03:48:58.449798+0000","last_became_peered":"2024-03-12T03:48:58.449798+0000","last_unstale":"2024-03-12T03:51:23.486447+0000","last_undegraded":"2024-03-12T03:51:23.486447+0000","last_fullsized":"2024-03-12T03:48:58.443857+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"269'10","reported_seq":583,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.485365+0000","last_change":"2024-03-12T03:50:39.504167+0000","last_active":"2024-03-12T03:51:23.485365+0000","last_peered":"2024-03-12T03:51:23.485365+0000","last_clean":"2024-03-12T03:51:23.485365+0000","last_became_active":"2024-03-12T03:46:25.286315+0000","last_became_peered":"2024-03-12T03:46:25.286315+0000","last_unstale":"2024-03-12T03:51:23.485365+0000","last_undegraded":"2024-03-12T03:51:23.485365+0000","last_fullsized":"2024-03-12T03:51:23.485365+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'10","last_scrub_stamp":"2024-03-12T03:50:39.504024+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:50:39.504024+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:54.158992+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":682,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.487043+0000","last_change":"2024-03-12T03:48:58.463096+0000","last_active":"2024-03-12T03:51:23.487043+0000","last_peered":"2024-03-12T03:51:23.487043+0000","last_clean":"2024-03-12T03:43:13.332505+0000","last_became_active":"2024-03-12T03:48:58.463096+0000","last_became_peered":"2024-03-12T03:48:58.463096+0000","last_unstale":"2024-03-12T03:51:23.487043+0000","last_undegraded":"2024-03-12T03:51:23.487043+0000","last_fullsized":"2024-03-12T03:48:58.443705+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.851271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.851271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":268,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.417555+0000","last_change":"2024-03-12T03:51:00.809146+0000","last_active":"2024-03-12T03:51:24.417555+0000","last_peered":"2024-03-12T03:51:24.417555+0000","last_clean":"2024-03-12T03:51:24.417555+0000","last_became_active":"2024-03-12T03:45:18.109983+0000","last_became_peered":"2024-03-12T03:45:18.109983+0000","last_unstale":"2024-03-12T03:51:24.417555+0000","last_undegraded":"2024-03-12T03:51:24.417555+0000","last_fullsized":"2024-03-12T03:51:24.417555+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:00.809033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:00.809033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:26.492749+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":805,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265440+0000","last_change":"2024-03-12T03:50:38.371213+0000","last_active":"2024-03-12T03:51:24.265440+0000","last_peered":"2024-03-12T03:51:24.265440+0000","last_clean":"2024-03-12T03:51:24.265440+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:51:24.265440+0000","last_undegraded":"2024-03-12T03:51:24.265440+0000","last_fullsized":"2024-03-12T03:51:24.265440+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:38.371102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:38.371102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:45.481431+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"269'31","reported_seq":768,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.487135+0000","last_change":"2024-03-12T03:49:53.291968+0000","last_active":"2024-03-12T03:51:23.487135+0000","last_peered":"2024-03-12T03:51:23.487135+0000","last_clean":"2024-03-12T03:51:23.487135+0000","last_became_active":"2024-03-12T03:45:18.107047+0000","last_became_peered":"2024-03-12T03:45:18.107047+0000","last_unstale":"2024-03-12T03:51:23.487135+0000","last_undegraded":"2024-03-12T03:51:23.487135+0000","last_fullsized":"2024-03-12T03:51:23.487135+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:49:53.291824+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:49:53.291824+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.090865+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":604,"reported_epoch":433,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:23.485503+0000","last_change":"2024-03-12T03:48:58.454467+0000","last_active":"2024-03-12T03:51:23.485503+0000","last_peered":"2024-03-12T03:51:23.485503+0000","last_clean":"2024-03-12T03:48:53.477110+0000","last_became_active":"2024-03-12T03:48:58.454467+0000","last_became_peered":"2024-03-12T03:48:58.454467+0000","last_unstale":"2024-03-12T03:51:23.485503+0000","last_undegraded":"2024-03-12T03:48:58.439427+0000","last_fullsized":"2024-03-12T03:48:58.439320+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:48:00.458460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.458460+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":393,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.265780+0000","last_change":"2024-03-12T03:48:58.461994+0000","last_active":"2024-03-12T03:51:24.265780+0000","last_peered":"2024-03-12T03:51:24.265780+0000","last_clean":"2024-03-12T03:39:31.213817+0000","last_became_active":"2024-03-12T03:48:58.461994+0000","last_became_peered":"2024-03-12T03:48:58.461994+0000","last_unstale":"2024-03-12T03:51:24.265780+0000","last_undegraded":"2024-03-12T03:51:24.265780+0000","last_fullsized":"2024-03-12T03:48:58.446456+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:09.160277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:09.160277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266197+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":801,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.708370+0000","last_change":"2024-03-12T03:50:53.919752+0000","last_active":"2024-03-12T03:51:26.708370+0000","last_peered":"2024-03-12T03:51:26.708370+0000","last_clean":"2024-03-12T03:51:26.708370+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:51:26.708370+0000","last_undegraded":"2024-03-12T03:51:26.708370+0000","last_fullsized":"2024-03-12T03:51:26.708370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:53.919661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.919661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:06.964625+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"269'27","reported_seq":599,"reported_epoch":433,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:24.265879+0000","last_change":"2024-03-12T03:48:58.462533+0000","last_active":"2024-03-12T03:51:24.265879+0000","last_peered":"2024-03-12T03:51:24.265879+0000","last_clean":"2024-03-12T03:43:13.333148+0000","last_became_active":"2024-03-12T03:48:58.462533+0000","last_became_peered":"2024-03-12T03:48:58.462533+0000","last_unstale":"2024-03-12T03:51:24.265879+0000","last_undegraded":"2024-03-12T03:48:58.446989+0000","last_fullsized":"2024-03-12T03:48:58.446871+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:15.795515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.795515+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"351'2","reported_seq":818,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.706156+0000","last_change":"2024-03-12T03:49:14.611362+0000","last_active":"2024-03-12T03:51:26.706156+0000","last_peered":"2024-03-12T03:51:26.706156+0000","last_clean":"2024-03-12T03:51:26.706156+0000","last_became_active":"2024-03-12T03:46:25.283344+0000","last_became_peered":"2024-03-12T03:46:25.283344+0000","last_unstale":"2024-03-12T03:51:26.706156+0000","last_undegraded":"2024-03-12T03:51:26.706156+0000","last_fullsized":"2024-03-12T03:51:26.706156+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'2","last_scrub_stamp":"2024-03-12T03:49:14.611213+0000","last_deep_scrub":"351'2","last_deep_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_clean_scrub_stamp":"2024-03-12T03:49:14.611213+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.982060+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'9","reported_seq":369,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418727+0000","last_change":"2024-03-12T03:50:46.889659+0000","last_active":"2024-03-12T03:51:24.418727+0000","last_peered":"2024-03-12T03:51:24.418727+0000","last_clean":"2024-03-12T03:51:24.418727+0000","last_became_active":"2024-03-12T03:45:18.108935+0000","last_became_peered":"2024-03-12T03:45:18.108935+0000","last_unstale":"2024-03-12T03:51:24.418727+0000","last_undegraded":"2024-03-12T03:51:24.418727+0000","last_fullsized":"2024-03-12T03:51:24.418727+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'9","last_scrub_stamp":"2024-03-12T03:50:46.889538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:46.889538+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:50.165482+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":814,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265807+0000","last_change":"2024-03-12T03:49:48.376454+0000","last_active":"2024-03-12T03:51:24.265807+0000","last_peered":"2024-03-12T03:51:24.265807+0000","last_clean":"2024-03-12T03:51:24.265807+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:51:24.265807+0000","last_undegraded":"2024-03-12T03:51:24.265807+0000","last_fullsized":"2024-03-12T03:51:24.265807+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:49:48.375999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:48.375999+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.266235+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"269'24","reported_seq":640,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486296+0000","last_change":"2024-03-12T03:50:05.585005+0000","last_active":"2024-03-12T03:51:23.486296+0000","last_peered":"2024-03-12T03:51:23.486296+0000","last_clean":"2024-03-12T03:51:23.486296+0000","last_became_active":"2024-03-12T03:46:25.283193+0000","last_became_peered":"2024-03-12T03:46:25.283193+0000","last_unstale":"2024-03-12T03:51:23.486296+0000","last_undegraded":"2024-03-12T03:51:23.486296+0000","last_fullsized":"2024-03-12T03:51:23.486296+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:50:05.584740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:05.584740+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.476261+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":825,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418361+0000","last_change":"2024-03-12T03:50:49.933782+0000","last_active":"2024-03-12T03:51:24.418361+0000","last_peered":"2024-03-12T03:51:24.418361+0000","last_clean":"2024-03-12T03:51:24.418361+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:51:24.418361+0000","last_undegraded":"2024-03-12T03:51:24.418361+0000","last_fullsized":"2024-03-12T03:51:24.418361+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:49.933653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:49.933653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:08.330902+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":819,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418458+0000","last_change":"2024-03-12T03:49:19.894833+0000","last_active":"2024-03-12T03:51:24.418458+0000","last_peered":"2024-03-12T03:51:24.418458+0000","last_clean":"2024-03-12T03:51:24.418458+0000","last_became_active":"2024-03-12T03:46:25.301691+0000","last_became_peered":"2024-03-12T03:46:25.301691+0000","last_unstale":"2024-03-12T03:51:24.418458+0000","last_undegraded":"2024-03-12T03:51:24.418458+0000","last_fullsized":"2024-03-12T03:51:24.418458+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:19.894587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:49:19.894587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731261+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":610,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.488012+0000","last_change":"2024-03-12T03:49:54.607508+0000","last_active":"2024-03-12T03:51:23.488012+0000","last_peered":"2024-03-12T03:51:23.488012+0000","last_clean":"2024-03-12T03:51:23.488012+0000","last_became_active":"2024-03-12T03:46:25.296764+0000","last_became_peered":"2024-03-12T03:46:25.296764+0000","last_unstale":"2024-03-12T03:51:23.488012+0000","last_undegraded":"2024-03-12T03:51:23.488012+0000","last_fullsized":"2024-03-12T03:51:23.488012+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:54.607394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.607394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603252+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"269'27","reported_seq":585,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.885597+0000","last_change":"2024-03-12T03:49:57.717659+0000","last_active":"2024-03-12T03:51:26.885597+0000","last_peered":"2024-03-12T03:51:26.885597+0000","last_clean":"2024-03-12T03:51:26.885597+0000","last_became_active":"2024-03-12T03:46:25.301700+0000","last_became_peered":"2024-03-12T03:46:25.301700+0000","last_unstale":"2024-03-12T03:51:26.885597+0000","last_undegraded":"2024-03-12T03:51:26.885597+0000","last_fullsized":"2024-03-12T03:51:26.885597+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:49:57.717507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:57.717507+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:31.696245+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":105,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:24.417841+0000","last_change":"2024-03-12T03:48:58.455613+0000","last_active":"2024-03-12T03:51:24.417841+0000","last_peered":"2024-03-12T03:51:24.417841+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:48:58.455613+0000","last_became_peered":"2024-03-12T03:48:58.455613+0000","last_unstale":"2024-03-12T03:51:24.417841+0000","last_undegraded":"2024-03-12T03:51:24.417841+0000","last_fullsized":"2024-03-12T03:48:58.445944+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.797296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.797296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":792,"reported_epoch":434,"state":"active+undersized","last_fresh":"2024-03-12T03:51:26.708183+0000","last_change":"2024-03-12T03:48:58.458438+0000","last_active":"2024-03-12T03:51:26.708183+0000","last_peered":"2024-03-12T03:51:26.708183+0000","last_clean":"2024-03-12T03:48:53.448127+0000","last_became_active":"2024-03-12T03:48:58.458438+0000","last_became_peered":"2024-03-12T03:48:58.458438+0000","last_unstale":"2024-03-12T03:51:26.708183+0000","last_undegraded":"2024-03-12T03:51:26.708183+0000","last_fullsized":"2024-03-12T03:48:58.440067+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.016030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.016030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.981916+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":794,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418313+0000","last_change":"2024-03-12T03:50:51.952305+0000","last_active":"2024-03-12T03:51:24.418313+0000","last_peered":"2024-03-12T03:51:24.418313+0000","last_clean":"2024-03-12T03:51:24.418313+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:51:24.418313+0000","last_undegraded":"2024-03-12T03:51:24.418313+0000","last_fullsized":"2024-03-12T03:51:24.418313+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:51.952189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:51.952189+0000","last_clean_scrub_stamp":"2024-03-12T03:50:51.952189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:21.592139+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"269'16","reported_seq":880,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.418079+0000","last_change":"2024-03-12T03:49:51.806758+0000","last_active":"2024-03-12T03:51:24.418079+0000","last_peered":"2024-03-12T03:51:24.418079+0000","last_clean":"2024-03-12T03:51:24.418079+0000","last_became_active":"2024-03-12T03:46:25.283025+0000","last_became_peered":"2024-03-12T03:46:25.283025+0000","last_unstale":"2024-03-12T03:51:24.418079+0000","last_undegraded":"2024-03-12T03:51:24.418079+0000","last_fullsized":"2024-03-12T03:51:24.418079+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'16","last_scrub_stamp":"2024-03-12T03:49:51.806651+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:49:51.806651+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":795,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486395+0000","last_change":"2024-03-12T03:50:02.319955+0000","last_active":"2024-03-12T03:51:23.486395+0000","last_peered":"2024-03-12T03:51:23.486395+0000","last_clean":"2024-03-12T03:51:23.486395+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:51:23.486395+0000","last_undegraded":"2024-03-12T03:51:23.486395+0000","last_fullsized":"2024-03-12T03:51:23.486395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:02.319837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:02.319837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:25.368545+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":797,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.265333+0000","last_change":"2024-03-12T03:51:01.307708+0000","last_active":"2024-03-12T03:51:24.265333+0000","last_peered":"2024-03-12T03:51:24.265333+0000","last_clean":"2024-03-12T03:51:24.265333+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:51:24.265333+0000","last_undegraded":"2024-03-12T03:51:24.265333+0000","last_fullsized":"2024-03-12T03:51:24.265333+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:01.307529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:51:01.307529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:03.635009+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":793,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486173+0000","last_change":"2024-03-12T03:49:46.378515+0000","last_active":"2024-03-12T03:51:23.486173+0000","last_peered":"2024-03-12T03:51:23.486173+0000","last_clean":"2024-03-12T03:51:23.486173+0000","last_became_active":"2024-03-12T03:46:25.293777+0000","last_became_peered":"2024-03-12T03:46:25.293777+0000","last_unstale":"2024-03-12T03:51:23.486173+0000","last_undegraded":"2024-03-12T03:51:23.486173+0000","last_fullsized":"2024-03-12T03:51:23.486173+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:46.378413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:46.378413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142730+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"345'33","reported_seq":870,"reported_epoch":433,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:24.418181+0000","last_change":"2024-03-12T03:48:58.457301+0000","last_active":"2024-03-12T03:51:24.418181+0000","last_peered":"2024-03-12T03:51:24.418181+0000","last_clean":"2024-03-12T03:48:53.446883+0000","last_became_active":"2024-03-12T03:48:58.457301+0000","last_became_peered":"2024-03-12T03:48:58.457301+0000","last_unstale":"2024-03-12T03:51:24.418181+0000","last_undegraded":"2024-03-12T03:48:58.444868+0000","last_fullsized":"2024-03-12T03:48:58.444750+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'33","last_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_deep_scrub":"345'33","last_deep_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_clean_scrub_stamp":"2024-03-12T03:47:02.110517+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731157+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":803,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486479+0000","last_change":"2024-03-12T03:50:41.172421+0000","last_active":"2024-03-12T03:51:23.486479+0000","last_peered":"2024-03-12T03:51:23.486479+0000","last_clean":"2024-03-12T03:51:23.486479+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:51:23.486479+0000","last_undegraded":"2024-03-12T03:51:23.486479+0000","last_fullsized":"2024-03-12T03:51:23.486479+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:41.172313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:50:41.172313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:47.192043+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":782,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.486570+0000","last_change":"2024-03-12T03:48:58.453634+0000","last_active":"2024-03-12T03:51:23.486570+0000","last_peered":"2024-03-12T03:51:23.486570+0000","last_clean":"2024-03-12T03:48:53.472261+0000","last_became_active":"2024-03-12T03:48:58.453634+0000","last_became_peered":"2024-03-12T03:48:58.453634+0000","last_unstale":"2024-03-12T03:51:23.486570+0000","last_undegraded":"2024-03-12T03:51:23.486570+0000","last_fullsized":"2024-03-12T03:48:58.444137+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:02.618599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_clean_scrub_stamp":"2024-03-12T03:46:02.618599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142546+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":108,"reported_epoch":434,"state":"active+undersized","last_fresh":"2024-03-12T03:51:26.705417+0000","last_change":"2024-03-12T03:48:58.463158+0000","last_active":"2024-03-12T03:51:26.705417+0000","last_peered":"2024-03-12T03:51:26.705417+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.463158+0000","last_became_peered":"2024-03-12T03:48:58.463158+0000","last_unstale":"2024-03-12T03:51:26.705417+0000","last_undegraded":"2024-03-12T03:51:26.705417+0000","last_fullsized":"2024-03-12T03:48:58.439250+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_clean_scrub_stamp":"2024-03-12T03:45:58.095196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.981900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"269'62","reported_seq":755,"reported_epoch":433,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:23.485705+0000","last_change":"2024-03-12T03:48:58.457134+0000","last_active":"2024-03-12T03:51:23.485705+0000","last_peered":"2024-03-12T03:51:23.485705+0000","last_clean":"2024-03-12T03:48:53.477142+0000","last_became_active":"2024-03-12T03:48:58.457134+0000","last_became_peered":"2024-03-12T03:48:58.457134+0000","last_unstale":"2024-03-12T03:51:23.485705+0000","last_undegraded":"2024-03-12T03:48:58.441299+0000","last_fullsized":"2024-03-12T03:48:58.441188+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:26.603109+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":635,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.487539+0000","last_change":"2024-03-12T03:50:37.469343+0000","last_active":"2024-03-12T03:51:23.487539+0000","last_peered":"2024-03-12T03:51:23.487539+0000","last_clean":"2024-03-12T03:51:23.487539+0000","last_became_active":"2024-03-12T03:46:25.279131+0000","last_became_peered":"2024-03-12T03:46:25.279131+0000","last_unstale":"2024-03-12T03:51:23.487539+0000","last_undegraded":"2024-03-12T03:51:23.487539+0000","last_fullsized":"2024-03-12T03:51:23.487539+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:37.469265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:50:37.469265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:53.254911+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":797,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.417641+0000","last_change":"2024-03-12T03:50:39.866882+0000","last_active":"2024-03-12T03:51:24.417641+0000","last_peered":"2024-03-12T03:51:24.417641+0000","last_clean":"2024-03-12T03:51:24.417641+0000","last_became_active":"2024-03-12T03:33:25.179951+0000","last_became_peered":"2024-03-12T03:33:25.179951+0000","last_unstale":"2024-03-12T03:51:24.417641+0000","last_undegraded":"2024-03-12T03:51:24.417641+0000","last_fullsized":"2024-03-12T03:51:24.417641+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:39.866768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:50:39.866768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:43.470676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":585,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:23.486285+0000","last_change":"2024-03-12T03:49:55.557456+0000","last_active":"2024-03-12T03:51:23.486285+0000","last_peered":"2024-03-12T03:51:23.486285+0000","last_clean":"2024-03-12T03:51:23.486285+0000","last_became_active":"2024-03-12T03:46:25.281081+0000","last_became_peered":"2024-03-12T03:46:25.281081+0000","last_unstale":"2024-03-12T03:51:23.486285+0000","last_undegraded":"2024-03-12T03:51:23.486285+0000","last_fullsized":"2024-03-12T03:51:23.486285+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:55.557335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:55.557335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.968328+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"269'26","reported_seq":843,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.417613+0000","last_change":"2024-03-12T03:50:57.888885+0000","last_active":"2024-03-12T03:51:24.417613+0000","last_peered":"2024-03-12T03:51:24.417613+0000","last_clean":"2024-03-12T03:51:24.417613+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:51:24.417613+0000","last_undegraded":"2024-03-12T03:51:24.417613+0000","last_fullsized":"2024-03-12T03:51:24.417613+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:50:57.888633+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:50:57.888633+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:23.716232+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":400,"reported_epoch":433,"state":"active+undersized","last_fresh":"2024-03-12T03:51:23.486931+0000","last_change":"2024-03-12T03:48:58.462711+0000","last_active":"2024-03-12T03:51:23.486931+0000","last_peered":"2024-03-12T03:51:23.486931+0000","last_clean":"2024-03-12T03:39:28.536135+0000","last_became_active":"2024-03-12T03:48:58.462711+0000","last_became_peered":"2024-03-12T03:48:58.462711+0000","last_unstale":"2024-03-12T03:51:23.486931+0000","last_undegraded":"2024-03-12T03:51:23.486931+0000","last_fullsized":"2024-03-12T03:48:58.443314+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:12.186146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:48:12.186146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:24.142603+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":806,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-12T03:51:26.706249+0000","last_change":"2024-03-12T03:50:04.535064+0000","last_active":"2024-03-12T03:51:26.706249+0000","last_peered":"2024-03-12T03:51:26.706249+0000","last_clean":"2024-03-12T03:51:26.706249+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:51:26.706249+0000","last_undegraded":"2024-03-12T03:51:26.706249+0000","last_fullsized":"2024-03-12T03:51:26.706249+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_clean_scrub_stamp":"2024-03-12T03:50:04.534937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:34.126590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":793,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-12T03:51:24.417677+0000","last_change":"2024-03-12T03:49:03.833199+0000","last_active":"2024-03-12T03:51:24.417677+0000","last_peered":"2024-03-12T03:51:24.417677+0000","last_clean":"2024-03-12T03:51:24.417677+0000","last_became_active":"2024-03-12T03:33:27.182865+0000","last_became_peered":"2024-03-12T03:33:27.182865+0000","last_unstale":"2024-03-12T03:51:24.417677+0000","last_undegraded":"2024-03-12T03:51:24.417677+0000","last_fullsized":"2024-03-12T03:51:24.417677+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:03.832786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_clean_scrub_stamp":"2024-03-12T03:49:03.832786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:28.731246+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277986,"num_write_kb":8895489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3205627904,"data_stored":3205627904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78065,"ondisk_log_size":78065,"up":38,"acting":38,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":1071,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":19,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":933560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2177,"ondisk_log_size":2177,"up":49,"acting":53,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1397,"num_read_kb":1122,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":2642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":83,"acting":83,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":87,"acting":87,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070686,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":778548,"kb_used_data":3240,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92966604,"statfs":{"total":95995035648,"available":95197802496,"internally_reserved":0,"allocated":3317760,"data_stored":1982043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":793901514},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332323,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":674076,"kb_used_data":2896,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93071076,"statfs":{"total":95995035648,"available":95304781824,"internally_reserved":0,"allocated":2965504,"data_stored":1652400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":687274438},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575100,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":788008,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92957144,"statfs":{"total":95995035648,"available":95188115456,"internally_reserved":0,"allocated":3039232,"data_stored":1661540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738595,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3725208,"kb_used_data":3102608,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":90019944,"statfs":{"total":95995035648,"available":92180422656,"internally_reserved":0,"allocated":3177070592,"data_stored":3175997927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411574,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":720156,"kb_used_data":3152,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93024996,"statfs":{"total":95995035648,"available":95257595904,"internally_reserved":0,"allocated":3227648,"data_stored":1916101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":330,"seq":1417339207741,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304636,"kb_used_data":3504,"kb_used_omap":13,"kb_used_meta":301106,"kb_avail":93440516,"statfs":{"total":95995035648,"available":95683088384,"internally_reserved":0,"allocated":3588096,"data_stored":2237103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13485,"internal_metadata":308333395},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953710,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647736,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93097416,"statfs":{"total":95995035648,"available":95331753984,"internally_reserved":0,"allocated":2994176,"data_stored":1653903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659368,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":869028,"kb_used_data":3480,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92876124,"statfs":{"total":95995035648,"available":95105150976,"internally_reserved":0,"allocated":3563520,"data_stored":2243191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3205627904,"data_stored":3205627904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:51:30.486 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-12T03:51:30.487 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-12T03:51:30.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-12T03:51:30.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:30.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:30.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:30.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:30.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:30.614 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:30.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:30.623 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:51:30.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:30.735 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:30.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:30.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:30.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:30.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:30.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:30.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:30.932 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:30 smithi169 ceph-mon[29344]: from='client.25652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:51:30.932 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:30 smithi169 ceph-mon[29344]: 1.b scrub starts 2024-03-12T03:51:30.932 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:30 smithi169 ceph-mon[29344]: 1.b scrub ok 2024-03-12T03:51:30.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:30 smithi169 ceph-mon[39949]: from='client.25652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:51:30.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:30 smithi169 ceph-mon[39949]: 1.b scrub starts 2024-03-12T03:51:30.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:30 smithi169 ceph-mon[39949]: 1.b scrub ok 2024-03-12T03:51:30.961 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:51:30.961 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:51:30.962 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:30 smithi173 ceph-mon[33544]: from='client.25652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:51:30.962 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:30 smithi173 ceph-mon[33544]: 1.b scrub starts 2024-03-12T03:51:30.962 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:30 smithi173 ceph-mon[33544]: 1.b scrub ok 2024-03-12T03:51:30.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:30.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:30.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:30.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:31.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:31.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:31.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:31.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:31.242 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:31.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:31.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:31.367 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:31.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:31.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:31.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:31.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:31.594 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:31.618 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:31.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:31.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:51:31.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-12T03:51:31.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:31.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:31.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:31.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:31.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:31.894 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:31.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:31.946 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:31 smithi173 ceph-mon[33544]: pgmap v880: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 763 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:31 smithi173 ceph-mon[33544]: 5.1 scrub starts 2024-03-12T03:51:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:31 smithi173 ceph-mon[33544]: 5.1 scrub ok 2024-03-12T03:51:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:31.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:32.006 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:31 smithi169 ceph-mon[39949]: pgmap v880: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 763 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:32.006 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:31 smithi169 ceph-mon[39949]: 5.1 scrub starts 2024-03-12T03:51:32.006 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:31 smithi169 ceph-mon[39949]: 5.1 scrub ok 2024-03-12T03:51:32.006 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:32.006 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:31 smithi169 ceph-mon[29344]: pgmap v880: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 763 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:32.007 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:31 smithi169 ceph-mon[29344]: 5.1 scrub starts 2024-03-12T03:51:32.007 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:31 smithi169 ceph-mon[29344]: 5.1 scrub ok 2024-03-12T03:51:32.007 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:32.007 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:51:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:51:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:51:32.020 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:32.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:32.047 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:32.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:32.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:32.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:32.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:32.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:32.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:32.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:32.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:32.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:32.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:32.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:32.499 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:32.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:32.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:32.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:32.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:32.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:32.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:32.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:32.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:32.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:32.875 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:32.899 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:32.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:33.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:33.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:33.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:33.073 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:32 smithi169 ceph-mon[29344]: 6.3 scrub starts 2024-03-12T03:51:33.073 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:32 smithi169 ceph-mon[29344]: 6.3 scrub ok 2024-03-12T03:51:33.073 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:32 smithi169 ceph-mon[29344]: 4.11 scrub starts 2024-03-12T03:51:33.073 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:32 smithi169 ceph-mon[29344]: 4.11 scrub ok 2024-03-12T03:51:33.073 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:32 smithi169 ceph-mon[39949]: 6.3 scrub starts 2024-03-12T03:51:33.073 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:32 smithi169 ceph-mon[39949]: 6.3 scrub ok 2024-03-12T03:51:33.073 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:32 smithi169 ceph-mon[39949]: 4.11 scrub starts 2024-03-12T03:51:33.073 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:32 smithi169 ceph-mon[39949]: 4.11 scrub ok 2024-03-12T03:51:33.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:33.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:33.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:33.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:33.178 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:33.192 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:32 smithi173 ceph-mon[33544]: 6.3 scrub starts 2024-03-12T03:51:33.192 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:32 smithi173 ceph-mon[33544]: 6.3 scrub ok 2024-03-12T03:51:33.192 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:32 smithi173 ceph-mon[33544]: 4.11 scrub starts 2024-03-12T03:51:33.192 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:32 smithi173 ceph-mon[33544]: 4.11 scrub ok 2024-03-12T03:51:33.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:33.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:33.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:33.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:33.405 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:33.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:33.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:51:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:51:33.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:33.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:33.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:33.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:33.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:33.659 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:33.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:33.747 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:51:33.747 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:51:33.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:33.782 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'unique_pool_3' removed 2024-03-12T03:51:33.782 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:33 smithi173 ceph-mon[33544]: pgmap v881: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:33.782 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:33 smithi173 ceph-mon[33544]: 3.13 scrub starts 2024-03-12T03:51:33.783 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:33 smithi173 ceph-mon[33544]: 3.13 scrub ok 2024-03-12T03:51:33.783 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:33 smithi173 ceph-mon[33544]: 6.1a scrub starts 2024-03-12T03:51:33.783 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:33 smithi173 ceph-mon[33544]: 6.1a scrub ok 2024-03-12T03:51:33.783 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:33 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3405011975' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:51:33.783 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:33 smithi173 ceph-mon[33544]: 5.a scrub starts 2024-03-12T03:51:33.783 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:33 smithi173 ceph-mon[33544]: 5.a scrub ok 2024-03-12T03:51:33.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:33.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:33.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:33.877 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[29344]: pgmap v881: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:33.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[29344]: 3.13 scrub starts 2024-03-12T03:51:33.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[29344]: 3.13 scrub ok 2024-03-12T03:51:33.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[29344]: 6.1a scrub starts 2024-03-12T03:51:33.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[29344]: 6.1a scrub ok 2024-03-12T03:51:33.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3405011975' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:51:33.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[29344]: 5.a scrub starts 2024-03-12T03:51:33.878 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[29344]: 5.a scrub ok 2024-03-12T03:51:33.878 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[39949]: pgmap v881: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 44 active+undersized, 102 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:33.878 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[39949]: 3.13 scrub starts 2024-03-12T03:51:33.878 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[39949]: 3.13 scrub ok 2024-03-12T03:51:33.878 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[39949]: 6.1a scrub starts 2024-03-12T03:51:33.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[39949]: 6.1a scrub ok 2024-03-12T03:51:33.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3405011975' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:51:33.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[39949]: 5.a scrub starts 2024-03-12T03:51:33.879 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:33 smithi169 ceph-mon[39949]: 5.a scrub ok 2024-03-12T03:51:33.892 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:33.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:33.918 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:33.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:34.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:34.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:34.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:34.147 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:34.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:34.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:34.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:34.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:34.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:34.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:34.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-12T03:51:34.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:34.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:34.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:34.483 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-12T03:51:34.484 INFO:tasks.radosbench:Beginning radosbench... 2024-03-12T03:51:34.484 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-12T03:51:34.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-12T03:51:34.531 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:34.555 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:34.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:34.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:34.626 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:34.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:34.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:34.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:34.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:34.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:34.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:34.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:34.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:34.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:34.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:34.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:34.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:35.029 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:35.029 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[29344]: 10.f scrub starts 2024-03-12T03:51:35.029 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[29344]: 10.f scrub ok 2024-03-12T03:51:35.029 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3405011975' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:51:35.029 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[29344]: osdmap e436: 8 total, 7 up, 8 in 2024-03-12T03:51:35.029 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[29344]: 10.0 scrub starts 2024-03-12T03:51:35.029 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[29344]: 10.0 scrub ok 2024-03-12T03:51:35.030 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:35.030 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[39949]: 10.f scrub starts 2024-03-12T03:51:35.030 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[39949]: 10.f scrub ok 2024-03-12T03:51:35.030 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3405011975' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:51:35.030 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[39949]: osdmap e436: 8 total, 7 up, 8 in 2024-03-12T03:51:35.030 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[39949]: 10.0 scrub starts 2024-03-12T03:51:35.030 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:34 smithi169 ceph-mon[39949]: 10.0 scrub ok 2024-03-12T03:51:35.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:35.047 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:35.047 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:34 smithi173 ceph-mon[33544]: 10.f scrub starts 2024-03-12T03:51:35.047 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:34 smithi173 ceph-mon[33544]: 10.f scrub ok 2024-03-12T03:51:35.047 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:34 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3405011975' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:51:35.047 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:34 smithi173 ceph-mon[33544]: osdmap e436: 8 total, 7 up, 8 in 2024-03-12T03:51:35.048 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:34 smithi173 ceph-mon[33544]: 10.0 scrub starts 2024-03-12T03:51:35.048 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:34 smithi173 ceph-mon[33544]: 10.0 scrub ok 2024-03-12T03:51:35.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:35.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:35.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:35.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:35.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:35.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:35.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:35.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:35.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:35.346 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:35.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:35.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:35.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:35.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:35.487 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:51:35.488 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:51:35.488 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-12T03:51:35.488 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.489174 2024-03-12T03:51:35.488 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 4 0.48917398989879735 2024-03-12T03:51:35.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:35.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:35.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:35.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:35.691 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:35.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:35.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:35.802 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:51:35.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:35.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:35.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:35.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:35.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[39949]: pgmap v883: 150 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 34 active+undersized, 96 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 546 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:35.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[39949]: 3.d scrub starts 2024-03-12T03:51:35.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[39949]: 3.d scrub ok 2024-03-12T03:51:35.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[39949]: 5.1c scrub starts 2024-03-12T03:51:35.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[39949]: 5.1c scrub ok 2024-03-12T03:51:35.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 73/690 objects degraded (10.580%), 16 pgs degraded, 50 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:35.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1635102303' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:35.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[29344]: pgmap v883: 150 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 34 active+undersized, 96 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 546 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[29344]: 3.d scrub starts 2024-03-12T03:51:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[29344]: 3.d scrub ok 2024-03-12T03:51:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[29344]: 5.1c scrub starts 2024-03-12T03:51:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[29344]: 5.1c scrub ok 2024-03-12T03:51:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 73/690 objects degraded (10.580%), 16 pgs degraded, 50 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1635102303' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:35.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:35 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:35.942 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:35.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:36.043 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:36.067 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:36.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:36.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:36.117 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:35 smithi173 ceph-mon[33544]: pgmap v883: 150 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 34 active+undersized, 96 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 546 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:35 smithi173 ceph-mon[33544]: 3.d scrub starts 2024-03-12T03:51:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:35 smithi173 ceph-mon[33544]: 3.d scrub ok 2024-03-12T03:51:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:35 smithi173 ceph-mon[33544]: 5.1c scrub starts 2024-03-12T03:51:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:35 smithi173 ceph-mon[33544]: 5.1c scrub ok 2024-03-12T03:51:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:35 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 73/690 objects degraded (10.580%), 16 pgs degraded, 50 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:36.118 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:35 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1635102303' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:36.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:35 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:36.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:36.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:36.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:36.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:36.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:36.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:36.320 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:36.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:36.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:36.446 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:36.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:36.547 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:36.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:36.571 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:36.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:36.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:36.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:36.697 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:36.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:36.715 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:51:36.715 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:51:36.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:36.822 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:36.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:36.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:36.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:36.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:36.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:36.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:36 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:51:36.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:36 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:51:36.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:36 smithi173 ceph-mon[33544]: osdmap e437: 8 total, 7 up, 8 in 2024-03-12T03:51:36.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:36 smithi173 ceph-mon[33544]: 6.1 scrub starts 2024-03-12T03:51:36.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:36 smithi173 ceph-mon[33544]: 6.1 scrub ok 2024-03-12T03:51:37.051 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:37.075 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:37.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:37.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:36 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:51:37.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:36 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:51:37.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:36 smithi169 ceph-mon[29344]: osdmap e437: 8 total, 7 up, 8 in 2024-03-12T03:51:37.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:36 smithi169 ceph-mon[29344]: 6.1 scrub starts 2024-03-12T03:51:37.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:36 smithi169 ceph-mon[29344]: 6.1 scrub ok 2024-03-12T03:51:37.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:36 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:51:37.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:36 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:51:37.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:36 smithi169 ceph-mon[39949]: osdmap e437: 8 total, 7 up, 8 in 2024-03-12T03:51:37.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:36 smithi169 ceph-mon[39949]: 6.1 scrub starts 2024-03-12T03:51:37.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:36 smithi169 ceph-mon[39949]: 6.1 scrub ok 2024-03-12T03:51:37.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:37.201 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:37.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:37.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:37.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:37.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:37.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:37.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:37.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-12T03:51:37.428 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:37.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:37.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:37.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:37.585 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:37.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:37.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:37.652 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:37.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:37.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:37.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:37.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:37.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:37.836 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:37.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:37.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:37.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:37.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:37.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:37.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:38.036 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:37 smithi169 ceph-mon[29344]: pgmap v885: 150 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 34 active+undersized, 96 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 531 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:38.036 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:37 smithi169 ceph-mon[29344]: 6.c scrub starts 2024-03-12T03:51:38.036 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:37 smithi169 ceph-mon[29344]: 6.c scrub ok 2024-03-12T03:51:38.037 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:37 smithi169 ceph-mon[39949]: pgmap v885: 150 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 34 active+undersized, 96 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 531 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:38.037 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:37 smithi169 ceph-mon[39949]: 6.c scrub starts 2024-03-12T03:51:38.037 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:37 smithi169 ceph-mon[39949]: 6.c scrub ok 2024-03-12T03:51:38.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:38.089 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:38.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:38.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:37 smithi173 ceph-mon[33544]: pgmap v885: 150 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 34 active+undersized, 96 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 531 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:38.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:37 smithi173 ceph-mon[33544]: 6.c scrub starts 2024-03-12T03:51:38.105 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:37 smithi173 ceph-mon[33544]: 6.c scrub ok 2024-03-12T03:51:38.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:38.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:38.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:38.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:38.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:38.342 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:38.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:38.443 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:38.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:38.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:38.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:38.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:38.484 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:51:38.515 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:51:38.534 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:51:38.555 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:51:38.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:38.575 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:51:38.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:38.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:38.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:38.595 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:51:38.615 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:51:38.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:51:38.615 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:51:38.615 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:51:38.615 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:51:38.616 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:51:38.616 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:51:38.616 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:51:38.616 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:51:38.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:38.718 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:38.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:38.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:38.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:38.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:38.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:38 smithi169 ceph-mon[29344]: 1.4 scrub starts 2024-03-12T03:51:38.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:38 smithi169 ceph-mon[29344]: 1.4 scrub ok 2024-03-12T03:51:38.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:38 smithi169 ceph-mon[29344]: 6.f scrub starts 2024-03-12T03:51:38.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:38 smithi169 ceph-mon[29344]: 6.f scrub ok 2024-03-12T03:51:38.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:38 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3320677943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:38.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:38 smithi169 ceph-mon[39949]: 1.4 scrub starts 2024-03-12T03:51:38.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:38 smithi169 ceph-mon[39949]: 1.4 scrub ok 2024-03-12T03:51:38.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:38 smithi169 ceph-mon[39949]: 6.f scrub starts 2024-03-12T03:51:38.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:38 smithi169 ceph-mon[39949]: 6.f scrub ok 2024-03-12T03:51:38.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:38 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3320677943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:38.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:38.970 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:38.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:38.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:39.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:39.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:39.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:39.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:39.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:38 smithi173 ceph-mon[33544]: 1.4 scrub starts 2024-03-12T03:51:39.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:38 smithi173 ceph-mon[33544]: 1.4 scrub ok 2024-03-12T03:51:39.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:38 smithi173 ceph-mon[33544]: 6.f scrub starts 2024-03-12T03:51:39.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:38 smithi173 ceph-mon[33544]: 6.f scrub ok 2024-03-12T03:51:39.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:38 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3320677943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:39.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:39.221 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:39.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:39.288 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:51:39.288 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:51:39.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:39.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:51:39.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:39.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:39.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:39.387 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:51:39.388 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:51:39.388 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-03-12T03:51:39.388 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:51:39.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:39.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:39.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:39.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:39.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:39.601 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:39.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:39.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:39.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:39.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:39.821 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'unique_pool_4' created 2024-03-12T03:51:39.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:39.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:39.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:39.954 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:39.979 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:39.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:39 smithi173 ceph-mon[33544]: pgmap v886: 150 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 34 active+undersized, 96 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:39 smithi173 ceph-mon[33544]: 6.15 deep-scrub starts 2024-03-12T03:51:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:39 smithi173 ceph-mon[33544]: 6.15 deep-scrub ok 2024-03-12T03:51:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:39 smithi173 ceph-mon[33544]: 4.1c scrub starts 2024-03-12T03:51:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:39 smithi173 ceph-mon[33544]: 4.1c scrub ok 2024-03-12T03:51:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:39 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2888414128' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-12T03:51:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:39 smithi173 ceph-mon[33544]: 4.4 scrub starts 2024-03-12T03:51:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:39 smithi173 ceph-mon[33544]: 4.4 scrub ok 2024-03-12T03:51:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:39 smithi173 ceph-mon[33544]: 6.7 scrub starts 2024-03-12T03:51:39.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:39 smithi173 ceph-mon[33544]: 6.7 scrub ok 2024-03-12T03:51:40.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:40.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-12T03:51:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:40.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:40.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:40.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[29344]: pgmap v886: 150 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 34 active+undersized, 96 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:40.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[29344]: 6.15 deep-scrub starts 2024-03-12T03:51:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[29344]: 6.15 deep-scrub ok 2024-03-12T03:51:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[29344]: 4.1c scrub starts 2024-03-12T03:51:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[29344]: 4.1c scrub ok 2024-03-12T03:51:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2888414128' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-12T03:51:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[29344]: 4.4 scrub starts 2024-03-12T03:51:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[29344]: 4.4 scrub ok 2024-03-12T03:51:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[29344]: 6.7 scrub starts 2024-03-12T03:51:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[29344]: 6.7 scrub ok 2024-03-12T03:51:40.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[39949]: pgmap v886: 150 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 34 active+undersized, 96 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:40.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[39949]: 6.15 deep-scrub starts 2024-03-12T03:51:40.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[39949]: 6.15 deep-scrub ok 2024-03-12T03:51:40.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[39949]: 4.1c scrub starts 2024-03-12T03:51:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[39949]: 4.1c scrub ok 2024-03-12T03:51:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2888414128' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-12T03:51:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[39949]: 4.4 scrub starts 2024-03-12T03:51:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[39949]: 4.4 scrub ok 2024-03-12T03:51:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[39949]: 6.7 scrub starts 2024-03-12T03:51:40.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:39 smithi169 ceph-mon[39949]: 6.7 scrub ok 2024-03-12T03:51:40.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:40.230 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:40.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:40.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:40.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:40.348 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:40.358 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:40.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:40.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:40.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:40.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:40.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:40.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:40.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-12T03:51:40.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:40.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:40.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:40.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:40.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:40.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:40.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:40.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:40.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:40.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:40.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:40.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:40.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:40.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:41.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:41.114 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:41.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:41.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:40 smithi169 ceph-mon[39949]: 5.10 deep-scrub starts 2024-03-12T03:51:41.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:40 smithi169 ceph-mon[39949]: 5.10 deep-scrub ok 2024-03-12T03:51:41.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:40 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2888414128' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-12T03:51:41.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:40 smithi169 ceph-mon[39949]: osdmap e438: 8 total, 7 up, 8 in 2024-03-12T03:51:41.126 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:40 smithi169 ceph-mon[29344]: 5.10 deep-scrub starts 2024-03-12T03:51:41.126 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:40 smithi169 ceph-mon[29344]: 5.10 deep-scrub ok 2024-03-12T03:51:41.126 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:40 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2888414128' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-12T03:51:41.126 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:40 smithi169 ceph-mon[29344]: osdmap e438: 8 total, 7 up, 8 in 2024-03-12T03:51:41.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:40 smithi173 ceph-mon[33544]: 5.10 deep-scrub starts 2024-03-12T03:51:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:40 smithi173 ceph-mon[33544]: 5.10 deep-scrub ok 2024-03-12T03:51:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:40 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2888414128' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-12T03:51:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:40 smithi173 ceph-mon[33544]: osdmap e438: 8 total, 7 up, 8 in 2024-03-12T03:51:41.240 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:41.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:41.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:41.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:51:41.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:41.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:41.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:41.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:41.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:41.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:41.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:41.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:41.527 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:41.582 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:41.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:41.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:41.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:41.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:41.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:41.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:41.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:41.834 INFO:teuthology.orchestra.run.smithi169.stderr:set osd.4 primary-affinity to 0.489174 (8320582) 2024-03-12T03:51:41.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:41.871 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:41.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:41.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:51:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:51:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[39949]: pgmap v888: 166 pgs: 16 unknown, 4 active+clean+remapped, 16 active+undersized+degraded, 34 active+undersized, 96 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 796 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[39949]: 4.14 scrub starts 2024-03-12T03:51:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[39949]: 4.14 scrub ok 2024-03-12T03:51:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[39949]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:51:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[39949]: osdmap e439: 8 total, 7 up, 8 in 2024-03-12T03:51:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1379121293' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.48917398989879735}]: dispatch 2024-03-12T03:51:41.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.48917398989879735}]: dispatch 2024-03-12T03:51:41.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[29344]: pgmap v888: 166 pgs: 16 unknown, 4 active+clean+remapped, 16 active+undersized+degraded, 34 active+undersized, 96 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 796 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:41.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[29344]: 4.14 scrub starts 2024-03-12T03:51:41.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[29344]: 4.14 scrub ok 2024-03-12T03:51:41.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[29344]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:51:41.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[29344]: osdmap e439: 8 total, 7 up, 8 in 2024-03-12T03:51:41.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1379121293' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.48917398989879735}]: dispatch 2024-03-12T03:51:41.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:41 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.48917398989879735}]: dispatch 2024-03-12T03:51:41.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:41.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:41.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:41.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:42.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:42.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:42.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:42.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:42.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:42.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:41 smithi173 ceph-mon[33544]: pgmap v888: 166 pgs: 16 unknown, 4 active+clean+remapped, 16 active+undersized+degraded, 34 active+undersized, 96 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 796 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:42.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:41 smithi173 ceph-mon[33544]: 4.14 scrub starts 2024-03-12T03:51:42.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:41 smithi173 ceph-mon[33544]: 4.14 scrub ok 2024-03-12T03:51:42.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:41 smithi173 ceph-mon[33544]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:51:42.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:41 smithi173 ceph-mon[33544]: osdmap e439: 8 total, 7 up, 8 in 2024-03-12T03:51:42.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:41 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1379121293' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.48917398989879735}]: dispatch 2024-03-12T03:51:42.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:41 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.48917398989879735}]: dispatch 2024-03-12T03:51:42.249 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:42.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:42.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:42.376 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:42.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:42.478 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:42.482 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:42.504 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:42.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:42.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:42.630 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:42.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:42.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:42.731 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:42.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:42.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:42.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:42.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:42.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:42.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:42.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:42.921 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[29344]: 6.b scrub starts 2024-03-12T03:51:42.921 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[29344]: 6.b scrub ok 2024-03-12T03:51:42.921 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.48917398989879735}]': finished 2024-03-12T03:51:42.921 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[29344]: osdmap e440: 8 total, 7 up, 8 in 2024-03-12T03:51:42.921 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[29344]: 3.16 scrub starts 2024-03-12T03:51:42.922 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[29344]: 3.16 scrub ok 2024-03-12T03:51:42.922 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[29344]: 5.d scrub starts 2024-03-12T03:51:42.922 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[29344]: 5.d scrub ok 2024-03-12T03:51:42.922 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[29344]: osdmap e441: 8 total, 7 up, 8 in 2024-03-12T03:51:42.922 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[39949]: 6.b scrub starts 2024-03-12T03:51:42.922 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[39949]: 6.b scrub ok 2024-03-12T03:51:42.922 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.48917398989879735}]': finished 2024-03-12T03:51:42.922 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[39949]: osdmap e440: 8 total, 7 up, 8 in 2024-03-12T03:51:42.922 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[39949]: 3.16 scrub starts 2024-03-12T03:51:42.922 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[39949]: 3.16 scrub ok 2024-03-12T03:51:42.922 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[39949]: 5.d scrub starts 2024-03-12T03:51:42.922 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[39949]: 5.d scrub ok 2024-03-12T03:51:42.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:42 smithi169 ceph-mon[39949]: osdmap e441: 8 total, 7 up, 8 in 2024-03-12T03:51:42.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:42.995 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:51:42.995 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:51:42.995 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:51:42.996 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:51:42.996 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:51:43.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:43.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:43.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:43.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:43.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:43.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:43.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:42 smithi173 ceph-mon[33544]: 6.b scrub starts 2024-03-12T03:51:43.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:42 smithi173 ceph-mon[33544]: 6.b scrub ok 2024-03-12T03:51:43.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:42 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.48917398989879735}]': finished 2024-03-12T03:51:43.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:42 smithi173 ceph-mon[33544]: osdmap e440: 8 total, 7 up, 8 in 2024-03-12T03:51:43.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:42 smithi173 ceph-mon[33544]: 3.16 scrub starts 2024-03-12T03:51:43.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:42 smithi173 ceph-mon[33544]: 3.16 scrub ok 2024-03-12T03:51:43.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:42 smithi173 ceph-mon[33544]: 5.d scrub starts 2024-03-12T03:51:43.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:42 smithi173 ceph-mon[33544]: 5.d scrub ok 2024-03-12T03:51:43.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:42 smithi173 ceph-mon[33544]: osdmap e441: 8 total, 7 up, 8 in 2024-03-12T03:51:43.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:43.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:43.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:43.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:43.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:43.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:43.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:43.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:43.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:43.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:43.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:43.586 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:51:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:51:43.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-12T03:51:43.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:43.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:43.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:43.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:43.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:43.783 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:43.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:43.830 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:43.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:43.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:43.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:44.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:44.035 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:44.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:44.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:44.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:44.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:44.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:43 smithi169 ceph-mon[39949]: pgmap v891: 166 pgs: 2 peering, 7 unknown, 4 active+clean+remapped, 16 active+undersized+degraded, 36 active+undersized, 101 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:44.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:43 smithi169 ceph-mon[39949]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-12T03:51:44.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:43 smithi169 ceph-mon[29344]: pgmap v891: 166 pgs: 2 peering, 7 unknown, 4 active+clean+remapped, 16 active+undersized+degraded, 36 active+undersized, 101 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:44.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:43 smithi169 ceph-mon[29344]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-12T03:51:44.222 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:43 smithi173 ceph-mon[33544]: pgmap v891: 166 pgs: 2 peering, 7 unknown, 4 active+clean+remapped, 16 active+undersized+degraded, 36 active+undersized, 101 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:44.222 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:43 smithi173 ceph-mon[33544]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-12T03:51:44.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:44.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:44.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:44.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:44.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:44.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:44.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:44.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:44.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:44.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:44.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:44.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:44.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:44.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:44.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:44.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:44.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:44.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:44.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:44.916 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:44.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:45.010 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:45.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:45.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:45.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:45.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:45.080 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:45.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:45.165 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:45.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:45.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:45.267 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:45.291 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:45.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:45.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:45.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:45.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:45.518 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:45.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:45.543 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:45.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:45.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:45.664 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:51:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:51:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:207226 8.5 8:afc51c0f:::benchmark_data_smithi169_106492_object73743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:25.224290+0000", 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 560.43644258999996, 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18384750899999999, 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 207226 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224290+0000", 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224290+0000", 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224291+0000", 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.204e-06 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224295+0000", 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.296e-06 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224303+0000", 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0160000000000008e-06 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.224311+0000", 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4599999999999997e-06 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.230995+0000", 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0066846989999999997 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.231191+0000", 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000196202 2024-03-12T03:51:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.231333+0000", 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000141361 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.235484+0000", 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041509989999999998 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.256368+0000", 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020884171 2024-03-12T03:51:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.407984+0000", 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.151616267 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.408078+0000", 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3574999999999998e-05 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:25.408137+0000", 2024-03-12T03:51:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9259000000000003e-05 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:210664 8.5 8:ac07f159:::benchmark_data_smithi169_106492_object77181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:29.292061+0000", 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 556.36867165299998, 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19662268099999999, 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:51:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 210664 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292061+0000", 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292061+0000", 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292062+0000", 2024-03-12T03:51:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.483e-06 2024-03-12T03:51:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292066+0000", 2024-03-12T03:51:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1309999999999999e-06 2024-03-12T03:51:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292076+0000", 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5689999999999994e-06 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.292082+0000", 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6760000000000001e-06 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.359067+0000", 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.066984848 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.359154+0000", 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7153999999999999e-05 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.359237+0000", 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3061999999999999e-05 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.365384+0000", 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006146744 2024-03-12T03:51:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.368955+0000", 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035707500000000001 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.488598+0000", 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.119642817 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.488634+0000", 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6406999999999999e-05 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.488683+0000", 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9039999999999998e-05 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24914.0:210936 8.5 8:a7cf8e7a:::benchmark_data_smithi169_106492_object77453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:42:29.560667+0000", 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 556.100065524, 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17761102000000001, 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24914", 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1426052812", 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 210936 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560667+0000", 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560667+0000", 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560668+0000", 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.201e-06 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560673+0000", 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8840000000000002e-06 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560685+0000", 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1907e-05 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.560691+0000", 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4670000000000001e-06 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.626200+0000", 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.065508937000000003 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.626307+0000", 2024-03-12T03:51:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107222 2024-03-12T03:51:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,7", 2024-03-12T03:51:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.626416+0000", 2024-03-12T03:51:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108634 2024-03-12T03:51:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.632883+0000", 2024-03-12T03:51:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00646724 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.634125+0000", 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012422920000000001 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.738168+0000", 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104042204 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.738216+0000", 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8018e-05 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:42:29.738278+0000", 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2014000000000006e-05 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:258442 9.1 9:8a150de2:::benchmark_data_smithi169_135229_object114227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:47:00.807043+0000", 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 284.85368877899998, 2024-03-12T03:51:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.030208796, 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 258442 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807043+0000", 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807043+0000", 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807045+0000", 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8300000000000001e-06 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807051+0000", 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.728e-06 2024-03-12T03:51:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807062+0000", 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1297e-05 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807078+0000", 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5784999999999999e-05 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807943+0000", 2024-03-12T03:51:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086520399999999995 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.808076+0000", 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000132724 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.808206+0000", 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013005100000000001 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.810938+0000", 2024-03-12T03:51:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027315830000000001 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.813484+0000", 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025461030000000001 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.836790+0000", 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.023306473000000001 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.836901+0000", 2024-03-12T03:51:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011044699999999999 2024-03-12T03:51:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.837252+0000", 2024-03-12T03:51:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035157100000000001 2024-03-12T03:51:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78615 10.e e383/366 10:7dfe4819:::benchmark_data_smithi169_162694_object78614:head v 383'4854, mlcod=383'4852)", 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.988349+0000", 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.67238317499999, 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.042519055, 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988349+0000", 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988349+0000", 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988350+0000", 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4499999999999995e-07 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988374+0000", 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4346999999999999e-05 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988385+0000", 2024-03-12T03:51:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0669000000000001e-05 2024-03-12T03:51:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988393+0000", 2024-03-12T03:51:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1259999999999998e-06 2024-03-12T03:51:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988481+0000", 2024-03-12T03:51:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7696999999999997e-05 2024-03-12T03:51:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988506+0000", 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4791999999999999e-05 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.990980+0000", 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024742309999999999 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030868+0000", 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039888248000000001 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78612 10.d e383/366 10:b3a4751d:::benchmark_data_smithi169_162694_object78611:head v 383'4904, mlcod=383'4902)", 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.988423+0000", 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.67230938399999, 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.042485090000000003, 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988423+0000", 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988423+0000", 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988423+0000", 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2500000000000001e-07 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988443+0000", 2024-03-12T03:51:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9653e-05 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988449+0000", 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5939999999999999e-06 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988455+0000", 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2120000000000002e-06 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988543+0000", 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8616999999999995e-05 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988562+0000", 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8893999999999999e-05 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991063+0000", 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025009260000000001 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030908+0000", 2024-03-12T03:51:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039844769000000002 2024-03-12T03:51:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78613 10.a e383/366 10:5ef8fbae:::benchmark_data_smithi169_162694_object78612:head v 383'4848, mlcod=383'4846)", 2024-03-12T03:51:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.988636+0000", 2024-03-12T03:51:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.67209611999999, 2024-03-12T03:51:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.042224314999999998, 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988636+0000", 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988636+0000", 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988637+0000", 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4600000000000005e-07 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988733+0000", 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6521000000000006e-05 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988742+0000", 2024-03-12T03:51:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1600000000000004e-06 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988749+0000", 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6409999999999996e-06 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988805+0000", 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6424000000000001e-05 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988823+0000", 2024-03-12T03:51:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7884e-05 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030828+0000", 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.042004527999999999 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030860+0000", 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2611000000000003e-05 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78614 10.f e383/366 10:fb4a5bd3:::benchmark_data_smithi169_162694_object78613:head v 383'4934, mlcod=383'4932)", 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.989287+0000", 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.671444777, 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041609650999999998, 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989287+0000", 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989287+0000", 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989288+0000", 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7199999999999998e-07 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989336+0000", 2024-03-12T03:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7648999999999997e-05 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989345+0000", 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3889999999999994e-06 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989352+0000", 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7410000000000003e-06 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989426+0000", 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3819999999999995e-05 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989444+0000", 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8717000000000001e-05 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030864+0000", 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041419639000000001 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030897+0000", 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3024000000000001e-05 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78618 10.7 e383/366 10:e87822aa:::benchmark_data_smithi169_162694_object78617:head v 383'4971, mlcod=383'4969)", 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.991518+0000", 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.66921367899999, 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.051062718999999999, 2024-03-12T03:51:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991518+0000", 2024-03-12T03:51:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991518+0000", 2024-03-12T03:51:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991519+0000", 2024-03-12T03:51:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1699999999999997e-07 2024-03-12T03:51:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991552+0000", 2024-03-12T03:51:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2230999999999997e-05 2024-03-12T03:51:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030814+0000", 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039262496000000001 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030828+0000", 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4209000000000001e-05 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030876+0000", 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8000000000000001e-05 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030898+0000", 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2101000000000001e-05 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.042488+0000", 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011589868 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.042581+0000", 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2997000000000004e-05 2024-03-12T03:51:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78634 10.a e383/366 10:57b48c2c:::benchmark_data_smithi169_162694_object78633:head v 383'4849, mlcod=383'4846)", 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.005400+0000", 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.65533214000001, 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.029465791000000002, 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.005400+0000", 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.005400+0000", 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.005401+0000", 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1039999999999999e-06 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.005443+0000", 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2264000000000001e-05 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030824+0000", 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025381048 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030835+0000", 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0369e-05 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030890+0000", 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4744999999999998e-05 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030908+0000", 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8261e-05 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.034823+0000", 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039151730000000001 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.034866+0000", 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2827000000000002e-05 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:79202 10.b e383/366 10:dbb2f93d:::benchmark_data_smithi169_162694_object79201:head v 383'4898, mlcod=383'4896)", 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.472218+0000", 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.188514029, 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026937955999999999, 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.472218+0000", 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.472218+0000", 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.472219+0000", 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.54e-07 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491702+0000", 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.019482816999999999 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491709+0000", 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7909999999999999e-06 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491716+0000", 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1609999999999997e-06 2024-03-12T03:51:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491764+0000", 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8424000000000003e-05 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491783+0000", 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8453000000000001e-05 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.499104+0000", 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0073216410000000003 2024-03-12T03:51:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.499156+0000", 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1915e-05 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:110617 10.a e389/384 10:5aa4436c:::benchmark_data_smithi169_162694_object110616:head v 389'6831, mlcod=389'6828)", 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:20.439153+0000", 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 145.22157936900001, 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027448368000000001, 2024-03-12T03:51:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439153+0000", 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439153+0000", 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439154+0000", 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1799999999999999e-06 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439180+0000", 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6109999999999999e-05 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439191+0000", 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1099e-05 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439203+0000", 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2242000000000001e-05 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.440972+0000", 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001768146 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.440997+0000", 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4956999999999998e-05 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.466546+0000", 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025549471000000001 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.466601+0000", 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5163e-05 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:114672 10.f e391/366 10:fe0e19ae:::benchmark_data_smithi169_162694_object114671:head v 391'7222, mlcod=391'7219)", 2024-03-12T03:51:45.726 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:23.949154+0000", 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 141.71157837499999, 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026819050000000001, 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949154+0000", 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949154+0000", 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949155+0000", 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.540000000000001e-07 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949190+0000", 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5561000000000001e-05 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949201+0000", 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0206e-05 2024-03-12T03:51:45.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949208+0000", 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8769999999999993e-06 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.950125+0000", 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091698799999999998 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.950147+0000", 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1537000000000001e-05 2024-03-12T03:51:45.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.975879+0000", 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025731878 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.975973+0000", 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4048999999999993e-05 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.730 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:137648 10.f e396/366 10:f39db279:::benchmark_data_smithi169_162694_object137647:head v 396'8687, mlcod=396'8684)", 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:42.087175+0000", 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 123.573556848, 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.030562865000000002, 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087175+0000", 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087175+0000", 2024-03-12T03:51:45.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087176+0000", 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7300000000000005e-07 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087211+0000", 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5135000000000002e-05 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087228+0000", 2024-03-12T03:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6532999999999999e-05 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087236+0000", 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3359999999999995e-06 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088257+0000", 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001021271 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088285+0000", 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7358000000000001e-05 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.117619+0000", 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.029334494999999999 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.117738+0000", 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011886400000000001 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:137650 10.7 e396/366 10:ea5697a5:::benchmark_data_smithi169_162694_object137649:head v 396'8710, mlcod=396'8708)", 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:42.087996+0000", 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 123.572735884, 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.030052637, 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087996+0000", 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087996+0000", 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087997+0000", 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3300000000000001e-07 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088032+0000", 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4813e-05 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088042+0000", 2024-03-12T03:51:45.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0505e-05 2024-03-12T03:51:45.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088050+0000", 2024-03-12T03:51:45.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.464e-06 2024-03-12T03:51:45.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088331+0000", 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028071999999999998 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088438+0000", 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107873 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.117985+0000", 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.029546161000000001 2024-03-12T03:51:45.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.118049+0000", 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4368000000000004e-05 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:137652 10.f e396/366 10:f0a37749:::benchmark_data_smithi169_162694_object137651:head v 396'8688, mlcod=396'8684)", 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:42.091280+0000", 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 123.569452313, 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026800033000000001, 2024-03-12T03:51:45.739 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.091280+0000", 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.091280+0000", 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.091280+0000", 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.01e-07 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.091316+0000", 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5309999999999999e-05 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.091325+0000", 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1749999999999994e-06 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.091330+0000", 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4759999999999997e-06 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.097939+0000", 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0066088589999999999 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.097958+0000", 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9100999999999999e-05 2024-03-12T03:51:45.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.118053+0000", 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020094615999999999 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.118080+0000", 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7095000000000001e-05 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:140402 10.b e397/384 10:dedafb12:::benchmark_data_smithi169_162694_object1407:head v 397'8640, mlcod=397'8638)", 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:43.998036+0000", 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.662696324, 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.034026572999999997, 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:43.998036+0000", 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:43.998036+0000", 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:43.998036+0000", 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.65e-07 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:43.998040+0000", 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7740000000000002e-06 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:43.998045+0000", 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0370000000000002e-06 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:43.998049+0000", 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1579999999999998e-06 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.000488+0000", 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024390480000000001 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.000540+0000", 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1650000000000002e-05 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.031911+0000", 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.031371688000000002 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.032062+0000", 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000150953 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.747 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:140408 10.b e397/384 10:deabafc9:::benchmark_data_smithi169_162694_object1413:head v 397'8641, mlcod=397'8638)", 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:44.003987+0000", 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.656745231, 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.028198746, 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.003987+0000", 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.003987+0000", 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.003988+0000", 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8599999999999997e-07 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.003997+0000", 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3719999999999994e-06 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.004012+0000", 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5160999999999999e-05 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.004024+0000", 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1907e-05 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.017637+0000", 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013612602999999999 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.017663+0000", 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6676e-05 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.032066+0000", 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014403009 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.032186+0000", 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119232 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:142066 10.0 e398/366 10:00965d4a:::benchmark_data_smithi169_162694_object3071:head v 398'8902, mlcod=398'8900)", 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:45.125283+0000", 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 120.53544890000001, 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.028954408000000001, 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.752 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.125283+0000", 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.125283+0000", 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.125284+0000", 2024-03-12T03:51:45.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2500000000000005e-07 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.125289+0000", 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4260000000000002e-06 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.125297+0000", 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5089999999999996e-06 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.125303+0000", 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9499999999999998e-06 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.127506+0000", 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022029829999999999 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.127527+0000", 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0910000000000001e-05 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.154151+0000", 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026624655000000001 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.154238+0000", 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6349999999999998e-05 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:260216 10.7 e422/366 10:eb55b62d:::benchmark_data_smithi169_162694_object121221:head v 422'16421, mlcod=422'16414)", 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:51:04.946729+0000", 2024-03-12T03:51:45.756 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 40.714003550000001, 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.028606256, 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.946729+0000", 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.946729+0000", 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.946729+0000", 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8299999999999997e-07 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.946735+0000", 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3859999999999998e-06 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.946747+0000", 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2150000000000001e-05 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.758 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.946753+0000", 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.438e-06 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.957653+0000", 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010899880000000001 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.957858+0000", 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020488400000000001 2024-03-12T03:51:45.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.975286+0000", 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.017428294 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.975335+0000", 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8640999999999997e-05 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:45.760 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:51:45.761 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:45.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:45.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:45.887 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:45.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:45.910 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:51:45.935 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:51:45.954 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:51:45.974 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:51:45.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:45 smithi173 ceph-mon[33544]: pgmap v893: 166 pgs: 4 activating, 2 peering, 4 active+clean+remapped, 16 active+undersized+degraded, 39 active+undersized, 101 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 466 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:45.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:45 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 73/690 objects degraded (10.580%), 16 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:45.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:45.993 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:51:46.011 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:46.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:46.013 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:51:46.033 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:51:46.034 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:51:46.034 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:51:46.034 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:51:46.034 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:51:46.034 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:51:46.034 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:51:46.034 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:51:46.034 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:51:46.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:46.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:46.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:46.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:45 smithi169 ceph-mon[29344]: pgmap v893: 166 pgs: 4 activating, 2 peering, 4 active+clean+remapped, 16 active+undersized+degraded, 39 active+undersized, 101 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 466 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:46.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:45 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 73/690 objects degraded (10.580%), 16 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:46.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:45 smithi169 ceph-mon[39949]: pgmap v893: 166 pgs: 4 activating, 2 peering, 4 active+clean+remapped, 16 active+undersized+degraded, 39 active+undersized, 101 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 466 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:46.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:45 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 73/690 objects degraded (10.580%), 16 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:46.241 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:46.266 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:46.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:46.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:46.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:46.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:46.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:46.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-12T03:51:46.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:46.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:46.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:46.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:46.648 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:46.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:46.685 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:46.751 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:46.774 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:46.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:51:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-12T03:51:46.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:46.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:46.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:47.001 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:47.026 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:47.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:47.091 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1070263459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:47.091 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4037441404' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:51:47.091 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:46 smithi173 ceph-mon[33544]: 6.19 scrub starts 2024-03-12T03:51:47.091 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:46 smithi173 ceph-mon[33544]: 6.19 scrub ok 2024-03-12T03:51:47.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:46 smithi173 ceph-mon[33544]: 4.c deep-scrub starts 2024-03-12T03:51:47.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:46 smithi173 ceph-mon[33544]: 4.c deep-scrub ok 2024-03-12T03:51:47.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:46 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:47.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:46 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:47.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2186243860' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:51:47.110 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1070263459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:47.110 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4037441404' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:51:47.110 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[29344]: 6.19 scrub starts 2024-03-12T03:51:47.110 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[29344]: 6.19 scrub ok 2024-03-12T03:51:47.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[29344]: 4.c deep-scrub starts 2024-03-12T03:51:47.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[29344]: 4.c deep-scrub ok 2024-03-12T03:51:47.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:47.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:47.111 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2186243860' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:51:47.111 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1070263459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:47.111 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4037441404' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:51:47.111 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[39949]: 6.19 scrub starts 2024-03-12T03:51:47.111 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[39949]: 6.19 scrub ok 2024-03-12T03:51:47.111 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[39949]: 4.c deep-scrub starts 2024-03-12T03:51:47.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[39949]: 4.c deep-scrub ok 2024-03-12T03:51:47.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:51:47.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:47.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2186243860' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:51:47.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:51:47.137 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:51:47.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:47.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:47.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:47.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:47.278 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:47.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:47.380 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:47.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:47.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:47.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:47.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:47.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:47.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:47.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:47.658 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:47.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:47.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:47.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:47.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:47.886 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:47.910 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:47.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:47.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[29344]: pgmap v894: 166 pgs: 4 activating, 2 peering, 4 active+clean+remapped, 16 active+undersized+degraded, 39 active+undersized, 101 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[29344]: 3.14 scrub starts 2024-03-12T03:51:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[29344]: 3.14 scrub ok 2024-03-12T03:51:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4037441404' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:51:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2186243860' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:51:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[29344]: osdmap e442: 8 total, 7 up, 8 in 2024-03-12T03:51:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[29344]: 4.1f scrub starts 2024-03-12T03:51:47.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[29344]: 4.1f scrub ok 2024-03-12T03:51:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[39949]: pgmap v894: 166 pgs: 4 activating, 2 peering, 4 active+clean+remapped, 16 active+undersized+degraded, 39 active+undersized, 101 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[39949]: 3.14 scrub starts 2024-03-12T03:51:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[39949]: 3.14 scrub ok 2024-03-12T03:51:47.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:47.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4037441404' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:51:47.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2186243860' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:51:47.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[39949]: osdmap e442: 8 total, 7 up, 8 in 2024-03-12T03:51:47.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[39949]: 4.1f scrub starts 2024-03-12T03:51:47.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:47 smithi169 ceph-mon[39949]: 4.1f scrub ok 2024-03-12T03:51:48.012 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:48.036 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:48.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:48.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:48.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:48.162 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:48.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:48.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:47 smithi173 ceph-mon[33544]: pgmap v894: 166 pgs: 4 activating, 2 peering, 4 active+clean+remapped, 16 active+undersized+degraded, 39 active+undersized, 101 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:48.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:47 smithi173 ceph-mon[33544]: 3.14 scrub starts 2024-03-12T03:51:48.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:47 smithi173 ceph-mon[33544]: 3.14 scrub ok 2024-03-12T03:51:48.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:47 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:48.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:47 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4037441404' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:51:48.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:47 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2186243860' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:51:48.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:47 smithi173 ceph-mon[33544]: osdmap e442: 8 total, 7 up, 8 in 2024-03-12T03:51:48.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:47 smithi173 ceph-mon[33544]: 4.1f scrub starts 2024-03-12T03:51:48.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:47 smithi173 ceph-mon[33544]: 4.1f scrub ok 2024-03-12T03:51:48.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:48.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:48.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:48.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:48.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:48.413 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:48.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:48.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:48.524 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:51:48.524 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:51:48.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:48.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:48.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:48.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:48.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:48.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:48.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:48.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:48.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:48.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:48.892 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:48.920 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:48.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:49.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:49.047 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:49.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:49.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:51:49.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:49.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:48 smithi169 ceph-mon[29344]: 3.15 scrub starts 2024-03-12T03:51:49.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:48 smithi169 ceph-mon[29344]: 3.15 scrub ok 2024-03-12T03:51:49.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:48 smithi169 ceph-mon[29344]: Health check update: Reduced data availability: 3 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-12T03:51:49.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:48 smithi169 ceph-mon[29344]: osdmap e443: 8 total, 7 up, 8 in 2024-03-12T03:51:49.165 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:48 smithi169 ceph-mon[39949]: 3.15 scrub starts 2024-03-12T03:51:49.165 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:48 smithi169 ceph-mon[39949]: 3.15 scrub ok 2024-03-12T03:51:49.165 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:48 smithi169 ceph-mon[39949]: Health check update: Reduced data availability: 3 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-12T03:51:49.165 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:48 smithi169 ceph-mon[39949]: osdmap e443: 8 total, 7 up, 8 in 2024-03-12T03:51:49.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:49.189 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:49.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:49.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:48 smithi173 ceph-mon[33544]: 3.15 scrub starts 2024-03-12T03:51:49.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:48 smithi173 ceph-mon[33544]: 3.15 scrub ok 2024-03-12T03:51:49.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:48 smithi173 ceph-mon[33544]: Health check update: Reduced data availability: 3 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-12T03:51:49.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:48 smithi173 ceph-mon[33544]: osdmap e443: 8 total, 7 up, 8 in 2024-03-12T03:51:49.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-12T03:51:49.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:49.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:49.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:49.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:49.385 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:49.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:49.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:49.440 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:49.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:49.465 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:49.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:49.565 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:49.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:49.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:49.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:49.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:49.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:49.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:49.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:49.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:49.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:49.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:49.944 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-12T03:51:49.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:49.947 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:49.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:49.948 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-12T03:51:49.948 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-12T03:51:50.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:50.073 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:50.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:50.085 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:hints = 1 2024-03-12T03:51:50.085 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-12T03:51:50.085 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Object prefix: benchmark_data_smithi169_185434 2024-03-12T03:51:50.085 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:51:50.085 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 0 0 0 0 0 0 - 0 2024-03-12T03:51:50.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:49 smithi169 ceph-mon[29344]: pgmap v896: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 39 active+undersized, 107 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 160 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:50.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:49 smithi169 ceph-mon[29344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs peering) 2024-03-12T03:51:50.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:49 smithi169 ceph-mon[29344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:51:50.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:49 smithi169 ceph-mon[39949]: pgmap v896: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 39 active+undersized, 107 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 160 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:50.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:49 smithi169 ceph-mon[39949]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs peering) 2024-03-12T03:51:50.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:49 smithi169 ceph-mon[39949]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:51:50.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:50.211 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:50.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:50.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:49 smithi173 ceph-mon[33544]: pgmap v896: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 39 active+undersized, 107 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 160 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:50.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:49 smithi173 ceph-mon[33544]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs peering) 2024-03-12T03:51:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:49 smithi173 ceph-mon[33544]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:51:50.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:50.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:51:50.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:50.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:50.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:50.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:50.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:50.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:50.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:50.591 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:50.638 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:50.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:50.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:50.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:50.766 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:50.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:50.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:50.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:50.919 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:50.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:51.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:51.084 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:51.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:51.086 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 1 15 3772 3757 234.777 234.812 0.00363069 0.0042458 2024-03-12T03:51:51.187 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:51.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:51.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:51.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:51.366 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:51.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:51.429 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:51:51.429 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:51:51.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:51.508 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:51.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:51.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:51.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:51.643 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:51.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:51.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:51.651 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:51:51.694 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:51:51.742 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:51:51.744 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:51.783 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:51:51.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:51.784 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:51.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:51.824 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:51:51.867 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:51:51.886 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:51.907 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:51:51.908 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:51:51.908 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:51:51.908 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:51:51.908 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:51:51.908 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:51:51.908 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:51:51.908 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:51:51.908 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:51:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:51.928 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:51.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:51.995 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:51 smithi169 ceph-mon[29344]: pgmap v898: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 39 active+undersized, 107 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 277 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:51.995 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:51 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1506309000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:51.996 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:51:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:51:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:51:51.996 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:51 smithi169 ceph-mon[39949]: pgmap v898: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 39 active+undersized, 107 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 277 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:51.996 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:51 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1506309000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:52.028 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:52.066 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:52.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:52.085 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 2 16 7227 7211 225.318 215.875 0.00330543 0.0044288 2024-03-12T03:51:52.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:52.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:52.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:51 smithi173 ceph-mon[33544]: pgmap v898: 166 pgs: 4 active+clean+remapped, 16 active+undersized+degraded, 39 active+undersized, 107 active+clean; 609 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 277 B/s rd, 0 op/s; 73/690 objects degraded (10.580%) 2024-03-12T03:51:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:51 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1506309000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:51:52.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:52.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:52.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:52.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:52.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-12T03:51:52.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:52.481 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:52.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:52.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:52.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:52.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:52.696 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:51:52.696 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:51:52.697 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-03-12T03:51:52.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:51:52.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:52.782 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:51:52.793 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:52.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:52.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:52.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:52.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:52.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:53.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:53.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:53.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:53.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:53.086 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 3 15 10580 10565 220.059 209.625 0.00388621 0.00453694 2024-03-12T03:51:53.171 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:53.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:52 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 883/10365 objects degraded (8.519%), 18 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:53.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:52 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 883/10365 objects degraded (8.519%), 18 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:53.191 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:52 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 883/10365 objects degraded (8.519%), 18 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:53.213 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:53.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:53.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:53.314 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:51:53.315 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:51:53.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:53.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:53.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:53.441 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:51:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:51:53.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:53.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:53.595 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:53.626 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:53.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:53.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:53.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:53.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:53.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:53.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:53.914 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:53.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:53.921 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:54.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:54.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:54.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:54.086 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 4 16 13525 13509 211.041 184 0.00561161 0.00473182 2024-03-12T03:51:54.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:53 smithi169 ceph-mon[29344]: pgmap v899: 166 pgs: 4 active+clean+remapped, 18 active+undersized+degraded, 37 active+undersized, 107 active+clean; 202 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 404 op/s; 883/10365 objects degraded (8.519%) 2024-03-12T03:51:54.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:53 smithi169 ceph-mon[39949]: pgmap v899: 166 pgs: 4 active+clean+remapped, 18 active+undersized+degraded, 37 active+undersized, 107 active+clean; 202 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 404 op/s; 883/10365 objects degraded (8.519%) 2024-03-12T03:51:54.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:54.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:54.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:54.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:54.198 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:53 smithi173 ceph-mon[33544]: pgmap v899: 166 pgs: 4 active+clean+remapped, 18 active+undersized+degraded, 37 active+undersized, 107 active+clean; 202 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 404 op/s; 883/10365 objects degraded (8.519%) 2024-03-12T03:51:54.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:54.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:54.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:54.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:54.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:54.464 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:54.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:54.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:54.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:54.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:54.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:54.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:54.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:54.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:54.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:54.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:54.956 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:51:54.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:55.014 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:55.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:55.051 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:54 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:55.052 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:54 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/133073799' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:55.052 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:54 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:55.086 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 5 16 16267 16251 203.105 171.375 0.00626495 0.00491496 2024-03-12T03:51:55.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:55.152 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:55.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:55.246 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:51:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:51:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:51:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:51:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:136482 9.1 9:8c6774bb:::benchmark_data_smithi169_135229_object136481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-12T03:51:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:17.113369+0000", 2024-03-12T03:51:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 397.91539772900001, 2024-03-12T03:51:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.020501748, 2024-03-12T03:51:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:51:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:51:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136482 2024-03-12T03:51:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.113369+0000", 2024-03-12T03:51:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.113369+0000", 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.113370+0000", 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2419999999999999e-06 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.113412+0000", 2024-03-12T03:51:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1980000000000001e-05 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.113422+0000", 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7079999999999992e-06 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.113430+0000", 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8679999999999999e-06 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.123492+0000", 2024-03-12T03:51:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010062786000000001 2024-03-12T03:51:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.123504+0000", 2024-03-12T03:51:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2065999999999999e-05 2024-03-12T03:51:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.117188+0000", 2024-03-12T03:51:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99368320499999996 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.118596+0000", 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00140876 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.118803+0000", 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020672999999999999 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.131479+0000", 2024-03-12T03:51:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012675906000000001 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.133719+0000", 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022399270000000001 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.133781+0000", 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1969000000000002e-05 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.133871+0000", 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9600999999999998e-05 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:136514 9.1 9:8a90bfc5:::benchmark_data_smithi169_135229_object136513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:17.162294+0000", 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 397.866472714, 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97167573699999998, 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136514 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162294+0000", 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162294+0000", 2024-03-12T03:51:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162294+0000", 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3499999999999996e-07 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162329+0000", 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4301999999999998e-05 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162337+0000", 2024-03-12T03:51:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9920000000000007e-06 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162343+0000", 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.37e-06 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162883+0000", 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054012899999999998 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.162898+0000", 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4354e-05 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.126948+0000", 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96405053799999996 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.127103+0000", 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000155227 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.127270+0000", 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016620299999999999 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.131536+0000", 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042664679999999998 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.133877+0000", 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023412929999999999 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.133911+0000", 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3680999999999998e-05 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.133970+0000", 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8545000000000003e-05 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:136534 9.1 9:8d1b151d:::benchmark_data_smithi169_135229_object136533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-12T03:51:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:17.178457+0000", 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 397.850309778, 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95566303200000002, 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136534 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178457+0000", 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178457+0000", 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178458+0000", 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1310000000000001e-06 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178501+0000", 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3600000000000003e-05 2024-03-12T03:51:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178512+0000", 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0899e-05 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178519+0000", 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0639999999999996e-06 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178572+0000", 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2095999999999997e-05 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.178579+0000", 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.943e-06 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.129362+0000", 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95078218400000003 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.129533+0000", 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017161699999999999 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.129723+0000", 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018984199999999999 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.134020+0000", 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042966050000000002 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.134037+0000", 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7572999999999999e-05 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.134071+0000", 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3800000000000002e-05 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.134120+0000", 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8677999999999997e-05 2024-03-12T03:51:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:136578 9.1 9:8d8890e7:::benchmark_data_smithi169_135229_object136577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:17.219682+0000", 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 397.809084316, 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91693437899999997, 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:51:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136578 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.219682+0000", 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.219682+0000", 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.219683+0000", 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.218e-06 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.219742+0000", 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8369999999999998e-05 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.219753+0000", 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1178999999999999e-05 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.219760+0000", 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9399999999999996e-06 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.220223+0000", 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00046343000000000003 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.220232+0000", 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0440000000000001e-06 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.131601+0000", 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91136816899999995 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.131751+0000", 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015015199999999999 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.131919+0000", 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016819200000000001 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.134165+0000", 2024-03-12T03:51:55.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002246315 2024-03-12T03:51:55.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.136416+0000", 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022507629999999998 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.136516+0000", 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9989999999999996e-05 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.136617+0000", 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100617 2024-03-12T03:51:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:136605 9.1 9:8a5baff4:::benchmark_data_smithi169_135229_object136604:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:17.244697+0000", 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 397.78406924900003, 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.903490817, 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:51:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136605 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.244697+0000", 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.244697+0000", 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.244698+0000", 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4600000000000003e-07 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.244742+0000", 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.401e-05 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.244757+0000", 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4918e-05 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.244765+0000", 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9149999999999996e-06 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.269257+0000", 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024491947999999999 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.269267+0000", 2024-03-12T03:51:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0022e-05 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.140509+0000", 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87124190700000004 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.140640+0000", 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013155 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.140792+0000", 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000152103 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.145679+0000", 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048862430000000002 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.148023+0000", 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023446180000000001 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.148086+0000", 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3010999999999997e-05 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.148188+0000", 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010172599999999999 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:136613 9.1 9:8d6f6f47:::benchmark_data_smithi169_135229_object136612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-12T03:51:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:45:17.254853+0000", 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 397.773913499, 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89725414599999997, 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136613 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.254853+0000", 2024-03-12T03:51:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.254853+0000", 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.254855+0000", 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7400000000000001e-06 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.254882+0000", 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7569999999999999e-05 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.254894+0000", 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1285e-05 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.254901+0000", 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5039999999999996e-06 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.272240+0000", 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.017338536000000002 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:17.272251+0000", 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1406000000000001e-05 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.143149+0000", 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87089840200000002 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.143280+0000", 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013056799999999999 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.143433+0000", 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000153187 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.147989+0000", 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045556110000000002 2024-03-12T03:51:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.151966+0000", 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039771989999999998 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.152018+0000", 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1496000000000003e-05 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:45:18.152107+0000", 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9642000000000007e-05 2024-03-12T03:51:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:29260 10.c 10:3cd68811:::benchmark_data_smithi169_162694_object29259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:24.397827+0000", 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 210.63093988399999, 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.079866952000000005, 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29260 2024-03-12T03:51:55.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.289 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.397827+0000", 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.397827+0000", 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.397828+0000", 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.19e-06 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.397884+0000", 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6209999999999999e-05 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.397895+0000", 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1336e-05 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.397903+0000", 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8620000000000003e-06 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.397973+0000", 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0090000000000001e-05 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.398104+0000", 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013018599999999999 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.398238+0000", 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000134303 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.399408+0000", 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011700199999999999 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.399750+0000", 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034221000000000002 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.477579+0000", 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.077828782999999999 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.477644+0000", 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5351999999999994e-05 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.477694+0000", 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9410000000000003e-05 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:29720 10.c 10:393b4ea3:::benchmark_data_smithi169_162694_object29719:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:24.691175+0000", 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 210.33759107399999, 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.058464862999999999, 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29720 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.691175+0000", 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.691175+0000", 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.691177+0000", 2024-03-12T03:51:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.159e-06 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.691230+0000", 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2967000000000003e-05 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.691243+0000", 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3743e-05 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.691250+0000", 2024-03-12T03:51:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2480000000000001e-06 2024-03-12T03:51:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.691294+0000", 2024-03-12T03:51:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4072000000000001e-05 2024-03-12T03:51:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.691434+0000", 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014041300000000001 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.691616+0000", 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018200599999999999 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.693596+0000", 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019797619999999999 2024-03-12T03:51:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.695695+0000", 2024-03-12T03:51:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020989400000000001 2024-03-12T03:51:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.749535+0000", 2024-03-12T03:51:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.053840387000000003 2024-03-12T03:51:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.749593+0000", 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8006e-05 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:24.749640+0000", 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7160000000000002e-05 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:30333 10.c 10:35a71850:::benchmark_data_smithi169_162694_object30332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:25.010096+0000", 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 210.018670507, 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.063263103000000001, 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 30333 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.010096+0000", 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.010096+0000", 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.010098+0000", 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.525e-06 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.010135+0000", 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7543000000000003e-05 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.010146+0000", 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1104000000000001e-05 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.010154+0000", 2024-03-12T03:51:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3540000000000002e-06 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.010207+0000", 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3934999999999998e-05 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.010340+0000", 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000132714 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-03-12T03:51:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.010473+0000", 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013277100000000001 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.011703+0000", 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012301219999999999 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.011972+0000", 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026940299999999998 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.073251+0000", 2024-03-12T03:51:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.061278759000000002 2024-03-12T03:51:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.073315+0000", 2024-03-12T03:51:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3285000000000001e-05 2024-03-12T03:51:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.073359+0000", 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4588000000000001e-05 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:31190 10.c 10:3edc7300:::benchmark_data_smithi169_162694_object31189:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:25.518411+0000", 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 209.51035568, 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.076445789, 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31190 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.518411+0000", 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.518411+0000", 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.518412+0000", 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.268e-06 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.518445+0000", 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3379e-05 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.518462+0000", 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6362e-05 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.518471+0000", 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6360000000000006e-06 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.518916+0000", 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044462499999999998 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.519053+0000", 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013640500000000001 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.519184+0000", 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013120300000000001 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.521109+0000", 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019254560000000001 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.522356+0000", 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012467509999999999 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.594739+0000", 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.072383092999999996 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.594806+0000", 2024-03-12T03:51:55.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6860999999999996e-05 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.594857+0000", 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.075e-05 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:31205 10.c 10:3f95a44b:::benchmark_data_smithi169_162694_object31204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:25.523862+0000", 2024-03-12T03:51:55.314 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 209.504904692, 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.075814907000000001, 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31205 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.523862+0000", 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.523862+0000", 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.523871+0000", 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7239999999999998e-06 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.523911+0000", 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0111000000000003e-05 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.523927+0000", 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6396e-05 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.523935+0000", 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2630000000000003e-06 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.524006+0000", 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0896999999999996e-05 2024-03-12T03:51:55.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.524180+0000", 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017394 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.524336+0000", 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015559300000000001 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.525904+0000", 2024-03-12T03:51:55.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001568236 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.527612+0000", 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017077220000000001 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.599552+0000", 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.071940237000000004 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.599624+0000", 2024-03-12T03:51:55.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1614999999999998e-05 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.599677+0000", 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3173000000000001e-05 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:31209 10.c 10:3a7bd5be:::benchmark_data_smithi169_162694_object31208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:25.526113+0000", 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 209.502653614, 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.073716553000000004, 2024-03-12T03:51:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31209 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.526113+0000", 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.526113+0000", 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.526114+0000", 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.359e-06 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.526151+0000", 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7020000000000001e-05 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.526167+0000", 2024-03-12T03:51:55.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6152999999999999e-05 2024-03-12T03:51:55.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.526177+0000", 2024-03-12T03:51:55.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0909999999999999e-06 2024-03-12T03:51:55.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.526232+0000", 2024-03-12T03:51:55.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5117000000000001e-05 2024-03-12T03:51:55.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.526383+0000", 2024-03-12T03:51:55.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015153700000000001 2024-03-12T03:51:55.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-03-12T03:51:55.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.526506+0000", 2024-03-12T03:51:55.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012288800000000001 2024-03-12T03:51:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.527728+0000", 2024-03-12T03:51:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012223760000000001 2024-03-12T03:51:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.528085+0000", 2024-03-12T03:51:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035616899999999998 2024-03-12T03:51:55.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.599732+0000", 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.071647257000000006 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.599791+0000", 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9091000000000001e-05 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:25.599829+0000", 2024-03-12T03:51:55.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8495000000000003e-05 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:33414 10.c 10:37ad20aa:::benchmark_data_smithi169_162694_object33413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:26.661244+0000", 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 208.367522947, 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.062107701000000001, 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:55.327 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 33414 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.661244+0000", 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.661244+0000", 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.661245+0000", 2024-03-12T03:51:55.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1790000000000001e-06 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.661282+0000", 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6854999999999999e-05 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.661292+0000", 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0444e-05 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.661300+0000", 2024-03-12T03:51:55.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8509999999999999e-06 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.661355+0000", 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5229999999999998e-05 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.661501+0000", 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014628500000000001 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-03-12T03:51:55.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.661646+0000", 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014428800000000001 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.663708+0000", 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020621049999999998 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.664178+0000", 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047055900000000001 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.723251+0000", 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.059072660999999999 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.723307+0000", 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5766000000000001e-05 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.723351+0000", 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4477999999999997e-05 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.332 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:33606 10.c 10:318ccf1a:::benchmark_data_smithi169_162694_object33605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:26.791954+0000", 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 208.23681251599999, 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.059085909999999998, 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 33606 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.791954+0000", 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.791954+0000", 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.791956+0000", 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.55e-06 2024-03-12T03:51:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.792034+0000", 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8906999999999995e-05 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.792051+0000", 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6799e-05 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.792062+0000", 2024-03-12T03:51:55.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0316000000000001e-05 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.792144+0000", 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2332999999999999e-05 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.792310+0000", 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016630799999999999 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-03-12T03:51:55.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.792443+0000", 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000132521 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.794240+0000", 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001797639 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.794470+0000", 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000229347 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.850789+0000", 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.056319180000000003 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.850992+0000", 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020284200000000001 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:26.851040+0000", 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8167999999999997e-05 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.338 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:75057 10.c 10:344ccc7f:::benchmark_data_smithi169_162694_object75056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:53.882924+0000", 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 181.14584255599999, 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.070096157000000006, 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75057 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.339 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.882924+0000", 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.882924+0000", 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.882925+0000", 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.139e-06 2024-03-12T03:51:55.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.883020+0000", 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4648000000000006e-05 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.883045+0000", 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5718e-05 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.883055+0000", 2024-03-12T03:51:55.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4089999999999992e-06 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.883132+0000", 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7039000000000006e-05 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.883291+0000", 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015941300000000001 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-03-12T03:51:55.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.883443+0000", 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015118 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.885650+0000", 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022072060000000002 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.885835+0000", 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018549699999999999 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.952916+0000", 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.067080441000000005 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.952975+0000", 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9713999999999998e-05 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:53.953020+0000", 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4753000000000003e-05 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.344 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.345 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14529.0:924 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e383)", 2024-03-12T03:51:55.345 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.582091+0000", 2024-03-12T03:51:55.345 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 177.446675285, 2024-03-12T03:51:55.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88087489399999996, 2024-03-12T03:51:55.345 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.345 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:51:55.345 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.345 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14529", 2024-03-12T03:51:55.345 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/951871534", 2024-03-12T03:51:55.345 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 924 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.582091+0000", 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.582091+0000", 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.582092+0000", 2024-03-12T03:51:55.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2440000000000001e-06 2024-03-12T03:51:55.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.582098+0000", 2024-03-12T03:51:55.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.801e-06 2024-03-12T03:51:55.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.582110+0000", 2024-03-12T03:51:55.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2218000000000001e-05 2024-03-12T03:51:55.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.582119+0000", 2024-03-12T03:51:55.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3860000000000007e-06 2024-03-12T03:51:55.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.582203+0000", 2024-03-12T03:51:55.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4081999999999999e-05 2024-03-12T03:51:55.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.582213+0000", 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0036e-05 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.462715+0000", 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88050184300000001 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.462841+0000", 2024-03-12T03:51:55.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000126019 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.462966+0000", 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000125265 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:111757 10.c 10:3f8355c3:::benchmark_data_smithi169_162694_object111756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:21.425072+0000", 2024-03-12T03:51:55.350 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 153.60369465299999, 2024-03-12T03:51:55.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.076759362999999997, 2024-03-12T03:51:55.351 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.351 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.351 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.351 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:55.351 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:55.351 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 111757 2024-03-12T03:51:55.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.351 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425072+0000", 2024-03-12T03:51:55.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425072+0000", 2024-03-12T03:51:55.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425073+0000", 2024-03-12T03:51:55.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1930000000000001e-06 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425121+0000", 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8335999999999999e-05 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425132+0000", 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0987e-05 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425139+0000", 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.37e-06 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425187+0000", 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8696999999999998e-05 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425334+0000", 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014704200000000001 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4", 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.425452+0000", 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117201 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.428082+0000", 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026299449999999999 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.501627+0000", 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.073545447999999999 2024-03-12T03:51:55.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.501733+0000", 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000105919 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.501831+0000", 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8225000000000002e-05 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:111770 10.c 10:3c3dd84c:::benchmark_data_smithi169_162694_object111769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:21.438771+0000", 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 153.589995902, 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.070066109000000001, 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 111770 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.438771+0000", 2024-03-12T03:51:55.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.438771+0000", 2024-03-12T03:51:55.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.438772+0000", 2024-03-12T03:51:55.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8619999999999999e-06 2024-03-12T03:51:55.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.438842+0000", 2024-03-12T03:51:55.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9046000000000003e-05 2024-03-12T03:51:55.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.438854+0000", 2024-03-12T03:51:55.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2761e-05 2024-03-12T03:51:55.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.438862+0000", 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8499999999999994e-06 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.438921+0000", 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8387000000000002e-05 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.439065+0000", 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014404000000000001 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4", 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.439221+0000", 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015669499999999999 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.441558+0000", 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023366860000000001 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.508652+0000", 2024-03-12T03:51:55.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.067094386000000006 2024-03-12T03:51:55.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.508721+0000", 2024-03-12T03:51:55.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9054000000000007e-05 2024-03-12T03:51:55.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:21.508837+0000", 2024-03-12T03:51:55.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011534199999999999 2024-03-12T03:51:55.363 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.363 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.363 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.363 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:114002 10.c 10:3a6f2b65:::benchmark_data_smithi169_162694_object114001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-12T03:51:55.363 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:23.319201+0000", 2024-03-12T03:51:55.363 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 151.70956546100001, 2024-03-12T03:51:55.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.067649508999999997, 2024-03-12T03:51:55.363 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.363 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:51:55.363 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 114002 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.319201+0000", 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.319201+0000", 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.319205+0000", 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4159999999999997e-06 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.319249+0000", 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3318000000000001e-05 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.319262+0000", 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.305e-05 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.319269+0000", 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5660000000000001e-06 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.319330+0000", 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0599e-05 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.319441+0000", 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011079200000000001 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4", 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.319556+0000", 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115117 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.321880+0000", 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023245100000000001 2024-03-12T03:51:55.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.386643+0000", 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.064762301999999994 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.386757+0000", 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114338 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.386851+0000", 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3500999999999998e-05 2024-03-12T03:51:55.368 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24359.0:3966 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94067641424640 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:51:41.845078+0000", 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 13.183688195, 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99126123399999999, 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24359", 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:51:55.369 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 3966 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845078+0000", 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845078+0000", 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845079+0000", 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.44e-07 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845083+0000", 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5899999999999999e-06 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845092+0000", 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3619999999999995e-06 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845099+0000", 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1199999999999996e-06 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845144+0000", 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5303999999999999e-05 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845151+0000", 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7129999999999999e-06 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:42.834504+0000", 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98935289400000004 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:42.834717+0000", 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000212845 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:42.836340+0000", 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001622762 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.373 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.374 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.374 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:51:55.374 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:51:55.374 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:51:55.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:55.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:51:55.376 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:54 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:55.376 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:54 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/133073799' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:55.376 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:54 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:55.376 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:54 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:55.376 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:54 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/133073799' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:55.376 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:54 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:51:55.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:55.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:55.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:55.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:55.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:55.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:55.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:55.688 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:55.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:55.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:51:55.827 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:55.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:51:55.929 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:55.967 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:55.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:55.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:56.069 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:56.086 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 6 15 19052 19037 198.271 174.125 0.00798365 0.00503915 2024-03-12T03:51:56.105 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:56.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:56.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:55 smithi173 ceph-mon[33544]: pgmap v900: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 515 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 3155/25362 objects degraded (12.440%) 2024-03-12T03:51:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:55 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:55 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:51:56.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:55 smithi173 ceph-mon[33544]: osdmap e444: 8 total, 7 up, 8 in 2024-03-12T03:51:56.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:56.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:56.307 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-12T03:51:56.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:56.346 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:55 smithi169 ceph-mon[39949]: pgmap v900: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 515 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 3155/25362 objects degraded (12.440%) 2024-03-12T03:51:56.347 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:55 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:56.347 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:55 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:51:56.347 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:55 smithi169 ceph-mon[39949]: osdmap e444: 8 total, 7 up, 8 in 2024-03-12T03:51:56.347 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:55 smithi169 ceph-mon[29344]: pgmap v900: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 515 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 3155/25362 objects degraded (12.440%) 2024-03-12T03:51:56.347 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:55 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:51:56.347 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:55 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:51:56.347 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:55 smithi169 ceph-mon[29344]: osdmap e444: 8 total, 7 up, 8 in 2024-03-12T03:51:56.390 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:56.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:56.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:56.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:56.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:56.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:56.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:56.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:56.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:56.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:56.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:56.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:56.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:56.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:56.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:56.927 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:56.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:57.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:57.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:57.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:57.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:57.073 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:56 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:51:57.073 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:56 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:51:57.073 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:56 smithi169 ceph-mon[29344]: pgmap v902: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 515 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s; 3155/25362 objects degraded (12.440%) 2024-03-12T03:51:57.074 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:56 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:51:57.074 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:56 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:51:57.074 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:56 smithi169 ceph-mon[39949]: pgmap v902: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 515 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s; 3155/25362 objects degraded (12.440%) 2024-03-12T03:51:57.086 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 7 16 21406 21390 190.954 147.062 0.00489086 0.0052307 2024-03-12T03:51:57.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:57.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:57.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:57.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:56 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:51:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:56 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:51:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:56 smithi173 ceph-mon[33544]: pgmap v902: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 515 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s; 3155/25362 objects degraded (12.440%) 2024-03-12T03:51:57.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:57.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:57.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:57.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:57.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:57.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:57.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:57.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:57.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:57.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:57.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:57.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:57.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:57.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:57.779 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:57.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:57.896 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:57.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:57.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:58.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:58.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:58.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:58.088 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 8 16 23836 23820 186.035 151.875 0.00704196 0.00536821 2024-03-12T03:51:58.096 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:51:58.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:58.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:58.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:58.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:57 smithi173 ceph-mon[33544]: osdmap e445: 8 total, 7 up, 8 in 2024-03-12T03:51:58.268 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:51:58.269 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:57 smithi169 ceph-mon[29344]: osdmap e445: 8 total, 7 up, 8 in 2024-03-12T03:51:58.269 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:57 smithi169 ceph-mon[39949]: osdmap e445: 8 total, 7 up, 8 in 2024-03-12T03:51:58.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:58.282 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:51:58.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:58.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:58.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:58.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:58.452 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:58.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:58.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:58.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:58.593 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:58.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:58.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:58.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:51:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:58.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:58.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:51:58.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:58.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:58.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:58.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:58.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:59.003 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:59.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:59.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:59.088 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 9 16 26363 26347 182.912 157.938 0.00249892 0.00546146 2024-03-12T03:51:59.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:59.156 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:59.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:59.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:58 smithi173 ceph-mon[33544]: pgmap v904: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s; 4914/55272 objects degraded (8.891%) 2024-03-12T03:51:59.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:58 smithi173 ceph-mon[33544]: from='client.25594 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:51:59.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:51:58 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 3155/25362 objects degraded (12.440%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:59.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:51:59.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:51:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:59.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:59.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:51:59.299 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:58 smithi169 ceph-mon[29344]: pgmap v904: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s; 4914/55272 objects degraded (8.891%) 2024-03-12T03:51:59.299 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:58 smithi169 ceph-mon[29344]: from='client.25594 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:51:59.299 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:51:58 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 3155/25362 objects degraded (12.440%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:59.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:58 smithi169 ceph-mon[39949]: pgmap v904: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s; 4914/55272 objects degraded (8.891%) 2024-03-12T03:51:59.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:58 smithi169 ceph-mon[39949]: from='client.25594 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:51:59.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:51:58 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 3155/25362 objects degraded (12.440%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:51:59.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:51:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:59.441 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:59.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:51:59.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:51:59.586 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:51:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:59.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:51:59.688 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:51:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:59.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:59.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:51:59.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:59.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:51:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:51:59.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:51:59.957 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":904,"stamp":"2024-03-12T03:51:58.205375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1192985211,"num_objects":18424,"num_object_clones":0,"num_object_copies":55272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4914,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18424,"num_whiteouts":0,"num_read":2525,"num_read_kb":2249,"num_write":19116,"num_write_kb":1164571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":938764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21439,"ondisk_log_size":21439,"up":439,"acting":443,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":506,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11744304,"kb_used_data":6248528,"kb_used_omap":24,"kb_used_meta":5494951,"kb_avail":738216912,"statfs":{"total":767960285184,"available":755934117888,"internally_reserved":0,"allocated":6398492672,"data_stored":6391494662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24644,"internal_metadata":5626830780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,4,6,4,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1192361984,"num_objects":18194,"num_object_clones":0,"num_object_copies":54582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4841,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18194,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":18194,"num_write_kb":1164416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004281"},"pg_stats":[{"pgid":"11.3","version":"444'1200","reported_seq":1221,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.506247+0000","last_change":"2024-03-12T03:51:40.889910+0000","last_active":"2024-03-12T03:51:56.506247+0000","last_peered":"2024-03-12T03:51:56.506247+0000","last_clean":"2024-03-12T03:51:56.506247+0000","last_became_active":"2024-03-12T03:51:40.889550+0000","last_became_peered":"2024-03-12T03:51:40.889550+0000","last_unstale":"2024-03-12T03:51:56.506247+0000","last_undegraded":"2024-03-12T03:51:56.506247+0000","last_fullsized":"2024-03-12T03:51:56.506247+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":1200,"log_dups_size":0,"ondisk_log_size":1200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:00.106245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78643200,"num_objects":1200,"num_object_clones":0,"num_object_copies":3600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1200,"num_write_kb":76800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1146,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.950506+0000","last_change":"2024-03-12T03:50:53.286652+0000","last_active":"2024-03-12T03:51:54.950506+0000","last_peered":"2024-03-12T03:51:54.950506+0000","last_clean":"2024-03-12T03:51:54.950506+0000","last_became_active":"2024-03-12T03:46:25.289005+0000","last_became_peered":"2024-03-12T03:46:25.289005+0000","last_unstale":"2024-03-12T03:51:54.950506+0000","last_undegraded":"2024-03-12T03:51:54.950506+0000","last_fullsized":"2024-03-12T03:51:54.950506+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:53.286515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.286515+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:21.938213+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"344'25","reported_seq":686,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494365+0000","last_change":"2024-03-12T03:51:46.582688+0000","last_active":"2024-03-12T03:51:48.494365+0000","last_peered":"2024-03-12T03:51:48.494365+0000","last_clean":"2024-03-12T03:51:48.494365+0000","last_became_active":"2024-03-12T03:51:42.838917+0000","last_became_peered":"2024-03-12T03:51:42.838917+0000","last_unstale":"2024-03-12T03:51:48.494365+0000","last_undegraded":"2024-03-12T03:51:48.494365+0000","last_fullsized":"2024-03-12T03:51:48.494365+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'25","last_scrub_stamp":"2024-03-12T03:51:46.582596+0000","last_deep_scrub":"344'25","last_deep_scrub_stamp":"2024-03-12T03:51:46.582596+0000","last_clean_scrub_stamp":"2024-03-12T03:51:46.582596+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:00.206615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":588,"reported_epoch":443,"state":"active+undersized","last_fresh":"2024-03-12T03:51:48.493937+0000","last_change":"2024-03-12T03:48:58.454102+0000","last_active":"2024-03-12T03:51:48.493937+0000","last_peered":"2024-03-12T03:51:48.493937+0000","last_clean":"2024-03-12T03:48:53.448575+0000","last_became_active":"2024-03-12T03:48:58.454102+0000","last_became_peered":"2024-03-12T03:48:58.454102+0000","last_unstale":"2024-03-12T03:51:48.493937+0000","last_undegraded":"2024-03-12T03:51:48.493937+0000","last_fullsized":"2024-03-12T03:48:58.443116+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:07.363439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:07.363439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:51.579568+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":716,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.955634+0000","last_change":"2024-03-12T03:51:42.650674+0000","last_active":"2024-03-12T03:51:54.955634+0000","last_peered":"2024-03-12T03:51:54.955634+0000","last_clean":"2024-03-12T03:51:54.955634+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:51:54.955634+0000","last_undegraded":"2024-03-12T03:51:54.955634+0000","last_fullsized":"2024-03-12T03:51:54.955634+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:51:42.650584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:42.650584+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:11.924689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":302,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.964351+0000","last_change":"2024-03-12T03:51:42.833356+0000","last_active":"2024-03-12T03:51:56.964351+0000","last_peered":"2024-03-12T03:51:56.964351+0000","last_clean":"2024-03-12T03:51:56.964351+0000","last_became_active":"2024-03-12T03:51:42.832276+0000","last_became_peered":"2024-03-12T03:51:42.832276+0000","last_unstale":"2024-03-12T03:51:56.964351+0000","last_undegraded":"2024-03-12T03:51:56.964351+0000","last_fullsized":"2024-03-12T03:51:56.964351+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:54.871895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.871895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"444'1322","reported_seq":1341,"reported_epoch":444,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:56.893412+0000","last_change":"2024-03-12T03:51:50.104059+0000","last_active":"2024-03-12T03:51:56.893412+0000","last_peered":"2024-03-12T03:51:56.893412+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:51:40.861166+0000","last_became_peered":"2024-03-12T03:51:40.861166+0000","last_unstale":"2024-03-12T03:51:56.893412+0000","last_undegraded":"2024-03-12T03:51:50.099763+0000","last_fullsized":"2024-03-12T03:51:40.821017+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":1322,"log_dups_size":0,"ondisk_log_size":1322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:42.828610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86638592,"num_objects":1322,"num_object_clones":0,"num_object_copies":3966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1322,"num_write_kb":84608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"269'31","reported_seq":560,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.955483+0000","last_change":"2024-03-12T03:51:42.832652+0000","last_active":"2024-03-12T03:51:54.955483+0000","last_peered":"2024-03-12T03:51:54.955483+0000","last_clean":"2024-03-12T03:51:54.955483+0000","last_became_active":"2024-03-12T03:51:42.832275+0000","last_became_peered":"2024-03-12T03:51:42.832275+0000","last_unstale":"2024-03-12T03:51:54.955483+0000","last_undegraded":"2024-03-12T03:51:54.955483+0000","last_fullsized":"2024-03-12T03:51:54.955483+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:50:01.871412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:01.871412+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":291,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.960860+0000","last_change":"2024-03-12T03:50:55.920975+0000","last_active":"2024-03-12T03:51:54.960860+0000","last_peered":"2024-03-12T03:51:54.960860+0000","last_clean":"2024-03-12T03:51:54.960860+0000","last_became_active":"2024-03-12T03:45:18.108377+0000","last_became_peered":"2024-03-12T03:45:18.108377+0000","last_unstale":"2024-03-12T03:51:54.960860+0000","last_undegraded":"2024-03-12T03:51:54.960860+0000","last_fullsized":"2024-03-12T03:51:54.960860+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:55.920704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:50:55.920704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:09.389962+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":816,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-03-12T03:51:56.963156+0000","last_change":"2024-03-12T03:48:58.457284+0000","last_active":"2024-03-12T03:51:56.963156+0000","last_peered":"2024-03-12T03:51:56.963156+0000","last_clean":"2024-03-12T03:48:53.449180+0000","last_became_active":"2024-03-12T03:48:58.457284+0000","last_became_peered":"2024-03-12T03:48:58.457284+0000","last_unstale":"2024-03-12T03:51:56.963156+0000","last_undegraded":"2024-03-12T03:51:56.963156+0000","last_fullsized":"2024-03-12T03:48:58.441935+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.054936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.054936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":573,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.955501+0000","last_change":"2024-03-12T03:51:37.847956+0000","last_active":"2024-03-12T03:51:54.955501+0000","last_peered":"2024-03-12T03:51:54.955501+0000","last_clean":"2024-03-12T03:51:54.955501+0000","last_became_active":"2024-03-12T03:46:25.293293+0000","last_became_peered":"2024-03-12T03:46:25.293293+0000","last_unstale":"2024-03-12T03:51:54.955501+0000","last_undegraded":"2024-03-12T03:51:54.955501+0000","last_fullsized":"2024-03-12T03:51:54.955501+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:37.847806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:37.847806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:47.499702+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"388'129","reported_seq":987,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.956180+0000","last_change":"2024-03-12T03:50:56.164839+0000","last_active":"2024-03-12T03:51:54.956180+0000","last_peered":"2024-03-12T03:51:54.956180+0000","last_clean":"2024-03-12T03:51:54.956180+0000","last_became_active":"2024-03-12T03:46:25.297913+0000","last_became_peered":"2024-03-12T03:46:25.297913+0000","last_unstale":"2024-03-12T03:51:54.956180+0000","last_undegraded":"2024-03-12T03:51:54.956180+0000","last_fullsized":"2024-03-12T03:51:54.956180+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"388'129","last_scrub_stamp":"2024-03-12T03:50:56.164705+0000","last_deep_scrub":"388'129","last_deep_scrub_stamp":"2024-03-12T03:49:17.168603+0000","last_clean_scrub_stamp":"2024-03-12T03:50:56.164705+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:06.277420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"443'738","reported_seq":755,"reported_epoch":443,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:53.563558+0000","last_change":"2024-03-12T03:51:50.105301+0000","last_active":"2024-03-12T03:51:53.563558+0000","last_peered":"2024-03-12T03:51:53.563558+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:51:40.888545+0000","last_became_peered":"2024-03-12T03:51:40.888545+0000","last_unstale":"2024-03-12T03:51:53.563558+0000","last_undegraded":"2024-03-12T03:51:50.099365+0000","last_fullsized":"2024-03-12T03:51:40.858217+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":738,"log_dups_size":0,"ondisk_log_size":738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:03.967469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48365568,"num_objects":738,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":738,"num_write_kb":47232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":977,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.964797+0000","last_change":"2024-03-12T03:51:30.025307+0000","last_active":"2024-03-12T03:51:56.964797+0000","last_peered":"2024-03-12T03:51:56.964797+0000","last_clean":"2024-03-12T03:51:56.964797+0000","last_became_active":"2024-03-12T03:50:58.842635+0000","last_became_peered":"2024-03-12T03:50:58.842635+0000","last_unstale":"2024-03-12T03:51:56.964797+0000","last_undegraded":"2024-03-12T03:51:56.964797+0000","last_fullsized":"2024-03-12T03:51:56.964797+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:30.025220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:51:30.025220+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:35.279704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"269'31","reported_seq":514,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.967858+0000","last_change":"2024-03-12T03:51:42.834522+0000","last_active":"2024-03-12T03:51:56.967858+0000","last_peered":"2024-03-12T03:51:56.967858+0000","last_clean":"2024-03-12T03:51:56.967858+0000","last_became_active":"2024-03-12T03:51:42.834094+0000","last_became_peered":"2024-03-12T03:51:42.834094+0000","last_unstale":"2024-03-12T03:51:56.967858+0000","last_undegraded":"2024-03-12T03:51:56.967858+0000","last_fullsized":"2024-03-12T03:51:56.967858+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:49:09.755469+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:49:09.755469+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":840,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.961102+0000","last_change":"2024-03-12T03:50:47.939438+0000","last_active":"2024-03-12T03:51:54.961102+0000","last_peered":"2024-03-12T03:51:54.961102+0000","last_clean":"2024-03-12T03:51:54.961102+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:51:54.961102+0000","last_undegraded":"2024-03-12T03:51:54.961102+0000","last_fullsized":"2024-03-12T03:51:54.961102+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:47.939203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_clean_scrub_stamp":"2024-03-12T03:50:47.939203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:07.451477+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":818,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.958474+0000","last_change":"2024-03-12T03:50:51.214639+0000","last_active":"2024-03-12T03:51:54.958474+0000","last_peered":"2024-03-12T03:51:54.958474+0000","last_clean":"2024-03-12T03:51:54.958474+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:51:54.958474+0000","last_undegraded":"2024-03-12T03:51:54.958474+0000","last_fullsized":"2024-03-12T03:51:54.958474+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:51.214503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:51.214503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:06.200621+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":611,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494243+0000","last_change":"2024-03-12T03:51:36.531246+0000","last_active":"2024-03-12T03:51:48.494243+0000","last_peered":"2024-03-12T03:51:48.494243+0000","last_clean":"2024-03-12T03:51:48.494243+0000","last_became_active":"2024-03-12T03:46:25.300814+0000","last_became_peered":"2024-03-12T03:46:25.300814+0000","last_unstale":"2024-03-12T03:51:48.494243+0000","last_undegraded":"2024-03-12T03:51:48.494243+0000","last_fullsized":"2024-03-12T03:51:48.494243+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:36.531169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:36.531169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:42.038087+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"445'1377","reported_seq":1401,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:57.829061+0000","last_change":"2024-03-12T03:51:40.860931+0000","last_active":"2024-03-12T03:51:57.829061+0000","last_peered":"2024-03-12T03:51:57.829061+0000","last_clean":"2024-03-12T03:51:57.829061+0000","last_became_active":"2024-03-12T03:51:40.860652+0000","last_became_peered":"2024-03-12T03:51:40.860652+0000","last_unstale":"2024-03-12T03:51:57.829061+0000","last_undegraded":"2024-03-12T03:51:57.829061+0000","last_fullsized":"2024-03-12T03:51:57.829061+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":1377,"log_dups_size":0,"ondisk_log_size":1377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:54.987451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90243072,"num_objects":1377,"num_object_clones":0,"num_object_copies":4131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1377,"num_write_kb":88128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":960,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.956661+0000","last_change":"2024-03-12T03:50:34.205833+0000","last_active":"2024-03-12T03:51:54.956661+0000","last_peered":"2024-03-12T03:51:54.956661+0000","last_clean":"2024-03-12T03:51:54.956661+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:51:54.956661+0000","last_undegraded":"2024-03-12T03:51:54.956661+0000","last_fullsized":"2024-03-12T03:51:54.956661+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:34.205711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:34.205711+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:00.651194+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":686,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.965104+0000","last_change":"2024-03-12T03:50:46.613513+0000","last_active":"2024-03-12T03:51:56.965104+0000","last_peered":"2024-03-12T03:51:56.965104+0000","last_clean":"2024-03-12T03:51:56.965104+0000","last_became_active":"2024-03-12T03:46:25.287373+0000","last_became_peered":"2024-03-12T03:46:25.287373+0000","last_unstale":"2024-03-12T03:51:56.965104+0000","last_undegraded":"2024-03-12T03:51:56.965104+0000","last_fullsized":"2024-03-12T03:51:56.965104+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:50:46.613361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:46.613361+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":202,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.955778+0000","last_change":"2024-03-12T03:51:42.832487+0000","last_active":"2024-03-12T03:51:54.955778+0000","last_peered":"2024-03-12T03:51:54.955778+0000","last_clean":"2024-03-12T03:48:53.446320+0000","last_became_active":"2024-03-12T03:51:42.832487+0000","last_became_peered":"2024-03-12T03:51:42.832487+0000","last_unstale":"2024-03-12T03:51:54.955778+0000","last_undegraded":"2024-03-12T03:51:54.955778+0000","last_fullsized":"2024-03-12T03:51:42.823170+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:16.876204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:16.876204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026936+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"441'10","reported_seq":43,"reported_epoch":444,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:54.957685+0000","last_change":"2024-03-12T03:51:42.832218+0000","last_active":"2024-03-12T03:51:54.957685+0000","last_peered":"2024-03-12T03:51:54.957685+0000","last_clean":"2024-03-12T03:33:28.357470+0000","last_became_active":"2024-03-12T03:51:42.832218+0000","last_became_peered":"2024-03-12T03:51:42.832218+0000","last_unstale":"2024-03-12T03:51:54.957685+0000","last_undegraded":"2024-03-12T03:51:42.824811+0000","last_fullsized":"2024-03-12T03:51:42.824692+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:47:53.964216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:53.964216+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":801,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.956954+0000","last_change":"2024-03-12T03:48:58.448113+0000","last_active":"2024-03-12T03:51:54.956954+0000","last_peered":"2024-03-12T03:51:54.956954+0000","last_clean":"2024-03-12T03:48:53.472090+0000","last_became_active":"2024-03-12T03:48:58.448113+0000","last_became_peered":"2024-03-12T03:48:58.448113+0000","last_unstale":"2024-03-12T03:51:54.956954+0000","last_undegraded":"2024-03-12T03:51:54.956954+0000","last_fullsized":"2024-03-12T03:48:58.441990+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:04.770746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:04.770746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026924+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"444'1264","reported_seq":1286,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.482952+0000","last_change":"2024-03-12T03:51:40.855404+0000","last_active":"2024-03-12T03:51:56.482952+0000","last_peered":"2024-03-12T03:51:56.482952+0000","last_clean":"2024-03-12T03:51:56.482952+0000","last_became_active":"2024-03-12T03:51:40.855216+0000","last_became_peered":"2024-03-12T03:51:40.855216+0000","last_unstale":"2024-03-12T03:51:56.482952+0000","last_undegraded":"2024-03-12T03:51:56.482952+0000","last_fullsized":"2024-03-12T03:51:56.482952+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":1264,"log_dups_size":0,"ondisk_log_size":1264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:41.033467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82837504,"num_objects":1264,"num_object_clones":0,"num_object_copies":3792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":830,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.961016+0000","last_change":"2024-03-12T03:51:30.617409+0000","last_active":"2024-03-12T03:51:54.961016+0000","last_peered":"2024-03-12T03:51:54.961016+0000","last_clean":"2024-03-12T03:51:54.961016+0000","last_became_active":"2024-03-12T03:46:25.286985+0000","last_became_peered":"2024-03-12T03:46:25.286985+0000","last_unstale":"2024-03-12T03:51:54.961016+0000","last_undegraded":"2024-03-12T03:51:54.961016+0000","last_fullsized":"2024-03-12T03:51:54.961016+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:30.617326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:30.617326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:57.949634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"269'42","reported_seq":676,"reported_epoch":443,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:48.493540+0000","last_change":"2024-03-12T03:48:58.463377+0000","last_active":"2024-03-12T03:51:48.493540+0000","last_peered":"2024-03-12T03:51:48.493540+0000","last_clean":"2024-03-12T03:48:53.447338+0000","last_became_active":"2024-03-12T03:48:58.463377+0000","last_became_peered":"2024-03-12T03:48:58.463377+0000","last_unstale":"2024-03-12T03:51:48.493540+0000","last_undegraded":"2024-03-12T03:48:58.440183+0000","last_fullsized":"2024-03-12T03:48:58.440082+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:47:13.347128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.347128+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:51.579616+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":592,"reported_epoch":443,"state":"active+undersized","last_fresh":"2024-03-12T03:51:48.494678+0000","last_change":"2024-03-12T03:48:58.464970+0000","last_active":"2024-03-12T03:51:48.494678+0000","last_peered":"2024-03-12T03:51:48.494678+0000","last_clean":"2024-03-12T03:48:53.448054+0000","last_became_active":"2024-03-12T03:48:58.464970+0000","last_became_peered":"2024-03-12T03:48:58.464970+0000","last_unstale":"2024-03-12T03:51:48.494678+0000","last_undegraded":"2024-03-12T03:51:48.494678+0000","last_fullsized":"2024-03-12T03:48:58.441302+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:05.639950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:05.639950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:51.579644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":601,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.493809+0000","last_change":"2024-03-12T03:51:42.841507+0000","last_active":"2024-03-12T03:51:48.493809+0000","last_peered":"2024-03-12T03:51:48.493809+0000","last_clean":"2024-03-12T03:51:48.493809+0000","last_became_active":"2024-03-12T03:51:42.840902+0000","last_became_peered":"2024-03-12T03:51:42.840902+0000","last_unstale":"2024-03-12T03:51:48.493809+0000","last_undegraded":"2024-03-12T03:51:48.493809+0000","last_fullsized":"2024-03-12T03:51:48.493809+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:14.836179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_clean_scrub_stamp":"2024-03-12T03:49:14.836179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:52.569307+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":975,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.962276+0000","last_change":"2024-03-12T03:50:48.924955+0000","last_active":"2024-03-12T03:51:54.962276+0000","last_peered":"2024-03-12T03:51:54.962276+0000","last_clean":"2024-03-12T03:51:54.962276+0000","last_became_active":"2024-03-12T03:44:19.373715+0000","last_became_peered":"2024-03-12T03:44:19.373715+0000","last_unstale":"2024-03-12T03:51:54.962276+0000","last_undegraded":"2024-03-12T03:51:54.962276+0000","last_fullsized":"2024-03-12T03:51:54.962276+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:48.924831+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:15.895638+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.924831+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:02.935590+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"444'1265","reported_seq":1287,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.360608+0000","last_change":"2024-03-12T03:51:40.862482+0000","last_active":"2024-03-12T03:51:56.360608+0000","last_peered":"2024-03-12T03:51:56.360608+0000","last_clean":"2024-03-12T03:51:56.360608+0000","last_became_active":"2024-03-12T03:51:40.861867+0000","last_became_peered":"2024-03-12T03:51:40.861867+0000","last_unstale":"2024-03-12T03:51:56.360608+0000","last_undegraded":"2024-03-12T03:51:56.360608+0000","last_fullsized":"2024-03-12T03:51:56.360608+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":1265,"log_dups_size":0,"ondisk_log_size":1265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:04.080777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82903040,"num_objects":1265,"num_object_clones":0,"num_object_copies":3795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1265,"num_write_kb":80960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"332'12","reported_seq":820,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.493704+0000","last_change":"2024-03-12T03:50:32.360621+0000","last_active":"2024-03-12T03:51:48.493704+0000","last_peered":"2024-03-12T03:51:48.493704+0000","last_clean":"2024-03-12T03:51:48.493704+0000","last_became_active":"2024-03-12T03:46:25.288674+0000","last_became_peered":"2024-03-12T03:46:25.288674+0000","last_unstale":"2024-03-12T03:51:48.493704+0000","last_undegraded":"2024-03-12T03:51:48.493704+0000","last_fullsized":"2024-03-12T03:51:48.493704+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'12","last_scrub_stamp":"2024-03-12T03:50:32.360502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:32.360502+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:52.569379+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"269'42","reported_seq":525,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.956061+0000","last_change":"2024-03-12T03:51:42.832600+0000","last_active":"2024-03-12T03:51:54.956061+0000","last_peered":"2024-03-12T03:51:54.956061+0000","last_clean":"2024-03-12T03:51:54.956061+0000","last_became_active":"2024-03-12T03:51:42.832229+0000","last_became_peered":"2024-03-12T03:51:42.832229+0000","last_unstale":"2024-03-12T03:51:54.956061+0000","last_undegraded":"2024-03-12T03:51:54.956061+0000","last_fullsized":"2024-03-12T03:51:54.956061+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:50:35.839859+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.839859+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.027084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"320'2","reported_seq":687,"reported_epoch":445,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:56.965167+0000","last_change":"2024-03-12T03:48:58.457502+0000","last_active":"2024-03-12T03:51:56.965167+0000","last_peered":"2024-03-12T03:51:56.965167+0000","last_clean":"2024-03-12T03:45:56.084290+0000","last_became_active":"2024-03-12T03:48:58.457502+0000","last_became_peered":"2024-03-12T03:48:58.457502+0000","last_unstale":"2024-03-12T03:51:56.965167+0000","last_undegraded":"2024-03-12T03:48:58.439950+0000","last_fullsized":"2024-03-12T03:48:58.439792+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'2","last_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_deep_scrub":"320'2","last_deep_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_clean_scrub_stamp":"2024-03-12T03:45:56.089247+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":836,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951215+0000","last_change":"2024-03-12T03:51:31.400543+0000","last_active":"2024-03-12T03:51:54.951215+0000","last_peered":"2024-03-12T03:51:54.951215+0000","last_clean":"2024-03-12T03:51:54.951215+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:51:54.951215+0000","last_undegraded":"2024-03-12T03:51:54.951215+0000","last_fullsized":"2024-03-12T03:51:54.951215+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:31.400418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:49:54.471078+0000","last_clean_scrub_stamp":"2024-03-12T03:51:31.400418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:59.747204+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":793,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.493915+0000","last_change":"2024-03-12T03:51:37.519315+0000","last_active":"2024-03-12T03:51:48.493915+0000","last_peered":"2024-03-12T03:51:48.493915+0000","last_clean":"2024-03-12T03:51:48.493915+0000","last_became_active":"2024-03-12T03:51:07.938356+0000","last_became_peered":"2024-03-12T03:51:07.938356+0000","last_unstale":"2024-03-12T03:51:48.493915+0000","last_undegraded":"2024-03-12T03:51:48.493915+0000","last_fullsized":"2024-03-12T03:51:48.493915+0000","mapping_epoch":330,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:37.519232+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_clean_scrub_stamp":"2024-03-12T03:51:37.519232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:02.132334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"444'1307","reported_seq":1329,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.908244+0000","last_change":"2024-03-12T03:51:40.889015+0000","last_active":"2024-03-12T03:51:56.908244+0000","last_peered":"2024-03-12T03:51:56.908244+0000","last_clean":"2024-03-12T03:51:56.908244+0000","last_became_active":"2024-03-12T03:51:40.888684+0000","last_became_peered":"2024-03-12T03:51:40.888684+0000","last_unstale":"2024-03-12T03:51:56.908244+0000","last_undegraded":"2024-03-12T03:51:56.908244+0000","last_fullsized":"2024-03-12T03:51:56.908244+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:00.047106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85655552,"num_objects":1307,"num_object_clones":0,"num_object_copies":3921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1307,"num_write_kb":83648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"385'9","reported_seq":188,"reported_epoch":444,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:54.956683+0000","last_change":"2024-03-12T03:48:58.451360+0000","last_active":"2024-03-12T03:51:54.956683+0000","last_peered":"2024-03-12T03:51:54.956683+0000","last_clean":"2024-03-12T03:33:28.360558+0000","last_became_active":"2024-03-12T03:48:58.451360+0000","last_became_peered":"2024-03-12T03:48:58.451360+0000","last_unstale":"2024-03-12T03:51:54.956683+0000","last_undegraded":"2024-03-12T03:48:58.442329+0000","last_fullsized":"2024-03-12T03:48:58.441949+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:48:14.138048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:14.138048+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"269'26","reported_seq":875,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.963966+0000","last_change":"2024-03-12T03:50:35.597476+0000","last_active":"2024-03-12T03:51:56.963966+0000","last_peered":"2024-03-12T03:51:56.963966+0000","last_clean":"2024-03-12T03:51:56.963966+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:51:56.963966+0000","last_undegraded":"2024-03-12T03:51:56.963966+0000","last_fullsized":"2024-03-12T03:51:56.963966+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:50:35.597225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.597225+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":848,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.957159+0000","last_change":"2024-03-12T03:50:48.222807+0000","last_active":"2024-03-12T03:51:54.957159+0000","last_peered":"2024-03-12T03:51:54.957159+0000","last_clean":"2024-03-12T03:51:54.957159+0000","last_became_active":"2024-03-12T03:46:25.294348+0000","last_became_peered":"2024-03-12T03:46:25.294348+0000","last_unstale":"2024-03-12T03:51:54.957159+0000","last_undegraded":"2024-03-12T03:51:54.957159+0000","last_fullsized":"2024-03-12T03:51:54.957159+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.222697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.222697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:14.336537+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":703,"reported_epoch":443,"state":"active+undersized","last_fresh":"2024-03-12T03:51:48.493346+0000","last_change":"2024-03-12T03:48:58.455443+0000","last_active":"2024-03-12T03:51:48.493346+0000","last_peered":"2024-03-12T03:51:48.493346+0000","last_clean":"2024-03-12T03:43:13.330724+0000","last_became_active":"2024-03-12T03:48:58.455443+0000","last_became_peered":"2024-03-12T03:48:58.455443+0000","last_unstale":"2024-03-12T03:51:48.493346+0000","last_undegraded":"2024-03-12T03:51:48.493346+0000","last_fullsized":"2024-03-12T03:48:58.439578+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:13.892760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.892760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:51.579583+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":570,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494937+0000","last_change":"2024-03-12T03:50:54.587747+0000","last_active":"2024-03-12T03:51:48.494937+0000","last_peered":"2024-03-12T03:51:48.494937+0000","last_clean":"2024-03-12T03:51:48.494937+0000","last_became_active":"2024-03-12T03:46:25.287360+0000","last_became_peered":"2024-03-12T03:46:25.287360+0000","last_unstale":"2024-03-12T03:51:48.494937+0000","last_undegraded":"2024-03-12T03:51:48.494937+0000","last_fullsized":"2024-03-12T03:51:48.494937+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:54.587604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:54.587604+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:20.215534+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"445'1456","reported_seq":1480,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:57.820682+0000","last_change":"2024-03-12T03:51:40.859739+0000","last_active":"2024-03-12T03:51:57.820682+0000","last_peered":"2024-03-12T03:51:57.820682+0000","last_clean":"2024-03-12T03:51:57.820682+0000","last_became_active":"2024-03-12T03:51:40.859434+0000","last_became_peered":"2024-03-12T03:51:40.859434+0000","last_unstale":"2024-03-12T03:51:57.820682+0000","last_undegraded":"2024-03-12T03:51:57.820682+0000","last_fullsized":"2024-03-12T03:51:57.820682+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":1456,"log_dups_size":0,"ondisk_log_size":1456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:47.982202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95420416,"num_objects":1456,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1456,"num_write_kb":93184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":130,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.957418+0000","last_change":"2024-03-12T03:48:58.449166+0000","last_active":"2024-03-12T03:51:54.957418+0000","last_peered":"2024-03-12T03:51:54.957418+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.449166+0000","last_became_peered":"2024-03-12T03:48:58.449166+0000","last_unstale":"2024-03-12T03:51:54.957418+0000","last_undegraded":"2024-03-12T03:51:54.957418+0000","last_fullsized":"2024-03-12T03:48:58.441301+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.179416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.179416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"269'35","reported_seq":628,"reported_epoch":444,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:54.956976+0000","last_change":"2024-03-12T03:48:58.454600+0000","last_active":"2024-03-12T03:51:54.956976+0000","last_peered":"2024-03-12T03:51:54.956976+0000","last_clean":"2024-03-12T03:43:28.848207+0000","last_became_active":"2024-03-12T03:48:58.454600+0000","last_became_peered":"2024-03-12T03:48:58.454600+0000","last_unstale":"2024-03-12T03:51:54.956976+0000","last_undegraded":"2024-03-12T03:48:58.443785+0000","last_fullsized":"2024-03-12T03:48:58.443677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-03-12T03:45:59.055318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.055318+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":47,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":849,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.963063+0000","last_change":"2024-03-12T03:49:51.465759+0000","last_active":"2024-03-12T03:51:56.963063+0000","last_peered":"2024-03-12T03:51:56.963063+0000","last_clean":"2024-03-12T03:51:56.963063+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:51:56.963063+0000","last_undegraded":"2024-03-12T03:51:56.963063+0000","last_fullsized":"2024-03-12T03:51:56.963063+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:51.465633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:49:51.465633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":838,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.962613+0000","last_change":"2024-03-12T03:51:36.030269+0000","last_active":"2024-03-12T03:51:56.962613+0000","last_peered":"2024-03-12T03:51:56.962613+0000","last_clean":"2024-03-12T03:51:56.962613+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:51:56.962613+0000","last_undegraded":"2024-03-12T03:51:56.962613+0000","last_fullsized":"2024-03-12T03:51:56.962613+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:36.030166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:36.030166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:41.226793+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":432,"reported_epoch":443,"state":"active+undersized","last_fresh":"2024-03-12T03:51:48.494375+0000","last_change":"2024-03-12T03:48:58.454186+0000","last_active":"2024-03-12T03:51:48.494375+0000","last_peered":"2024-03-12T03:51:48.494375+0000","last_clean":"2024-03-12T03:34:01.860694+0000","last_became_active":"2024-03-12T03:48:58.454186+0000","last_became_peered":"2024-03-12T03:48:58.454186+0000","last_unstale":"2024-03-12T03:51:48.494375+0000","last_undegraded":"2024-03-12T03:51:48.494375+0000","last_fullsized":"2024-03-12T03:48:58.443407+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:04.857986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:04.857986+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:49.548039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"443'743","reported_seq":760,"reported_epoch":443,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:53.558636+0000","last_change":"2024-03-12T03:51:50.121091+0000","last_active":"2024-03-12T03:51:53.558636+0000","last_peered":"2024-03-12T03:51:53.558636+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:51:40.888468+0000","last_became_peered":"2024-03-12T03:51:40.888468+0000","last_unstale":"2024-03-12T03:51:53.558636+0000","last_undegraded":"2024-03-12T03:51:50.116896+0000","last_fullsized":"2024-03-12T03:51:40.858247+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:50.677251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48693248,"num_objects":743,"num_object_clones":0,"num_object_copies":2229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":743,"num_write_kb":47552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":805,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.958572+0000","last_change":"2024-03-12T03:48:58.467253+0000","last_active":"2024-03-12T03:51:54.958572+0000","last_peered":"2024-03-12T03:51:54.958572+0000","last_clean":"2024-03-12T03:48:53.472859+0000","last_became_active":"2024-03-12T03:48:58.467253+0000","last_became_peered":"2024-03-12T03:48:58.467253+0000","last_unstale":"2024-03-12T03:51:54.958572+0000","last_undegraded":"2024-03-12T03:51:54.958572+0000","last_fullsized":"2024-03-12T03:48:58.444374+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:00.577693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:46:00.577693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026825+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"269'24","reported_seq":651,"reported_epoch":445,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:56.969944+0000","last_change":"2024-03-12T03:48:58.449415+0000","last_active":"2024-03-12T03:51:56.969944+0000","last_peered":"2024-03-12T03:51:56.969944+0000","last_clean":"2024-03-12T03:43:28.788554+0000","last_became_active":"2024-03-12T03:48:58.449415+0000","last_became_peered":"2024-03-12T03:48:58.449415+0000","last_unstale":"2024-03-12T03:51:56.969944+0000","last_undegraded":"2024-03-12T03:48:58.441112+0000","last_fullsized":"2024-03-12T03:48:58.441001+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:48:17.191968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:17.191968+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":1944,"reported_epoch":444,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:54.950814+0000","last_change":"2024-03-12T03:48:58.462253+0000","last_active":"2024-03-12T03:51:54.950814+0000","last_peered":"2024-03-12T03:51:54.950814+0000","last_clean":"2024-03-12T03:48:56.580904+0000","last_became_active":"2024-03-12T03:48:58.462253+0000","last_became_peered":"2024-03-12T03:48:58.462253+0000","last_unstale":"2024-03-12T03:51:54.950814+0000","last_undegraded":"2024-03-12T03:48:58.445825+0000","last_fullsized":"2024-03-12T03:48:58.445711+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:47:13.591375+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.591375+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.294143+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1106,"num_read_kb":1101,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":705,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.961197+0000","last_change":"2024-03-12T03:48:58.464609+0000","last_active":"2024-03-12T03:51:54.961197+0000","last_peered":"2024-03-12T03:51:54.961197+0000","last_clean":"2024-03-12T03:43:13.332262+0000","last_became_active":"2024-03-12T03:48:58.464609+0000","last_became_peered":"2024-03-12T03:48:58.464609+0000","last_unstale":"2024-03-12T03:51:54.961197+0000","last_undegraded":"2024-03-12T03:51:54.961197+0000","last_fullsized":"2024-03-12T03:48:58.443632+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:58.050985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.050985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.630659+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":835,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.962746+0000","last_change":"2024-03-12T03:49:13.598817+0000","last_active":"2024-03-12T03:51:56.962746+0000","last_peered":"2024-03-12T03:51:56.962746+0000","last_clean":"2024-03-12T03:51:56.962746+0000","last_became_active":"2024-03-12T03:46:25.302256+0000","last_became_peered":"2024-03-12T03:46:25.302256+0000","last_unstale":"2024-03-12T03:51:56.962746+0000","last_undegraded":"2024-03-12T03:51:56.962746+0000","last_fullsized":"2024-03-12T03:51:56.962746+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:13.598692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:13.598692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":969,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.965040+0000","last_change":"2024-03-12T03:51:11.966626+0000","last_active":"2024-03-12T03:51:56.965040+0000","last_peered":"2024-03-12T03:51:56.965040+0000","last_clean":"2024-03-12T03:51:56.965040+0000","last_became_active":"2024-03-12T03:51:11.965655+0000","last_became_peered":"2024-03-12T03:51:11.965655+0000","last_unstale":"2024-03-12T03:51:56.965040+0000","last_undegraded":"2024-03-12T03:51:56.965040+0000","last_fullsized":"2024-03-12T03:51:56.965040+0000","mapping_epoch":154,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:50.467976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.467976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"443'774","reported_seq":793,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:53.571541+0000","last_change":"2024-03-12T03:51:40.888952+0000","last_active":"2024-03-12T03:51:53.571541+0000","last_peered":"2024-03-12T03:51:53.571541+0000","last_clean":"2024-03-12T03:51:53.571541+0000","last_became_active":"2024-03-12T03:51:40.888646+0000","last_became_peered":"2024-03-12T03:51:40.888646+0000","last_unstale":"2024-03-12T03:51:53.571541+0000","last_undegraded":"2024-03-12T03:51:53.571541+0000","last_fullsized":"2024-03-12T03:51:53.571541+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":774,"log_dups_size":0,"ondisk_log_size":774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:00.148445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50724864,"num_objects":774,"num_object_clones":0,"num_object_copies":2322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":774,"num_write_kb":49536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":832,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951809+0000","last_change":"2024-03-12T03:50:48.205459+0000","last_active":"2024-03-12T03:51:54.951809+0000","last_peered":"2024-03-12T03:51:54.951809+0000","last_clean":"2024-03-12T03:51:54.951809+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:51:54.951809+0000","last_undegraded":"2024-03-12T03:51:54.951809+0000","last_fullsized":"2024-03-12T03:51:54.951809+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.205352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.205352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:17.162359+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"315'29","reported_seq":984,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:55.036321+0000","last_change":"2024-03-12T03:50:45.902898+0000","last_active":"2024-03-12T03:51:55.036321+0000","last_peered":"2024-03-12T03:51:55.036321+0000","last_clean":"2024-03-12T03:51:55.036321+0000","last_became_active":"2024-03-12T03:45:18.109048+0000","last_became_peered":"2024-03-12T03:45:18.109048+0000","last_unstale":"2024-03-12T03:51:55.036321+0000","last_undegraded":"2024-03-12T03:51:55.036321+0000","last_fullsized":"2024-03-12T03:51:55.036321+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'29","last_scrub_stamp":"2024-03-12T03:50:45.902742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:45.902742+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.630693+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":596,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.493664+0000","last_change":"2024-03-12T03:50:33.413216+0000","last_active":"2024-03-12T03:51:48.493664+0000","last_peered":"2024-03-12T03:51:48.493664+0000","last_clean":"2024-03-12T03:51:48.493664+0000","last_became_active":"2024-03-12T03:46:25.292535+0000","last_became_peered":"2024-03-12T03:46:25.292535+0000","last_unstale":"2024-03-12T03:51:48.493664+0000","last_undegraded":"2024-03-12T03:51:48.493664+0000","last_fullsized":"2024-03-12T03:51:48.493664+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:33.413098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:33.413098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:56.872026+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1103,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.950668+0000","last_change":"2024-03-12T03:48:58.452024+0000","last_active":"2024-03-12T03:51:54.950668+0000","last_peered":"2024-03-12T03:51:54.950668+0000","last_clean":"2024-03-12T03:48:53.448868+0000","last_became_active":"2024-03-12T03:48:58.452024+0000","last_became_peered":"2024-03-12T03:48:58.452024+0000","last_unstale":"2024-03-12T03:51:54.950668+0000","last_undegraded":"2024-03-12T03:51:54.950668+0000","last_fullsized":"2024-03-12T03:48:58.444796+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_clean_scrub_stamp":"2024-03-12T03:48:11.435684+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.294208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":824,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.957958+0000","last_change":"2024-03-12T03:51:39.169999+0000","last_active":"2024-03-12T03:51:54.957958+0000","last_peered":"2024-03-12T03:51:54.957958+0000","last_clean":"2024-03-12T03:51:54.957958+0000","last_became_active":"2024-03-12T03:46:25.295668+0000","last_became_peered":"2024-03-12T03:46:25.295668+0000","last_unstale":"2024-03-12T03:51:54.957958+0000","last_undegraded":"2024-03-12T03:51:54.957958+0000","last_fullsized":"2024-03-12T03:51:54.957958+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.169871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.169871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:03.084086+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"443'743","reported_seq":763,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:53.539232+0000","last_change":"2024-03-12T03:51:40.888948+0000","last_active":"2024-03-12T03:51:53.539232+0000","last_peered":"2024-03-12T03:51:53.539232+0000","last_clean":"2024-03-12T03:51:53.539232+0000","last_became_active":"2024-03-12T03:51:40.888540+0000","last_became_peered":"2024-03-12T03:51:40.888540+0000","last_unstale":"2024-03-12T03:51:53.539232+0000","last_undegraded":"2024-03-12T03:51:53.539232+0000","last_fullsized":"2024-03-12T03:51:53.539232+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:04.601254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48693248,"num_objects":743,"num_object_clones":0,"num_object_copies":2229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":743,"num_write_kb":47552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":563,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.956239+0000","last_change":"2024-03-12T03:50:59.748930+0000","last_active":"2024-03-12T03:51:54.956239+0000","last_peered":"2024-03-12T03:51:54.956239+0000","last_clean":"2024-03-12T03:51:54.956239+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:51:54.956239+0000","last_undegraded":"2024-03-12T03:51:54.956239+0000","last_fullsized":"2024-03-12T03:51:54.956239+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:59.748792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.748792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:08.499819+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"269'23","reported_seq":865,"reported_epoch":445,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:56.964714+0000","last_change":"2024-03-12T03:48:58.451116+0000","last_active":"2024-03-12T03:51:56.964714+0000","last_peered":"2024-03-12T03:51:56.964714+0000","last_clean":"2024-03-12T03:48:53.448806+0000","last_became_active":"2024-03-12T03:48:58.451116+0000","last_became_peered":"2024-03-12T03:48:58.451116+0000","last_unstale":"2024-03-12T03:51:56.964714+0000","last_undegraded":"2024-03-12T03:48:58.440926+0000","last_fullsized":"2024-03-12T03:48:58.440811+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'23","last_scrub_stamp":"2024-03-12T03:45:59.998395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.998395+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":552,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.956846+0000","last_change":"2024-03-12T03:48:58.469230+0000","last_active":"2024-03-12T03:51:54.956846+0000","last_peered":"2024-03-12T03:51:54.956846+0000","last_clean":"2024-03-12T03:48:53.446422+0000","last_became_active":"2024-03-12T03:48:58.469230+0000","last_became_peered":"2024-03-12T03:48:58.469230+0000","last_unstale":"2024-03-12T03:51:54.956846+0000","last_undegraded":"2024-03-12T03:51:54.956846+0000","last_fullsized":"2024-03-12T03:48:58.446209+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:06.903506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.903506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":850,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.964547+0000","last_change":"2024-03-12T03:50:41.599042+0000","last_active":"2024-03-12T03:51:56.964547+0000","last_peered":"2024-03-12T03:51:56.964547+0000","last_clean":"2024-03-12T03:51:56.964547+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:51:56.964547+0000","last_undegraded":"2024-03-12T03:51:56.964547+0000","last_fullsized":"2024-03-12T03:51:56.964547+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:41.598934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:50:41.598934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:57.904020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":836,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951515+0000","last_change":"2024-03-12T03:50:44.293342+0000","last_active":"2024-03-12T03:51:54.951515+0000","last_peered":"2024-03-12T03:51:54.951515+0000","last_clean":"2024-03-12T03:51:54.951515+0000","last_became_active":"2024-03-12T03:46:25.284168+0000","last_became_peered":"2024-03-12T03:46:25.284168+0000","last_unstale":"2024-03-12T03:51:54.951515+0000","last_undegraded":"2024-03-12T03:51:54.951515+0000","last_fullsized":"2024-03-12T03:51:54.951515+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:50:44.293155+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:49:18.522103+0000","last_clean_scrub_stamp":"2024-03-12T03:50:44.293155+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:00.958822+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1118,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.951124+0000","last_change":"2024-03-12T03:48:58.455161+0000","last_active":"2024-03-12T03:51:54.951124+0000","last_peered":"2024-03-12T03:51:54.951124+0000","last_clean":"2024-03-12T03:48:53.449188+0000","last_became_active":"2024-03-12T03:48:58.455161+0000","last_became_peered":"2024-03-12T03:48:58.455161+0000","last_unstale":"2024-03-12T03:51:54.951124+0000","last_undegraded":"2024-03-12T03:51:54.951124+0000","last_fullsized":"2024-03-12T03:48:58.444402+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:01.456424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:01.456424+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.294224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"443'791","reported_seq":808,"reported_epoch":443,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:53.564910+0000","last_change":"2024-03-12T03:51:50.089118+0000","last_active":"2024-03-12T03:51:53.564910+0000","last_peered":"2024-03-12T03:51:53.564910+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:51:40.861779+0000","last_became_peered":"2024-03-12T03:51:40.861779+0000","last_unstale":"2024-03-12T03:51:53.564910+0000","last_undegraded":"2024-03-12T03:51:50.086761+0000","last_fullsized":"2024-03-12T03:51:40.858211+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:40.108172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51838976,"num_objects":791,"num_object_clones":0,"num_object_copies":2373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":791,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":791,"num_write_kb":50624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":791}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":130,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-03-12T03:51:56.963710+0000","last_change":"2024-03-12T03:48:58.458916+0000","last_active":"2024-03-12T03:51:56.963710+0000","last_peered":"2024-03-12T03:51:56.963710+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.458916+0000","last_became_peered":"2024-03-12T03:48:58.458916+0000","last_unstale":"2024-03-12T03:51:56.963710+0000","last_undegraded":"2024-03-12T03:51:56.963710+0000","last_fullsized":"2024-03-12T03:48:58.440354+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:06.864708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.864708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"382'54","reported_seq":218,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.962636+0000","last_change":"2024-03-12T03:51:38.936343+0000","last_active":"2024-03-12T03:51:56.962636+0000","last_peered":"2024-03-12T03:51:56.962636+0000","last_clean":"2024-03-12T03:51:56.962636+0000","last_became_active":"2024-03-12T03:48:48.462111+0000","last_became_peered":"2024-03-12T03:48:48.462111+0000","last_unstale":"2024-03-12T03:51:56.962636+0000","last_undegraded":"2024-03-12T03:51:56.962636+0000","last_fullsized":"2024-03-12T03:51:56.962636+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'54","last_scrub_stamp":"2024-03-12T03:51:38.936190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:38.936190+0000","objects_scrubbed":11,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:59.909110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":58,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":823,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.961398+0000","last_change":"2024-03-12T03:48:58.453277+0000","last_active":"2024-03-12T03:51:54.961398+0000","last_peered":"2024-03-12T03:51:54.961398+0000","last_clean":"2024-03-12T03:48:53.447092+0000","last_became_active":"2024-03-12T03:48:58.453277+0000","last_became_peered":"2024-03-12T03:48:58.453277+0000","last_unstale":"2024-03-12T03:51:54.961398+0000","last_undegraded":"2024-03-12T03:51:54.961398+0000","last_fullsized":"2024-03-12T03:48:58.443592+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.648824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.648824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.630643+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":837,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.957304+0000","last_change":"2024-03-12T03:50:58.107026+0000","last_active":"2024-03-12T03:51:54.957304+0000","last_peered":"2024-03-12T03:51:54.957304+0000","last_clean":"2024-03-12T03:51:54.957304+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:51:54.957304+0000","last_undegraded":"2024-03-12T03:51:54.957304+0000","last_fullsized":"2024-03-12T03:51:54.957304+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:50:58.106746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:50:58.106746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:24.247584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":636,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494522+0000","last_change":"2024-03-12T03:50:40.511469+0000","last_active":"2024-03-12T03:51:48.494522+0000","last_peered":"2024-03-12T03:51:48.494522+0000","last_clean":"2024-03-12T03:51:48.494522+0000","last_became_active":"2024-03-12T03:46:25.284416+0000","last_became_peered":"2024-03-12T03:46:25.284416+0000","last_unstale":"2024-03-12T03:51:48.494522+0000","last_undegraded":"2024-03-12T03:51:48.494522+0000","last_fullsized":"2024-03-12T03:51:48.494522+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:40.511391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:40.511391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:06.001248+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":960,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.955937+0000","last_change":"2024-03-12T03:48:58.460115+0000","last_active":"2024-03-12T03:51:54.955937+0000","last_peered":"2024-03-12T03:51:54.955937+0000","last_clean":"2024-03-12T03:48:53.471868+0000","last_became_active":"2024-03-12T03:48:58.460115+0000","last_became_peered":"2024-03-12T03:48:58.460115+0000","last_unstale":"2024-03-12T03:51:54.955937+0000","last_undegraded":"2024-03-12T03:51:54.955937+0000","last_fullsized":"2024-03-12T03:48:58.445534+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:06.690873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.690873+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"444'1301","reported_seq":1323,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.500391+0000","last_change":"2024-03-12T03:51:40.888869+0000","last_active":"2024-03-12T03:51:56.500391+0000","last_peered":"2024-03-12T03:51:56.500391+0000","last_clean":"2024-03-12T03:51:56.500391+0000","last_became_active":"2024-03-12T03:51:40.888578+0000","last_became_peered":"2024-03-12T03:51:40.888578+0000","last_unstale":"2024-03-12T03:51:56.500391+0000","last_undegraded":"2024-03-12T03:51:56.500391+0000","last_fullsized":"2024-03-12T03:51:56.500391+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":1301,"log_dups_size":0,"ondisk_log_size":1301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:52.079859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85262336,"num_objects":1301,"num_object_clones":0,"num_object_copies":3903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1301,"num_write_kb":83264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":379,"reported_epoch":443,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:51:48.494300+0000","last_change":"2024-03-12T03:51:00.633662+0000","last_active":"2024-03-12T03:51:48.494300+0000","last_peered":"2024-03-12T03:51:48.494300+0000","last_clean":"2024-03-12T03:51:48.494300+0000","last_became_active":"2024-03-12T03:50:58.843997+0000","last_became_peered":"2024-03-12T03:50:58.843997+0000","last_unstale":"2024-03-12T03:51:48.494300+0000","last_undegraded":"2024-03-12T03:51:48.494300+0000","last_fullsized":"2024-03-12T03:51:48.494300+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:00.633453+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.924742+0000","last_clean_scrub_stamp":"2024-03-12T03:51:00.633453+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:21.187556+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"269'26","reported_seq":609,"reported_epoch":445,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:56.965141+0000","last_change":"2024-03-12T03:48:58.462328+0000","last_active":"2024-03-12T03:51:56.965141+0000","last_peered":"2024-03-12T03:51:56.965141+0000","last_clean":"2024-03-12T03:43:28.842466+0000","last_became_active":"2024-03-12T03:48:58.462328+0000","last_became_peered":"2024-03-12T03:48:58.462328+0000","last_unstale":"2024-03-12T03:51:56.965141+0000","last_undegraded":"2024-03-12T03:48:58.442501+0000","last_fullsized":"2024-03-12T03:48:58.442398+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:15.774076+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.774076+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":836,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.957271+0000","last_change":"2024-03-12T03:51:28.049718+0000","last_active":"2024-03-12T03:51:54.957271+0000","last_peered":"2024-03-12T03:51:54.957271+0000","last_clean":"2024-03-12T03:51:54.957271+0000","last_became_active":"2024-03-12T03:46:25.303784+0000","last_became_peered":"2024-03-12T03:46:25.303784+0000","last_unstale":"2024-03-12T03:51:54.957271+0000","last_undegraded":"2024-03-12T03:51:54.957271+0000","last_fullsized":"2024-03-12T03:51:54.957271+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:28.049638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:51:28.049638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:45.711056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":565,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.956113+0000","last_change":"2024-03-12T03:51:33.662176+0000","last_active":"2024-03-12T03:51:54.956113+0000","last_peered":"2024-03-12T03:51:54.956113+0000","last_clean":"2024-03-12T03:51:54.956113+0000","last_became_active":"2024-03-12T03:46:25.284556+0000","last_became_peered":"2024-03-12T03:46:25.284556+0000","last_unstale":"2024-03-12T03:51:54.956113+0000","last_undegraded":"2024-03-12T03:51:54.956113+0000","last_fullsized":"2024-03-12T03:51:54.956113+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:33.662099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:33.662099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:42.786905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":410,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.950517+0000","last_change":"2024-03-12T03:48:58.464502+0000","last_active":"2024-03-12T03:51:54.950517+0000","last_peered":"2024-03-12T03:51:54.950517+0000","last_clean":"2024-03-12T03:39:28.535672+0000","last_became_active":"2024-03-12T03:48:58.464502+0000","last_became_peered":"2024-03-12T03:48:58.464502+0000","last_unstale":"2024-03-12T03:51:54.950517+0000","last_undegraded":"2024-03-12T03:51:54.950517+0000","last_fullsized":"2024-03-12T03:48:58.446034+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:55.192996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:55.192996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.294097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"444'1358","reported_seq":1380,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.491284+0000","last_change":"2024-03-12T03:51:40.888852+0000","last_active":"2024-03-12T03:51:56.491284+0000","last_peered":"2024-03-12T03:51:56.491284+0000","last_clean":"2024-03-12T03:51:56.491284+0000","last_became_active":"2024-03-12T03:51:40.888537+0000","last_became_peered":"2024-03-12T03:51:40.888537+0000","last_unstale":"2024-03-12T03:51:56.491284+0000","last_undegraded":"2024-03-12T03:51:56.491284+0000","last_fullsized":"2024-03-12T03:51:56.491284+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":1358,"log_dups_size":0,"ondisk_log_size":1358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:53.079103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88997888,"num_objects":1358,"num_object_clones":0,"num_object_copies":4074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1358,"num_write_kb":86912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":708,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494371+0000","last_change":"2024-03-12T03:50:59.586938+0000","last_active":"2024-03-12T03:51:48.494371+0000","last_peered":"2024-03-12T03:51:48.494371+0000","last_clean":"2024-03-12T03:51:48.494371+0000","last_became_active":"2024-03-12T03:50:58.840022+0000","last_became_peered":"2024-03-12T03:50:58.840022+0000","last_unstale":"2024-03-12T03:51:48.494371+0000","last_undegraded":"2024-03-12T03:51:48.494371+0000","last_fullsized":"2024-03-12T03:51:48.494371+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.586800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.586800+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:24.596398+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"269'51","reported_seq":821,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.963034+0000","last_change":"2024-03-12T03:49:53.484809+0000","last_active":"2024-03-12T03:51:56.963034+0000","last_peered":"2024-03-12T03:51:56.963034+0000","last_clean":"2024-03-12T03:51:56.963034+0000","last_became_active":"2024-03-12T03:45:18.106503+0000","last_became_peered":"2024-03-12T03:45:18.106503+0000","last_unstale":"2024-03-12T03:51:56.963034+0000","last_undegraded":"2024-03-12T03:51:56.963034+0000","last_fullsized":"2024-03-12T03:51:56.963034+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'51","last_scrub_stamp":"2024-03-12T03:49:53.484646+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:49:53.484646+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":836,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951376+0000","last_change":"2024-03-12T03:51:34.290875+0000","last_active":"2024-03-12T03:51:54.951376+0000","last_peered":"2024-03-12T03:51:54.951376+0000","last_clean":"2024-03-12T03:51:54.951376+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:51:54.951376+0000","last_undegraded":"2024-03-12T03:51:54.951376+0000","last_fullsized":"2024-03-12T03:51:54.951376+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:34.290763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:51:34.290763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:45.217984+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":542,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.956535+0000","last_change":"2024-03-12T03:48:58.459230+0000","last_active":"2024-03-12T03:51:54.956535+0000","last_peered":"2024-03-12T03:51:54.956535+0000","last_clean":"2024-03-12T03:48:53.446740+0000","last_became_active":"2024-03-12T03:48:58.459230+0000","last_became_peered":"2024-03-12T03:48:58.459230+0000","last_unstale":"2024-03-12T03:51:54.956535+0000","last_undegraded":"2024-03-12T03:51:54.956535+0000","last_fullsized":"2024-03-12T03:48:58.447475+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.954731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.954731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":846,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.950569+0000","last_change":"2024-03-12T03:50:42.276895+0000","last_active":"2024-03-12T03:51:54.950569+0000","last_peered":"2024-03-12T03:51:54.950569+0000","last_clean":"2024-03-12T03:51:54.950569+0000","last_became_active":"2024-03-12T03:46:25.292122+0000","last_became_peered":"2024-03-12T03:46:25.292122+0000","last_unstale":"2024-03-12T03:51:54.950569+0000","last_undegraded":"2024-03-12T03:51:54.950569+0000","last_fullsized":"2024-03-12T03:51:54.950569+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:42.276562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:42.276562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:08.411799+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"444'1247","reported_seq":1265,"reported_epoch":444,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:56.897427+0000","last_change":"2024-03-12T03:51:50.088041+0000","last_active":"2024-03-12T03:51:56.897427+0000","last_peered":"2024-03-12T03:51:56.897427+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:51:40.824119+0000","last_became_peered":"2024-03-12T03:51:40.824119+0000","last_unstale":"2024-03-12T03:51:56.897427+0000","last_undegraded":"2024-03-12T03:51:50.087161+0000","last_fullsized":"2024-03-12T03:51:40.821411+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":1247,"log_dups_size":0,"ondisk_log_size":1247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:44.215101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81723392,"num_objects":1247,"num_object_clones":0,"num_object_copies":3741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1247,"num_write_kb":79808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1247}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":780,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.493741+0000","last_change":"2024-03-12T03:51:27.598751+0000","last_active":"2024-03-12T03:51:48.493741+0000","last_peered":"2024-03-12T03:51:48.493741+0000","last_clean":"2024-03-12T03:51:48.493741+0000","last_became_active":"2024-03-12T03:50:58.836091+0000","last_became_peered":"2024-03-12T03:50:58.836091+0000","last_unstale":"2024-03-12T03:51:48.493741+0000","last_undegraded":"2024-03-12T03:51:48.493741+0000","last_fullsized":"2024-03-12T03:51:48.493741+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:27.598607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:51:27.598607+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:29.106930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":558,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.955352+0000","last_change":"2024-03-12T03:48:58.461069+0000","last_active":"2024-03-12T03:51:54.955352+0000","last_peered":"2024-03-12T03:51:54.955352+0000","last_clean":"2024-03-12T03:48:53.445898+0000","last_became_active":"2024-03-12T03:48:58.461069+0000","last_became_peered":"2024-03-12T03:48:58.461069+0000","last_unstale":"2024-03-12T03:51:54.955352+0000","last_undegraded":"2024-03-12T03:51:54.955352+0000","last_fullsized":"2024-03-12T03:48:58.444677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.981014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.981014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"269'30","reported_seq":533,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.962367+0000","last_change":"2024-03-12T03:51:42.837070+0000","last_active":"2024-03-12T03:51:56.962367+0000","last_peered":"2024-03-12T03:51:56.962367+0000","last_clean":"2024-03-12T03:51:56.962367+0000","last_became_active":"2024-03-12T03:51:42.836618+0000","last_became_peered":"2024-03-12T03:51:42.836618+0000","last_unstale":"2024-03-12T03:51:56.962367+0000","last_undegraded":"2024-03-12T03:51:56.962367+0000","last_fullsized":"2024-03-12T03:51:56.962367+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'30","last_scrub_stamp":"2024-03-12T03:49:16.863622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:16.863622+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":836,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951135+0000","last_change":"2024-03-12T03:50:39.308944+0000","last_active":"2024-03-12T03:51:54.951135+0000","last_peered":"2024-03-12T03:51:54.951135+0000","last_clean":"2024-03-12T03:51:54.951135+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:51:54.951135+0000","last_undegraded":"2024-03-12T03:51:54.951135+0000","last_fullsized":"2024-03-12T03:51:54.951135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:39.308832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:50:39.308832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.888490+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":559,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494056+0000","last_change":"2024-03-12T03:51:41.674084+0000","last_active":"2024-03-12T03:51:48.494056+0000","last_peered":"2024-03-12T03:51:48.494056+0000","last_clean":"2024-03-12T03:51:48.494056+0000","last_became_active":"2024-03-12T03:46:25.280784+0000","last_became_peered":"2024-03-12T03:46:25.280784+0000","last_unstale":"2024-03-12T03:51:48.494056+0000","last_undegraded":"2024-03-12T03:51:48.494056+0000","last_fullsized":"2024-03-12T03:51:48.494056+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:41.673933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:41.673933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:04.398728+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"444'1308","reported_seq":1329,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.722151+0000","last_change":"2024-03-12T03:51:40.859665+0000","last_active":"2024-03-12T03:51:56.722151+0000","last_peered":"2024-03-12T03:51:56.722151+0000","last_clean":"2024-03-12T03:51:56.722151+0000","last_became_active":"2024-03-12T03:51:40.859390+0000","last_became_peered":"2024-03-12T03:51:40.859390+0000","last_unstale":"2024-03-12T03:51:56.722151+0000","last_undegraded":"2024-03-12T03:51:56.722151+0000","last_fullsized":"2024-03-12T03:51:56.722151+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:40.938836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85721088,"num_objects":1308,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1308,"num_write_kb":83712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":715,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.962276+0000","last_change":"2024-03-12T03:50:58.872833+0000","last_active":"2024-03-12T03:51:54.962276+0000","last_peered":"2024-03-12T03:51:54.962276+0000","last_clean":"2024-03-12T03:51:54.962276+0000","last_became_active":"2024-03-12T03:50:58.850740+0000","last_became_peered":"2024-03-12T03:50:58.850740+0000","last_unstale":"2024-03-12T03:51:54.962276+0000","last_undegraded":"2024-03-12T03:51:54.962276+0000","last_fullsized":"2024-03-12T03:51:54.962276+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:58.872716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:58.872716+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:05.807093+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":1052,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951126+0000","last_change":"2024-03-12T03:50:37.369892+0000","last_active":"2024-03-12T03:51:54.951126+0000","last_peered":"2024-03-12T03:51:54.951126+0000","last_clean":"2024-03-12T03:51:54.951126+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:51:54.951126+0000","last_undegraded":"2024-03-12T03:51:54.951126+0000","last_fullsized":"2024-03-12T03:51:54.951126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:50:37.369758+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:50:37.369758+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.294285+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"269'48","reported_seq":898,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.955860+0000","last_change":"2024-03-12T03:50:47.265447+0000","last_active":"2024-03-12T03:51:54.955860+0000","last_peered":"2024-03-12T03:51:54.955860+0000","last_clean":"2024-03-12T03:51:54.955860+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:51:54.955860+0000","last_undegraded":"2024-03-12T03:51:54.955860+0000","last_fullsized":"2024-03-12T03:51:54.955860+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'48","last_scrub_stamp":"2024-03-12T03:50:47.265298+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:50:47.265298+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:02.448314+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":827,"reported_epoch":444,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:54.951430+0000","last_change":"2024-03-12T03:48:58.456357+0000","last_active":"2024-03-12T03:51:54.951430+0000","last_peered":"2024-03-12T03:51:54.951430+0000","last_clean":"2024-03-12T03:48:53.450296+0000","last_became_active":"2024-03-12T03:48:58.456357+0000","last_became_peered":"2024-03-12T03:48:58.456357+0000","last_unstale":"2024-03-12T03:51:54.951430+0000","last_undegraded":"2024-03-12T03:48:58.445880+0000","last_fullsized":"2024-03-12T03:48:58.445769+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:47:05.588582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:05.588582+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.294128+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":801,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.956866+0000","last_change":"2024-03-12T03:48:58.452652+0000","last_active":"2024-03-12T03:51:54.956866+0000","last_peered":"2024-03-12T03:51:54.956866+0000","last_clean":"2024-03-12T03:48:53.472055+0000","last_became_active":"2024-03-12T03:48:58.452652+0000","last_became_peered":"2024-03-12T03:48:58.452652+0000","last_unstale":"2024-03-12T03:51:54.956866+0000","last_undegraded":"2024-03-12T03:51:54.956866+0000","last_fullsized":"2024-03-12T03:48:58.441674+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:59.567890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.567890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026800+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":374,"reported_epoch":444,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:51:54.950569+0000","last_change":"2024-03-12T03:50:59.305188+0000","last_active":"2024-03-12T03:51:54.950569+0000","last_peered":"2024-03-12T03:51:54.950569+0000","last_clean":"2024-03-12T03:51:54.950569+0000","last_became_active":"2024-03-12T03:50:58.836425+0000","last_became_peered":"2024-03-12T03:50:58.836425+0000","last_unstale":"2024-03-12T03:51:54.950569+0000","last_undegraded":"2024-03-12T03:51:54.950569+0000","last_fullsized":"2024-03-12T03:51:54.950569+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.304950+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:50:06.384808+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.304950+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:24.443886+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,1],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"269'18","reported_seq":622,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494704+0000","last_change":"2024-03-12T03:50:48.504074+0000","last_active":"2024-03-12T03:51:48.494704+0000","last_peered":"2024-03-12T03:51:48.494704+0000","last_clean":"2024-03-12T03:51:48.494704+0000","last_became_active":"2024-03-12T03:46:25.288573+0000","last_became_peered":"2024-03-12T03:46:25.288573+0000","last_unstale":"2024-03-12T03:51:48.494704+0000","last_undegraded":"2024-03-12T03:51:48.494704+0000","last_fullsized":"2024-03-12T03:51:48.494704+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-03-12T03:50:48.503336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.503336+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:06.681325+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":130,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.955988+0000","last_change":"2024-03-12T03:48:58.452359+0000","last_active":"2024-03-12T03:51:54.955988+0000","last_peered":"2024-03-12T03:51:54.955988+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:48:58.452359+0000","last_became_peered":"2024-03-12T03:48:58.452359+0000","last_unstale":"2024-03-12T03:51:54.955988+0000","last_undegraded":"2024-03-12T03:51:54.955988+0000","last_fullsized":"2024-03-12T03:48:58.441273+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:10.400135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.400135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":827,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.957507+0000","last_change":"2024-03-12T03:50:42.134792+0000","last_active":"2024-03-12T03:51:54.957507+0000","last_peered":"2024-03-12T03:51:54.957507+0000","last_clean":"2024-03-12T03:51:54.957507+0000","last_became_active":"2024-03-12T03:46:25.293521+0000","last_became_peered":"2024-03-12T03:46:25.293521+0000","last_unstale":"2024-03-12T03:51:54.957507+0000","last_undegraded":"2024-03-12T03:51:54.957507+0000","last_fullsized":"2024-03-12T03:51:54.957507+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:42.134482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:42.134482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.027112+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":823,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951027+0000","last_change":"2024-03-12T03:51:38.361542+0000","last_active":"2024-03-12T03:51:54.951027+0000","last_peered":"2024-03-12T03:51:54.951027+0000","last_clean":"2024-03-12T03:51:54.951027+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:51:54.951027+0000","last_undegraded":"2024-03-12T03:51:54.951027+0000","last_fullsized":"2024-03-12T03:51:54.951027+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:38.361430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:38.361430+0000","last_clean_scrub_stamp":"2024-03-12T03:51:38.361430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:44.373604+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"269'27","reported_seq":748,"reported_epoch":443,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:48.494668+0000","last_change":"2024-03-12T03:48:58.462615+0000","last_active":"2024-03-12T03:51:48.494668+0000","last_peered":"2024-03-12T03:51:48.494668+0000","last_clean":"2024-03-12T03:43:13.333279+0000","last_became_active":"2024-03-12T03:48:58.462615+0000","last_became_peered":"2024-03-12T03:48:58.462615+0000","last_unstale":"2024-03-12T03:51:48.494668+0000","last_undegraded":"2024-03-12T03:48:58.443892+0000","last_fullsized":"2024-03-12T03:48:58.443761+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:48:06.104297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.104297+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:51.579676+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":827,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.950895+0000","last_change":"2024-03-12T03:50:35.416894+0000","last_active":"2024-03-12T03:51:54.950895+0000","last_peered":"2024-03-12T03:51:54.950895+0000","last_clean":"2024-03-12T03:51:54.950895+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:51:54.950895+0000","last_undegraded":"2024-03-12T03:51:54.950895+0000","last_fullsized":"2024-03-12T03:51:54.950895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:35.416780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.416780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.856705+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":585,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494867+0000","last_change":"2024-03-12T03:50:47.530660+0000","last_active":"2024-03-12T03:51:48.494867+0000","last_peered":"2024-03-12T03:51:48.494867+0000","last_clean":"2024-03-12T03:51:48.494867+0000","last_became_active":"2024-03-12T03:46:25.286866+0000","last_became_peered":"2024-03-12T03:46:25.286866+0000","last_unstale":"2024-03-12T03:51:48.494867+0000","last_undegraded":"2024-03-12T03:51:48.494867+0000","last_fullsized":"2024-03-12T03:51:48.494867+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:47.530547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:50:47.530547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:12.449208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":555,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.956924+0000","last_change":"2024-03-12T03:50:37.807983+0000","last_active":"2024-03-12T03:51:54.956924+0000","last_peered":"2024-03-12T03:51:54.956924+0000","last_clean":"2024-03-12T03:51:54.956924+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:51:54.956924+0000","last_undegraded":"2024-03-12T03:51:54.956924+0000","last_fullsized":"2024-03-12T03:51:54.956924+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:37.807855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:37.807855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":844,"reported_epoch":444,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:51:54.950440+0000","last_change":"2024-03-12T03:50:57.227680+0000","last_active":"2024-03-12T03:51:54.950440+0000","last_peered":"2024-03-12T03:51:54.950440+0000","last_clean":"2024-03-12T03:51:54.950440+0000","last_became_active":"2024-03-12T03:50:56.823948+0000","last_became_peered":"2024-03-12T03:50:56.823948+0000","last_unstale":"2024-03-12T03:51:54.950440+0000","last_undegraded":"2024-03-12T03:51:54.950440+0000","last_fullsized":"2024-03-12T03:51:54.950440+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:57.227429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:57.227429+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:27.157222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"269'21","reported_seq":885,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.957599+0000","last_change":"2024-03-12T03:51:01.099499+0000","last_active":"2024-03-12T03:51:54.957599+0000","last_peered":"2024-03-12T03:51:54.957599+0000","last_clean":"2024-03-12T03:51:54.957599+0000","last_became_active":"2024-03-12T03:46:25.298007+0000","last_became_peered":"2024-03-12T03:46:25.298007+0000","last_unstale":"2024-03-12T03:51:54.957599+0000","last_undegraded":"2024-03-12T03:51:54.957599+0000","last_fullsized":"2024-03-12T03:51:54.957599+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'21","last_scrub_stamp":"2024-03-12T03:51:01.099350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:01.099350+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:04.580700+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":704,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.950615+0000","last_change":"2024-03-12T03:48:58.462149+0000","last_active":"2024-03-12T03:51:54.950615+0000","last_peered":"2024-03-12T03:51:54.950615+0000","last_clean":"2024-03-12T03:43:13.331412+0000","last_became_active":"2024-03-12T03:48:58.462149+0000","last_became_peered":"2024-03-12T03:48:58.462149+0000","last_unstale":"2024-03-12T03:51:54.950615+0000","last_undegraded":"2024-03-12T03:51:54.950615+0000","last_fullsized":"2024-03-12T03:48:58.443909+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.294156+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"332'14","reported_seq":838,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494697+0000","last_change":"2024-03-12T03:50:53.568427+0000","last_active":"2024-03-12T03:51:48.494697+0000","last_peered":"2024-03-12T03:51:48.494697+0000","last_clean":"2024-03-12T03:51:48.494697+0000","last_became_active":"2024-03-12T03:46:25.300463+0000","last_became_peered":"2024-03-12T03:46:25.300463+0000","last_unstale":"2024-03-12T03:51:48.494697+0000","last_undegraded":"2024-03-12T03:51:48.494697+0000","last_fullsized":"2024-03-12T03:51:48.494697+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'14","last_scrub_stamp":"2024-03-12T03:50:53.568307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.568307+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:17.071409+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":308,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.955388+0000","last_change":"2024-03-12T03:51:42.834189+0000","last_active":"2024-03-12T03:51:54.955388+0000","last_peered":"2024-03-12T03:51:54.955388+0000","last_clean":"2024-03-12T03:51:54.955388+0000","last_became_active":"2024-03-12T03:51:42.833867+0000","last_became_peered":"2024-03-12T03:51:42.833867+0000","last_unstale":"2024-03-12T03:51:54.955388+0000","last_undegraded":"2024-03-12T03:51:54.955388+0000","last_fullsized":"2024-03-12T03:51:54.955388+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:04.042226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:04.042226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":775,"reported_epoch":444,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:51:54.955761+0000","last_change":"2024-03-12T03:50:59.098952+0000","last_active":"2024-03-12T03:51:54.955761+0000","last_peered":"2024-03-12T03:51:54.955761+0000","last_clean":"2024-03-12T03:51:54.955761+0000","last_became_active":"2024-03-12T03:50:58.849455+0000","last_became_peered":"2024-03-12T03:50:58.849455+0000","last_unstale":"2024-03-12T03:51:54.955761+0000","last_undegraded":"2024-03-12T03:51:54.955761+0000","last_fullsized":"2024-03-12T03:51:54.955761+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.098619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.098619+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:22.927231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,3],"avail_no_missing":["5","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"403'39","reported_seq":692,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494160+0000","last_change":"2024-03-12T03:51:40.548114+0000","last_active":"2024-03-12T03:51:48.494160+0000","last_peered":"2024-03-12T03:51:48.494160+0000","last_clean":"2024-03-12T03:51:48.494160+0000","last_became_active":"2024-03-12T03:46:25.299798+0000","last_became_peered":"2024-03-12T03:46:25.299798+0000","last_unstale":"2024-03-12T03:51:48.494160+0000","last_undegraded":"2024-03-12T03:51:48.494160+0000","last_fullsized":"2024-03-12T03:51:48.494160+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'39","last_scrub_stamp":"2024-03-12T03:51:40.547967+0000","last_deep_scrub":"403'39","last_deep_scrub_stamp":"2024-03-12T03:50:06.535772+0000","last_clean_scrub_stamp":"2024-03-12T03:51:40.547967+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:59.794998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":862,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951700+0000","last_change":"2024-03-12T03:51:32.285686+0000","last_active":"2024-03-12T03:51:54.951700+0000","last_peered":"2024-03-12T03:51:54.951700+0000","last_clean":"2024-03-12T03:51:54.951700+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:51:54.951700+0000","last_undegraded":"2024-03-12T03:51:54.951700+0000","last_fullsized":"2024-03-12T03:51:54.951700+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:51:32.285604+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:51:32.285604+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:38.152150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"385'9","reported_seq":1036,"reported_epoch":444,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:54.957337+0000","last_change":"2024-03-12T03:48:58.448464+0000","last_active":"2024-03-12T03:51:54.957337+0000","last_peered":"2024-03-12T03:51:54.957337+0000","last_clean":"2024-03-12T03:48:56.813639+0000","last_became_active":"2024-03-12T03:48:58.448464+0000","last_became_peered":"2024-03-12T03:48:58.448464+0000","last_unstale":"2024-03-12T03:51:54.957337+0000","last_undegraded":"2024-03-12T03:48:58.442644+0000","last_fullsized":"2024-03-12T03:48:58.442315+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:53.458170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.458170+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026814+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":692,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.950914+0000","last_change":"2024-03-12T03:48:58.463189+0000","last_active":"2024-03-12T03:51:54.950914+0000","last_peered":"2024-03-12T03:51:54.950914+0000","last_clean":"2024-03-12T03:43:13.332853+0000","last_became_active":"2024-03-12T03:48:58.463189+0000","last_became_peered":"2024-03-12T03:48:58.463189+0000","last_unstale":"2024-03-12T03:51:54.950914+0000","last_undegraded":"2024-03-12T03:51:54.950914+0000","last_fullsized":"2024-03-12T03:48:58.445060+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.079569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.079569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.294175+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"269'19","reported_seq":862,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.961583+0000","last_change":"2024-03-12T03:51:01.797152+0000","last_active":"2024-03-12T03:51:54.961583+0000","last_peered":"2024-03-12T03:51:54.961583+0000","last_clean":"2024-03-12T03:51:54.961583+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:51:54.961583+0000","last_undegraded":"2024-03-12T03:51:54.961583+0000","last_fullsized":"2024-03-12T03:51:54.961583+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'19","last_scrub_stamp":"2024-03-12T03:51:01.797018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:01.797018+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:06.011630+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951619+0000","last_change":"2024-03-12T03:51:46.344521+0000","last_active":"2024-03-12T03:51:54.951619+0000","last_peered":"2024-03-12T03:51:54.951619+0000","last_clean":"2024-03-12T03:51:54.951619+0000","last_became_active":"2024-03-12T03:51:42.833922+0000","last_became_peered":"2024-03-12T03:51:42.833922+0000","last_unstale":"2024-03-12T03:51:54.951619+0000","last_undegraded":"2024-03-12T03:51:54.951619+0000","last_fullsized":"2024-03-12T03:51:54.951619+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:46.344443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_clean_scrub_stamp":"2024-03-12T03:51:46.344443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:08.297714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":838,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.963288+0000","last_change":"2024-03-12T03:50:48.723164+0000","last_active":"2024-03-12T03:51:56.963288+0000","last_peered":"2024-03-12T03:51:56.963288+0000","last_clean":"2024-03-12T03:51:56.963288+0000","last_became_active":"2024-03-12T03:46:25.288050+0000","last_became_peered":"2024-03-12T03:46:25.288050+0000","last_unstale":"2024-03-12T03:51:56.963288+0000","last_undegraded":"2024-03-12T03:51:56.963288+0000","last_fullsized":"2024-03-12T03:51:56.963288+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.723073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.723073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.423812+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":596,"reported_epoch":443,"state":"active+undersized","last_fresh":"2024-03-12T03:51:48.493566+0000","last_change":"2024-03-12T03:48:58.456788+0000","last_active":"2024-03-12T03:51:48.493566+0000","last_peered":"2024-03-12T03:51:48.493566+0000","last_clean":"2024-03-12T03:48:53.447742+0000","last_became_active":"2024-03-12T03:48:58.456788+0000","last_became_peered":"2024-03-12T03:48:58.456788+0000","last_unstale":"2024-03-12T03:51:48.493566+0000","last_undegraded":"2024-03-12T03:51:48.493566+0000","last_fullsized":"2024-03-12T03:48:58.442213+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.433184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.433184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:51.579630+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"269'15","reported_seq":418,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.956540+0000","last_change":"2024-03-12T03:49:56.329520+0000","last_active":"2024-03-12T03:51:54.956540+0000","last_peered":"2024-03-12T03:51:54.956540+0000","last_clean":"2024-03-12T03:51:54.956540+0000","last_became_active":"2024-03-12T03:43:01.037348+0000","last_became_peered":"2024-03-12T03:43:01.037348+0000","last_unstale":"2024-03-12T03:51:54.956540+0000","last_undegraded":"2024-03-12T03:51:54.956540+0000","last_fullsized":"2024-03-12T03:51:54.956540+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'15","last_scrub_stamp":"2024-03-12T03:49:56.329393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:56.329393+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026992+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":853,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951074+0000","last_change":"2024-03-12T03:51:47.306023+0000","last_active":"2024-03-12T03:51:54.951074+0000","last_peered":"2024-03-12T03:51:54.951074+0000","last_clean":"2024-03-12T03:51:54.951074+0000","last_became_active":"2024-03-12T03:46:25.283602+0000","last_became_peered":"2024-03-12T03:46:25.283602+0000","last_unstale":"2024-03-12T03:51:54.951074+0000","last_undegraded":"2024-03-12T03:51:54.951074+0000","last_fullsized":"2024-03-12T03:51:54.951074+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:47.305919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:51:47.305919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:54.712288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":608,"reported_epoch":443,"state":"active+undersized","last_fresh":"2024-03-12T03:51:48.493854+0000","last_change":"2024-03-12T03:48:58.460970+0000","last_active":"2024-03-12T03:51:48.493854+0000","last_peered":"2024-03-12T03:51:48.493854+0000","last_clean":"2024-03-12T03:48:53.448497+0000","last_became_active":"2024-03-12T03:48:58.460970+0000","last_became_peered":"2024-03-12T03:48:58.460970+0000","last_unstale":"2024-03-12T03:51:48.493854+0000","last_undegraded":"2024-03-12T03:51:48.493854+0000","last_fullsized":"2024-03-12T03:48:58.444135+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_clean_scrub_stamp":"2024-03-12T03:48:10.625582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:51.579690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":128,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.956011+0000","last_change":"2024-03-12T03:48:58.453783+0000","last_active":"2024-03-12T03:51:54.956011+0000","last_peered":"2024-03-12T03:51:54.956011+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.453783+0000","last_became_peered":"2024-03-12T03:48:58.453783+0000","last_unstale":"2024-03-12T03:51:54.956011+0000","last_undegraded":"2024-03-12T03:51:54.956011+0000","last_fullsized":"2024-03-12T03:48:58.441749+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:13.228117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:48:13.228117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"269'26","reported_seq":632,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.493725+0000","last_change":"2024-03-12T03:51:31.509254+0000","last_active":"2024-03-12T03:51:48.493725+0000","last_peered":"2024-03-12T03:51:48.493725+0000","last_clean":"2024-03-12T03:51:48.493725+0000","last_became_active":"2024-03-12T03:46:25.287095+0000","last_became_peered":"2024-03-12T03:46:25.287095+0000","last_unstale":"2024-03-12T03:51:48.493725+0000","last_undegraded":"2024-03-12T03:51:48.493725+0000","last_fullsized":"2024-03-12T03:51:48.493725+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:51:31.509156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:31.509156+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:41.197038+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":833,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.957595+0000","last_change":"2024-03-12T03:51:42.097053+0000","last_active":"2024-03-12T03:51:54.957595+0000","last_peered":"2024-03-12T03:51:54.957595+0000","last_clean":"2024-03-12T03:51:54.957595+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:51:54.957595+0000","last_undegraded":"2024-03-12T03:51:54.957595+0000","last_fullsized":"2024-03-12T03:51:54.957595+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:42.096972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:51:42.096972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:57.485503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":827,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951375+0000","last_change":"2024-03-12T03:51:39.276331+0000","last_active":"2024-03-12T03:51:54.951375+0000","last_peered":"2024-03-12T03:51:54.951375+0000","last_clean":"2024-03-12T03:51:54.951375+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:51:54.951375+0000","last_undegraded":"2024-03-12T03:51:54.951375+0000","last_fullsized":"2024-03-12T03:51:54.951375+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.276250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.276250+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.276250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:51.293047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":623,"reported_epoch":443,"state":"active+undersized","last_fresh":"2024-03-12T03:51:48.494132+0000","last_change":"2024-03-12T03:48:58.449798+0000","last_active":"2024-03-12T03:51:48.494132+0000","last_peered":"2024-03-12T03:51:48.494132+0000","last_clean":"2024-03-12T03:48:53.448767+0000","last_became_active":"2024-03-12T03:48:58.449798+0000","last_became_peered":"2024-03-12T03:48:58.449798+0000","last_unstale":"2024-03-12T03:51:48.494132+0000","last_undegraded":"2024-03-12T03:51:48.494132+0000","last_fullsized":"2024-03-12T03:48:58.443857+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:51.579532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"269'10","reported_seq":603,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.493159+0000","last_change":"2024-03-12T03:50:39.504167+0000","last_active":"2024-03-12T03:51:48.493159+0000","last_peered":"2024-03-12T03:51:48.493159+0000","last_clean":"2024-03-12T03:51:48.493159+0000","last_became_active":"2024-03-12T03:46:25.286315+0000","last_became_peered":"2024-03-12T03:46:25.286315+0000","last_unstale":"2024-03-12T03:51:48.493159+0000","last_undegraded":"2024-03-12T03:51:48.493159+0000","last_fullsized":"2024-03-12T03:51:48.493159+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'10","last_scrub_stamp":"2024-03-12T03:50:39.504024+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:50:39.504024+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:54.158992+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":704,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.958288+0000","last_change":"2024-03-12T03:48:58.463096+0000","last_active":"2024-03-12T03:51:54.958288+0000","last_peered":"2024-03-12T03:51:54.958288+0000","last_clean":"2024-03-12T03:43:13.332505+0000","last_became_active":"2024-03-12T03:48:58.463096+0000","last_became_peered":"2024-03-12T03:48:58.463096+0000","last_unstale":"2024-03-12T03:51:54.958288+0000","last_undegraded":"2024-03-12T03:51:54.958288+0000","last_fullsized":"2024-03-12T03:48:58.443705+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.851271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.851271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":290,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.960914+0000","last_change":"2024-03-12T03:51:00.809146+0000","last_active":"2024-03-12T03:51:54.960914+0000","last_peered":"2024-03-12T03:51:54.960914+0000","last_clean":"2024-03-12T03:51:54.960914+0000","last_became_active":"2024-03-12T03:45:18.109983+0000","last_became_peered":"2024-03-12T03:45:18.109983+0000","last_unstale":"2024-03-12T03:51:54.960914+0000","last_undegraded":"2024-03-12T03:51:54.960914+0000","last_fullsized":"2024-03-12T03:51:54.960914+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:00.809033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:00.809033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:26.492749+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":827,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951356+0000","last_change":"2024-03-12T03:50:38.371213+0000","last_active":"2024-03-12T03:51:54.951356+0000","last_peered":"2024-03-12T03:51:54.951356+0000","last_clean":"2024-03-12T03:51:54.951356+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:51:54.951356+0000","last_undegraded":"2024-03-12T03:51:54.951356+0000","last_fullsized":"2024-03-12T03:51:54.951356+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:38.371102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:38.371102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.294255+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"269'31","reported_seq":799,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.958468+0000","last_change":"2024-03-12T03:51:47.045902+0000","last_active":"2024-03-12T03:51:54.958468+0000","last_peered":"2024-03-12T03:51:54.958468+0000","last_clean":"2024-03-12T03:51:54.958468+0000","last_became_active":"2024-03-12T03:45:18.107047+0000","last_became_peered":"2024-03-12T03:45:18.107047+0000","last_unstale":"2024-03-12T03:51:54.958468+0000","last_undegraded":"2024-03-12T03:51:54.958468+0000","last_fullsized":"2024-03-12T03:51:54.958468+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:51:47.045769+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:51:47.045769+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:04.653195+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":624,"reported_epoch":443,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:48.493291+0000","last_change":"2024-03-12T03:48:58.454467+0000","last_active":"2024-03-12T03:51:48.493291+0000","last_peered":"2024-03-12T03:51:48.493291+0000","last_clean":"2024-03-12T03:48:53.477110+0000","last_became_active":"2024-03-12T03:48:58.454467+0000","last_became_peered":"2024-03-12T03:48:58.454467+0000","last_unstale":"2024-03-12T03:51:48.493291+0000","last_undegraded":"2024-03-12T03:48:58.439427+0000","last_fullsized":"2024-03-12T03:48:58.439320+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:48:00.458460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.458460+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:51.579660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":415,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.951235+0000","last_change":"2024-03-12T03:48:58.461994+0000","last_active":"2024-03-12T03:51:54.951235+0000","last_peered":"2024-03-12T03:51:54.951235+0000","last_clean":"2024-03-12T03:39:31.213817+0000","last_became_active":"2024-03-12T03:48:58.461994+0000","last_became_peered":"2024-03-12T03:48:58.461994+0000","last_unstale":"2024-03-12T03:51:54.951235+0000","last_undegraded":"2024-03-12T03:51:54.951235+0000","last_fullsized":"2024-03-12T03:48:58.446456+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:09.160277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:09.160277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.294189+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":824,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.963848+0000","last_change":"2024-03-12T03:50:53.919752+0000","last_active":"2024-03-12T03:51:56.963848+0000","last_peered":"2024-03-12T03:51:56.963848+0000","last_clean":"2024-03-12T03:51:56.963848+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:51:56.963848+0000","last_undegraded":"2024-03-12T03:51:56.963848+0000","last_fullsized":"2024-03-12T03:51:56.963848+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:53.919661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.919661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:06.964625+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"269'27","reported_seq":621,"reported_epoch":444,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:54.951515+0000","last_change":"2024-03-12T03:48:58.462533+0000","last_active":"2024-03-12T03:51:54.951515+0000","last_peered":"2024-03-12T03:51:54.951515+0000","last_clean":"2024-03-12T03:43:13.333148+0000","last_became_active":"2024-03-12T03:48:58.462533+0000","last_became_peered":"2024-03-12T03:48:58.462533+0000","last_unstale":"2024-03-12T03:51:54.951515+0000","last_undegraded":"2024-03-12T03:48:58.446989+0000","last_fullsized":"2024-03-12T03:48:58.446871+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:15.795515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.795515+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.294113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"351'2","reported_seq":840,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.962485+0000","last_change":"2024-03-12T03:49:14.611362+0000","last_active":"2024-03-12T03:51:56.962485+0000","last_peered":"2024-03-12T03:51:56.962485+0000","last_clean":"2024-03-12T03:51:56.962485+0000","last_became_active":"2024-03-12T03:46:25.283344+0000","last_became_peered":"2024-03-12T03:46:25.283344+0000","last_unstale":"2024-03-12T03:51:56.962485+0000","last_undegraded":"2024-03-12T03:51:56.962485+0000","last_fullsized":"2024-03-12T03:51:56.962485+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'2","last_scrub_stamp":"2024-03-12T03:49:14.611213+0000","last_deep_scrub":"351'2","last_deep_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_clean_scrub_stamp":"2024-03-12T03:49:14.611213+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"441'10","reported_seq":49,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951702+0000","last_change":"2024-03-12T03:51:42.834336+0000","last_active":"2024-03-12T03:51:54.951702+0000","last_peered":"2024-03-12T03:51:54.951702+0000","last_clean":"2024-03-12T03:51:54.951702+0000","last_became_active":"2024-03-12T03:51:42.833920+0000","last_became_peered":"2024-03-12T03:51:42.833920+0000","last_unstale":"2024-03-12T03:51:54.951702+0000","last_undegraded":"2024-03-12T03:51:54.951702+0000","last_fullsized":"2024-03-12T03:51:54.951702+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'9","last_scrub_stamp":"2024-03-12T03:50:46.889538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:46.889538+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:56.292809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":844,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.951714+0000","last_change":"2024-03-12T03:51:28.318948+0000","last_active":"2024-03-12T03:51:54.951714+0000","last_peered":"2024-03-12T03:51:54.951714+0000","last_clean":"2024-03-12T03:51:54.951714+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:51:54.951714+0000","last_undegraded":"2024-03-12T03:51:54.951714+0000","last_fullsized":"2024-03-12T03:51:54.951714+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:51:28.318860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:28.318860+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:51.019893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"269'24","reported_seq":661,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494237+0000","last_change":"2024-03-12T03:50:05.585005+0000","last_active":"2024-03-12T03:51:48.494237+0000","last_peered":"2024-03-12T03:51:48.494237+0000","last_clean":"2024-03-12T03:51:48.494237+0000","last_became_active":"2024-03-12T03:46:25.283193+0000","last_became_peered":"2024-03-12T03:46:25.283193+0000","last_unstale":"2024-03-12T03:51:48.494237+0000","last_undegraded":"2024-03-12T03:51:48.494237+0000","last_fullsized":"2024-03-12T03:51:48.494237+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:50:05.584740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:05.584740+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:52.569346+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":847,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.961683+0000","last_change":"2024-03-12T03:50:49.933782+0000","last_active":"2024-03-12T03:51:54.961683+0000","last_peered":"2024-03-12T03:51:54.961683+0000","last_clean":"2024-03-12T03:51:54.961683+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:51:54.961683+0000","last_undegraded":"2024-03-12T03:51:54.961683+0000","last_fullsized":"2024-03-12T03:51:54.961683+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:49.933653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:49.933653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:08.330902+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":604,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494467+0000","last_change":"2024-03-12T03:51:42.841557+0000","last_active":"2024-03-12T03:51:48.494467+0000","last_peered":"2024-03-12T03:51:48.494467+0000","last_clean":"2024-03-12T03:51:48.494467+0000","last_became_active":"2024-03-12T03:51:42.841245+0000","last_became_peered":"2024-03-12T03:51:42.841245+0000","last_unstale":"2024-03-12T03:51:48.494467+0000","last_undegraded":"2024-03-12T03:51:48.494467+0000","last_fullsized":"2024-03-12T03:51:48.494467+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:34.600111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:34.600111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:57.735269+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":638,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494931+0000","last_change":"2024-03-12T03:51:28.687582+0000","last_active":"2024-03-12T03:51:48.494931+0000","last_peered":"2024-03-12T03:51:48.494931+0000","last_clean":"2024-03-12T03:51:48.494931+0000","last_became_active":"2024-03-12T03:46:25.296764+0000","last_became_peered":"2024-03-12T03:46:25.296764+0000","last_unstale":"2024-03-12T03:51:48.494931+0000","last_undegraded":"2024-03-12T03:51:48.494931+0000","last_fullsized":"2024-03-12T03:51:48.494931+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:28.687482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:28.687482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:49.176343+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"269'27","reported_seq":613,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.956759+0000","last_change":"2024-03-12T03:51:38.702128+0000","last_active":"2024-03-12T03:51:54.956759+0000","last_peered":"2024-03-12T03:51:54.956759+0000","last_clean":"2024-03-12T03:51:54.956759+0000","last_became_active":"2024-03-12T03:46:25.301700+0000","last_became_peered":"2024-03-12T03:46:25.301700+0000","last_unstale":"2024-03-12T03:51:54.956759+0000","last_undegraded":"2024-03-12T03:51:54.956759+0000","last_fullsized":"2024-03-12T03:51:54.956759+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:51:38.702022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:38.702022+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:56.648527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.951806+0000","last_change":"2024-03-12T03:51:42.833539+0000","last_active":"2024-03-12T03:51:54.951806+0000","last_peered":"2024-03-12T03:51:54.951806+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:51:42.833539+0000","last_became_peered":"2024-03-12T03:51:42.833539+0000","last_unstale":"2024-03-12T03:51:54.951806+0000","last_undegraded":"2024-03-12T03:51:54.951806+0000","last_fullsized":"2024-03-12T03:51:42.829920+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.797296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.797296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.294082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":814,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-03-12T03:51:56.963668+0000","last_change":"2024-03-12T03:48:58.458438+0000","last_active":"2024-03-12T03:51:56.963668+0000","last_peered":"2024-03-12T03:51:56.963668+0000","last_clean":"2024-03-12T03:48:53.448127+0000","last_became_active":"2024-03-12T03:48:58.458438+0000","last_became_peered":"2024-03-12T03:48:58.458438+0000","last_unstale":"2024-03-12T03:51:56.963668+0000","last_undegraded":"2024-03-12T03:51:56.963668+0000","last_fullsized":"2024-03-12T03:48:58.440067+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.016030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.016030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":817,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.962258+0000","last_change":"2024-03-12T03:50:51.952305+0000","last_active":"2024-03-12T03:51:54.962258+0000","last_peered":"2024-03-12T03:51:54.962258+0000","last_clean":"2024-03-12T03:51:54.962258+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:51:54.962258+0000","last_undegraded":"2024-03-12T03:51:54.962258+0000","last_fullsized":"2024-03-12T03:51:54.962258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:51.952189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:51.952189+0000","last_clean_scrub_stamp":"2024-03-12T03:50:51.952189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:21.592139+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"269'16","reported_seq":661,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.493654+0000","last_change":"2024-03-12T03:51:42.838878+0000","last_active":"2024-03-12T03:51:48.493654+0000","last_peered":"2024-03-12T03:51:48.493654+0000","last_clean":"2024-03-12T03:51:48.493654+0000","last_became_active":"2024-03-12T03:51:42.838579+0000","last_became_peered":"2024-03-12T03:51:42.838579+0000","last_unstale":"2024-03-12T03:51:48.493654+0000","last_undegraded":"2024-03-12T03:51:48.493654+0000","last_fullsized":"2024-03-12T03:51:48.493654+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'16","last_scrub_stamp":"2024-03-12T03:49:51.806651+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:49:51.806651+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:52.569327+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":817,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.957460+0000","last_change":"2024-03-12T03:50:02.319955+0000","last_active":"2024-03-12T03:51:54.957460+0000","last_peered":"2024-03-12T03:51:54.957460+0000","last_clean":"2024-03-12T03:51:54.957460+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:51:54.957460+0000","last_undegraded":"2024-03-12T03:51:54.957460+0000","last_fullsized":"2024-03-12T03:51:54.957460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:02.319837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:02.319837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.027023+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":819,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.950871+0000","last_change":"2024-03-12T03:51:01.307708+0000","last_active":"2024-03-12T03:51:54.950871+0000","last_peered":"2024-03-12T03:51:54.950871+0000","last_clean":"2024-03-12T03:51:54.950871+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:51:54.950871+0000","last_undegraded":"2024-03-12T03:51:54.950871+0000","last_fullsized":"2024-03-12T03:51:54.950871+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:01.307529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:51:01.307529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:03.635009+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":823,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.956495+0000","last_change":"2024-03-12T03:51:46.135297+0000","last_active":"2024-03-12T03:51:54.956495+0000","last_peered":"2024-03-12T03:51:54.956495+0000","last_clean":"2024-03-12T03:51:54.956495+0000","last_became_active":"2024-03-12T03:46:25.293777+0000","last_became_peered":"2024-03-12T03:46:25.293777+0000","last_unstale":"2024-03-12T03:51:54.956495+0000","last_undegraded":"2024-03-12T03:51:54.956495+0000","last_fullsized":"2024-03-12T03:51:54.956495+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:46.135156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:46.135156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:02.988360+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"345'33","reported_seq":691,"reported_epoch":443,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:48.493961+0000","last_change":"2024-03-12T03:51:42.838076+0000","last_active":"2024-03-12T03:51:48.493961+0000","last_peered":"2024-03-12T03:51:48.493961+0000","last_clean":"2024-03-12T03:47:02.086867+0000","last_became_active":"2024-03-12T03:51:42.838076+0000","last_became_peered":"2024-03-12T03:51:42.838076+0000","last_unstale":"2024-03-12T03:51:48.493961+0000","last_undegraded":"2024-03-12T03:51:42.833048+0000","last_fullsized":"2024-03-12T03:51:42.832909+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'33","last_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_deep_scrub":"345'33","last_deep_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_clean_scrub_stamp":"2024-03-12T03:47:02.110517+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:51.579600+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":825,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.957252+0000","last_change":"2024-03-12T03:50:41.172421+0000","last_active":"2024-03-12T03:51:54.957252+0000","last_peered":"2024-03-12T03:51:54.957252+0000","last_clean":"2024-03-12T03:51:54.957252+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:51:54.957252+0000","last_undegraded":"2024-03-12T03:51:54.957252+0000","last_fullsized":"2024-03-12T03:51:54.957252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:41.172313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:50:41.172313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.027051+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":804,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.957184+0000","last_change":"2024-03-12T03:48:58.453634+0000","last_active":"2024-03-12T03:51:54.957184+0000","last_peered":"2024-03-12T03:51:54.957184+0000","last_clean":"2024-03-12T03:48:53.472261+0000","last_became_active":"2024-03-12T03:48:58.453634+0000","last_became_peered":"2024-03-12T03:48:58.453634+0000","last_unstale":"2024-03-12T03:51:54.957184+0000","last_undegraded":"2024-03-12T03:51:54.957184+0000","last_fullsized":"2024-03-12T03:48:58.444137+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:02.618599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_clean_scrub_stamp":"2024-03-12T03:46:02.618599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":130,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-03-12T03:51:56.964016+0000","last_change":"2024-03-12T03:48:58.463158+0000","last_active":"2024-03-12T03:51:56.964016+0000","last_peered":"2024-03-12T03:51:56.964016+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.463158+0000","last_became_peered":"2024-03-12T03:48:58.463158+0000","last_unstale":"2024-03-12T03:51:56.964016+0000","last_undegraded":"2024-03-12T03:51:56.964016+0000","last_fullsized":"2024-03-12T03:48:58.439250+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_clean_scrub_stamp":"2024-03-12T03:45:58.095196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"269'62","reported_seq":775,"reported_epoch":443,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:51:48.493294+0000","last_change":"2024-03-12T03:48:58.457134+0000","last_active":"2024-03-12T03:51:48.493294+0000","last_peered":"2024-03-12T03:51:48.493294+0000","last_clean":"2024-03-12T03:48:53.477142+0000","last_became_active":"2024-03-12T03:48:58.457134+0000","last_became_peered":"2024-03-12T03:48:58.457134+0000","last_unstale":"2024-03-12T03:51:48.493294+0000","last_undegraded":"2024-03-12T03:48:58.441299+0000","last_fullsized":"2024-03-12T03:48:58.441188+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:51.579552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":655,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.494678+0000","last_change":"2024-03-12T03:50:37.469343+0000","last_active":"2024-03-12T03:51:48.494678+0000","last_peered":"2024-03-12T03:51:48.494678+0000","last_clean":"2024-03-12T03:51:48.494678+0000","last_became_active":"2024-03-12T03:46:25.279131+0000","last_became_peered":"2024-03-12T03:46:25.279131+0000","last_unstale":"2024-03-12T03:51:48.494678+0000","last_undegraded":"2024-03-12T03:51:48.494678+0000","last_fullsized":"2024-03-12T03:51:48.494678+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:37.469265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:50:37.469265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:53.254911+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":20,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.964259+0000","last_change":"2024-03-12T03:51:42.832852+0000","last_active":"2024-03-12T03:51:56.964259+0000","last_peered":"2024-03-12T03:51:56.964259+0000","last_clean":"2024-03-12T03:51:56.964259+0000","last_became_active":"2024-03-12T03:51:42.832427+0000","last_became_peered":"2024-03-12T03:51:42.832427+0000","last_unstale":"2024-03-12T03:51:56.964259+0000","last_undegraded":"2024-03-12T03:51:56.964259+0000","last_fullsized":"2024-03-12T03:51:56.964259+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:39.866768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:50:39.866768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:07.996602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":606,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-12T03:51:48.493985+0000","last_change":"2024-03-12T03:49:55.557456+0000","last_active":"2024-03-12T03:51:48.493985+0000","last_peered":"2024-03-12T03:51:48.493985+0000","last_clean":"2024-03-12T03:51:48.493985+0000","last_became_active":"2024-03-12T03:46:25.281081+0000","last_became_peered":"2024-03-12T03:46:25.281081+0000","last_unstale":"2024-03-12T03:51:48.493985+0000","last_undegraded":"2024-03-12T03:51:48.493985+0000","last_fullsized":"2024-03-12T03:51:48.493985+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:55.557335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:55.557335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:52.569361+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"269'26","reported_seq":865,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.960729+0000","last_change":"2024-03-12T03:50:57.888885+0000","last_active":"2024-03-12T03:51:54.960729+0000","last_peered":"2024-03-12T03:51:54.960729+0000","last_clean":"2024-03-12T03:51:54.960729+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:51:54.960729+0000","last_undegraded":"2024-03-12T03:51:54.960729+0000","last_fullsized":"2024-03-12T03:51:54.960729+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:50:57.888633+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:50:57.888633+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:23.716232+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":422,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-03-12T03:51:54.958124+0000","last_change":"2024-03-12T03:48:58.462711+0000","last_active":"2024-03-12T03:51:54.958124+0000","last_peered":"2024-03-12T03:51:54.958124+0000","last_clean":"2024-03-12T03:39:28.536135+0000","last_became_active":"2024-03-12T03:48:58.462711+0000","last_became_peered":"2024-03-12T03:48:58.462711+0000","last_unstale":"2024-03-12T03:51:54.958124+0000","last_undegraded":"2024-03-12T03:51:54.958124+0000","last_fullsized":"2024-03-12T03:48:58.443314+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:12.186146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:48:12.186146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:51:59.026887+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":828,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-12T03:51:56.964081+0000","last_change":"2024-03-12T03:50:04.535064+0000","last_active":"2024-03-12T03:51:56.964081+0000","last_peered":"2024-03-12T03:51:56.964081+0000","last_clean":"2024-03-12T03:51:56.964081+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:51:56.964081+0000","last_undegraded":"2024-03-12T03:51:56.964081+0000","last_fullsized":"2024-03-12T03:51:56.964081+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_clean_scrub_stamp":"2024-03-12T03:50:04.534937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":19,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-12T03:51:54.956766+0000","last_change":"2024-03-12T03:51:42.833194+0000","last_active":"2024-03-12T03:51:54.956766+0000","last_peered":"2024-03-12T03:51:54.956766+0000","last_clean":"2024-03-12T03:51:54.956766+0000","last_became_active":"2024-03-12T03:51:42.832894+0000","last_became_peered":"2024-03-12T03:51:42.832894+0000","last_unstale":"2024-03-12T03:51:54.956766+0000","last_undegraded":"2024-03-12T03:51:54.956766+0000","last_fullsized":"2024-03-12T03:51:54.956766+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:32.662924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_clean_scrub_stamp":"2024-03-12T03:51:32.662924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:49.792164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1192361984,"num_objects":18194,"num_object_clones":0,"num_object_copies":54582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4841,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18194,"num_write_kb":1164416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3444899840,"data_stored":3444899840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18194,"ondisk_log_size":18194,"up":43,"acting":43,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1106,"num_read_kb":1101,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":19,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":933560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2177,"ondisk_log_size":2177,"up":49,"acting":53,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1393,"num_read_kb":1118,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":2642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":83,"acting":83,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":87,"acting":87,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070692,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1265668,"kb_used_data":476292,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":92479484,"statfs":{"total":95995035648,"available":94698991616,"internally_reserved":0,"allocated":487723008,"data_stored":486886927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":808122826},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332329,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1255788,"kb_used_data":570540,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92489364,"statfs":{"total":95995035648,"available":94709108736,"internally_reserved":0,"allocated":584232960,"data_stored":583419492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":701495750},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575106,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1505960,"kb_used_data":666600,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92239192,"statfs":{"total":95995035648,"available":94452932608,"internally_reserved":0,"allocated":682598400,"data_stored":681725590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":859306437},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738595,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3725208,"kb_used_data":3102608,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":90019944,"statfs":{"total":95995035648,"available":92180422656,"internally_reserved":0,"allocated":3177070592,"data_stored":3175997927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411580,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1233140,"kb_used_data":489772,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92512012,"statfs":{"total":95995035648,"available":94732300288,"internally_reserved":0,"allocated":501526528,"data_stored":500714617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":330,"seq":1417339207747,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638536,"kb_used_data":337408,"kb_used_omap":13,"kb_used_meta":301106,"kb_avail":93106616,"statfs":{"total":95995035648,"available":95341174784,"internally_reserved":0,"allocated":345505792,"data_stored":344576047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13485,"internal_metadata":308333395},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953716,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939728,"kb_used_data":290760,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92805424,"statfs":{"total":95995035648,"available":95032754176,"internally_reserved":0,"allocated":297738240,"data_stored":296897603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659374,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1180276,"kb_used_data":314548,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92564876,"statfs":{"total":95995035648,"available":94786433024,"internally_reserved":0,"allocated":322097152,"data_stored":321276459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":753467392,"data_stored":753467392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":513998848,"data_stored":513998848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":344719360,"data_stored":344719360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":329449472,"data_stored":329449472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":345636864,"data_stored":345636864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":659030016,"data_stored":659030016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":498597888,"data_stored":498597888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:51:59.964 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-12T03:51:59.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool set unique_pool_4 pg_num 26 2024-03-12T03:51:59.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:51:59.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:00.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:00.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:00.089 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 10 16 28742 28726 179.48 148.688 0.0122591 0.0055665 2024-03-12T03:52:00.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:00.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:00.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:00.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:00.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:00.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:00.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:00.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:00.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:00.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:00.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:00.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:00.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:00.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:00.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:00.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:00.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:00.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:00.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:00.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:00.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:00.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:00.954 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:00.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:01.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:01.089 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 11 16 30944 30928 175.674 137.625 0.0152947 0.00568374 2024-03-12T03:52:01.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:01.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:01.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:01.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:01.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:01.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:01.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:01.362 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:01.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:01.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:01.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:01.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:01.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:01.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:01.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:01.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:01.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:01.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:01.631 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:01 smithi169 ceph-mon[29344]: pgmap v905: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 148 MiB/s wr, 2.37k op/s; 7276/67260 objects degraded (10.818%) 2024-03-12T03:52:01.632 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:01 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:52:01.632 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:01 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:52:01.632 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:01 smithi169 ceph-mon[39949]: pgmap v905: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 148 MiB/s wr, 2.37k op/s; 7276/67260 objects degraded (10.818%) 2024-03-12T03:52:01.632 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:01 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:52:01.632 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:01 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:52:01.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:01 smithi173 ceph-mon[33544]: pgmap v905: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 148 MiB/s wr, 2.37k op/s; 7276/67260 objects degraded (10.818%) 2024-03-12T03:52:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:01 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:52:01.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:01 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:52:01.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:01.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:01.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:01.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:01.897 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:01.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:01.924 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:52:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:52:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:52:01.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:02.038 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:02.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:02.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:02.089 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 12 16 33229 33213 172.935 142.812 0.00378386 0.00577759 2024-03-12T03:52:02.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:02.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:02.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:02.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:02.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:02.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:02.306 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:02.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:02.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:02.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:02.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:02.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:02.547 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:02.584 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:02.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:02.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:02 smithi169 ceph-mon[29344]: osdmap e446: 8 total, 7 up, 8 in 2024-03-12T03:52:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:02 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:52:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:02 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:02 smithi169 ceph-mon[39949]: osdmap e446: 8 total, 7 up, 8 in 2024-03-12T03:52:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:02 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:52:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:02 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:02.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:02.718 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:02.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:02.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:02 smithi173 ceph-mon[33544]: osdmap e446: 8 total, 7 up, 8 in 2024-03-12T03:52:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:02 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:52:02.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:02 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:02.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:02.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:02.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:02.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:02.954 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:02.988 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:02.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:03.089 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 13 16 35443 35427 170.276 138.375 0.00425833 0.00586803 2024-03-12T03:52:03.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:03.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:03.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:03.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:03.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:03.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:03.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:03.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:03.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:03.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:03.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:03.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:03.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:03.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:03.625 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:03.666 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:03.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:03.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:03.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:52:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:52:03.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:03.819 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:03.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:03.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:03.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:03.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:03 smithi173 ceph-mon[33544]: pgmap v907: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 177 MiB/s wr, 2.83k op/s; 8758/87075 objects degraded (10.058%) 2024-03-12T03:52:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:03 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 8758/87075 objects degraded (10.058%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:03 smithi173 ceph-mon[33544]: osdmap e447: 8 total, 7 up, 8 in 2024-03-12T03:52:04.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:04.089 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 14 16 37501 37485 167.301 128.625 0.00544235 0.00597228 2024-03-12T03:52:04.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:04.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:04.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:03 smithi169 ceph-mon[29344]: pgmap v907: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 177 MiB/s wr, 2.83k op/s; 8758/87075 objects degraded (10.058%) 2024-03-12T03:52:04.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:03 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 8758/87075 objects degraded (10.058%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:04.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:03 smithi169 ceph-mon[29344]: osdmap e447: 8 total, 7 up, 8 in 2024-03-12T03:52:04.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:03 smithi169 ceph-mon[39949]: pgmap v907: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 177 MiB/s wr, 2.83k op/s; 8758/87075 objects degraded (10.058%) 2024-03-12T03:52:04.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:03 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 8758/87075 objects degraded (10.058%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:04.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:03 smithi169 ceph-mon[39949]: osdmap e447: 8 total, 7 up, 8 in 2024-03-12T03:52:04.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:04.234 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:04.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:04.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:04.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:04.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:04.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:04.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:04.509 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:04.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:04.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:04.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:04.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:04.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:04.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:04.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:04.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:04.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:04.959 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:04.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:04 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:52:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:04 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:52:05.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:05.090 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 15 16 39241 39225 163.398 108.75 0.00275779 0.00611469 2024-03-12T03:52:05.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:05.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:05.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:05.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:04 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:52:05.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:04 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:52:05.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:05.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:04 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:52:05.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:04 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:52:05.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:05.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:05.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:05.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:05.369 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:05.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:05.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:05.510 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:05.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:05.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:05.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:05.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:05.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:05.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:05.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:05.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:05.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:05.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:05.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:05.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:05 smithi173 ceph-mon[33544]: pgmap v909: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 706 B/s rd, 150 MiB/s wr, 2.39k op/s; 10946/102432 objects degraded (10.686%) 2024-03-12T03:52:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:05 smithi173 ceph-mon[33544]: osdmap e448: 8 total, 7 up, 8 in 2024-03-12T03:52:06.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:06.046 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:06.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:06.089 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 16 16 41261 41245 161.076 126.25 0.00466452 0.00620346 2024-03-12T03:52:06.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:06.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:05 smithi169 ceph-mon[29344]: pgmap v909: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 706 B/s rd, 150 MiB/s wr, 2.39k op/s; 10946/102432 objects degraded (10.686%) 2024-03-12T03:52:06.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:05 smithi169 ceph-mon[29344]: osdmap e448: 8 total, 7 up, 8 in 2024-03-12T03:52:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:05 smithi169 ceph-mon[39949]: pgmap v909: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 706 B/s rd, 150 MiB/s wr, 2.39k op/s; 10946/102432 objects degraded (10.686%) 2024-03-12T03:52:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:05 smithi169 ceph-mon[39949]: osdmap e448: 8 total, 7 up, 8 in 2024-03-12T03:52:06.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:06.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:06.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:06.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:06.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:06.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:06.424 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:06.480 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:06.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:06.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:06.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:06.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:06.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:06.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:06.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:06.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:06.854 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:06.908 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:06.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:07.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:07.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:07.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:07.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:07.089 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 17 16 42715 42699 156.946 90.875 0.00248145 0.00633204 2024-03-12T03:52:07.135 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:07.158 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:07.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:06 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:52:07.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:06 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3474030009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:52:07.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:06 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:52:07.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:06 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3474030009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:52:07.234 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:52:07.234 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:52:07.234 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:52:07.234 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:52:07.234 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:52:07.234 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:52:07.235 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:52:07.235 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:52:07.235 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:52:07.235 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:52:07.235 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:52:07.235 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:52:07.235 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:52:07.235 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:52:07.235 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:52:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:06 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:52:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:06 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3474030009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:52:07.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:07.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:07.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:07.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:07.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:07.434 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:07.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:07.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:07.570 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:07.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:07.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:07.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:07.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:07.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:07.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:07.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:07.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:07.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:07 smithi169 ceph-mon[29344]: pgmap v911: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 122 MiB/s wr, 1.95k op/s; 10946/102432 objects degraded (10.686%) 2024-03-12T03:52:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:07 smithi169 ceph-mon[39949]: pgmap v911: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 122 MiB/s wr, 1.95k op/s; 10946/102432 objects degraded (10.686%) 2024-03-12T03:52:07.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:07.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:07.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:08.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:08.089 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 18 15 44495 44480 154.41 111.312 0.00684825 0.00647126 2024-03-12T03:52:08.127 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:08.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:08.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:08.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:08.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:07 smithi173 ceph-mon[33544]: pgmap v911: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 122 MiB/s wr, 1.95k op/s; 10946/102432 objects degraded (10.686%) 2024-03-12T03:52:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:08.264 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:08.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:08.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:08.389 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:08.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:08.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:08.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:08.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:08.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:08.625 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:08.670 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:08.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:08.732 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:52:08.733 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:52:08.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:08.792 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:08 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 10946/102432 objects degraded (10.686%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:08.792 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:08 smithi169 ceph-mon[29344]: osdmap e449: 8 total, 7 up, 8 in 2024-03-12T03:52:08.792 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:08 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 10946/102432 objects degraded (10.686%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:08.792 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:08 smithi169 ceph-mon[39949]: osdmap e449: 8 total, 7 up, 8 in 2024-03-12T03:52:08.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:08.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:08.846 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:52:08.847 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:52:08.906 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:08.946 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:08.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:09.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:09.086 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:09.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:09.089 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 19 16 46107 46091 151.583 100.688 0.0133574 0.00659212 2024-03-12T03:52:09.188 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:09.227 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:09.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:09.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:08 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 10946/102432 objects degraded (10.686%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:09.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:08 smithi173 ceph-mon[33544]: osdmap e449: 8 total, 7 up, 8 in 2024-03-12T03:52:09.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:09.360 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:09.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:09.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:09.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-12T03:52:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:09.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:09.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:09.538 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:52:09.542 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:52:09.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:09.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:09.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:09.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:09.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:09.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:09.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:09.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:09.896 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:09.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:09.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:10.009 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:09 smithi169 ceph-mon[29344]: pgmap v912: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 166 MiB/s wr, 2.65k op/s; 12186/122376 objects degraded (9.958%) 2024-03-12T03:52:10.010 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:09 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:52:10.010 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:09 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:52:10.010 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:09 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:52:10.010 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:09 smithi169 ceph-mon[39949]: pgmap v912: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 166 MiB/s wr, 2.65k op/s; 12186/122376 objects degraded (9.958%) 2024-03-12T03:52:10.010 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:09 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:52:10.010 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:09 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:52:10.010 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:09 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:52:10.041 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:10.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:10.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:10.090 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:52:10.088367+0000 min lat: 0.00121451 max lat: 0.238103 avg lat: 0.00666678 2024-03-12T03:52:10.090 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:52:10.090 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 20 16 47986 47970 149.875 117.438 0.0140903 0.00666678 2024-03-12T03:52:10.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:10.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:10.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:10.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:09 smithi173 ceph-mon[33544]: pgmap v912: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 166 MiB/s wr, 2.65k op/s; 12186/122376 objects degraded (9.958%) 2024-03-12T03:52:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:09 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:52:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:09 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:52:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:09 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:52:10.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:10.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:10.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:10.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:10.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:10.452 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:10.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:10.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:10.597 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:10.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:10.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:10.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:10.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:10.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:10.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:52:10.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:10.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:10.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:11.016 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:11.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:11.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:11.090 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 21 16 49891 49875 148.401 119.062 0.00585311 0.00673273 2024-03-12T03:52:11.118 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:11.171 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:11.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:11.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:11.320 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:11.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:11.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:11.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:11.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:11.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:11.597 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:11.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:11.671 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:52:11.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:11.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:11.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:11.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:11.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:11.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:11.877 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:11.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:11.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:11 smithi169 ceph-mon[29344]: pgmap v914: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 762 B/s rd, 122 MiB/s wr, 1.95k op/s; 13901/130824 objects degraded (10.626%) 2024-03-12T03:52:11.924 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:52:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:52:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:52:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:11 smithi169 ceph-mon[39949]: pgmap v914: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 762 B/s rd, 122 MiB/s wr, 1.95k op/s; 13901/130824 objects degraded (10.626%) 2024-03-12T03:52:11.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:12.024 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:12.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:12.090 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 22 16 51731 51715 146.883 115 0.00609478 0.00680387 2024-03-12T03:52:12.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:12.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:12.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:12.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:11 smithi173 ceph-mon[33544]: pgmap v914: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 762 B/s rd, 122 MiB/s wr, 1.95k op/s; 13901/130824 objects degraded (10.626%) 2024-03-12T03:52:12.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:12.301 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:12.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:12.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:12.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:12.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:12.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:12.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:12.581 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:12.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:12.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:12.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:12.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:12.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:12.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:12.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:12.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:12.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:12.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:12.988 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:12.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:13.088 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:13.092 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 23 16 53523 53507 145.366 112 0.0115404 0.0068742 2024-03-12T03:52:13.127 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:13.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:13.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:13.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:13.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:13.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:13.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:13.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:13.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:13.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:13.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:13.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:13.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:13.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:13.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:13.684 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:13.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:13.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:52:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:52:13.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:13.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:13.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:13.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:13.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:13.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:14.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:14.090 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:13 smithi169 ceph-mon[29344]: pgmap v915: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s; 14992/145392 objects degraded (10.311%) 2024-03-12T03:52:14.091 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:13 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 14992/145392 objects degraded (10.311%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:14.091 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:13 smithi169 ceph-mon[39949]: pgmap v915: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s; 14992/145392 objects degraded (10.311%) 2024-03-12T03:52:14.091 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:13 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 14992/145392 objects degraded (10.311%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:14.091 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 24 16 55242 55226 143.785 107.438 0.0122513 0.00695023 2024-03-12T03:52:14.093 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:14.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:14.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:14.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:14.226 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:14.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:14.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:14.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:13 smithi173 ceph-mon[33544]: pgmap v915: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s; 14992/145392 objects degraded (10.311%) 2024-03-12T03:52:14.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:13 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 14992/145392 objects degraded (10.311%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:14.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:14.377 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:14.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:14.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:14.520 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:14.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:14.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:14.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:14.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:14.664 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:14.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:14.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:14.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:14.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:14.890 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:14 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:14.891 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:14 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:14.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:14.920 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:52:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:14.951 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:14.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:15.005 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:52:15.006 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:52:15.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:15.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:15.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:15.091 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 25 16 56777 56761 141.871 95.9375 0.0055464 0.00704393 2024-03-12T03:52:15.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:15.223 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:15.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:15.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:14 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:15.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:15.362 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:15.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:15.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:15.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:15.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:15.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:15.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:15.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:15.739 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:15.775 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:15.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:15.818 INFO:teuthology.orchestra.run.smithi169.stderr:set pool 11 pg_num to 26 2024-03-12T03:52:15.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:15.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:15.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:15.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:16.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:16.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:16.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:16.091 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 26 15 58524 58509 140.616 109.25 0.015941 0.00710724 2024-03-12T03:52:16.108 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:15 smithi169 ceph-mon[29344]: pgmap v916: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 16723/157533 objects degraded (10.616%) 2024-03-12T03:52:16.108 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:15 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:52:16.108 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:15 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:52:16.108 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:15 smithi169 ceph-mon[29344]: osdmap e450: 8 total, 7 up, 8 in 2024-03-12T03:52:16.108 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:15 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3194156985' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-12T03:52:16.109 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:15 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-12T03:52:16.109 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:15 smithi169 ceph-mon[39949]: pgmap v916: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 16723/157533 objects degraded (10.616%) 2024-03-12T03:52:16.109 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:15 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:52:16.109 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:15 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:52:16.109 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:15 smithi169 ceph-mon[39949]: osdmap e450: 8 total, 7 up, 8 in 2024-03-12T03:52:16.109 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:15 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3194156985' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-12T03:52:16.109 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:15 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-12T03:52:16.156 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:16.195 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:16.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:16.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:15 smithi173 ceph-mon[33544]: pgmap v916: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 16723/157533 objects degraded (10.616%) 2024-03-12T03:52:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:15 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:52:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:15 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:52:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:15 smithi173 ceph-mon[33544]: osdmap e450: 8 total, 7 up, 8 in 2024-03-12T03:52:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:15 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3194156985' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-12T03:52:16.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:15 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-12T03:52:16.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:16.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:16.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:16.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:16.431 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:16.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:16.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:16.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:16.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:16.609 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:16.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:16.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:16.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:16.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:16.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:16.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:16.813 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:52:16.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:16.892 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:16.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:16.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:16.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:17.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:17.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:17.091 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 27 16 59720 59704 138.174 74.6875 0.0289859 0.00717335 2024-03-12T03:52:17.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:17.165 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:17.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:17.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:17.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:16 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-12T03:52:17.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:16 smithi169 ceph-mon[39949]: osdmap e451: 8 total, 7 up, 8 in 2024-03-12T03:52:17.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:16 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-12T03:52:17.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:16 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-12T03:52:17.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:16 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1347306343' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:52:17.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:16 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:16 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-12T03:52:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:16 smithi169 ceph-mon[29344]: osdmap e451: 8 total, 7 up, 8 in 2024-03-12T03:52:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:16 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-12T03:52:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:16 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-12T03:52:17.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:16 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1347306343' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:52:17.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:16 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:17.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:16 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-12T03:52:17.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:16 smithi173 ceph-mon[33544]: osdmap e451: 8 total, 7 up, 8 in 2024-03-12T03:52:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:16 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-12T03:52:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:16 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-12T03:52:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:16 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1347306343' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:52:17.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:16 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:17.267 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:17.292 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:17.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:17.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:17.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:17.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:17.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:17.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:17.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:17.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:17.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:17.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:17.672 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:17.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:17.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:17.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:17.798 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:17.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:17.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:17.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:17.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:18.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:18.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:18.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:18.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:18.091 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 28 16 60166 60150 134.235 27.875 0.00611145 0.00744524 2024-03-12T03:52:18.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:17 smithi169 ceph-mon[29344]: pgmap v919: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 663 B/s rd, 95 MiB/s wr, 1.52k op/s; 16723/157533 objects degraded (10.616%) 2024-03-12T03:52:18.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:17 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:52:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:17 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-12T03:52:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:17 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1347306343' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:52:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:17 smithi169 ceph-mon[29344]: osdmap e452: 8 total, 7 up, 8 in 2024-03-12T03:52:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:17 smithi169 ceph-mon[39949]: pgmap v919: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 663 B/s rd, 95 MiB/s wr, 1.52k op/s; 16723/157533 objects degraded (10.616%) 2024-03-12T03:52:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:17 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:52:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:17 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-12T03:52:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:17 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1347306343' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:52:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:17 smithi169 ceph-mon[39949]: osdmap e452: 8 total, 7 up, 8 in 2024-03-12T03:52:18.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:18.211 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:18.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:18.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:17 smithi173 ceph-mon[33544]: pgmap v919: 166 pgs: 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 107 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 663 B/s rd, 95 MiB/s wr, 1.52k op/s; 16723/157533 objects degraded (10.616%) 2024-03-12T03:52:18.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:17 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:52:18.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:17 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-12T03:52:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:17 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1347306343' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:52:18.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:17 smithi173 ceph-mon[33544]: osdmap e452: 8 total, 7 up, 8 in 2024-03-12T03:52:18.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:18.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:18.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:18.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:18.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:18.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:18.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:18.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-12T03:52:18.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:18.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:18.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:18.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:18.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:18.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:18.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:18.896 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:18.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:18.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:19.021 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:19.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:19.045 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:52:19.091 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 29 16 61538 61522 132.563 85.75 0.00383653 0.00748143 2024-03-12T03:52:19.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:19.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:19.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:19.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[29344]: 6.12 scrub starts 2024-03-12T03:52:19.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[29344]: 6.12 scrub ok 2024-03-12T03:52:19.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[29344]: osdmap e453: 8 total, 7 up, 8 in 2024-03-12T03:52:19.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[29344]: 11.7 scrub starts 2024-03-12T03:52:19.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[29344]: 11.0 scrub starts 2024-03-12T03:52:19.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-12T03:52:19.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-12T03:52:19.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 16723/157533 objects degraded (10.616%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:19.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[29344]: 11.0 scrub ok 2024-03-12T03:52:19.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[39949]: 6.12 scrub starts 2024-03-12T03:52:19.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[39949]: 6.12 scrub ok 2024-03-12T03:52:19.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[39949]: osdmap e453: 8 total, 7 up, 8 in 2024-03-12T03:52:19.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[39949]: 11.7 scrub starts 2024-03-12T03:52:19.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[39949]: 11.0 scrub starts 2024-03-12T03:52:19.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-12T03:52:19.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-12T03:52:19.172 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 16723/157533 objects degraded (10.616%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:19.172 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:18 smithi169 ceph-mon[39949]: 11.0 scrub ok 2024-03-12T03:52:19.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:18 smithi173 ceph-mon[33544]: 6.12 scrub starts 2024-03-12T03:52:19.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:18 smithi173 ceph-mon[33544]: 6.12 scrub ok 2024-03-12T03:52:19.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:18 smithi173 ceph-mon[33544]: osdmap e453: 8 total, 7 up, 8 in 2024-03-12T03:52:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:18 smithi173 ceph-mon[33544]: 11.7 scrub starts 2024-03-12T03:52:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:18 smithi173 ceph-mon[33544]: 11.0 scrub starts 2024-03-12T03:52:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:18 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-12T03:52:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:18 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-12T03:52:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:18 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 16723/157533 objects degraded (10.616%), 21 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:18 smithi173 ceph-mon[33544]: 11.0 scrub ok 2024-03-12T03:52:19.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:19.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:19.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:19.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:19.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:19.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:19.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:19.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:19.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:19.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:19.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:19.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:19.655 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:19.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:19.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:19.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:19.785 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:19.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:19.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:19.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:52:19.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:19.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:19.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:20.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:20.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:20.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:20.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:20.091 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 30 16 61538 61522 128.145 0 - 0.00748143 2024-03-12T03:52:20.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[39949]: pgmap v922: 176 pgs: 3 peering, 9 unknown, 4 active+clean+remapped, 19 active+undersized+degraded, 34 active+undersized, 107 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 10341/174255 objects degraded (5.934%) 2024-03-12T03:52:20.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[39949]: 11.3 scrub starts 2024-03-12T03:52:20.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[39949]: 11.7 scrub ok 2024-03-12T03:52:20.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[39949]: 11.8 scrub starts 2024-03-12T03:52:20.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[39949]: 11.3 scrub ok 2024-03-12T03:52:20.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-12T03:52:20.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[39949]: osdmap e454: 8 total, 7 up, 8 in 2024-03-12T03:52:20.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[39949]: 4.8 scrub starts 2024-03-12T03:52:20.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[39949]: 4.8 scrub ok 2024-03-12T03:52:20.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[39949]: 11.8 scrub ok 2024-03-12T03:52:20.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[39949]: 5.17 scrub starts 2024-03-12T03:52:20.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[39949]: 5.17 scrub ok 2024-03-12T03:52:20.119 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[29344]: pgmap v922: 176 pgs: 3 peering, 9 unknown, 4 active+clean+remapped, 19 active+undersized+degraded, 34 active+undersized, 107 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 10341/174255 objects degraded (5.934%) 2024-03-12T03:52:20.120 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[29344]: 11.3 scrub starts 2024-03-12T03:52:20.120 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[29344]: 11.7 scrub ok 2024-03-12T03:52:20.120 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[29344]: 11.8 scrub starts 2024-03-12T03:52:20.120 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[29344]: 11.3 scrub ok 2024-03-12T03:52:20.120 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-12T03:52:20.120 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[29344]: osdmap e454: 8 total, 7 up, 8 in 2024-03-12T03:52:20.120 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[29344]: 4.8 scrub starts 2024-03-12T03:52:20.120 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[29344]: 4.8 scrub ok 2024-03-12T03:52:20.120 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[29344]: 11.8 scrub ok 2024-03-12T03:52:20.120 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[29344]: 5.17 scrub starts 2024-03-12T03:52:20.120 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:19 smithi169 ceph-mon[29344]: 5.17 scrub ok 2024-03-12T03:52:20.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:20.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:20.163 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:20.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:19 smithi173 ceph-mon[33544]: pgmap v922: 176 pgs: 3 peering, 9 unknown, 4 active+clean+remapped, 19 active+undersized+degraded, 34 active+undersized, 107 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 10341/174255 objects degraded (5.934%) 2024-03-12T03:52:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:19 smithi173 ceph-mon[33544]: 11.3 scrub starts 2024-03-12T03:52:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:19 smithi173 ceph-mon[33544]: 11.7 scrub ok 2024-03-12T03:52:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:19 smithi173 ceph-mon[33544]: 11.8 scrub starts 2024-03-12T03:52:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:19 smithi173 ceph-mon[33544]: 11.3 scrub ok 2024-03-12T03:52:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:19 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-12T03:52:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:19 smithi173 ceph-mon[33544]: osdmap e454: 8 total, 7 up, 8 in 2024-03-12T03:52:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:19 smithi173 ceph-mon[33544]: 4.8 scrub starts 2024-03-12T03:52:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:19 smithi173 ceph-mon[33544]: 4.8 scrub ok 2024-03-12T03:52:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:19 smithi173 ceph-mon[33544]: 11.8 scrub ok 2024-03-12T03:52:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:19 smithi173 ceph-mon[33544]: 5.17 scrub starts 2024-03-12T03:52:20.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:19 smithi173 ceph-mon[33544]: 5.17 scrub ok 2024-03-12T03:52:20.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:20.290 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:20.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:20.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:20.418 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:20.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:20.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:20.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:20.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:20.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:20.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:20.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:20.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:20.770 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:20.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:20.794 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:20.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:20.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:20.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:20.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:20.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:21.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:21.096 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 31 16 61541 61525 124.016 0.09375 2.18674 0.00758845 2024-03-12T03:52:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:21.097 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:21.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:21.101 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:20 smithi169 ceph-mon[29344]: 11.5 scrub starts 2024-03-12T03:52:21.101 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:20 smithi169 ceph-mon[29344]: 11.5 scrub ok 2024-03-12T03:52:21.102 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:20 smithi169 ceph-mon[29344]: 4.5 scrub starts 2024-03-12T03:52:21.102 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:20 smithi169 ceph-mon[29344]: 4.5 scrub ok 2024-03-12T03:52:21.102 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:20 smithi169 ceph-mon[29344]: osdmap e455: 8 total, 7 up, 8 in 2024-03-12T03:52:21.102 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:20 smithi169 ceph-mon[39949]: 11.5 scrub starts 2024-03-12T03:52:21.102 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:20 smithi169 ceph-mon[39949]: 11.5 scrub ok 2024-03-12T03:52:21.102 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:20 smithi169 ceph-mon[39949]: 4.5 scrub starts 2024-03-12T03:52:21.102 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:20 smithi169 ceph-mon[39949]: 4.5 scrub ok 2024-03-12T03:52:21.102 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:20 smithi169 ceph-mon[39949]: osdmap e455: 8 total, 7 up, 8 in 2024-03-12T03:52:21.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:21.226 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:52:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:20 smithi173 ceph-mon[33544]: 11.5 scrub starts 2024-03-12T03:52:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:20 smithi173 ceph-mon[33544]: 11.5 scrub ok 2024-03-12T03:52:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:20 smithi173 ceph-mon[33544]: 4.5 scrub starts 2024-03-12T03:52:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:20 smithi173 ceph-mon[33544]: 4.5 scrub ok 2024-03-12T03:52:21.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:20 smithi173 ceph-mon[33544]: osdmap e455: 8 total, 7 up, 8 in 2024-03-12T03:52:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:21.255 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:21.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:21.271 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:52:21.317 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:52:21.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:21.357 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:52:21.388 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:21.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:21.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:21.398 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:52:21.444 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:52:21.489 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:52:21.489 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:52:21.489 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:52:21.489 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:52:21.489 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:52:21.489 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:52:21.489 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:52:21.489 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:52:21.489 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:52:21.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:21.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:21.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:21.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:21.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:21.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:21.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:21.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:21.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:21.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:21.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:21.826 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:52:21.826 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:52:21.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:52:21.827 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:52:21.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.827 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:217886 9.c 9.6a680d4c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e330)", 2024-03-12T03:52:21.827 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:46:24.296644+0000", 2024-03-12T03:52:21.827 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 357.52274699200001, 2024-03-12T03:52:21.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0069382339999999, 2024-03-12T03:52:21.827 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.827 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:52:21.827 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.827 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.827 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.827 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 217886 2024-03-12T03:52:21.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.828 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296644+0000", 2024-03-12T03:52:21.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296644+0000", 2024-03-12T03:52:21.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296645+0000", 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5149999999999999e-06 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296651+0000", 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9499999999999998e-06 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296662+0000", 2024-03-12T03:52:21.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.066e-05 2024-03-12T03:52:21.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296679+0000", 2024-03-12T03:52:21.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7268000000000001e-05 2024-03-12T03:52:21.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.303754+0000", 2024-03-12T03:52:21.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0070746569999999998 2024-03-12T03:52:21.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.831 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.303761+0000", 2024-03-12T03:52:21.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2069999999999998e-06 2024-03-12T03:52:21.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.831 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.303573+0000", 2024-03-12T03:52:21.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.99981134699999996 2024-03-12T03:52:21.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.832 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.303582+0000", 2024-03-12T03:52:21.832 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6299999999999993e-06 2024-03-12T03:52:21.832 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.832 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.832 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.832 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.832 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:217888 9.c 9.eefddbcc (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e330)", 2024-03-12T03:52:21.832 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:46:24.296692+0000", 2024-03-12T03:52:21.832 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 357.52269874900003, 2024-03-12T03:52:21.832 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0069225639999999, 2024-03-12T03:52:21.832 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.833 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:52:21.833 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.833 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.833 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.833 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 217888 2024-03-12T03:52:21.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.833 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.833 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.833 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.833 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296692+0000", 2024-03-12T03:52:21.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296692+0000", 2024-03-12T03:52:21.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296693+0000", 2024-03-12T03:52:21.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7000000000000005e-07 2024-03-12T03:52:21.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296696+0000", 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1109999999999999e-06 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296704+0000", 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8290000000000008e-06 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296710+0000", 2024-03-12T03:52:21.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8760000000000002e-06 2024-03-12T03:52:21.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.836 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.303783+0000", 2024-03-12T03:52:21.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0070734689999999998 2024-03-12T03:52:21.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.836 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.303789+0000", 2024-03-12T03:52:21.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6699999999999999e-06 2024-03-12T03:52:21.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.303606+0000", 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.99981741400000002 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.303615+0000", 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6249999999999996e-06 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.837 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.838 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:217897 9.2 9.2e08ee2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e330)", 2024-03-12T03:52:21.838 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:46:24.296736+0000", 2024-03-12T03:52:21.838 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 357.52265463700002, 2024-03-12T03:52:21.838 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97172771000000002, 2024-03-12T03:52:21.838 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.838 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:52:21.838 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.838 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.838 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.838 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 217897 2024-03-12T03:52:21.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.838 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296736+0000", 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296736+0000", 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296737+0000", 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5900000000000003e-07 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296739+0000", 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6280000000000001e-06 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296744+0000", 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1220000000000003e-06 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296749+0000", 2024-03-12T03:52:21.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8130000000000003e-06 2024-03-12T03:52:21.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.301390+0000", 2024-03-12T03:52:21.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046407280000000002 2024-03-12T03:52:21.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.301395+0000", 2024-03-12T03:52:21.842 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3580000000000002e-06 2024-03-12T03:52:21.842 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.842 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.842 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.842 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.268456+0000", 2024-03-12T03:52:21.842 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96706071599999999 2024-03-12T03:52:21.842 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.842 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.842 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.842 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.268464+0000", 2024-03-12T03:52:21.842 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9859999999999994e-06 2024-03-12T03:52:21.842 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:217901 9.8 9.787aefa8 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e330)", 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:46:24.296755+0000", 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 357.522636078, 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97650482800000005, 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.843 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.844 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 217901 2024-03-12T03:52:21.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.844 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296755+0000", 2024-03-12T03:52:21.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296755+0000", 2024-03-12T03:52:21.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296755+0000", 2024-03-12T03:52:21.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5999999999999999e-07 2024-03-12T03:52:21.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296757+0000", 2024-03-12T03:52:21.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0980000000000001e-06 2024-03-12T03:52:21.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.846 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.846 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296763+0000", 2024-03-12T03:52:21.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6230000000000002e-06 2024-03-12T03:52:21.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.846 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.846 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296768+0000", 2024-03-12T03:52:21.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0289999999999996e-06 2024-03-12T03:52:21.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.846 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.300824+0000", 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040562020000000001 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.301384+0000", 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00055981800000000001 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.273250+0000", 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97186606600000003 2024-03-12T03:52:21.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.848 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.273260+0000", 2024-03-12T03:52:21.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6320000000000003e-06 2024-03-12T03:52:21.848 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.848 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.848 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.848 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:217902 9.8 9.5e8518c8 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e330)", 2024-03-12T03:52:21.848 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:46:24.296774+0000", 2024-03-12T03:52:21.848 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 357.52261714899998, 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97783939099999995, 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 217902 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296774+0000", 2024-03-12T03:52:21.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296774+0000", 2024-03-12T03:52:21.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296774+0000", 2024-03-12T03:52:21.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.72e-07 2024-03-12T03:52:21.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296776+0000", 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.153e-06 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296782+0000", 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3329999999999996e-06 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.296786+0000", 2024-03-12T03:52:21.851 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7339999999999999e-06 2024-03-12T03:52:21.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.301450+0000", 2024-03-12T03:52:21.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046631609999999999 2024-03-12T03:52:21.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.301457+0000", 2024-03-12T03:52:21.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7069999999999992e-06 2024-03-12T03:52:21.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.274604+0000", 2024-03-12T03:52:21.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97314708999999999 2024-03-12T03:52:21.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.274613+0000", 2024-03-12T03:52:21.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8410000000000006e-06 2024-03-12T03:52:21.853 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.853 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.853 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:217911 9.8 9.d554e018 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-03-12T03:52:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:46:24.302527+0000", 2024-03-12T03:52:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 357.516863441, 2024-03-12T03:52:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97211743799999994, 2024-03-12T03:52:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:52:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 217911 2024-03-12T03:52:21.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.855 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.302527+0000", 2024-03-12T03:52:21.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.302527+0000", 2024-03-12T03:52:21.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.302529+0000", 2024-03-12T03:52:21.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9130000000000001e-06 2024-03-12T03:52:21.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.302534+0000", 2024-03-12T03:52:21.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2869999999999998e-06 2024-03-12T03:52:21.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.302543+0000", 2024-03-12T03:52:21.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.098e-06 2024-03-12T03:52:21.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.302551+0000", 2024-03-12T03:52:21.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5909999999999996e-06 2024-03-12T03:52:21.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.302602+0000", 2024-03-12T03:52:21.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0717999999999998e-05 2024-03-12T03:52:21.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.302608+0000", 2024-03-12T03:52:21.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4940000000000003e-06 2024-03-12T03:52:21.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.274638+0000", 2024-03-12T03:52:21.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97203068299999995 2024-03-12T03:52:21.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.274645+0000", 2024-03-12T03:52:21.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6540000000000002e-06 2024-03-12T03:52:21.860 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.860 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.860 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.860 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:217915 9.8 9.b43d4028 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:46:24.306492+0000", 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 357.51289858299998, 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97599264600000002, 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 217915 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.306492+0000", 2024-03-12T03:52:21.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.306492+0000", 2024-03-12T03:52:21.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.306494+0000", 2024-03-12T03:52:21.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4980000000000001e-06 2024-03-12T03:52:21.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.306498+0000", 2024-03-12T03:52:21.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4170000000000002e-06 2024-03-12T03:52:21.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.306507+0000", 2024-03-12T03:52:21.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5450000000000003e-06 2024-03-12T03:52:21.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.306516+0000", 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5659999999999996e-06 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.308898+0000", 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023819470000000001 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:24.308908+0000", 2024-03-12T03:52:21.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7650000000000005e-06 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.279299+0000", 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97039127000000003 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:46:25.282485+0000", 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0031856380000000002 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.865 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.866 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261758 9.1 9:85260111:::benchmark_data_smithi169_135229_object117543:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.866 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.410205+0000", 2024-03-12T03:52:21.866 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.409186052, 2024-03-12T03:52:21.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0706916660000001, 2024-03-12T03:52:21.866 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.866 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:52:21.866 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.866 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.866 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.866 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261758 2024-03-12T03:52:21.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.867 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410205+0000", 2024-03-12T03:52:21.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410205+0000", 2024-03-12T03:52:21.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410206+0000", 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5799999999999998e-07 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410211+0000", 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7169999999999999e-06 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410220+0000", 2024-03-12T03:52:21.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2930000000000006e-06 2024-03-12T03:52:21.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410228+0000", 2024-03-12T03:52:21.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6899999999999992e-06 2024-03-12T03:52:21.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410291+0000", 2024-03-12T03:52:21.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3311000000000004e-05 2024-03-12T03:52:21.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.870 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-03-12T03:52:21.870 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410297+0000", 2024-03-12T03:52:21.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6290000000000004e-06 2024-03-12T03:52:21.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.870 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.870 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415951+0000", 2024-03-12T03:52:21.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056530599999999997 2024-03-12T03:52:21.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415960+0000", 2024-03-12T03:52:21.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3470000000000005e-06 2024-03-12T03:52:21.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.480813+0000", 2024-03-12T03:52:21.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0648534999999999 2024-03-12T03:52:21.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.480897+0000", 2024-03-12T03:52:21.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3160999999999999e-05 2024-03-12T03:52:21.872 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.872 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.872 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.872 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261761 9.1 9:8da2e467:::benchmark_data_smithi169_135229_object117546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.872 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.415027+0000", 2024-03-12T03:52:21.872 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.40436396799998, 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.084977831, 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261761 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415027+0000", 2024-03-12T03:52:21.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415027+0000", 2024-03-12T03:52:21.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415028+0000", 2024-03-12T03:52:21.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.424e-06 2024-03-12T03:52:21.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415034+0000", 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6450000000000002e-06 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415043+0000", 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3030000000000005e-06 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415051+0000", 2024-03-12T03:52:21.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2400000000000001e-06 2024-03-12T03:52:21.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.876 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.416244+0000", 2024-03-12T03:52:21.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001193345 2024-03-12T03:52:21.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.876 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.416253+0000", 2024-03-12T03:52:21.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7790000000000001e-06 2024-03-12T03:52:21.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.481315+0000", 2024-03-12T03:52:21.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0650625730000001 2024-03-12T03:52:21.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:52:21.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.481432+0000", 2024-03-12T03:52:21.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011627 2024-03-12T03:52:21.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:52:21.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.481880+0000", 2024-03-12T03:52:21.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00044876900000000001 2024-03-12T03:52:21.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.493626+0000", 2024-03-12T03:52:21.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011745267 2024-03-12T03:52:21.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:52:21.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.496512+0000", 2024-03-12T03:52:21.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028865090000000002 2024-03-12T03:52:21.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.499861+0000", 2024-03-12T03:52:21.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033486789999999998 2024-03-12T03:52:21.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:52:21.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.499933+0000", 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2706000000000005e-05 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.500005+0000", 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1322000000000006e-05 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261765 9.1 9:8ad8db76:::benchmark_data_smithi169_135229_object117550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.880 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.417087+0000", 2024-03-12T03:52:21.881 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.40230392000001, 2024-03-12T03:52:21.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0890983059999999, 2024-03-12T03:52:21.881 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.881 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:52:21.881 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.881 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.881 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.881 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261765 2024-03-12T03:52:21.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.881 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417087+0000", 2024-03-12T03:52:21.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417087+0000", 2024-03-12T03:52:21.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417088+0000", 2024-03-12T03:52:21.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7199999999999998e-07 2024-03-12T03:52:21.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417092+0000", 2024-03-12T03:52:21.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9570000000000004e-06 2024-03-12T03:52:21.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417102+0000", 2024-03-12T03:52:21.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0461e-05 2024-03-12T03:52:21.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417110+0000", 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8129999999999996e-06 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417160+0000", 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9874999999999999e-05 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417168+0000", 2024-03-12T03:52:21.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0830000000000004e-06 2024-03-12T03:52:21.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.491925+0000", 2024-03-12T03:52:21.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0747573530000001 2024-03-12T03:52:21.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:52:21.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.492007+0000", 2024-03-12T03:52:21.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1908000000000003e-05 2024-03-12T03:52:21.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.492086+0000", 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8860000000000001e-05 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.496568+0000", 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004481652 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:52:21.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.498024+0000", 2024-03-12T03:52:21.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014564669999999999 2024-03-12T03:52:21.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.887 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.887 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506074+0000", 2024-03-12T03:52:21.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080501300000000008 2024-03-12T03:52:21.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.887 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:52:21.887 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506130+0000", 2024-03-12T03:52:21.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5436999999999998e-05 2024-03-12T03:52:21.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.888 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506185+0000", 2024-03-12T03:52:21.888 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5538000000000003e-05 2024-03-12T03:52:21.888 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.888 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.888 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.888 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261783 9.1 9:898fe7fd:::benchmark_data_smithi169_135229_object117568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.888 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.437041+0000", 2024-03-12T03:52:21.889 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.38235037700002, 2024-03-12T03:52:21.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0694922330000001, 2024-03-12T03:52:21.889 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.889 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:52:21.889 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.889 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.889 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.889 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261783 2024-03-12T03:52:21.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.890 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437041+0000", 2024-03-12T03:52:21.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437041+0000", 2024-03-12T03:52:21.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.891 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437041+0000", 2024-03-12T03:52:21.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0999999999999999e-07 2024-03-12T03:52:21.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.891 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437044+0000", 2024-03-12T03:52:21.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9220000000000001e-06 2024-03-12T03:52:21.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.891 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437050+0000", 2024-03-12T03:52:21.892 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4289999999999998e-06 2024-03-12T03:52:21.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.892 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:52:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:52:21.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437056+0000", 2024-03-12T03:52:21.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7620000000000001e-06 2024-03-12T03:52:21.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437144+0000", 2024-03-12T03:52:21.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8048999999999997e-05 2024-03-12T03:52:21.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437153+0000", 2024-03-12T03:52:21.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8300000000000002e-06 2024-03-12T03:52:21.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.493683+0000", 2024-03-12T03:52:21.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0565300179999999 2024-03-12T03:52:21.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:52:21.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.493780+0000", 2024-03-12T03:52:21.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7338999999999998e-05 2024-03-12T03:52:21.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:52:21.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.493902+0000", 2024-03-12T03:52:21.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121987 2024-03-12T03:52:21.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.498030+0000", 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004127716 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.500102+0000", 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0020721350000000001 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506322+0000", 2024-03-12T03:52:21.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062200040000000003 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506462+0000", 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013965099999999999 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506533+0000", 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0881000000000001e-05 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.897 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.898 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261785 9.1 9:86b9b242:::benchmark_data_smithi169_135229_object117570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.898 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.437071+0000", 2024-03-12T03:52:21.898 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.38232019499998, 2024-03-12T03:52:21.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.070147272, 2024-03-12T03:52:21.898 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.898 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:52:21.898 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.898 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.898 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.898 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261785 2024-03-12T03:52:21.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.899 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.899 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437071+0000", 2024-03-12T03:52:21.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.899 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437071+0000", 2024-03-12T03:52:21.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437071+0000", 2024-03-12T03:52:21.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9500000000000003e-07 2024-03-12T03:52:21.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437074+0000", 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9799999999999998e-06 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437081+0000", 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3620000000000002e-06 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437087+0000", 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9649999999999996e-06 2024-03-12T03:52:21.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437175+0000", 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8422000000000001e-05 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437181+0000", 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9359999999999996e-06 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.495018+0000", 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0578373510000001 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.495118+0000", 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9735000000000006e-05 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.495234+0000", 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011618200000000001 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.500119+0000", 2024-03-12T03:52:21.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048847059999999999 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506031+0000", 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0059123939999999996 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506677+0000", 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00064582699999999997 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506726+0000", 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9202000000000001e-05 2024-03-12T03:52:21.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507218+0000", 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00049171499999999999 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261796 9.1 9:8f8556f2:::benchmark_data_smithi169_135229_object117581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.456145+0000", 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.36324559100001, 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.05128215, 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.905 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:52:21.906 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.906 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.906 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.906 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261796 2024-03-12T03:52:21.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.906 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456145+0000", 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456145+0000", 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456147+0000", 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3939999999999999e-06 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456152+0000", 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.49e-06 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456167+0000", 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4620999999999999e-05 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456176+0000", 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8009999999999993e-06 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.908 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456235+0000", 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9247999999999999e-05 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456243+0000", 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4880000000000007e-06 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.496616+0000", 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.040372536 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:52:21.910 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.496720+0000", 2024-03-12T03:52:21.910 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000104215 2024-03-12T03:52:21.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.910 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:52:21.910 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.496797+0000", 2024-03-12T03:52:21.910 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6518000000000002e-05 2024-03-12T03:52:21.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.910 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:52:21.910 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.503848+0000", 2024-03-12T03:52:21.910 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0070512819999999999 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506248+0000", 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024002730000000001 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507334+0000", 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0010858440000000001 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:52:21.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507384+0000", 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9749000000000001e-05 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507427+0000", 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3690999999999999e-05 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261804 9.1 9:88b23f87:::benchmark_data_smithi169_135229_object117589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.460244+0000", 2024-03-12T03:52:21.912 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.35914696399999, 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0476733279999999, 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261804 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460244+0000", 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460244+0000", 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460245+0000", 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5340000000000001e-06 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.914 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460251+0000", 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.169e-06 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460261+0000", 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0654e-05 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460270+0000", 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2060000000000007e-06 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460372+0000", 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000102212 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460381+0000", 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5759999999999995e-06 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.498055+0000", 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0376738430000001 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.498145+0000", 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9747999999999995e-05 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.498237+0000", 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2366000000000003e-05 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506628+0000", 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083906250000000005 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506642+0000", 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4623999999999999e-05 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507621+0000", 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00097875799999999993 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507879+0000", 2024-03-12T03:52:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00025774899999999999 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507917+0000", 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8263999999999998e-05 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261806 9.1 9:8d72a8bd:::benchmark_data_smithi169_135229_object117591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.460561+0000", 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.358829523, 2024-03-12T03:52:21.919 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.047474284, 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261806 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460561+0000", 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460561+0000", 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460562+0000", 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0999999999999999e-07 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460565+0000", 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0790000000000001e-06 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460571+0000", 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2519999999999996e-06 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460577+0000", 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6370000000000004e-06 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460623+0000", 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6378999999999998e-05 2024-03-12T03:52:21.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460631+0000", 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9619999999999993e-06 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.500157+0000", 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0395256470000001 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.500271+0000", 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000113964 2024-03-12T03:52:21.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.500388+0000", 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116964 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507233+0000", 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068449100000000001 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.924 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507274+0000", 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1075000000000002e-05 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507962+0000", 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00068805199999999998 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508004+0000", 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1888999999999998e-05 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508036+0000", 2024-03-12T03:52:21.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1964000000000001e-05 2024-03-12T03:52:21.926 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.926 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.926 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.926 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261826 9.1 9:8dda3b7f:::benchmark_data_smithi169_135229_object117611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.926 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.482519+0000", 2024-03-12T03:52:21.926 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.33687239699998, 2024-03-12T03:52:21.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0256452170000001, 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261826 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482519+0000", 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482519+0000", 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482572+0000", 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3956000000000003e-05 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482579+0000", 2024-03-12T03:52:21.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7290000000000003e-06 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482591+0000", 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1640999999999999e-05 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482599+0000", 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8099999999999998e-06 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482685+0000", 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6614999999999997e-05 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482693+0000", 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2060000000000007e-06 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.503955+0000", 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.021261559 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.931 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:52:21.931 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.504048+0000", 2024-03-12T03:52:21.931 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2949999999999996e-05 2024-03-12T03:52:21.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.931 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:52:21.931 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.504132+0000", 2024-03-12T03:52:21.931 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3923000000000003e-05 2024-03-12T03:52:21.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507238+0000", 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0031059769999999998 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507568+0000", 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00032969500000000002 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508093+0000", 2024-03-12T03:52:21.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052585900000000005 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508131+0000", 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7798e-05 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508164+0000", 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2499000000000002e-05 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261854 9.1 9:827d7e25:::benchmark_data_smithi169_135229_object117639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.504936+0000", 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.31445448699998, 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0172641549999999, 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261854 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.504936+0000", 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.504936+0000", 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.504938+0000", 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.161e-06 2024-03-12T03:52:21.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:21.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.504941+0000", 2024-03-12T03:52:21.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8589999999999998e-06 2024-03-12T03:52:21.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.504950+0000", 2024-03-12T03:52:21.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1929999999999993e-06 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.504957+0000", 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2640000000000002e-06 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.505095+0000", 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013761399999999999 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.505105+0000", 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0474e-05 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508192+0000", 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0030875100000001 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508397+0000", 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000204976 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508499+0000", 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101908 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.516871+0000", 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083716069999999997 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.516943+0000", 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2513000000000002e-05 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522079+0000", 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051355070000000001 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522141+0000", 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1593000000000005e-05 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522201+0000", 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9976000000000003e-05 2024-03-12T03:52:21.941 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261863 9.1 9:8e1fde2b:::benchmark_data_smithi169_135229_object117648:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.512523+0000", 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.30686800400002, 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0097642929999999, 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.942 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261863 2024-03-12T03:52:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512523+0000", 2024-03-12T03:52:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512523+0000", 2024-03-12T03:52:21.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512524+0000", 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.362e-06 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512529+0000", 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1379999999999998e-06 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512542+0000", 2024-03-12T03:52:21.944 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3050999999999999e-05 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512550+0000", 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4560000000000003e-06 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512612+0000", 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2089999999999997e-05 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512621+0000", 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5019999999999992e-06 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.510476+0000", 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.99785585899999996 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.510571+0000", 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4543000000000006e-05 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.510687+0000", 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115807 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.516909+0000", 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062220299999999999 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.521991+0000", 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050824800000000003 2024-03-12T03:52:21.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522208+0000", 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00021681000000000001 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522249+0000", 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1390999999999997e-05 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522287+0000", 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7774000000000001e-05 2024-03-12T03:52:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261865 9.1 9:87df5a68:::benchmark_data_smithi169_135229_object117650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.515742+0000", 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.30364843500001, 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0066701549999999, 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.949 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261865 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.515742+0000", 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.515742+0000", 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.515744+0000", 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1400000000000001e-06 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.515748+0000", 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7299999999999996e-06 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.515758+0000", 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9059999999999997e-06 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.515766+0000", 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9060000000000002e-06 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.515822+0000", 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5544000000000002e-05 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.515840+0000", 2024-03-12T03:52:21.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8737999999999999e-05 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.513643+0000", 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.99780255399999995 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.513759+0000", 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011632 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:52:21.953 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.513887+0000", 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012784999999999999 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.516980+0000", 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0030926579999999999 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522293+0000", 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053132880000000002 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.954 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522333+0000", 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9823999999999997e-05 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522377+0000", 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.456e-05 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522413+0000", 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5136999999999999e-05 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.955 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261868 9.1 9:87e1d5c9:::benchmark_data_smithi169_135229_object117653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.517760+0000", 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 318.30163134600002, 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0123262909999999, 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261868 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.956 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517760+0000", 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517760+0000", 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517760+0000", 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4799999999999997e-07 2024-03-12T03:52:21.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517764+0000", 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9430000000000002e-06 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517774+0000", 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2769999999999994e-06 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517779+0000", 2024-03-12T03:52:21.958 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6350000000000003e-06 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517818+0000", 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8349999999999997e-05 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517823+0000", 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5260000000000001e-06 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.517013+0000", 2024-03-12T03:52:21.959 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.99919013499999998 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.517253+0000", 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00023998300000000001 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.517365+0000", 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112104 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522065+0000", 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046995939999999996 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522298+0000", 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00023326400000000001 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.529995+0000", 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076970169999999996 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:52:21.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.530044+0000", 2024-03-12T03:52:21.962 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8473000000000002e-05 2024-03-12T03:52:21.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:52:21.962 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:52:21.962 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:52:21.962 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.530086+0000", 2024-03-12T03:52:21.962 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2141999999999997e-05 2024-03-12T03:52:21.962 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.962 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.962 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.962 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:52:21.962 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:52:21.962 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:52:21.963 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:52:21.963 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:52:21.963 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-03-12T03:52:21.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:52:22.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:22.006 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:22.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:22.093 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 32 16 62368 62352 121.754 51.6875 0.00525381 0.00820715 2024-03-12T03:52:22.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:22.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:22.161 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:22.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:22.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[29344]: pgmap v925: 176 pgs: 3 peering, 6 unknown, 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 108 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s; 11590/180573 objects degraded (6.418%) 2024-03-12T03:52:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[29344]: 11.4 scrub starts 2024-03-12T03:52:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[29344]: 11.4 scrub ok 2024-03-12T03:52:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[29344]: osdmap e456: 8 total, 7 up, 8 in 2024-03-12T03:52:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[29344]: 4.12 scrub starts 2024-03-12T03:52:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[29344]: 4.12 scrub ok 2024-03-12T03:52:22.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2978684652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:52:22.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[39949]: pgmap v925: 176 pgs: 3 peering, 6 unknown, 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 108 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s; 11590/180573 objects degraded (6.418%) 2024-03-12T03:52:22.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[39949]: 11.4 scrub starts 2024-03-12T03:52:22.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[39949]: 11.4 scrub ok 2024-03-12T03:52:22.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[39949]: osdmap e456: 8 total, 7 up, 8 in 2024-03-12T03:52:22.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[39949]: 4.12 scrub starts 2024-03-12T03:52:22.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[39949]: 4.12 scrub ok 2024-03-12T03:52:22.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:21 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2978684652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:52:22.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:21 smithi173 ceph-mon[33544]: pgmap v925: 176 pgs: 3 peering, 6 unknown, 4 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 108 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s; 11590/180573 objects degraded (6.418%) 2024-03-12T03:52:22.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:21 smithi173 ceph-mon[33544]: 11.4 scrub starts 2024-03-12T03:52:22.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:21 smithi173 ceph-mon[33544]: 11.4 scrub ok 2024-03-12T03:52:22.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:21 smithi173 ceph-mon[33544]: osdmap e456: 8 total, 7 up, 8 in 2024-03-12T03:52:22.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:21 smithi173 ceph-mon[33544]: 4.12 scrub starts 2024-03-12T03:52:22.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:21 smithi173 ceph-mon[33544]: 4.12 scrub ok 2024-03-12T03:52:22.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:21 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2978684652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:52:22.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:22.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:22.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:22.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:22.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:22.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:22.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:22.528 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:22.569 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:22.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:22.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:22.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:22.712 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:22.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:22.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:22.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:22.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:22.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:22.947 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:22.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:22.971 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:22.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:23.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:23.093 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 33 16 63318 63302 119.864 59.375 0.00802631 0.00833766 2024-03-12T03:52:23.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:23.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:22 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:52:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:22 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:52:23.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:22 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:52:23.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:22 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:52:23.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:23.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:22 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:52:23.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:22 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:52:23.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:23.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:23.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:23.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:23.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:23.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:23.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:23.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:23.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:23.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:23.648 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:23.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:23.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:23.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:52:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:52:23.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:23.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:23.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:23.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:23.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:23.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:23.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:23.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:24.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:24.049 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:24.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:24.093 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 34 16 64294 64278 118.132 61 0.0138349 0.00844881 2024-03-12T03:52:24.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:24.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:23 smithi169 ceph-mon[29344]: pgmap v927: 176 pgs: 1 peering, 3 unknown, 4 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 23 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 108 active+clean; 3.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 21456/173307 objects degraded (12.380%); 7589/173307 objects misplaced (4.379%); 1.0 MiB/s, 16 objects/s recovering 2024-03-12T03:52:24.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:23 smithi169 ceph-mon[29344]: osdmap e457: 8 total, 7 up, 8 in 2024-03-12T03:52:24.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:23 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 21456/173307 objects degraded (12.380%), 25 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:23 smithi169 ceph-mon[39949]: pgmap v927: 176 pgs: 1 peering, 3 unknown, 4 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 23 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 108 active+clean; 3.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 21456/173307 objects degraded (12.380%); 7589/173307 objects misplaced (4.379%); 1.0 MiB/s, 16 objects/s recovering 2024-03-12T03:52:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:23 smithi169 ceph-mon[39949]: osdmap e457: 8 total, 7 up, 8 in 2024-03-12T03:52:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:23 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 21456/173307 objects degraded (12.380%), 25 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:24.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:24.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:24.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:23 smithi173 ceph-mon[33544]: pgmap v927: 176 pgs: 1 peering, 3 unknown, 4 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 23 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 108 active+clean; 3.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 21456/173307 objects degraded (12.380%); 7589/173307 objects misplaced (4.379%); 1.0 MiB/s, 16 objects/s recovering 2024-03-12T03:52:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:23 smithi173 ceph-mon[33544]: osdmap e457: 8 total, 7 up, 8 in 2024-03-12T03:52:24.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:23 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 21456/173307 objects degraded (12.380%), 25 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:24.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:24.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:24.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:24.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:24.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:24.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:24.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:24.549 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:24.589 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:24.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:24.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:24.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:24.721 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:24.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:24.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:24.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:24.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:24.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:24.977 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:24.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:24.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:25.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:25.093 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 35 16 65256 65240 116.475 60.125 0.00623613 0.00858018 2024-03-12T03:52:25.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:25.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:24 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:52:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:24 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:52:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:25.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:24 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:52:25.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:24 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:52:25.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:24 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:52:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:24 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:52:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:25.242 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:25.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:25.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:25.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:25.372 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:25.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:25.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:25.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:25.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:25.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:25.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:25.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:25.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:25.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:25.780 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:25.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:25.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:25.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:25.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:26.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:26.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:26.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:26.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:26.093 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 36 16 66088 66072 114.684 52 0.00839919 0.00871231 2024-03-12T03:52:26.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:26.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:25 smithi169 ceph-mon[29344]: pgmap v929: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 952 B/s rd, 0 op/s; 35136/189294 objects degraded (18.562%); 24601/189294 objects misplaced (12.996%); 3.0 MiB/s, 48 objects/s recovering 2024-03-12T03:52:26.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:25 smithi169 ceph-mon[29344]: osdmap e458: 8 total, 7 up, 8 in 2024-03-12T03:52:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:25 smithi169 ceph-mon[39949]: pgmap v929: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 952 B/s rd, 0 op/s; 35136/189294 objects degraded (18.562%); 24601/189294 objects misplaced (12.996%); 3.0 MiB/s, 48 objects/s recovering 2024-03-12T03:52:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:25 smithi169 ceph-mon[39949]: osdmap e458: 8 total, 7 up, 8 in 2024-03-12T03:52:26.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:26.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:26.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:25 smithi173 ceph-mon[33544]: pgmap v929: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 952 B/s rd, 0 op/s; 35136/189294 objects degraded (18.562%); 24601/189294 objects misplaced (12.996%); 3.0 MiB/s, 48 objects/s recovering 2024-03-12T03:52:26.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:25 smithi173 ceph-mon[33544]: osdmap e458: 8 total, 7 up, 8 in 2024-03-12T03:52:26.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:26.299 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:52:26.300 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:52:26.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:26.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:26.412 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:26.443 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:26.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:26.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:26.577 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:26.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:26.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:26.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:26.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:26.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:26.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:26.833 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:26.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:26.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:26.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:26.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:26.979 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:26.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:27.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:27.093 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 37 16 66947 66931 113.036 53.6875 0.00659528 0.00884141 2024-03-12T03:52:27.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:27.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:27.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:27.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:27.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:27.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:27.396 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:27.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:27.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:27.499 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:27.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:27.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:27.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:27.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:27.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:27.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:27.760 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:27.783 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:27.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:27.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:27.929 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:27.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:27.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:28.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:28.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:28.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:28.093 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 38 16 67843 67827 111.535 56 0.00416078 0.00896004 2024-03-12T03:52:28.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:28.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:27 smithi169 ceph-mon[29344]: pgmap v931: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 35136/189294 objects degraded (18.562%); 24601/189294 objects misplaced (12.996%); 2.7 MiB/s, 43 objects/s recovering 2024-03-12T03:52:28.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:27 smithi169 ceph-mon[39949]: pgmap v931: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 35136/189294 objects degraded (18.562%); 24601/189294 objects misplaced (12.996%); 2.7 MiB/s, 43 objects/s recovering 2024-03-12T03:52:28.194 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:28.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:28.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:27 smithi173 ceph-mon[33544]: pgmap v931: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 35136/189294 objects degraded (18.562%); 24601/189294 objects misplaced (12.996%); 2.7 MiB/s, 43 objects/s recovering 2024-03-12T03:52:28.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:28.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:52:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:28.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:28.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:28.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:28.500 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:28.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:28.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:28.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:28.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:28.734 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:28.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:28.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:28.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:28.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:28.915 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:28.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:28.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:29.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:29.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:29.041 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:29.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:29.093 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 39 16 68600 68584 109.888 47.3125 0.0112027 0.00909314 2024-03-12T03:52:29.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:29.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:28 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 35136/189294 objects degraded (18.562%), 30 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:29.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:28 smithi169 ceph-mon[39949]: osdmap e459: 8 total, 7 up, 8 in 2024-03-12T03:52:29.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:28 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 35136/189294 objects degraded (18.562%), 30 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:29.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:28 smithi169 ceph-mon[29344]: osdmap e459: 8 total, 7 up, 8 in 2024-03-12T03:52:29.192 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:29.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:29.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:28 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 35136/189294 objects degraded (18.562%), 30 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:29.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:28 smithi173 ceph-mon[33544]: osdmap e459: 8 total, 7 up, 8 in 2024-03-12T03:52:29.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:29.334 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:29.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:29.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:29.487 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:29.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:29.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:29.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:29.613 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:29.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:29.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:29.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:29.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:29.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:29.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:29.894 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:29.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:29.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:29 smithi173 ceph-mon[33544]: pgmap v932: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 35678/198471 objects degraded (17.976%); 23857/198471 objects misplaced (12.020%); 7.1 MiB/s, 114 objects/s recovering 2024-03-12T03:52:29.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:30.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:30.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:30.093 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:52:30.092200+0000 min lat: 0.00121451 max lat: 2.38513 avg lat: 0.00920796 2024-03-12T03:52:30.094 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:52:30.094 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 40 16 69496 69480 108.541 56 0.0271983 0.00920796 2024-03-12T03:52:30.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:30.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:30.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:29 smithi169 ceph-mon[29344]: pgmap v932: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 35678/198471 objects degraded (17.976%); 23857/198471 objects misplaced (12.020%); 7.1 MiB/s, 114 objects/s recovering 2024-03-12T03:52:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:29 smithi169 ceph-mon[39949]: pgmap v932: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 35678/198471 objects degraded (17.976%); 23857/198471 objects misplaced (12.020%); 7.1 MiB/s, 114 objects/s recovering 2024-03-12T03:52:30.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:30.291 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:30.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:30.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:30.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:30.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:30.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:30.518 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:30.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:30.560 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:30.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:30.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:30.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:30.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:30.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:30.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:30.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:30.970 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:30.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:31.072 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:31.096 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 41 16 70270 70254 107.067 48.375 0.0162433 0.009333 2024-03-12T03:52:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:31.137 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:31.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:31.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:31.270 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:31.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:31.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:31.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:31.407 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:31.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:31.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:31.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:31.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:31.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:31.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:31.687 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:31.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:31.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:31.831 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:31.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:31.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:31.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:31.968 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:31.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:32.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:32.096 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 42 16 71262 71246 105.994 62 0.00780569 0.00942973 2024-03-12T03:52:32.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:32.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:32.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:32.110 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:31 smithi169 ceph-mon[29344]: pgmap v934: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 696 B/s rd, 51 MiB/s wr, 817 op/s; 36226/202680 objects degraded (17.873%); 23477/202680 objects misplaced (11.583%); 8.7 MiB/s, 139 objects/s recovering 2024-03-12T03:52:32.110 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:31 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:32.110 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:31 smithi169 ceph-mon[39949]: pgmap v934: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 696 B/s rd, 51 MiB/s wr, 817 op/s; 36226/202680 objects degraded (17.873%); 23477/202680 objects misplaced (11.583%); 8.7 MiB/s, 139 objects/s recovering 2024-03-12T03:52:32.110 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:31 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:32.111 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:52:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:52:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:52:32.122 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:52:32.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-12T03:52:32.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:32.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:32.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:32.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:31 smithi173 ceph-mon[33544]: pgmap v934: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 696 B/s rd, 51 MiB/s wr, 817 op/s; 36226/202680 objects degraded (17.873%); 23477/202680 objects misplaced (11.583%); 8.7 MiB/s, 139 objects/s recovering 2024-03-12T03:52:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:31 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:32.320 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:52:32.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:32.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:32.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:32.377 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:52:32.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:32.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:32.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:32.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:32.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:32.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:32.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:32.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:32.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:32.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:32.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:32.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:32.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:33.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:33.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:33.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:33.096 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 43 16 72136 72120 104.8 54.625 0.0138142 0.0095373 2024-03-12T03:52:33.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:33.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:33.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:33.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:33.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:33.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:33.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:33.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:33.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:33.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:33.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:33.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:33.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:33.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:33.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:33.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:33.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:33.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:52:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:52:33.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:33.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:33.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:33.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:33.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:33.959 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:33.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:34.061 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:34.097 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 44 16 73097 73081 103.781 60.0625 0.00700478 0.00963021 2024-03-12T03:52:34.102 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:34.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:34.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:33 smithi169 ceph-mon[29344]: pgmap v935: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 812 op/s; 36632/208785 objects degraded (17.545%); 22933/208785 objects misplaced (10.984%); 9.7 MiB/s, 154 objects/s recovering 2024-03-12T03:52:34.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:33 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:52:34.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:33 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:52:34.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:33 smithi169 ceph-mon[29344]: osdmap e460: 8 total, 7 up, 8 in 2024-03-12T03:52:34.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:33 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 36632/208785 objects degraded (17.545%), 30 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:34.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:33 smithi169 ceph-mon[29344]: osdmap e461: 8 total, 7 up, 8 in 2024-03-12T03:52:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:33 smithi169 ceph-mon[39949]: pgmap v935: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 812 op/s; 36632/208785 objects degraded (17.545%); 22933/208785 objects misplaced (10.984%); 9.7 MiB/s, 154 objects/s recovering 2024-03-12T03:52:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:33 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:52:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:33 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:52:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:33 smithi169 ceph-mon[39949]: osdmap e460: 8 total, 7 up, 8 in 2024-03-12T03:52:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:33 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 36632/208785 objects degraded (17.545%), 30 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:34.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:33 smithi169 ceph-mon[39949]: osdmap e461: 8 total, 7 up, 8 in 2024-03-12T03:52:34.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:34.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:34.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:34.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:33 smithi173 ceph-mon[33544]: pgmap v935: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 812 op/s; 36632/208785 objects degraded (17.545%); 22933/208785 objects misplaced (10.984%); 9.7 MiB/s, 154 objects/s recovering 2024-03-12T03:52:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:33 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:52:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:33 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:52:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:33 smithi173 ceph-mon[33544]: osdmap e460: 8 total, 7 up, 8 in 2024-03-12T03:52:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:33 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 36632/208785 objects degraded (17.545%), 30 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:33 smithi173 ceph-mon[33544]: osdmap e461: 8 total, 7 up, 8 in 2024-03-12T03:52:34.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:34.360 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:34.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:34.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:34.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:34.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:34.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:34.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:34.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:34.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:34.755 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:34.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:34.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:34.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:34.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:34.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:34.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:35.017 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:35.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:35.097 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 45 16 74038 74022 102.782 58.8125 0.01087 0.0097242 2024-03-12T03:52:35.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:35.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:35.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:35.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:35.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:35.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:35.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:35.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:35.443 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:35.443 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:35.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:35.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:35.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:35.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:35.615 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:35.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:35.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:35.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:35.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:35.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:35.883 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:35.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:35.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:36.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:36.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:36.097 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 46 16 74936 74920 101.768 56.125 0.00474321 0.00982103 2024-03-12T03:52:36.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:36.145 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:36.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:36.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:36.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:35 smithi169 ceph-mon[29344]: pgmap v938: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 963 op/s; 37329/215802 objects degraded (17.298%); 22336/215802 objects misplaced (10.350%); 11 MiB/s, 182 objects/s recovering 2024-03-12T03:52:36.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:35 smithi169 ceph-mon[39949]: pgmap v938: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 963 op/s; 37329/215802 objects degraded (17.298%); 22336/215802 objects misplaced (10.350%); 11 MiB/s, 182 objects/s recovering 2024-03-12T03:52:36.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:35 smithi173 ceph-mon[33544]: pgmap v938: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 963 op/s; 37329/215802 objects degraded (17.298%); 22336/215802 objects misplaced (10.350%); 11 MiB/s, 182 objects/s recovering 2024-03-12T03:52:36.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:36.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:36.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:36.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:36.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:36.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:36.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:36.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:36.554 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:36.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:36.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:36.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:36.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:36.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:36.837 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:36.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:36.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:36.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:36.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:36.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:37.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:37.097 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 47 16 75860 75844 100.831 57.75 0.017601 0.00991168 2024-03-12T03:52:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:37.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:37.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:37.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:37.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:37.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:37.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:37.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:37.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:37.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:37.515 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:37.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:37.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:37.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:37.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:37.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:37.752 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:37.786 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:37.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:37.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:37.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:37.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:38.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:38.065 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:38.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:38.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:38.097 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 48 16 76835 76819 100 60.9375 0.0144457 0.00999471 2024-03-12T03:52:38.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:38.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:38.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:38.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:37 smithi173 ceph-mon[33544]: pgmap v939: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 663 B/s rd, 47 MiB/s wr, 749 op/s; 37329/215802 objects degraded (17.298%); 22336/215802 objects misplaced (10.350%); 8.9 MiB/s, 141 objects/s recovering 2024-03-12T03:52:38.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:38.344 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:38.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:38.352 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:37 smithi169 ceph-mon[29344]: pgmap v939: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 663 B/s rd, 47 MiB/s wr, 749 op/s; 37329/215802 objects degraded (17.298%); 22336/215802 objects misplaced (10.350%); 8.9 MiB/s, 141 objects/s recovering 2024-03-12T03:52:38.353 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:37 smithi169 ceph-mon[39949]: pgmap v939: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 663 B/s rd, 47 MiB/s wr, 749 op/s; 37329/215802 objects degraded (17.298%); 22336/215802 objects misplaced (10.350%); 8.9 MiB/s, 141 objects/s recovering 2024-03-12T03:52:38.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:38.476 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:38.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:38.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:38.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:38.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:38.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:38.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:38.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:38.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:38.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:38.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:38.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:38.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:38.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:38.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:38.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:39.019 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:39.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:39.097 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 49 16 77769 77753 99.1508 58.375 0.00954786 0.0100805 2024-03-12T03:52:39.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:39.164 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:39.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:38 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 37329/215802 objects degraded (17.298%), 30 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:39.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:39.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:39.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:39.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:39.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:38 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 37329/215802 objects degraded (17.298%), 30 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:39.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:38 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 37329/215802 objects degraded (17.298%), 30 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:39.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:39.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:39.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:39.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:39.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:39.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:39.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:39.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:39.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:39.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:39.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:39.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:39.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:39.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:39.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:40.073 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:40.097 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 50 16 78608 78592 98.2165 52.4375 0.0130571 0.0101765 2024-03-12T03:52:40.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:40.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:40.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:40.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:40.238 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:40.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:40.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:40.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:40.373 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:40.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:40.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:39 smithi169 ceph-mon[29344]: pgmap v940: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 950 op/s; 37909/225474 objects degraded (16.813%); 21444/225474 objects misplaced (9.511%); 12 MiB/s, 191 objects/s recovering 2024-03-12T03:52:40.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:39 smithi169 ceph-mon[39949]: pgmap v940: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 950 op/s; 37909/225474 objects degraded (16.813%); 21444/225474 objects misplaced (9.511%); 12 MiB/s, 191 objects/s recovering 2024-03-12T03:52:40.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:39 smithi173 ceph-mon[33544]: pgmap v940: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 950 op/s; 37909/225474 objects degraded (16.813%); 21444/225474 objects misplaced (9.511%); 12 MiB/s, 191 objects/s recovering 2024-03-12T03:52:40.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:40.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:40.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:40.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:40.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:40.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:40.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:40.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:40.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:40.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:40.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:40.916 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:40.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:41.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:41.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:41.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:41.097 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 51 16 79551 79535 97.4464 58.9375 0.00546671 0.0102559 2024-03-12T03:52:41.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:41.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:41.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:41.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:41.314 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:41.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:41.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:41.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:41 smithi169 ceph-mon[29344]: pgmap v941: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 877 op/s; 38478/229839 objects degraded (16.741%); 21028/229839 objects misplaced (9.149%); 11 MiB/s, 175 objects/s recovering 2024-03-12T03:52:41.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:41 smithi169 ceph-mon[39949]: pgmap v941: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 877 op/s; 38478/229839 objects degraded (16.741%); 21028/229839 objects misplaced (9.149%); 11 MiB/s, 175 objects/s recovering 2024-03-12T03:52:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:41.443 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:41.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:41.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:41 smithi173 ceph-mon[33544]: pgmap v941: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 877 op/s; 38478/229839 objects degraded (16.741%); 21028/229839 objects misplaced (9.149%); 11 MiB/s, 175 objects/s recovering 2024-03-12T03:52:41.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:41.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:41.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:41.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:41.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:41.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:41.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:41.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:41.842 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:41.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:41.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:41.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:41.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:41.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:42.083 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:42.098 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 52 16 80291 80275 96.4611 46.25 0.0113499 0.0103611 2024-03-12T03:52:42.125 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:42.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:42.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:42.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:52:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:52:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:52:42.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:42.254 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:42.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:42.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:42.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:42.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:42.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:42.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:42.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:42.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:42.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:42.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:42.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:42.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:42.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:42.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:42.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:42.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:42.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:42.925 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:42.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:43.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:43.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:43.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:43.098 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 53 16 81149 81133 95.6529 53.625 0.00982701 0.0104496 2024-03-12T03:52:43.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:43.195 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:43.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:43.296 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:43.328 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:43.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:43.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:43.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:43.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:43.564 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:43.595 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:43.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:43.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:43.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:43 smithi169 ceph-mon[29344]: pgmap v942: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 874 op/s; 38856/235989 objects degraded (16.465%); 20290/235989 objects misplaced (8.598%); 12 MiB/s, 193 objects/s recovering 2024-03-12T03:52:43.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:43 smithi169 ceph-mon[39949]: pgmap v942: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 874 op/s; 38856/235989 objects degraded (16.465%); 20290/235989 objects misplaced (8.598%); 12 MiB/s, 193 objects/s recovering 2024-03-12T03:52:43.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:43.725 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:43.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:43.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:43.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:43 smithi173 ceph-mon[33544]: pgmap v942: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 874 op/s; 38856/235989 objects degraded (16.465%); 20290/235989 objects misplaced (8.598%); 12 MiB/s, 193 objects/s recovering 2024-03-12T03:52:43.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:52:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:52:43.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:43.856 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:43.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:43.957 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:43.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:43.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:44.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:44.099 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 54 16 81809 81793 94.6456 41.25 0.00595894 0.01056 2024-03-12T03:52:44.126 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:44.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:44.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:44.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:44.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:44.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:44.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:44.402 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:44.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:44.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:44.528 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:44.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:44.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:44.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:44.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:44 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 38856/235989 objects degraded (16.465%), 30 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:44.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:44.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:44 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 38856/235989 objects degraded (16.465%), 30 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:44.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:44.675 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:44.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:44.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:44 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 38856/235989 objects degraded (16.465%), 30 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:44.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:44.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:44.819 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:44.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:44.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:44.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:44.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:44.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:45.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:45.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:45.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:45.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:45.099 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 55 16 82739 82723 93.9805 58.125 0.0155442 0.0106347 2024-03-12T03:52:45.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:45.236 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:45.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:45.306 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:45 smithi169 ceph-mon[29344]: pgmap v943: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 954 B/s rd, 61 MiB/s wr, 971 op/s; 39545/242835 objects degraded (16.285%); 19612/242835 objects misplaced (8.076%); 13 MiB/s, 211 objects/s recovering 2024-03-12T03:52:45.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:45.384 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:45.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:45.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:45.486 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:45.533 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:45.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:45.599 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:45 smithi169 ceph-mon[39949]: pgmap v943: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 954 B/s rd, 61 MiB/s wr, 971 op/s; 39545/242835 objects degraded (16.285%); 19612/242835 objects misplaced (8.076%); 13 MiB/s, 211 objects/s recovering 2024-03-12T03:52:45.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:45.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:45.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:45 smithi173 ceph-mon[33544]: pgmap v943: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 954 B/s rd, 61 MiB/s wr, 971 op/s; 39545/242835 objects degraded (16.285%); 19612/242835 objects misplaced (8.076%); 13 MiB/s, 211 objects/s recovering 2024-03-12T03:52:45.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:45.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:45.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:45.917 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:45.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:46.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:46.099 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 56 16 83761 83745 93.4429 63.875 0.00900596 0.0106966 2024-03-12T03:52:46.100 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:46.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:46.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:46.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:46.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:46.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:46.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:46.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:46.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:46.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:46.506 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:46.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:46.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:46.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:46.644 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:46.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:46.745 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:46.787 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:46.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:46.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:46.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:46.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:46.930 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:46.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:47.031 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:47.091 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:47.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:47.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:47.100 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 57 16 84830 84814 92.9755 66.8125 0.0170127 0.0107503 2024-03-12T03:52:47.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:47.260 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:47.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:47.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:47.362 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:47.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:47.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:47.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:47.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:47.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:47.619 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:47 smithi169 ceph-mon[29344]: pgmap v944: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 47 MiB/s wr, 751 op/s; 39545/242835 objects degraded (16.285%); 19612/242835 objects misplaced (8.076%); 10 MiB/s, 167 objects/s recovering 2024-03-12T03:52:47.620 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:47 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:47.620 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:47 smithi169 ceph-mon[39949]: pgmap v944: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 47 MiB/s wr, 751 op/s; 39545/242835 objects degraded (16.285%); 19612/242835 objects misplaced (8.076%); 10 MiB/s, 167 objects/s recovering 2024-03-12T03:52:47.620 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:47 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:47.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:47.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:47.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:47.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:47 smithi173 ceph-mon[33544]: pgmap v944: 176 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 47 MiB/s wr, 751 op/s; 39545/242835 objects degraded (16.285%); 19612/242835 objects misplaced (8.076%); 10 MiB/s, 167 objects/s recovering 2024-03-12T03:52:47.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:47 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:47.768 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:47.818 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:47.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:47.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:47.920 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:47.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:47.961 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:47.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:48.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:48.103 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 58 16 85742 85726 92.3547 57 0.0145331 0.0108225 2024-03-12T03:52:48.113 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:48.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:48.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:48.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:48.243 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:48.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:48.260 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:52:48.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:48.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:48.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:48.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:48.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:48.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:48.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:48.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:48.625 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:52:48.630 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:52:48.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:48.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:48.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:48.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:48.777 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:48.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:48.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:48.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:48.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:48.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:49.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:49.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:49.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:49.099 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 59 16 86569 86553 91.6655 51.6875 0.0214653 0.0109034 2024-03-12T03:52:49.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:49.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:49.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:49.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:49.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:49.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:49.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:49.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:49.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:49.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:49.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:49.480 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[39949]: pgmap v945: 176 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 38209/252276 objects degraded (15.146%); 20328/252276 objects misplaced (8.058%); 15 MiB/s, 243 objects/s recovering 2024-03-12T03:52:49.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[39949]: osdmap e462: 8 total, 7 up, 8 in 2024-03-12T03:52:49.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 38209/252276 objects degraded (15.146%), 29 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:49.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[39949]: osdmap e463: 8 total, 7 up, 8 in 2024-03-12T03:52:49.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/645735460' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:52:49.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:52:49.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[39949]: from='client.25615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:52:49.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[29344]: pgmap v945: 176 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 38209/252276 objects degraded (15.146%); 20328/252276 objects misplaced (8.058%); 15 MiB/s, 243 objects/s recovering 2024-03-12T03:52:49.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[29344]: osdmap e462: 8 total, 7 up, 8 in 2024-03-12T03:52:49.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 38209/252276 objects degraded (15.146%), 29 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:49.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[29344]: osdmap e463: 8 total, 7 up, 8 in 2024-03-12T03:52:49.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/645735460' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:52:49.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:52:49.483 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:49 smithi169 ceph-mon[29344]: from='client.25615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:52:49.530 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:52:49.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:49.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:49.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:49.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:49.712 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:49.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:49.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:49 smithi173 ceph-mon[33544]: pgmap v945: 176 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 38209/252276 objects degraded (15.146%); 20328/252276 objects misplaced (8.058%); 15 MiB/s, 243 objects/s recovering 2024-03-12T03:52:49.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:49 smithi173 ceph-mon[33544]: osdmap e462: 8 total, 7 up, 8 in 2024-03-12T03:52:49.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:49 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 38209/252276 objects degraded (15.146%), 29 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:49.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:49 smithi173 ceph-mon[33544]: osdmap e463: 8 total, 7 up, 8 in 2024-03-12T03:52:49.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:49 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/645735460' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:52:49.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:49 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:52:49.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:49 smithi173 ceph-mon[33544]: from='client.25615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:52:49.814 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:49.874 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:49.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:49.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:50.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:50.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:50.100 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:52:50.098494+0000 min lat: 0.00121451 max lat: 2.38513 avg lat: 0.010968 2024-03-12T03:52:50.100 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:52:50.100 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 60 16 87519 87503 91.1273 59.375 0.00876844 0.010968 2024-03-12T03:52:50.111 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:50.150 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:50.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:50.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:50.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:50.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:50.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:50.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:50.433 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:50.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:50.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:50.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:50.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:50.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:50.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:50 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:52:50.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:50 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:52:50.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:50 smithi169 ceph-mon[39949]: osdmap e464: 8 total, 7 up, 8 in 2024-03-12T03:52:50.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:50 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:52:50.652 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:50 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:52:50.653 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:50 smithi169 ceph-mon[29344]: osdmap e464: 8 total, 7 up, 8 in 2024-03-12T03:52:50.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:50.699 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:50.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:50 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:52:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:50 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:52:50.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:50 smithi173 ceph-mon[33544]: osdmap e464: 8 total, 7 up, 8 in 2024-03-12T03:52:50.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:50.840 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:50.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:50.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:50.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:50.984 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:50.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:50.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:51.087 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:51.100 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 61 16 88457 88441 90.5944 58.625 0.00979802 0.0110325 2024-03-12T03:52:51.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:51.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:51.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:51.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:51.266 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:51.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:51.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:51.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:51.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:51.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:51.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:51.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:51.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:51.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:51.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:51.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:51.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:51.780 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:51 smithi169 ceph-mon[29344]: pgmap v949: 176 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 48 MiB/s wr, 770 op/s; 38824/256701 objects degraded (15.124%); 19845/256701 objects misplaced (7.731%); 13 MiB/s, 206 objects/s recovering 2024-03-12T03:52:51.780 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:51 smithi169 ceph-mon[39949]: pgmap v949: 176 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 48 MiB/s wr, 770 op/s; 38824/256701 objects degraded (15.124%); 19845/256701 objects misplaced (7.731%); 13 MiB/s, 206 objects/s recovering 2024-03-12T03:52:51.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:51.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:51.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:51.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:51.925 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:51.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:51 smithi173 ceph-mon[33544]: pgmap v949: 176 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 48 MiB/s wr, 770 op/s; 38824/256701 objects degraded (15.124%); 19845/256701 objects misplaced (7.731%); 13 MiB/s, 206 objects/s recovering 2024-03-12T03:52:52.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:52.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:52.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:52.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:52.100 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 62 16 89277 89261 89.9599 51.25 0.012847 0.0111107 2024-03-12T03:52:52.164 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:52.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:52:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:52:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:52:52.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:52.203 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:52.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:52.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:52.351 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:52.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:52.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:52.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:52.497 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:52.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:52.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:52.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:52.665 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:52.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:52.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:52.791 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:52.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:52.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:52.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:52.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:52.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:52.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:53.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:53.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:53.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:53.100 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 63 16 90198 90182 89.4456 57.5625 0.0163434 0.0111744 2024-03-12T03:52:53.164 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:53.197 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:53.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:53.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:53.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:53.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:53.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:53.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:53.472 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:53.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:53.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:53.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:53.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:53.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:53.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:53.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:53.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:52:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:52:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:53 smithi173 ceph-mon[33544]: pgmap v950: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 36458/263034 objects degraded (13.861%); 22202/263034 objects misplaced (8.441%); 15 MiB/s, 243 objects/s recovering 2024-03-12T03:52:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:53 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 36458/263034 objects degraded (13.861%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:53.745 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:53.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:53.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:53.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:53.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:53.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:53 smithi169 ceph-mon[29344]: pgmap v950: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 36458/263034 objects degraded (13.861%); 22202/263034 objects misplaced (8.441%); 15 MiB/s, 243 objects/s recovering 2024-03-12T03:52:53.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:53 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 36458/263034 objects degraded (13.861%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:53.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:53 smithi169 ceph-mon[39949]: pgmap v950: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 36458/263034 objects degraded (13.861%); 22202/263034 objects misplaced (8.441%); 15 MiB/s, 243 objects/s recovering 2024-03-12T03:52:53.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:53 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 36458/263034 objects degraded (13.861%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:53.979 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:54.025 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:54.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:54.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:54.100 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 64 16 91109 91093 88.9377 56.9375 0.00875591 0.0112385 2024-03-12T03:52:54.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:54.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:54.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:54.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:54.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:54.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:54.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:54.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:54.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:54.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:54.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:54.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:54.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:54.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:54.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:54.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:54.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:54.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:54.917 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:54 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:54 smithi169 ceph-mon[29344]: osdmap e465: 8 total, 7 up, 8 in 2024-03-12T03:52:54.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:54 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:54 smithi169 ceph-mon[39949]: osdmap e465: 8 total, 7 up, 8 in 2024-03-12T03:52:54.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:54.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:54.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:54 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:54 smithi173 ceph-mon[33544]: osdmap e465: 8 total, 7 up, 8 in 2024-03-12T03:52:55.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:55.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:55.072 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:55.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:55.100 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 65 16 91939 91923 88.3675 51.875 0.00342853 0.0112994 2024-03-12T03:52:55.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:55.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:55.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:55.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:55.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:55.350 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:55.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:55.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:55.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:55.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:55.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:55.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:55.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:55.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:55.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:55.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:55.765 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:55.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:55.818 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:55 smithi169 ceph-mon[29344]: pgmap v952: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 866 B/s rd, 63 MiB/s wr, 1.01k op/s; 37080/270141 objects degraded (13.726%); 21071/270141 objects misplaced (7.800%); 13 MiB/s, 214 objects/s recovering 2024-03-12T03:52:55.819 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:55 smithi169 ceph-mon[39949]: pgmap v952: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 866 B/s rd, 63 MiB/s wr, 1.01k op/s; 37080/270141 objects degraded (13.726%); 21071/270141 objects misplaced (7.800%); 13 MiB/s, 214 objects/s recovering 2024-03-12T03:52:55.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:55.894 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:55.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:55.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:55 smithi173 ceph-mon[33544]: pgmap v952: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 866 B/s rd, 63 MiB/s wr, 1.01k op/s; 37080/270141 objects degraded (13.726%); 21071/270141 objects misplaced (7.800%); 13 MiB/s, 214 objects/s recovering 2024-03-12T03:52:55.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:56.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:56.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:56.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:56.100 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 66 16 92870 92854 87.9102 58.1875 0.0123506 0.0113702 2024-03-12T03:52:56.132 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:56.156 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:56.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:56.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:56.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:56.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:56.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:56.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:56.441 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:56.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:56.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:56.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:56.590 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:56.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:56.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:56.721 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:56.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:56.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:56.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:56.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:56.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:56.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:56.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:57.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:57.102 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 67 16 93770 93754 87.4353 56.25 0.0225754 0.0114321 2024-03-12T03:52:57.126 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:57.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:57.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:57.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:57.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:57.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:57.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:57.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:57.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:57.402 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:57.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:57.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:57.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:57.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:57.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:57.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:57.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:57.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:57.759 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:57.802 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:57.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:57.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:57 smithi169 ceph-mon[29344]: pgmap v953: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 663 B/s rd, 48 MiB/s wr, 771 op/s; 37080/270141 objects degraded (13.726%); 21071/270141 objects misplaced (7.800%); 10 MiB/s, 164 objects/s recovering 2024-03-12T03:52:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:57 smithi169 ceph-mon[39949]: pgmap v953: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 663 B/s rd, 48 MiB/s wr, 771 op/s; 37080/270141 objects degraded (13.726%); 21071/270141 objects misplaced (7.800%); 10 MiB/s, 164 objects/s recovering 2024-03-12T03:52:57.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:57.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:57 smithi173 ceph-mon[33544]: pgmap v953: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 663 B/s rd, 48 MiB/s wr, 771 op/s; 37080/270141 objects degraded (13.726%); 21071/270141 objects misplaced (7.800%); 10 MiB/s, 164 objects/s recovering 2024-03-12T03:52:58.037 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:58.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:58.078 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:58.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:58.102 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 68 16 94689 94673 86.9942 57.4375 0.0150474 0.01149 2024-03-12T03:52:58.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:52:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:58.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:58.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:52:58.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:58.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:58.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:58.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:58.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:58.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:58.482 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:58.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:58.583 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:52:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:58.632 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:58.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:52:58.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:52:58.776 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:58.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:52:58.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:52:58.913 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:58.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:52:58.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:58 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 37080/270141 objects degraded (13.726%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:58 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 37080/270141 objects degraded (13.726%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:58 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 37080/270141 objects degraded (13.726%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:52:59.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:59.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:59.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:59.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:59.102 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 69 16 95607 95591 86.5649 57.375 0.0100617 0.0115467 2024-03-12T03:52:59.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:59.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:59.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:59.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:52:59.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:59.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:52:59.426 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:59.469 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:59.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:59.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:59.616 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:59.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:59.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:52:59.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:52:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:59.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:52:59.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:52:59.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:52:59 smithi173 ceph-mon[33544]: pgmap v954: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 54 MiB/s wr, 866 op/s; 37519/279357 objects degraded (13.430%); 20139/279357 objects misplaced (7.209%); 11 MiB/s, 175 objects/s recovering 2024-03-12T03:52:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:52:59.899 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:52:59.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:52:59.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:52:59 smithi169 ceph-mon[29344]: pgmap v954: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 54 MiB/s wr, 866 op/s; 37519/279357 objects degraded (13.430%); 20139/279357 objects misplaced (7.209%); 11 MiB/s, 175 objects/s recovering 2024-03-12T03:52:59.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:52:59 smithi169 ceph-mon[39949]: pgmap v954: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 54 MiB/s wr, 866 op/s; 37519/279357 objects degraded (13.430%); 20139/279357 objects misplaced (7.209%); 11 MiB/s, 175 objects/s recovering 2024-03-12T03:53:00.000 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:00.046 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:00.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:00.102 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 70 16 96610 96594 86.2238 62.6875 0.0125437 0.0115924 2024-03-12T03:53:00.148 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:00.191 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:00.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:00.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:00.318 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:00.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:00.419 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:00.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:00.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:00.553 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:00.577 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:00.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:00.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:00.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:00.713 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:00.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:00.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:00.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:00.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:00.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:00.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:00.989 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:00.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:01.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:01.102 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 71 16 97386 97370 85.6924 48.5 0.00519124 0.0116642 2024-03-12T03:53:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:01.133 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:01.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:01.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:01.280 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:01.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:01.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:01.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:01.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:01.520 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:01.551 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:01.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:01.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:01.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:01.695 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:01.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:01.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:01.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:01.832 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:01 smithi169 ceph-mon[29344]: pgmap v955: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 896 op/s; 38037/283575 objects degraded (13.413%); 19595/283575 objects misplaced (6.910%); 12 MiB/s, 189 objects/s recovering 2024-03-12T03:53:01.833 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:01 smithi169 ceph-mon[39949]: pgmap v955: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 896 op/s; 38037/283575 objects degraded (13.413%); 19595/283575 objects misplaced (6.910%); 12 MiB/s, 189 objects/s recovering 2024-03-12T03:53:01.837 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:01.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:01.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:01.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:01.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:01.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:01 smithi173 ceph-mon[33544]: pgmap v955: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 896 op/s; 38037/283575 objects degraded (13.413%); 19595/283575 objects misplaced (6.910%); 12 MiB/s, 189 objects/s recovering 2024-03-12T03:53:02.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:02.103 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 72 16 98296 98280 85.2919 56.875 0.0126181 0.0117187 2024-03-12T03:53:02.106 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:02.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:02.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:53:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:53:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:53:02.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:02.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:02.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:02.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:02.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:02.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:02.466 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:02.507 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:02.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:02.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:02.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:02.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:02.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:02.774 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:02.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:02.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:02.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:02.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:03.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:03.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:03.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:03.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:03.074 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:02 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:53:03.074 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:02 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:03.075 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:02 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:53:03.075 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:02 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:03.103 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 73 16 99017 99001 84.7408 45.0625 0.0213528 0.0117957 2024-03-12T03:53:03.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:03.176 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:03.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:03.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:02 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:53:03.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:02 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:03.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:03.324 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:03.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:03.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:03.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:03.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:03.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:03.606 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:03.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:03.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:03.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:53:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:53:03.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:03.747 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:03.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:03.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:03.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:03.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:03.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:03.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:04.015 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:04.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:04.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:04.103 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 74 16 100028 100012 84.4495 63.1875 0.00926386 0.0118369 2024-03-12T03:53:04.116 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:04.165 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:04.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:04.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:04.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:03 smithi169 ceph-mon[29344]: pgmap v956: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 901 op/s; 38356/290049 objects degraded (13.224%); 19293/290049 objects misplaced (6.652%); 12 MiB/s, 189 objects/s recovering 2024-03-12T03:53:04.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:03 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 38356/290049 objects degraded (13.224%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:04.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:03 smithi169 ceph-mon[39949]: pgmap v956: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 901 op/s; 38356/290049 objects degraded (13.224%); 19293/290049 objects misplaced (6.652%); 12 MiB/s, 189 objects/s recovering 2024-03-12T03:53:04.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:03 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 38356/290049 objects degraded (13.224%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:04.220 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":947,"stamp":"2024-03-12T03:52:48.519761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5496537723,"num_objects":84092,"num_object_clones":0,"num_object_copies":252276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38209,"num_objects_misplaced":20328,"num_objects_unfound":0,"num_objects_dirty":84092,"num_whiteouts":0,"num_read":2575,"num_read_kb":2299,"num_write":73353,"num_write_kb":4635675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5046,"num_bytes_recovered":319640332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118438,"ondisk_log_size":118438,"up":465,"acting":463,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":553,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":24465060,"kb_used_data":18402212,"kb_used_omap":24,"kb_used_meta":6062759,"kb_avail":725496156,"statfs":{"total":767960285184,"available":742908063744,"internally_reserved":0,"allocated":18843865088,"data_stored":18838602915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24644,"internal_metadata":6208266172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,5,9,7,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":490143744,"num_objects":7479,"num_object_clones":0,"num_object_copies":22437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-269,"num_objects_misplaced":-700,"num_objects_unfound":0,"num_objects_dirty":7479,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7479,"num_write_kb":478656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1953,"num_bytes_recovered":127991808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.289890"},"pg_stats":[{"pgid":"11.3","version":"461'4452","reported_seq":4542,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:46.447405+0000","last_change":"2024-03-12T03:52:18.834008+0000","last_active":"2024-03-12T03:52:46.447405+0000","last_peered":"2024-03-12T03:52:46.447405+0000","last_clean":"2024-03-12T03:52:46.447405+0000","last_became_active":"2024-03-12T03:52:17.865581+0000","last_became_peered":"2024-03-12T03:52:17.865581+0000","last_unstale":"2024-03-12T03:52:46.447405+0000","last_undegraded":"2024-03-12T03:52:46.447405+0000","last_fullsized":"2024-03-12T03:52:46.447405+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'3733","last_scrub_stamp":"2024-03-12T03:52:18.833641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:52:18.833641+0000","objects_scrubbed":1885,"log_size":4452,"log_dups_size":0,"ondisk_log_size":4452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:36.920158+0000","scrub_duration":439,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171376640,"num_objects":2615,"num_object_clones":0,"num_object_copies":7845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":730,"num_write_kb":46720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1180,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.525798+0000","last_change":"2024-03-12T03:50:53.286652+0000","last_active":"2024-03-12T03:52:33.525798+0000","last_peered":"2024-03-12T03:52:33.525798+0000","last_clean":"2024-03-12T03:52:33.525798+0000","last_became_active":"2024-03-12T03:46:25.289005+0000","last_became_peered":"2024-03-12T03:46:25.289005+0000","last_unstale":"2024-03-12T03:52:33.525798+0000","last_undegraded":"2024-03-12T03:52:33.525798+0000","last_fullsized":"2024-03-12T03:52:33.525798+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:53.286515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.286515+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"344'25","reported_seq":722,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.522582+0000","last_change":"2024-03-12T03:51:46.582688+0000","last_active":"2024-03-12T03:52:33.522582+0000","last_peered":"2024-03-12T03:52:33.522582+0000","last_clean":"2024-03-12T03:52:33.522582+0000","last_became_active":"2024-03-12T03:51:42.838917+0000","last_became_peered":"2024-03-12T03:51:42.838917+0000","last_unstale":"2024-03-12T03:52:33.522582+0000","last_undegraded":"2024-03-12T03:52:33.522582+0000","last_fullsized":"2024-03-12T03:52:33.522582+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'25","last_scrub_stamp":"2024-03-12T03:51:46.582596+0000","last_deep_scrub":"344'25","last_deep_scrub_stamp":"2024-03-12T03:51:46.582596+0000","last_clean_scrub_stamp":"2024-03-12T03:51:46.582596+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:00.206615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":624,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.527525+0000","last_change":"2024-03-12T03:48:58.454102+0000","last_active":"2024-03-12T03:52:33.527525+0000","last_peered":"2024-03-12T03:52:33.527525+0000","last_clean":"2024-03-12T03:48:53.448575+0000","last_became_active":"2024-03-12T03:48:58.454102+0000","last_became_peered":"2024-03-12T03:48:58.454102+0000","last_unstale":"2024-03-12T03:52:33.527525+0000","last_undegraded":"2024-03-12T03:52:33.527525+0000","last_fullsized":"2024-03-12T03:48:58.443116+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:07.363439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:07.363439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":760,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:41.855417+0000","last_change":"2024-03-12T03:51:42.650674+0000","last_active":"2024-03-12T03:52:41.855417+0000","last_peered":"2024-03-12T03:52:41.855417+0000","last_clean":"2024-03-12T03:52:41.855417+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:52:41.855417+0000","last_undegraded":"2024-03-12T03:52:41.855417+0000","last_fullsized":"2024-03-12T03:52:41.855417+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:51:42.650584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:42.650584+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:11.924689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":334,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.528096+0000","last_change":"2024-03-12T03:51:42.833356+0000","last_active":"2024-03-12T03:52:33.528096+0000","last_peered":"2024-03-12T03:52:33.528096+0000","last_clean":"2024-03-12T03:52:33.528096+0000","last_became_active":"2024-03-12T03:51:42.832276+0000","last_became_peered":"2024-03-12T03:51:42.832276+0000","last_unstale":"2024-03-12T03:52:33.528096+0000","last_undegraded":"2024-03-12T03:52:33.528096+0000","last_fullsized":"2024-03-12T03:52:33.528096+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:54.871895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.871895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"461'4540","reported_seq":4599,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:46.999864+0000","last_change":"2024-03-12T03:52:17.842637+0000","last_active":"2024-03-12T03:52:46.999864+0000","last_peered":"2024-03-12T03:52:46.999864+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:52:17.842637+0000","last_became_peered":"2024-03-12T03:52:17.842637+0000","last_unstale":"2024-03-12T03:52:46.999864+0000","last_undegraded":"2024-03-12T03:52:17.830421+0000","last_fullsized":"2024-03-12T03:52:17.830244+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:51.087084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174424064,"num_objects":2661,"num_object_clones":0,"num_object_copies":7983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2661,"num_write_kb":170336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":2661}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"269'31","reported_seq":594,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.547035+0000","last_change":"2024-03-12T03:51:42.832652+0000","last_active":"2024-03-12T03:52:33.547035+0000","last_peered":"2024-03-12T03:52:33.547035+0000","last_clean":"2024-03-12T03:52:33.547035+0000","last_became_active":"2024-03-12T03:51:42.832275+0000","last_became_peered":"2024-03-12T03:51:42.832275+0000","last_unstale":"2024-03-12T03:52:33.547035+0000","last_undegraded":"2024-03-12T03:52:33.547035+0000","last_fullsized":"2024-03-12T03:52:33.547035+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:50:01.871412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:01.871412+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":325,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.537505+0000","last_change":"2024-03-12T03:50:55.920975+0000","last_active":"2024-03-12T03:52:33.537505+0000","last_peered":"2024-03-12T03:52:33.537505+0000","last_clean":"2024-03-12T03:52:33.537505+0000","last_became_active":"2024-03-12T03:45:18.108377+0000","last_became_peered":"2024-03-12T03:45:18.108377+0000","last_unstale":"2024-03-12T03:52:33.537505+0000","last_undegraded":"2024-03-12T03:52:33.537505+0000","last_fullsized":"2024-03-12T03:52:33.537505+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:55.920704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:50:55.920704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":848,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.533209+0000","last_change":"2024-03-12T03:48:58.457284+0000","last_active":"2024-03-12T03:52:33.533209+0000","last_peered":"2024-03-12T03:52:33.533209+0000","last_clean":"2024-03-12T03:48:53.449180+0000","last_became_active":"2024-03-12T03:48:58.457284+0000","last_became_peered":"2024-03-12T03:48:58.457284+0000","last_unstale":"2024-03-12T03:52:33.533209+0000","last_undegraded":"2024-03-12T03:52:33.533209+0000","last_fullsized":"2024-03-12T03:48:58.441935+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.054936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.054936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":607,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.544472+0000","last_change":"2024-03-12T03:51:37.847956+0000","last_active":"2024-03-12T03:52:33.544472+0000","last_peered":"2024-03-12T03:52:33.544472+0000","last_clean":"2024-03-12T03:52:33.544472+0000","last_became_active":"2024-03-12T03:46:25.293293+0000","last_became_peered":"2024-03-12T03:46:25.293293+0000","last_unstale":"2024-03-12T03:52:33.544472+0000","last_undegraded":"2024-03-12T03:52:33.544472+0000","last_fullsized":"2024-03-12T03:52:33.544472+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:37.847806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:37.847806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:47.499702+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"388'129","reported_seq":1021,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.508684+0000","last_change":"2024-03-12T03:50:56.164839+0000","last_active":"2024-03-12T03:52:33.508684+0000","last_peered":"2024-03-12T03:52:33.508684+0000","last_clean":"2024-03-12T03:52:33.508684+0000","last_became_active":"2024-03-12T03:46:25.297913+0000","last_became_peered":"2024-03-12T03:46:25.297913+0000","last_unstale":"2024-03-12T03:52:33.508684+0000","last_undegraded":"2024-03-12T03:52:33.508684+0000","last_fullsized":"2024-03-12T03:52:33.508684+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"388'129","last_scrub_stamp":"2024-03-12T03:50:56.164705+0000","last_deep_scrub":"388'129","last_deep_scrub_stamp":"2024-03-12T03:49:17.168603+0000","last_clean_scrub_stamp":"2024-03-12T03:50:56.164705+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"461'4381","reported_seq":4440,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:43.568403+0000","last_change":"2024-03-12T03:52:17.853813+0000","last_active":"2024-03-12T03:52:43.568403+0000","last_peered":"2024-03-12T03:52:43.568403+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:52:17.853813+0000","last_became_peered":"2024-03-12T03:52:17.853813+0000","last_unstale":"2024-03-12T03:52:43.568403+0000","last_undegraded":"2024-03-12T03:52:17.833553+0000","last_fullsized":"2024-03-12T03:52:17.833423+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:45.789531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166625280,"num_objects":2542,"num_object_clones":0,"num_object_copies":7626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2542,"num_write_kb":162720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":1009,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.536430+0000","last_change":"2024-03-12T03:51:30.025307+0000","last_active":"2024-03-12T03:52:33.536430+0000","last_peered":"2024-03-12T03:52:33.536430+0000","last_clean":"2024-03-12T03:52:33.536430+0000","last_became_active":"2024-03-12T03:50:58.842635+0000","last_became_peered":"2024-03-12T03:50:58.842635+0000","last_unstale":"2024-03-12T03:52:33.536430+0000","last_undegraded":"2024-03-12T03:52:33.536430+0000","last_fullsized":"2024-03-12T03:52:33.536430+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:30.025220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:51:30.025220+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:35.279704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"269'31","reported_seq":546,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.528681+0000","last_change":"2024-03-12T03:51:42.834522+0000","last_active":"2024-03-12T03:52:33.528681+0000","last_peered":"2024-03-12T03:52:33.528681+0000","last_clean":"2024-03-12T03:52:33.528681+0000","last_became_active":"2024-03-12T03:51:42.834094+0000","last_became_peered":"2024-03-12T03:51:42.834094+0000","last_unstale":"2024-03-12T03:52:33.528681+0000","last_undegraded":"2024-03-12T03:52:33.528681+0000","last_fullsized":"2024-03-12T03:52:33.528681+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:49:09.755469+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:49:09.755469+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":874,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.537123+0000","last_change":"2024-03-12T03:50:47.939438+0000","last_active":"2024-03-12T03:52:33.537123+0000","last_peered":"2024-03-12T03:52:33.537123+0000","last_clean":"2024-03-12T03:52:33.537123+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:52:33.537123+0000","last_undegraded":"2024-03-12T03:52:33.537123+0000","last_fullsized":"2024-03-12T03:52:33.537123+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:47.939203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_clean_scrub_stamp":"2024-03-12T03:50:47.939203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":852,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.511683+0000","last_change":"2024-03-12T03:50:51.214639+0000","last_active":"2024-03-12T03:52:33.511683+0000","last_peered":"2024-03-12T03:52:33.511683+0000","last_clean":"2024-03-12T03:52:33.511683+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:52:33.511683+0000","last_undegraded":"2024-03-12T03:52:33.511683+0000","last_fullsized":"2024-03-12T03:52:33.511683+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:51.214503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:51.214503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":647,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.522501+0000","last_change":"2024-03-12T03:51:36.531246+0000","last_active":"2024-03-12T03:52:33.522501+0000","last_peered":"2024-03-12T03:52:33.522501+0000","last_clean":"2024-03-12T03:52:33.522501+0000","last_became_active":"2024-03-12T03:46:25.300814+0000","last_became_peered":"2024-03-12T03:46:25.300814+0000","last_unstale":"2024-03-12T03:52:33.522501+0000","last_undegraded":"2024-03-12T03:52:33.522501+0000","last_fullsized":"2024-03-12T03:52:33.522501+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:36.531169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:36.531169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:42.038087+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"461'4498","reported_seq":4589,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:47.975778+0000","last_change":"2024-03-12T03:52:18.576917+0000","last_active":"2024-03-12T03:52:47.975778+0000","last_peered":"2024-03-12T03:52:47.975778+0000","last_clean":"2024-03-12T03:52:47.975778+0000","last_became_active":"2024-03-12T03:52:17.844536+0000","last_became_peered":"2024-03-12T03:52:17.844536+0000","last_unstale":"2024-03-12T03:52:47.975778+0000","last_undegraded":"2024-03-12T03:52:47.975778+0000","last_fullsized":"2024-03-12T03:52:47.975778+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'3720","last_scrub_stamp":"2024-03-12T03:52:18.576087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:52:18.576087+0000","objects_scrubbed":1876,"log_size":4498,"log_dups_size":0,"ondisk_log_size":4498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:26.728197+0000","scrub_duration":521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175243264,"num_objects":2674,"num_object_clones":0,"num_object_copies":8022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":798,"num_write_kb":51072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":994,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.510897+0000","last_change":"2024-03-12T03:50:34.205833+0000","last_active":"2024-03-12T03:52:33.510897+0000","last_peered":"2024-03-12T03:52:33.510897+0000","last_clean":"2024-03-12T03:52:33.510897+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:52:33.510897+0000","last_undegraded":"2024-03-12T03:52:33.510897+0000","last_fullsized":"2024-03-12T03:52:33.510897+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:34.205711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:34.205711+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":718,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.537602+0000","last_change":"2024-03-12T03:50:46.613513+0000","last_active":"2024-03-12T03:52:33.537602+0000","last_peered":"2024-03-12T03:52:33.537602+0000","last_clean":"2024-03-12T03:52:33.537602+0000","last_became_active":"2024-03-12T03:46:25.287373+0000","last_became_peered":"2024-03-12T03:46:25.287373+0000","last_unstale":"2024-03-12T03:52:33.537602+0000","last_undegraded":"2024-03-12T03:52:33.537602+0000","last_fullsized":"2024-03-12T03:52:33.537602+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:50:46.613361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:46.613361+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":236,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.507883+0000","last_change":"2024-03-12T03:51:42.832487+0000","last_active":"2024-03-12T03:52:33.507883+0000","last_peered":"2024-03-12T03:52:33.507883+0000","last_clean":"2024-03-12T03:48:53.446320+0000","last_became_active":"2024-03-12T03:51:42.832487+0000","last_became_peered":"2024-03-12T03:51:42.832487+0000","last_unstale":"2024-03-12T03:52:33.507883+0000","last_undegraded":"2024-03-12T03:52:33.507883+0000","last_fullsized":"2024-03-12T03:51:42.823170+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:16.876204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:16.876204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"441'10","reported_seq":87,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:36.856022+0000","last_change":"2024-03-12T03:51:42.832218+0000","last_active":"2024-03-12T03:52:36.856022+0000","last_peered":"2024-03-12T03:52:36.856022+0000","last_clean":"2024-03-12T03:33:28.357470+0000","last_became_active":"2024-03-12T03:51:42.832218+0000","last_became_peered":"2024-03-12T03:51:42.832218+0000","last_unstale":"2024-03-12T03:52:36.856022+0000","last_undegraded":"2024-03-12T03:51:42.824811+0000","last_fullsized":"2024-03-12T03:51:42.824692+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:47:53.964216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:53.964216+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":835,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.511371+0000","last_change":"2024-03-12T03:48:58.448113+0000","last_active":"2024-03-12T03:52:33.511371+0000","last_peered":"2024-03-12T03:52:33.511371+0000","last_clean":"2024-03-12T03:48:53.472090+0000","last_became_active":"2024-03-12T03:48:58.448113+0000","last_became_peered":"2024-03-12T03:48:58.448113+0000","last_unstale":"2024-03-12T03:52:33.511371+0000","last_undegraded":"2024-03-12T03:52:33.511371+0000","last_fullsized":"2024-03-12T03:48:58.441990+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:04.770746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:04.770746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"461'5219","reported_seq":5278,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:46.450113+0000","last_change":"2024-03-12T03:51:40.855404+0000","last_active":"2024-03-12T03:52:46.450113+0000","last_peered":"2024-03-12T03:52:46.450113+0000","last_clean":"2024-03-12T03:52:46.450113+0000","last_became_active":"2024-03-12T03:51:40.855216+0000","last_became_peered":"2024-03-12T03:51:40.855216+0000","last_unstale":"2024-03-12T03:52:46.450113+0000","last_undegraded":"2024-03-12T03:52:46.450113+0000","last_fullsized":"2024-03-12T03:52:46.450113+0000","mapping_epoch":438,"log_start":"444'1200","ondisk_log_start":"444'1200","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4019,"log_dups_size":0,"ondisk_log_size":4019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:03.432188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342032384,"num_objects":5219,"num_object_clones":0,"num_object_copies":15657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5219,"num_write_kb":334016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":865,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.537012+0000","last_change":"2024-03-12T03:51:30.617409+0000","last_active":"2024-03-12T03:52:33.537012+0000","last_peered":"2024-03-12T03:52:33.537012+0000","last_clean":"2024-03-12T03:52:33.537012+0000","last_became_active":"2024-03-12T03:46:25.286985+0000","last_became_peered":"2024-03-12T03:46:25.286985+0000","last_unstale":"2024-03-12T03:52:33.537012+0000","last_undegraded":"2024-03-12T03:52:33.537012+0000","last_fullsized":"2024-03-12T03:52:33.537012+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:30.617326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:30.617326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:57.949634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"269'42","reported_seq":713,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:33.531481+0000","last_change":"2024-03-12T03:48:58.463377+0000","last_active":"2024-03-12T03:52:33.531481+0000","last_peered":"2024-03-12T03:52:33.531481+0000","last_clean":"2024-03-12T03:48:53.447338+0000","last_became_active":"2024-03-12T03:48:58.463377+0000","last_became_peered":"2024-03-12T03:48:58.463377+0000","last_unstale":"2024-03-12T03:52:33.531481+0000","last_undegraded":"2024-03-12T03:48:58.440183+0000","last_fullsized":"2024-03-12T03:48:58.440082+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:47:13.347128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.347128+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":628,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.526380+0000","last_change":"2024-03-12T03:48:58.464970+0000","last_active":"2024-03-12T03:52:33.526380+0000","last_peered":"2024-03-12T03:52:33.526380+0000","last_clean":"2024-03-12T03:48:53.448054+0000","last_became_active":"2024-03-12T03:48:58.464970+0000","last_became_peered":"2024-03-12T03:48:58.464970+0000","last_unstale":"2024-03-12T03:52:33.526380+0000","last_undegraded":"2024-03-12T03:52:33.526380+0000","last_fullsized":"2024-03-12T03:48:58.441302+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:05.639950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:05.639950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":637,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524780+0000","last_change":"2024-03-12T03:51:42.841507+0000","last_active":"2024-03-12T03:52:33.524780+0000","last_peered":"2024-03-12T03:52:33.524780+0000","last_clean":"2024-03-12T03:52:33.524780+0000","last_became_active":"2024-03-12T03:51:42.840902+0000","last_became_peered":"2024-03-12T03:51:42.840902+0000","last_unstale":"2024-03-12T03:52:33.524780+0000","last_undegraded":"2024-03-12T03:52:33.524780+0000","last_fullsized":"2024-03-12T03:52:33.524780+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:14.836179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_clean_scrub_stamp":"2024-03-12T03:49:14.836179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":1010,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.538369+0000","last_change":"2024-03-12T03:50:48.924955+0000","last_active":"2024-03-12T03:52:33.538369+0000","last_peered":"2024-03-12T03:52:33.538369+0000","last_clean":"2024-03-12T03:52:33.538369+0000","last_became_active":"2024-03-12T03:44:19.373715+0000","last_became_peered":"2024-03-12T03:44:19.373715+0000","last_unstale":"2024-03-12T03:52:33.538369+0000","last_undegraded":"2024-03-12T03:52:33.538369+0000","last_fullsized":"2024-03-12T03:52:33.538369+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:48.924831+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:15.895638+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.924831+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"461'5196","reported_seq":5255,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:46.364089+0000","last_change":"2024-03-12T03:51:40.862482+0000","last_active":"2024-03-12T03:52:46.364089+0000","last_peered":"2024-03-12T03:52:46.364089+0000","last_clean":"2024-03-12T03:52:46.364089+0000","last_became_active":"2024-03-12T03:51:40.861867+0000","last_became_peered":"2024-03-12T03:51:40.861867+0000","last_unstale":"2024-03-12T03:52:46.364089+0000","last_undegraded":"2024-03-12T03:52:46.364089+0000","last_fullsized":"2024-03-12T03:52:46.364089+0000","mapping_epoch":438,"log_start":"444'1100","ondisk_log_start":"444'1100","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4096,"log_dups_size":0,"ondisk_log_size":4096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:55.718738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340525056,"num_objects":5196,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5196,"num_write_kb":332544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"332'12","reported_seq":867,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:36.854435+0000","last_change":"2024-03-12T03:50:32.360621+0000","last_active":"2024-03-12T03:52:36.854435+0000","last_peered":"2024-03-12T03:52:36.854435+0000","last_clean":"2024-03-12T03:52:36.854435+0000","last_became_active":"2024-03-12T03:46:25.288674+0000","last_became_peered":"2024-03-12T03:46:25.288674+0000","last_unstale":"2024-03-12T03:52:36.854435+0000","last_undegraded":"2024-03-12T03:52:36.854435+0000","last_fullsized":"2024-03-12T03:52:36.854435+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'12","last_scrub_stamp":"2024-03-12T03:50:32.360502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:32.360502+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"269'42","reported_seq":559,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.510008+0000","last_change":"2024-03-12T03:51:42.832600+0000","last_active":"2024-03-12T03:52:33.510008+0000","last_peered":"2024-03-12T03:52:33.510008+0000","last_clean":"2024-03-12T03:52:33.510008+0000","last_became_active":"2024-03-12T03:51:42.832229+0000","last_became_peered":"2024-03-12T03:51:42.832229+0000","last_unstale":"2024-03-12T03:52:33.510008+0000","last_undegraded":"2024-03-12T03:52:33.510008+0000","last_fullsized":"2024-03-12T03:52:33.510008+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:50:35.839859+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.839859+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"320'2","reported_seq":719,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:33.528288+0000","last_change":"2024-03-12T03:48:58.457502+0000","last_active":"2024-03-12T03:52:33.528288+0000","last_peered":"2024-03-12T03:52:33.528288+0000","last_clean":"2024-03-12T03:45:56.084290+0000","last_became_active":"2024-03-12T03:48:58.457502+0000","last_became_peered":"2024-03-12T03:48:58.457502+0000","last_unstale":"2024-03-12T03:52:33.528288+0000","last_undegraded":"2024-03-12T03:48:58.439950+0000","last_fullsized":"2024-03-12T03:48:58.439792+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'2","last_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_deep_scrub":"320'2","last_deep_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_clean_scrub_stamp":"2024-03-12T03:45:56.089247+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":871,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.523528+0000","last_change":"2024-03-12T03:51:31.400543+0000","last_active":"2024-03-12T03:52:33.523528+0000","last_peered":"2024-03-12T03:52:33.523528+0000","last_clean":"2024-03-12T03:52:33.523528+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:52:33.523528+0000","last_undegraded":"2024-03-12T03:52:33.523528+0000","last_fullsized":"2024-03-12T03:52:33.523528+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:31.400418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:49:54.471078+0000","last_clean_scrub_stamp":"2024-03-12T03:51:31.400418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:59.747204+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":830,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.522280+0000","last_change":"2024-03-12T03:51:37.519315+0000","last_active":"2024-03-12T03:52:33.522280+0000","last_peered":"2024-03-12T03:52:33.522280+0000","last_clean":"2024-03-12T03:52:33.522280+0000","last_became_active":"2024-03-12T03:51:07.938356+0000","last_became_peered":"2024-03-12T03:51:07.938356+0000","last_unstale":"2024-03-12T03:52:33.522280+0000","last_undegraded":"2024-03-12T03:52:33.522280+0000","last_fullsized":"2024-03-12T03:52:33.522280+0000","mapping_epoch":330,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:37.519232+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_clean_scrub_stamp":"2024-03-12T03:51:37.519232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:02.132334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"461'5299","reported_seq":5358,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:47.074879+0000","last_change":"2024-03-12T03:51:40.889015+0000","last_active":"2024-03-12T03:52:47.074879+0000","last_peered":"2024-03-12T03:52:47.074879+0000","last_clean":"2024-03-12T03:52:47.074879+0000","last_became_active":"2024-03-12T03:51:40.888684+0000","last_became_peered":"2024-03-12T03:51:40.888684+0000","last_unstale":"2024-03-12T03:52:47.074879+0000","last_undegraded":"2024-03-12T03:52:47.074879+0000","last_fullsized":"2024-03-12T03:52:47.074879+0000","mapping_epoch":438,"log_start":"443'100","ondisk_log_start":"443'100","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:51.736313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347275264,"num_objects":5299,"num_object_clones":0,"num_object_copies":15897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5299,"num_write_kb":339136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"385'9","reported_seq":232,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:36.856041+0000","last_change":"2024-03-12T03:48:58.451360+0000","last_active":"2024-03-12T03:52:36.856041+0000","last_peered":"2024-03-12T03:52:36.856041+0000","last_clean":"2024-03-12T03:33:28.360558+0000","last_became_active":"2024-03-12T03:48:58.451360+0000","last_became_peered":"2024-03-12T03:48:58.451360+0000","last_unstale":"2024-03-12T03:52:36.856041+0000","last_undegraded":"2024-03-12T03:48:58.442329+0000","last_fullsized":"2024-03-12T03:48:58.441949+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:48:14.138048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:14.138048+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"269'26","reported_seq":908,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.528011+0000","last_change":"2024-03-12T03:50:35.597476+0000","last_active":"2024-03-12T03:52:33.528011+0000","last_peered":"2024-03-12T03:52:33.528011+0000","last_clean":"2024-03-12T03:52:33.528011+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:52:33.528011+0000","last_undegraded":"2024-03-12T03:52:33.528011+0000","last_fullsized":"2024-03-12T03:52:33.528011+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:50:35.597225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.597225+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":883,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.511543+0000","last_change":"2024-03-12T03:50:48.222807+0000","last_active":"2024-03-12T03:52:33.511543+0000","last_peered":"2024-03-12T03:52:33.511543+0000","last_clean":"2024-03-12T03:52:33.511543+0000","last_became_active":"2024-03-12T03:46:25.294348+0000","last_became_peered":"2024-03-12T03:46:25.294348+0000","last_unstale":"2024-03-12T03:52:33.511543+0000","last_undegraded":"2024-03-12T03:52:33.511543+0000","last_fullsized":"2024-03-12T03:52:33.511543+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.222697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.222697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":739,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.531201+0000","last_change":"2024-03-12T03:48:58.455443+0000","last_active":"2024-03-12T03:52:33.531201+0000","last_peered":"2024-03-12T03:52:33.531201+0000","last_clean":"2024-03-12T03:43:13.330724+0000","last_became_active":"2024-03-12T03:48:58.455443+0000","last_became_peered":"2024-03-12T03:48:58.455443+0000","last_unstale":"2024-03-12T03:52:33.531201+0000","last_undegraded":"2024-03-12T03:52:33.531201+0000","last_fullsized":"2024-03-12T03:48:58.439578+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:13.892760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.892760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":606,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.527043+0000","last_change":"2024-03-12T03:50:54.587747+0000","last_active":"2024-03-12T03:52:33.527043+0000","last_peered":"2024-03-12T03:52:33.527043+0000","last_clean":"2024-03-12T03:52:33.527043+0000","last_became_active":"2024-03-12T03:46:25.287360+0000","last_became_peered":"2024-03-12T03:46:25.287360+0000","last_unstale":"2024-03-12T03:52:33.527043+0000","last_undegraded":"2024-03-12T03:52:33.527043+0000","last_fullsized":"2024-03-12T03:52:33.527043+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:54.587604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:54.587604+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"461'5360","reported_seq":5419,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:48.061342+0000","last_change":"2024-03-12T03:51:40.859739+0000","last_active":"2024-03-12T03:52:48.061342+0000","last_peered":"2024-03-12T03:52:48.061342+0000","last_clean":"2024-03-12T03:52:48.061342+0000","last_became_active":"2024-03-12T03:51:40.859434+0000","last_became_peered":"2024-03-12T03:51:40.859434+0000","last_unstale":"2024-03-12T03:52:48.061342+0000","last_undegraded":"2024-03-12T03:52:48.061342+0000","last_fullsized":"2024-03-12T03:52:48.061342+0000","mapping_epoch":438,"log_start":"444'1100","ondisk_log_start":"444'1100","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:01.523543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351272960,"num_objects":5360,"num_object_clones":0,"num_object_copies":16080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5360,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":164,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.511766+0000","last_change":"2024-03-12T03:48:58.449166+0000","last_active":"2024-03-12T03:52:33.511766+0000","last_peered":"2024-03-12T03:52:33.511766+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.449166+0000","last_became_peered":"2024-03-12T03:48:58.449166+0000","last_unstale":"2024-03-12T03:52:33.511766+0000","last_undegraded":"2024-03-12T03:52:33.511766+0000","last_fullsized":"2024-03-12T03:48:58.441301+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.179416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.179416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"269'35","reported_seq":662,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:33.511370+0000","last_change":"2024-03-12T03:48:58.454600+0000","last_active":"2024-03-12T03:52:33.511370+0000","last_peered":"2024-03-12T03:52:33.511370+0000","last_clean":"2024-03-12T03:43:28.848207+0000","last_became_active":"2024-03-12T03:48:58.454600+0000","last_became_peered":"2024-03-12T03:48:58.454600+0000","last_unstale":"2024-03-12T03:52:33.511370+0000","last_undegraded":"2024-03-12T03:48:58.443785+0000","last_fullsized":"2024-03-12T03:48:58.443677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-03-12T03:45:59.055318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.055318+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":47,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":882,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.533125+0000","last_change":"2024-03-12T03:49:51.465759+0000","last_active":"2024-03-12T03:52:33.533125+0000","last_peered":"2024-03-12T03:52:33.533125+0000","last_clean":"2024-03-12T03:52:33.533125+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:52:33.533125+0000","last_undegraded":"2024-03-12T03:52:33.533125+0000","last_fullsized":"2024-03-12T03:52:33.533125+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:51.465633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:49:51.465633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":870,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524548+0000","last_change":"2024-03-12T03:51:36.030269+0000","last_active":"2024-03-12T03:52:33.524548+0000","last_peered":"2024-03-12T03:52:33.524548+0000","last_clean":"2024-03-12T03:52:33.524548+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:52:33.524548+0000","last_undegraded":"2024-03-12T03:52:33.524548+0000","last_fullsized":"2024-03-12T03:52:33.524548+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:36.030166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:36.030166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:41.226793+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":468,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.525334+0000","last_change":"2024-03-12T03:48:58.454186+0000","last_active":"2024-03-12T03:52:33.525334+0000","last_peered":"2024-03-12T03:52:33.525334+0000","last_clean":"2024-03-12T03:34:01.860694+0000","last_became_active":"2024-03-12T03:48:58.454186+0000","last_became_peered":"2024-03-12T03:48:58.454186+0000","last_unstale":"2024-03-12T03:52:33.525334+0000","last_undegraded":"2024-03-12T03:52:33.525334+0000","last_fullsized":"2024-03-12T03:48:58.443407+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:04.857986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:04.857986+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"461'4501","reported_seq":4560,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:43.551683+0000","last_change":"2024-03-12T03:52:17.854261+0000","last_active":"2024-03-12T03:52:43.551683+0000","last_peered":"2024-03-12T03:52:43.551683+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:52:17.854261+0000","last_became_peered":"2024-03-12T03:52:17.854261+0000","last_unstale":"2024-03-12T03:52:43.551683+0000","last_undegraded":"2024-03-12T03:52:17.834203+0000","last_fullsized":"2024-03-12T03:52:17.834063+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:45.789623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169803776,"num_objects":2591,"num_object_clones":0,"num_object_copies":7773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2591,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2591,"num_write_kb":165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2591}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":839,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.511762+0000","last_change":"2024-03-12T03:48:58.467253+0000","last_active":"2024-03-12T03:52:33.511762+0000","last_peered":"2024-03-12T03:52:33.511762+0000","last_clean":"2024-03-12T03:48:53.472859+0000","last_became_active":"2024-03-12T03:48:58.467253+0000","last_became_peered":"2024-03-12T03:48:58.467253+0000","last_unstale":"2024-03-12T03:52:33.511762+0000","last_undegraded":"2024-03-12T03:52:33.511762+0000","last_fullsized":"2024-03-12T03:48:58.444374+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:00.577693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:46:00.577693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"269'24","reported_seq":683,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:33.528845+0000","last_change":"2024-03-12T03:48:58.449415+0000","last_active":"2024-03-12T03:52:33.528845+0000","last_peered":"2024-03-12T03:52:33.528845+0000","last_clean":"2024-03-12T03:43:28.788554+0000","last_became_active":"2024-03-12T03:48:58.449415+0000","last_became_peered":"2024-03-12T03:48:58.449415+0000","last_unstale":"2024-03-12T03:52:33.528845+0000","last_undegraded":"2024-03-12T03:48:58.441112+0000","last_fullsized":"2024-03-12T03:48:58.441001+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:48:17.191968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:17.191968+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":2028,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:44.976358+0000","last_change":"2024-03-12T03:48:58.462253+0000","last_active":"2024-03-12T03:52:44.976358+0000","last_peered":"2024-03-12T03:52:44.976358+0000","last_clean":"2024-03-12T03:48:56.580904+0000","last_became_active":"2024-03-12T03:48:58.462253+0000","last_became_peered":"2024-03-12T03:48:58.462253+0000","last_unstale":"2024-03-12T03:52:44.976358+0000","last_undegraded":"2024-03-12T03:48:58.445825+0000","last_fullsized":"2024-03-12T03:48:58.445711+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:47:13.591375+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.591375+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1156,"num_read_kb":1151,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":739,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.537214+0000","last_change":"2024-03-12T03:48:58.464609+0000","last_active":"2024-03-12T03:52:33.537214+0000","last_peered":"2024-03-12T03:52:33.537214+0000","last_clean":"2024-03-12T03:43:13.332262+0000","last_became_active":"2024-03-12T03:48:58.464609+0000","last_became_peered":"2024-03-12T03:48:58.464609+0000","last_unstale":"2024-03-12T03:52:33.537214+0000","last_undegraded":"2024-03-12T03:52:33.537214+0000","last_fullsized":"2024-03-12T03:48:58.443632+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:58.050985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.050985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":868,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.531797+0000","last_change":"2024-03-12T03:49:13.598817+0000","last_active":"2024-03-12T03:52:33.531797+0000","last_peered":"2024-03-12T03:52:33.531797+0000","last_clean":"2024-03-12T03:52:33.531797+0000","last_became_active":"2024-03-12T03:46:25.302256+0000","last_became_peered":"2024-03-12T03:46:25.302256+0000","last_unstale":"2024-03-12T03:52:33.531797+0000","last_undegraded":"2024-03-12T03:52:33.531797+0000","last_fullsized":"2024-03-12T03:52:33.531797+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:13.598692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:13.598692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1002,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.536731+0000","last_change":"2024-03-12T03:51:11.966626+0000","last_active":"2024-03-12T03:52:33.536731+0000","last_peered":"2024-03-12T03:52:33.536731+0000","last_clean":"2024-03-12T03:52:33.536731+0000","last_became_active":"2024-03-12T03:51:11.965655+0000","last_became_peered":"2024-03-12T03:51:11.965655+0000","last_unstale":"2024-03-12T03:52:33.536731+0000","last_undegraded":"2024-03-12T03:52:33.536731+0000","last_fullsized":"2024-03-12T03:52:33.536731+0000","mapping_epoch":154,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:50.467976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.467976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"461'5192","reported_seq":5251,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:43.567328+0000","last_change":"2024-03-12T03:51:40.888952+0000","last_active":"2024-03-12T03:52:43.567328+0000","last_peered":"2024-03-12T03:52:43.567328+0000","last_clean":"2024-03-12T03:52:43.567328+0000","last_became_active":"2024-03-12T03:51:40.888646+0000","last_became_peered":"2024-03-12T03:51:40.888646+0000","last_unstale":"2024-03-12T03:52:43.567328+0000","last_undegraded":"2024-03-12T03:52:43.567328+0000","last_fullsized":"2024-03-12T03:52:43.567328+0000","mapping_epoch":438,"log_start":"444'1200","ondisk_log_start":"444'1200","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:51.206700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340262912,"num_objects":5192,"num_object_clones":0,"num_object_copies":15576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5192,"num_write_kb":332288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":867,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524491+0000","last_change":"2024-03-12T03:50:48.205459+0000","last_active":"2024-03-12T03:52:33.524491+0000","last_peered":"2024-03-12T03:52:33.524491+0000","last_clean":"2024-03-12T03:52:33.524491+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:52:33.524491+0000","last_undegraded":"2024-03-12T03:52:33.524491+0000","last_fullsized":"2024-03-12T03:52:33.524491+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.205352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.205352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"315'29","reported_seq":1036,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:36.854395+0000","last_change":"2024-03-12T03:52:19.658496+0000","last_active":"2024-03-12T03:52:36.854395+0000","last_peered":"2024-03-12T03:52:36.854395+0000","last_clean":"2024-03-12T03:52:36.854395+0000","last_became_active":"2024-03-12T03:45:18.109048+0000","last_became_peered":"2024-03-12T03:45:18.109048+0000","last_unstale":"2024-03-12T03:52:36.854395+0000","last_undegraded":"2024-03-12T03:52:36.854395+0000","last_fullsized":"2024-03-12T03:52:36.854395+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'29","last_scrub_stamp":"2024-03-12T03:52:19.658412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:52:19.658412+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:39.220370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":633,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524725+0000","last_change":"2024-03-12T03:50:33.413216+0000","last_active":"2024-03-12T03:52:33.524725+0000","last_peered":"2024-03-12T03:52:33.524725+0000","last_clean":"2024-03-12T03:52:33.524725+0000","last_became_active":"2024-03-12T03:46:25.292535+0000","last_became_peered":"2024-03-12T03:46:25.292535+0000","last_unstale":"2024-03-12T03:52:33.524725+0000","last_undegraded":"2024-03-12T03:52:33.524725+0000","last_fullsized":"2024-03-12T03:52:33.524725+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:33.413098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:33.413098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1137,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.524007+0000","last_change":"2024-03-12T03:48:58.452024+0000","last_active":"2024-03-12T03:52:33.524007+0000","last_peered":"2024-03-12T03:52:33.524007+0000","last_clean":"2024-03-12T03:48:53.448868+0000","last_became_active":"2024-03-12T03:48:58.452024+0000","last_became_peered":"2024-03-12T03:48:58.452024+0000","last_unstale":"2024-03-12T03:52:33.524007+0000","last_undegraded":"2024-03-12T03:52:33.524007+0000","last_fullsized":"2024-03-12T03:48:58.444796+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_clean_scrub_stamp":"2024-03-12T03:48:11.435684+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":858,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.511187+0000","last_change":"2024-03-12T03:51:39.169999+0000","last_active":"2024-03-12T03:52:33.511187+0000","last_peered":"2024-03-12T03:52:33.511187+0000","last_clean":"2024-03-12T03:52:33.511187+0000","last_became_active":"2024-03-12T03:46:25.295668+0000","last_became_peered":"2024-03-12T03:46:25.295668+0000","last_unstale":"2024-03-12T03:52:33.511187+0000","last_undegraded":"2024-03-12T03:52:33.511187+0000","last_fullsized":"2024-03-12T03:52:33.511187+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.169871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.169871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:03.084086+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"461'4462","reported_seq":4554,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:43.524404+0000","last_change":"2024-03-12T03:52:19.002532+0000","last_active":"2024-03-12T03:52:43.524404+0000","last_peered":"2024-03-12T03:52:43.524404+0000","last_clean":"2024-03-12T03:52:43.524404+0000","last_became_active":"2024-03-12T03:52:17.842035+0000","last_became_peered":"2024-03-12T03:52:17.842035+0000","last_unstale":"2024-03-12T03:52:43.524404+0000","last_undegraded":"2024-03-12T03:52:43.524404+0000","last_fullsized":"2024-03-12T03:52:43.524404+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'3776","last_scrub_stamp":"2024-03-12T03:52:19.002413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:52:19.002413+0000","objects_scrubbed":1905,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:25.408042+0000","scrub_duration":316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169869312,"num_objects":2592,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":687,"num_write_kb":43968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":598,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.544729+0000","last_change":"2024-03-12T03:50:59.748930+0000","last_active":"2024-03-12T03:52:33.544729+0000","last_peered":"2024-03-12T03:52:33.544729+0000","last_clean":"2024-03-12T03:52:33.544729+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:52:33.544729+0000","last_undegraded":"2024-03-12T03:52:33.544729+0000","last_fullsized":"2024-03-12T03:52:33.544729+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:59.748792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.748792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"269'23","reported_seq":897,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:33.537373+0000","last_change":"2024-03-12T03:48:58.451116+0000","last_active":"2024-03-12T03:52:33.537373+0000","last_peered":"2024-03-12T03:52:33.537373+0000","last_clean":"2024-03-12T03:48:53.448806+0000","last_became_active":"2024-03-12T03:48:58.451116+0000","last_became_peered":"2024-03-12T03:48:58.451116+0000","last_unstale":"2024-03-12T03:52:33.537373+0000","last_undegraded":"2024-03-12T03:48:58.440926+0000","last_fullsized":"2024-03-12T03:48:58.440811+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'23","last_scrub_stamp":"2024-03-12T03:45:59.998395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.998395+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":587,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.544952+0000","last_change":"2024-03-12T03:48:58.469230+0000","last_active":"2024-03-12T03:52:33.544952+0000","last_peered":"2024-03-12T03:52:33.544952+0000","last_clean":"2024-03-12T03:48:53.446422+0000","last_became_active":"2024-03-12T03:48:58.469230+0000","last_became_peered":"2024-03-12T03:48:58.469230+0000","last_unstale":"2024-03-12T03:52:33.544952+0000","last_undegraded":"2024-03-12T03:52:33.544952+0000","last_fullsized":"2024-03-12T03:48:58.446209+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:06.903506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.903506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":882,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.543508+0000","last_change":"2024-03-12T03:50:41.599042+0000","last_active":"2024-03-12T03:52:33.543508+0000","last_peered":"2024-03-12T03:52:33.543508+0000","last_clean":"2024-03-12T03:52:33.543508+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:52:33.543508+0000","last_undegraded":"2024-03-12T03:52:33.543508+0000","last_fullsized":"2024-03-12T03:52:33.543508+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:41.598934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:50:41.598934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":871,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.525405+0000","last_change":"2024-03-12T03:50:44.293342+0000","last_active":"2024-03-12T03:52:33.525405+0000","last_peered":"2024-03-12T03:52:33.525405+0000","last_clean":"2024-03-12T03:52:33.525405+0000","last_became_active":"2024-03-12T03:46:25.284168+0000","last_became_peered":"2024-03-12T03:46:25.284168+0000","last_unstale":"2024-03-12T03:52:33.525405+0000","last_undegraded":"2024-03-12T03:52:33.525405+0000","last_fullsized":"2024-03-12T03:52:33.525405+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:50:44.293155+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:49:18.522103+0000","last_clean_scrub_stamp":"2024-03-12T03:50:44.293155+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1153,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.524242+0000","last_change":"2024-03-12T03:48:58.455161+0000","last_active":"2024-03-12T03:52:33.524242+0000","last_peered":"2024-03-12T03:52:33.524242+0000","last_clean":"2024-03-12T03:48:53.449188+0000","last_became_active":"2024-03-12T03:48:58.455161+0000","last_became_peered":"2024-03-12T03:48:58.455161+0000","last_unstale":"2024-03-12T03:52:33.524242+0000","last_undegraded":"2024-03-12T03:52:33.524242+0000","last_fullsized":"2024-03-12T03:48:58.444402+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:01.456424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:01.456424+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"461'5125","reported_seq":5181,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:43.642668+0000","last_change":"2024-03-12T03:51:50.089118+0000","last_active":"2024-03-12T03:52:43.642668+0000","last_peered":"2024-03-12T03:52:43.642668+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:51:40.861779+0000","last_became_peered":"2024-03-12T03:51:40.861779+0000","last_unstale":"2024-03-12T03:52:43.642668+0000","last_undegraded":"2024-03-12T03:51:50.086761+0000","last_fullsized":"2024-03-12T03:51:40.858211+0000","mapping_epoch":438,"log_start":"444'1200","ondisk_log_start":"444'1200","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:05.364487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335872000,"num_objects":5125,"num_object_clones":0,"num_object_copies":15375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5125,"num_write_kb":328000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":5125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":162,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.531445+0000","last_change":"2024-03-12T03:48:58.458916+0000","last_active":"2024-03-12T03:52:33.531445+0000","last_peered":"2024-03-12T03:52:33.531445+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.458916+0000","last_became_peered":"2024-03-12T03:48:58.458916+0000","last_unstale":"2024-03-12T03:52:33.531445+0000","last_undegraded":"2024-03-12T03:52:33.531445+0000","last_fullsized":"2024-03-12T03:48:58.440354+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:06.864708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.864708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"382'54","reported_seq":260,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:41.858221+0000","last_change":"2024-03-12T03:51:38.936343+0000","last_active":"2024-03-12T03:52:41.858221+0000","last_peered":"2024-03-12T03:52:41.858221+0000","last_clean":"2024-03-12T03:52:41.858221+0000","last_became_active":"2024-03-12T03:48:48.462111+0000","last_became_peered":"2024-03-12T03:48:48.462111+0000","last_unstale":"2024-03-12T03:52:41.858221+0000","last_undegraded":"2024-03-12T03:52:41.858221+0000","last_fullsized":"2024-03-12T03:52:41.858221+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'54","last_scrub_stamp":"2024-03-12T03:51:38.936190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:38.936190+0000","objects_scrubbed":11,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:59.909110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":58,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":858,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.538054+0000","last_change":"2024-03-12T03:48:58.453277+0000","last_active":"2024-03-12T03:52:33.538054+0000","last_peered":"2024-03-12T03:52:33.538054+0000","last_clean":"2024-03-12T03:48:53.447092+0000","last_became_active":"2024-03-12T03:48:58.453277+0000","last_became_peered":"2024-03-12T03:48:58.453277+0000","last_unstale":"2024-03-12T03:52:33.538054+0000","last_undegraded":"2024-03-12T03:52:33.538054+0000","last_fullsized":"2024-03-12T03:48:58.443592+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.648824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.648824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":872,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.511667+0000","last_change":"2024-03-12T03:50:58.107026+0000","last_active":"2024-03-12T03:52:33.511667+0000","last_peered":"2024-03-12T03:52:33.511667+0000","last_clean":"2024-03-12T03:52:33.511667+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:52:33.511667+0000","last_undegraded":"2024-03-12T03:52:33.511667+0000","last_fullsized":"2024-03-12T03:52:33.511667+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:50:58.106746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:50:58.106746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":672,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.525585+0000","last_change":"2024-03-12T03:50:40.511469+0000","last_active":"2024-03-12T03:52:33.525585+0000","last_peered":"2024-03-12T03:52:33.525585+0000","last_clean":"2024-03-12T03:52:33.525585+0000","last_became_active":"2024-03-12T03:46:25.284416+0000","last_became_peered":"2024-03-12T03:46:25.284416+0000","last_unstale":"2024-03-12T03:52:33.525585+0000","last_undegraded":"2024-03-12T03:52:33.525585+0000","last_fullsized":"2024-03-12T03:52:33.525585+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:40.511391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:40.511391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":995,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.509903+0000","last_change":"2024-03-12T03:48:58.460115+0000","last_active":"2024-03-12T03:52:33.509903+0000","last_peered":"2024-03-12T03:52:33.509903+0000","last_clean":"2024-03-12T03:48:53.471868+0000","last_became_active":"2024-03-12T03:48:58.460115+0000","last_became_peered":"2024-03-12T03:48:58.460115+0000","last_unstale":"2024-03-12T03:52:33.509903+0000","last_undegraded":"2024-03-12T03:52:33.509903+0000","last_fullsized":"2024-03-12T03:48:58.445534+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:06.690873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.690873+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"461'4527","reported_seq":4622,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:46.502030+0000","last_change":"2024-03-12T03:52:20.532953+0000","last_active":"2024-03-12T03:52:46.502030+0000","last_peered":"2024-03-12T03:52:46.502030+0000","last_clean":"2024-03-12T03:52:46.502030+0000","last_became_active":"2024-03-12T03:52:17.865616+0000","last_became_peered":"2024-03-12T03:52:17.865616+0000","last_unstale":"2024-03-12T03:52:46.502030+0000","last_undegraded":"2024-03-12T03:52:46.502030+0000","last_fullsized":"2024-03-12T03:52:46.502030+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'3769","last_scrub_stamp":"2024-03-12T03:52:20.532779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:52:20.532779+0000","objects_scrubbed":1944,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:23.686247+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177078272,"num_objects":2702,"num_object_clones":0,"num_object_copies":8106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":758,"num_write_kb":48512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":415,"reported_epoch":461,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:52:33.525070+0000","last_change":"2024-03-12T03:51:00.633662+0000","last_active":"2024-03-12T03:52:33.525070+0000","last_peered":"2024-03-12T03:52:33.525070+0000","last_clean":"2024-03-12T03:52:33.525070+0000","last_became_active":"2024-03-12T03:50:58.843997+0000","last_became_peered":"2024-03-12T03:50:58.843997+0000","last_unstale":"2024-03-12T03:52:33.525070+0000","last_undegraded":"2024-03-12T03:52:33.525070+0000","last_fullsized":"2024-03-12T03:52:33.525070+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:00.633453+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.924742+0000","last_clean_scrub_stamp":"2024-03-12T03:51:00.633453+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"269'26","reported_seq":641,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:33.536782+0000","last_change":"2024-03-12T03:48:58.462328+0000","last_active":"2024-03-12T03:52:33.536782+0000","last_peered":"2024-03-12T03:52:33.536782+0000","last_clean":"2024-03-12T03:43:28.842466+0000","last_became_active":"2024-03-12T03:48:58.462328+0000","last_became_peered":"2024-03-12T03:48:58.462328+0000","last_unstale":"2024-03-12T03:52:33.536782+0000","last_undegraded":"2024-03-12T03:48:58.442501+0000","last_fullsized":"2024-03-12T03:48:58.442398+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:15.774076+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.774076+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":870,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.509439+0000","last_change":"2024-03-12T03:51:28.049718+0000","last_active":"2024-03-12T03:52:33.509439+0000","last_peered":"2024-03-12T03:52:33.509439+0000","last_clean":"2024-03-12T03:52:33.509439+0000","last_became_active":"2024-03-12T03:46:25.303784+0000","last_became_peered":"2024-03-12T03:46:25.303784+0000","last_unstale":"2024-03-12T03:52:33.509439+0000","last_undegraded":"2024-03-12T03:52:33.509439+0000","last_fullsized":"2024-03-12T03:52:33.509439+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:28.049638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:51:28.049638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:45.711056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":600,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.547318+0000","last_change":"2024-03-12T03:51:33.662176+0000","last_active":"2024-03-12T03:52:33.547318+0000","last_peered":"2024-03-12T03:52:33.547318+0000","last_clean":"2024-03-12T03:52:33.547318+0000","last_became_active":"2024-03-12T03:46:25.284556+0000","last_became_peered":"2024-03-12T03:46:25.284556+0000","last_unstale":"2024-03-12T03:52:33.547318+0000","last_undegraded":"2024-03-12T03:52:33.547318+0000","last_fullsized":"2024-03-12T03:52:33.547318+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:33.662099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:33.662099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:42.786905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":444,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.526067+0000","last_change":"2024-03-12T03:48:58.464502+0000","last_active":"2024-03-12T03:52:33.526067+0000","last_peered":"2024-03-12T03:52:33.526067+0000","last_clean":"2024-03-12T03:39:28.535672+0000","last_became_active":"2024-03-12T03:48:58.464502+0000","last_became_peered":"2024-03-12T03:48:58.464502+0000","last_unstale":"2024-03-12T03:52:33.526067+0000","last_undegraded":"2024-03-12T03:52:33.526067+0000","last_fullsized":"2024-03-12T03:48:58.446034+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:55.192996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:55.192996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"461'4585","reported_seq":4676,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:46.502735+0000","last_change":"2024-03-12T03:52:19.581816+0000","last_active":"2024-03-12T03:52:46.502735+0000","last_peered":"2024-03-12T03:52:46.502735+0000","last_clean":"2024-03-12T03:52:46.502735+0000","last_became_active":"2024-03-12T03:52:17.874643+0000","last_became_peered":"2024-03-12T03:52:17.874643+0000","last_unstale":"2024-03-12T03:52:46.502735+0000","last_undegraded":"2024-03-12T03:52:46.502735+0000","last_fullsized":"2024-03-12T03:52:46.502735+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'3868","last_scrub_stamp":"2024-03-12T03:52:19.581698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:52:19.581698+0000","objects_scrubbed":1911,"log_size":4585,"log_dups_size":0,"ondisk_log_size":4585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:38.458206+0000","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172228608,"num_objects":2628,"num_object_clones":0,"num_object_copies":7884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":717,"num_write_kb":45888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":744,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.526195+0000","last_change":"2024-03-12T03:50:59.586938+0000","last_active":"2024-03-12T03:52:33.526195+0000","last_peered":"2024-03-12T03:52:33.526195+0000","last_clean":"2024-03-12T03:52:33.526195+0000","last_became_active":"2024-03-12T03:50:58.840022+0000","last_became_peered":"2024-03-12T03:50:58.840022+0000","last_unstale":"2024-03-12T03:52:33.526195+0000","last_undegraded":"2024-03-12T03:52:33.526195+0000","last_fullsized":"2024-03-12T03:52:33.526195+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.586800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.586800+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"269'51","reported_seq":853,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.527713+0000","last_change":"2024-03-12T03:49:53.484809+0000","last_active":"2024-03-12T03:52:33.527713+0000","last_peered":"2024-03-12T03:52:33.527713+0000","last_clean":"2024-03-12T03:52:33.527713+0000","last_became_active":"2024-03-12T03:45:18.106503+0000","last_became_peered":"2024-03-12T03:45:18.106503+0000","last_unstale":"2024-03-12T03:52:33.527713+0000","last_undegraded":"2024-03-12T03:52:33.527713+0000","last_fullsized":"2024-03-12T03:52:33.527713+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'51","last_scrub_stamp":"2024-03-12T03:49:53.484646+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:49:53.484646+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":870,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524773+0000","last_change":"2024-03-12T03:51:34.290875+0000","last_active":"2024-03-12T03:52:33.524773+0000","last_peered":"2024-03-12T03:52:33.524773+0000","last_clean":"2024-03-12T03:52:33.524773+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:52:33.524773+0000","last_undegraded":"2024-03-12T03:52:33.524773+0000","last_fullsized":"2024-03-12T03:52:33.524773+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:34.290763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:51:34.290763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:45.217984+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":576,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.543824+0000","last_change":"2024-03-12T03:48:58.459230+0000","last_active":"2024-03-12T03:52:33.543824+0000","last_peered":"2024-03-12T03:52:33.543824+0000","last_clean":"2024-03-12T03:48:53.446740+0000","last_became_active":"2024-03-12T03:48:58.459230+0000","last_became_peered":"2024-03-12T03:48:58.459230+0000","last_unstale":"2024-03-12T03:52:33.543824+0000","last_undegraded":"2024-03-12T03:52:33.543824+0000","last_fullsized":"2024-03-12T03:48:58.447475+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.954731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.954731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":880,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.523574+0000","last_change":"2024-03-12T03:50:42.276895+0000","last_active":"2024-03-12T03:52:33.523574+0000","last_peered":"2024-03-12T03:52:33.523574+0000","last_clean":"2024-03-12T03:52:33.523574+0000","last_became_active":"2024-03-12T03:46:25.292122+0000","last_became_peered":"2024-03-12T03:46:25.292122+0000","last_unstale":"2024-03-12T03:52:33.523574+0000","last_undegraded":"2024-03-12T03:52:33.523574+0000","last_fullsized":"2024-03-12T03:52:33.523574+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:42.276562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:42.276562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"461'4415","reported_seq":4474,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:46.968669+0000","last_change":"2024-03-12T03:52:17.847287+0000","last_active":"2024-03-12T03:52:46.968669+0000","last_peered":"2024-03-12T03:52:46.968669+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:52:17.847287+0000","last_became_peered":"2024-03-12T03:52:17.847287+0000","last_unstale":"2024-03-12T03:52:46.968669+0000","last_undegraded":"2024-03-12T03:52:17.831115+0000","last_fullsized":"2024-03-12T03:52:17.830980+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:51.087109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168886272,"num_objects":2577,"num_object_clones":0,"num_object_copies":7731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2577,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":164928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":2577}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":816,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.522133+0000","last_change":"2024-03-12T03:51:27.598751+0000","last_active":"2024-03-12T03:52:33.522133+0000","last_peered":"2024-03-12T03:52:33.522133+0000","last_clean":"2024-03-12T03:52:33.522133+0000","last_became_active":"2024-03-12T03:50:58.836091+0000","last_became_peered":"2024-03-12T03:50:58.836091+0000","last_unstale":"2024-03-12T03:52:33.522133+0000","last_undegraded":"2024-03-12T03:52:33.522133+0000","last_fullsized":"2024-03-12T03:52:33.522133+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:27.598607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:51:27.598607+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":593,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.544193+0000","last_change":"2024-03-12T03:48:58.461069+0000","last_active":"2024-03-12T03:52:33.544193+0000","last_peered":"2024-03-12T03:52:33.544193+0000","last_clean":"2024-03-12T03:48:53.445898+0000","last_became_active":"2024-03-12T03:48:58.461069+0000","last_became_peered":"2024-03-12T03:48:58.461069+0000","last_unstale":"2024-03-12T03:52:33.544193+0000","last_undegraded":"2024-03-12T03:52:33.544193+0000","last_fullsized":"2024-03-12T03:48:58.444677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.981014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.981014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"269'30","reported_seq":565,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524254+0000","last_change":"2024-03-12T03:51:42.837070+0000","last_active":"2024-03-12T03:52:33.524254+0000","last_peered":"2024-03-12T03:52:33.524254+0000","last_clean":"2024-03-12T03:52:33.524254+0000","last_became_active":"2024-03-12T03:51:42.836618+0000","last_became_peered":"2024-03-12T03:51:42.836618+0000","last_unstale":"2024-03-12T03:52:33.524254+0000","last_undegraded":"2024-03-12T03:52:33.524254+0000","last_fullsized":"2024-03-12T03:52:33.524254+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'30","last_scrub_stamp":"2024-03-12T03:49:16.863622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:16.863622+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":870,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.525823+0000","last_change":"2024-03-12T03:50:39.308944+0000","last_active":"2024-03-12T03:52:33.525823+0000","last_peered":"2024-03-12T03:52:33.525823+0000","last_clean":"2024-03-12T03:52:33.525823+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:52:33.525823+0000","last_undegraded":"2024-03-12T03:52:33.525823+0000","last_fullsized":"2024-03-12T03:52:33.525823+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:39.308832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:50:39.308832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":595,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.527797+0000","last_change":"2024-03-12T03:51:41.674084+0000","last_active":"2024-03-12T03:52:33.527797+0000","last_peered":"2024-03-12T03:52:33.527797+0000","last_clean":"2024-03-12T03:52:33.527797+0000","last_became_active":"2024-03-12T03:46:25.280784+0000","last_became_peered":"2024-03-12T03:46:25.280784+0000","last_unstale":"2024-03-12T03:52:33.527797+0000","last_undegraded":"2024-03-12T03:52:33.527797+0000","last_fullsized":"2024-03-12T03:52:33.527797+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:41.673933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:41.673933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:04.398728+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"461'4484","reported_seq":4576,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:46.725325+0000","last_change":"2024-03-12T03:52:18.432284+0000","last_active":"2024-03-12T03:52:46.725325+0000","last_peered":"2024-03-12T03:52:46.725325+0000","last_clean":"2024-03-12T03:52:46.725325+0000","last_became_active":"2024-03-12T03:52:17.849018+0000","last_became_peered":"2024-03-12T03:52:17.849018+0000","last_unstale":"2024-03-12T03:52:46.725325+0000","last_undegraded":"2024-03-12T03:52:46.725325+0000","last_fullsized":"2024-03-12T03:52:46.725325+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'3756","last_scrub_stamp":"2024-03-12T03:52:18.431993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:52:18.431993+0000","objects_scrubbed":1910,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:26.640432+0000","scrub_duration":472,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174063616,"num_objects":2656,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":746,"num_write_kb":47744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":749,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.538311+0000","last_change":"2024-03-12T03:50:58.872833+0000","last_active":"2024-03-12T03:52:33.538311+0000","last_peered":"2024-03-12T03:52:33.538311+0000","last_clean":"2024-03-12T03:52:33.538311+0000","last_became_active":"2024-03-12T03:50:58.850740+0000","last_became_peered":"2024-03-12T03:50:58.850740+0000","last_unstale":"2024-03-12T03:52:33.538311+0000","last_undegraded":"2024-03-12T03:52:33.538311+0000","last_fullsized":"2024-03-12T03:52:33.538311+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:58.872716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:58.872716+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":1096,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:36.854431+0000","last_change":"2024-03-12T03:50:37.369892+0000","last_active":"2024-03-12T03:52:36.854431+0000","last_peered":"2024-03-12T03:52:36.854431+0000","last_clean":"2024-03-12T03:52:36.854431+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:52:36.854431+0000","last_undegraded":"2024-03-12T03:52:36.854431+0000","last_fullsized":"2024-03-12T03:52:36.854431+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:50:37.369758+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:50:37.369758+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"269'48","reported_seq":940,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.508168+0000","last_change":"2024-03-12T03:52:18.945880+0000","last_active":"2024-03-12T03:52:33.508168+0000","last_peered":"2024-03-12T03:52:33.508168+0000","last_clean":"2024-03-12T03:52:33.508168+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:52:33.508168+0000","last_undegraded":"2024-03-12T03:52:33.508168+0000","last_fullsized":"2024-03-12T03:52:33.508168+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'48","last_scrub_stamp":"2024-03-12T03:52:18.945763+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:52:18.945763+0000","objects_scrubbed":9,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:20.060348+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":861,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:33.525445+0000","last_change":"2024-03-12T03:48:58.456357+0000","last_active":"2024-03-12T03:52:33.525445+0000","last_peered":"2024-03-12T03:52:33.525445+0000","last_clean":"2024-03-12T03:48:53.450296+0000","last_became_active":"2024-03-12T03:48:58.456357+0000","last_became_peered":"2024-03-12T03:48:58.456357+0000","last_unstale":"2024-03-12T03:52:33.525445+0000","last_undegraded":"2024-03-12T03:48:58.445880+0000","last_fullsized":"2024-03-12T03:48:58.445769+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:47:05.588582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:05.588582+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":835,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.511092+0000","last_change":"2024-03-12T03:48:58.452652+0000","last_active":"2024-03-12T03:52:33.511092+0000","last_peered":"2024-03-12T03:52:33.511092+0000","last_clean":"2024-03-12T03:48:53.472055+0000","last_became_active":"2024-03-12T03:48:58.452652+0000","last_became_peered":"2024-03-12T03:48:58.452652+0000","last_unstale":"2024-03-12T03:52:33.511092+0000","last_undegraded":"2024-03-12T03:52:33.511092+0000","last_fullsized":"2024-03-12T03:48:58.441674+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:59.567890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.567890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.18","version":"461'4390","reported_seq":4457,"reported_epoch":461,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-12T03:52:43.574792+0000","last_change":"2024-03-12T03:52:21.215524+0000","last_active":"2024-03-12T03:52:43.574792+0000","last_peered":"2024-03-12T03:52:43.574792+0000","last_clean":"2024-03-12T03:52:18.828320+0000","last_became_active":"2024-03-12T03:52:21.192592+0000","last_became_peered":"2024-03-12T03:52:21.192592+0000","last_unstale":"2024-03-12T03:52:43.574792+0000","last_undegraded":"2024-03-12T03:52:21.029651+0000","last_fullsized":"2024-03-12T03:52:20.855058+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":452,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165478400,"num_objects":2525,"num_object_clones":0,"num_object_copies":7575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1927,"num_objects_misplaced":1927,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2525,"num_write_kb":161600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[3,6],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1927},{"shards":"3,6","objects":598}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":408,"reported_epoch":461,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:52:33.523452+0000","last_change":"2024-03-12T03:50:59.305188+0000","last_active":"2024-03-12T03:52:33.523452+0000","last_peered":"2024-03-12T03:52:33.523452+0000","last_clean":"2024-03-12T03:52:33.523452+0000","last_became_active":"2024-03-12T03:50:58.836425+0000","last_became_peered":"2024-03-12T03:50:58.836425+0000","last_unstale":"2024-03-12T03:52:33.523452+0000","last_undegraded":"2024-03-12T03:52:33.523452+0000","last_fullsized":"2024-03-12T03:52:33.523452+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.304950+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:50:06.384808+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.304950+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,1],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"269'18","reported_seq":658,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.526549+0000","last_change":"2024-03-12T03:50:48.504074+0000","last_active":"2024-03-12T03:52:33.526549+0000","last_peered":"2024-03-12T03:52:33.526549+0000","last_clean":"2024-03-12T03:52:33.526549+0000","last_became_active":"2024-03-12T03:46:25.288573+0000","last_became_peered":"2024-03-12T03:46:25.288573+0000","last_unstale":"2024-03-12T03:52:33.526549+0000","last_undegraded":"2024-03-12T03:52:33.526549+0000","last_fullsized":"2024-03-12T03:52:33.526549+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-03-12T03:50:48.503336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.503336+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":164,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.508330+0000","last_change":"2024-03-12T03:48:58.452359+0000","last_active":"2024-03-12T03:52:33.508330+0000","last_peered":"2024-03-12T03:52:33.508330+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:48:58.452359+0000","last_became_peered":"2024-03-12T03:48:58.452359+0000","last_unstale":"2024-03-12T03:52:33.508330+0000","last_undegraded":"2024-03-12T03:52:33.508330+0000","last_fullsized":"2024-03-12T03:48:58.441273+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:10.400135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.400135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":862,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.510545+0000","last_change":"2024-03-12T03:50:42.134792+0000","last_active":"2024-03-12T03:52:33.510545+0000","last_peered":"2024-03-12T03:52:33.510545+0000","last_clean":"2024-03-12T03:52:33.510545+0000","last_became_active":"2024-03-12T03:46:25.293521+0000","last_became_peered":"2024-03-12T03:46:25.293521+0000","last_unstale":"2024-03-12T03:52:33.510545+0000","last_undegraded":"2024-03-12T03:52:33.510545+0000","last_fullsized":"2024-03-12T03:52:33.510545+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:42.134482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:42.134482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":858,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524559+0000","last_change":"2024-03-12T03:51:38.361542+0000","last_active":"2024-03-12T03:52:33.524559+0000","last_peered":"2024-03-12T03:52:33.524559+0000","last_clean":"2024-03-12T03:52:33.524559+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:52:33.524559+0000","last_undegraded":"2024-03-12T03:52:33.524559+0000","last_fullsized":"2024-03-12T03:52:33.524559+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:38.361430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:38.361430+0000","last_clean_scrub_stamp":"2024-03-12T03:51:38.361430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.19","version":"461'4541","reported_seq":6374,"reported_epoch":461,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-12T03:52:43.716721+0000","last_change":"2024-03-12T03:52:21.105153+0000","last_active":"2024-03-12T03:52:43.716721+0000","last_peered":"2024-03-12T03:52:43.716721+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:52:21.087300+0000","last_became_peered":"2024-03-12T03:52:21.087300+0000","last_unstale":"2024-03-12T03:52:43.716721+0000","last_undegraded":"2024-03-12T03:52:21.016255+0000","last_fullsized":"2024-03-12T03:52:20.855244+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":452,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4541,"log_dups_size":0,"ondisk_log_size":4541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172425216,"num_objects":2631,"num_object_clones":0,"num_object_copies":7893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2631,"num_objects_misplaced":2858,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2631,"num_write_kb":168384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":591,"num_bytes_recovered":38731776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[3,2],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"2,3","objects":1202},{"shards":"3,6","objects":1429}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"269'27","reported_seq":784,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:33.525946+0000","last_change":"2024-03-12T03:48:58.462615+0000","last_active":"2024-03-12T03:52:33.525946+0000","last_peered":"2024-03-12T03:52:33.525946+0000","last_clean":"2024-03-12T03:43:13.333279+0000","last_became_active":"2024-03-12T03:48:58.462615+0000","last_became_peered":"2024-03-12T03:48:58.462615+0000","last_unstale":"2024-03-12T03:52:33.525946+0000","last_undegraded":"2024-03-12T03:48:58.443892+0000","last_fullsized":"2024-03-12T03:48:58.443761+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:48:06.104297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.104297+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":862,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.526297+0000","last_change":"2024-03-12T03:50:35.416894+0000","last_active":"2024-03-12T03:52:33.526297+0000","last_peered":"2024-03-12T03:52:33.526297+0000","last_clean":"2024-03-12T03:52:33.526297+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:52:33.526297+0000","last_undegraded":"2024-03-12T03:52:33.526297+0000","last_fullsized":"2024-03-12T03:52:33.526297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:35.416780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.416780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":629,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.526693+0000","last_change":"2024-03-12T03:52:19.693182+0000","last_active":"2024-03-12T03:52:33.526693+0000","last_peered":"2024-03-12T03:52:33.526693+0000","last_clean":"2024-03-12T03:52:33.526693+0000","last_became_active":"2024-03-12T03:46:25.286866+0000","last_became_peered":"2024-03-12T03:46:25.286866+0000","last_unstale":"2024-03-12T03:52:33.526693+0000","last_undegraded":"2024-03-12T03:52:33.526693+0000","last_fullsized":"2024-03-12T03:52:33.526693+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:52:19.693090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:52:19.693090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:28.404737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":589,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.555292+0000","last_change":"2024-03-12T03:50:37.807983+0000","last_active":"2024-03-12T03:52:33.555292+0000","last_peered":"2024-03-12T03:52:33.555292+0000","last_clean":"2024-03-12T03:52:33.555292+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:52:33.555292+0000","last_undegraded":"2024-03-12T03:52:33.555292+0000","last_fullsized":"2024-03-12T03:52:33.555292+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:37.807855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:37.807855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":878,"reported_epoch":461,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:52:33.523353+0000","last_change":"2024-03-12T03:50:57.227680+0000","last_active":"2024-03-12T03:52:33.523353+0000","last_peered":"2024-03-12T03:52:33.523353+0000","last_clean":"2024-03-12T03:52:33.523353+0000","last_became_active":"2024-03-12T03:50:56.823948+0000","last_became_peered":"2024-03-12T03:50:56.823948+0000","last_unstale":"2024-03-12T03:52:33.523353+0000","last_undegraded":"2024-03-12T03:52:33.523353+0000","last_fullsized":"2024-03-12T03:52:33.523353+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:57.227429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:57.227429+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"269'21","reported_seq":920,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.512973+0000","last_change":"2024-03-12T03:51:01.099499+0000","last_active":"2024-03-12T03:52:33.512973+0000","last_peered":"2024-03-12T03:52:33.512973+0000","last_clean":"2024-03-12T03:52:33.512973+0000","last_became_active":"2024-03-12T03:46:25.298007+0000","last_became_peered":"2024-03-12T03:46:25.298007+0000","last_unstale":"2024-03-12T03:52:33.512973+0000","last_undegraded":"2024-03-12T03:52:33.512973+0000","last_fullsized":"2024-03-12T03:52:33.512973+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'21","last_scrub_stamp":"2024-03-12T03:51:01.099350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:01.099350+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":738,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.523424+0000","last_change":"2024-03-12T03:48:58.462149+0000","last_active":"2024-03-12T03:52:33.523424+0000","last_peered":"2024-03-12T03:52:33.523424+0000","last_clean":"2024-03-12T03:43:13.331412+0000","last_became_active":"2024-03-12T03:48:58.462149+0000","last_became_peered":"2024-03-12T03:48:58.462149+0000","last_unstale":"2024-03-12T03:52:33.523424+0000","last_undegraded":"2024-03-12T03:52:33.523424+0000","last_fullsized":"2024-03-12T03:48:58.443909+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"332'14","reported_seq":884,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:36.854532+0000","last_change":"2024-03-12T03:50:53.568427+0000","last_active":"2024-03-12T03:52:36.854532+0000","last_peered":"2024-03-12T03:52:36.854532+0000","last_clean":"2024-03-12T03:52:36.854532+0000","last_became_active":"2024-03-12T03:46:25.300463+0000","last_became_peered":"2024-03-12T03:46:25.300463+0000","last_unstale":"2024-03-12T03:52:36.854532+0000","last_undegraded":"2024-03-12T03:52:36.854532+0000","last_fullsized":"2024-03-12T03:52:36.854532+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'14","last_scrub_stamp":"2024-03-12T03:50:53.568307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.568307+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":342,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.544376+0000","last_change":"2024-03-12T03:51:42.834189+0000","last_active":"2024-03-12T03:52:33.544376+0000","last_peered":"2024-03-12T03:52:33.544376+0000","last_clean":"2024-03-12T03:52:33.544376+0000","last_became_active":"2024-03-12T03:51:42.833867+0000","last_became_peered":"2024-03-12T03:51:42.833867+0000","last_unstale":"2024-03-12T03:52:33.544376+0000","last_undegraded":"2024-03-12T03:52:33.544376+0000","last_fullsized":"2024-03-12T03:52:33.544376+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:04.042226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:04.042226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":809,"reported_epoch":461,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:52:33.507883+0000","last_change":"2024-03-12T03:50:59.098952+0000","last_active":"2024-03-12T03:52:33.507883+0000","last_peered":"2024-03-12T03:52:33.507883+0000","last_clean":"2024-03-12T03:52:33.507883+0000","last_became_active":"2024-03-12T03:50:58.849455+0000","last_became_peered":"2024-03-12T03:50:58.849455+0000","last_unstale":"2024-03-12T03:52:33.507883+0000","last_undegraded":"2024-03-12T03:52:33.507883+0000","last_fullsized":"2024-03-12T03:52:33.507883+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.098619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.098619+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,3],"avail_no_missing":["5","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"403'39","reported_seq":728,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.522422+0000","last_change":"2024-03-12T03:51:40.548114+0000","last_active":"2024-03-12T03:52:33.522422+0000","last_peered":"2024-03-12T03:52:33.522422+0000","last_clean":"2024-03-12T03:52:33.522422+0000","last_became_active":"2024-03-12T03:46:25.299798+0000","last_became_peered":"2024-03-12T03:46:25.299798+0000","last_unstale":"2024-03-12T03:52:33.522422+0000","last_undegraded":"2024-03-12T03:52:33.522422+0000","last_fullsized":"2024-03-12T03:52:33.522422+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'39","last_scrub_stamp":"2024-03-12T03:51:40.547967+0000","last_deep_scrub":"403'39","last_deep_scrub_stamp":"2024-03-12T03:50:06.535772+0000","last_clean_scrub_stamp":"2024-03-12T03:51:40.547967+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:59.794998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":897,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.523633+0000","last_change":"2024-03-12T03:51:32.285686+0000","last_active":"2024-03-12T03:52:33.523633+0000","last_peered":"2024-03-12T03:52:33.523633+0000","last_clean":"2024-03-12T03:52:33.523633+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:52:33.523633+0000","last_undegraded":"2024-03-12T03:52:33.523633+0000","last_fullsized":"2024-03-12T03:52:33.523633+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:51:32.285604+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:51:32.285604+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:38.152150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"385'9","reported_seq":1081,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:41.855275+0000","last_change":"2024-03-12T03:48:58.448464+0000","last_active":"2024-03-12T03:52:41.855275+0000","last_peered":"2024-03-12T03:52:41.855275+0000","last_clean":"2024-03-12T03:48:56.813639+0000","last_became_active":"2024-03-12T03:48:58.448464+0000","last_became_peered":"2024-03-12T03:48:58.448464+0000","last_unstale":"2024-03-12T03:52:41.855275+0000","last_undegraded":"2024-03-12T03:48:58.442644+0000","last_fullsized":"2024-03-12T03:48:58.442315+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:53.458170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.458170+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":726,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.524751+0000","last_change":"2024-03-12T03:48:58.463189+0000","last_active":"2024-03-12T03:52:33.524751+0000","last_peered":"2024-03-12T03:52:33.524751+0000","last_clean":"2024-03-12T03:43:13.332853+0000","last_became_active":"2024-03-12T03:48:58.463189+0000","last_became_peered":"2024-03-12T03:48:58.463189+0000","last_unstale":"2024-03-12T03:52:33.524751+0000","last_undegraded":"2024-03-12T03:52:33.524751+0000","last_fullsized":"2024-03-12T03:48:58.445060+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.079569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.079569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"269'19","reported_seq":897,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.538382+0000","last_change":"2024-03-12T03:51:01.797152+0000","last_active":"2024-03-12T03:52:33.538382+0000","last_peered":"2024-03-12T03:52:33.538382+0000","last_clean":"2024-03-12T03:52:33.538382+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:52:33.538382+0000","last_undegraded":"2024-03-12T03:52:33.538382+0000","last_fullsized":"2024-03-12T03:52:33.538382+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'19","last_scrub_stamp":"2024-03-12T03:51:01.797018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:01.797018+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":60,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524107+0000","last_change":"2024-03-12T03:51:46.344521+0000","last_active":"2024-03-12T03:52:33.524107+0000","last_peered":"2024-03-12T03:52:33.524107+0000","last_clean":"2024-03-12T03:52:33.524107+0000","last_became_active":"2024-03-12T03:51:42.833922+0000","last_became_peered":"2024-03-12T03:51:42.833922+0000","last_unstale":"2024-03-12T03:52:33.524107+0000","last_undegraded":"2024-03-12T03:52:33.524107+0000","last_fullsized":"2024-03-12T03:52:33.524107+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:46.344443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_clean_scrub_stamp":"2024-03-12T03:51:46.344443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:08.297714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":871,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.527923+0000","last_change":"2024-03-12T03:50:48.723164+0000","last_active":"2024-03-12T03:52:33.527923+0000","last_peered":"2024-03-12T03:52:33.527923+0000","last_clean":"2024-03-12T03:52:33.527923+0000","last_became_active":"2024-03-12T03:46:25.288050+0000","last_became_peered":"2024-03-12T03:46:25.288050+0000","last_unstale":"2024-03-12T03:52:33.527923+0000","last_undegraded":"2024-03-12T03:52:33.527923+0000","last_fullsized":"2024-03-12T03:52:33.527923+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.723073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.723073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":632,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.542027+0000","last_change":"2024-03-12T03:48:58.456788+0000","last_active":"2024-03-12T03:52:33.542027+0000","last_peered":"2024-03-12T03:52:33.542027+0000","last_clean":"2024-03-12T03:48:53.447742+0000","last_became_active":"2024-03-12T03:48:58.456788+0000","last_became_peered":"2024-03-12T03:48:58.456788+0000","last_unstale":"2024-03-12T03:52:33.542027+0000","last_undegraded":"2024-03-12T03:52:33.542027+0000","last_fullsized":"2024-03-12T03:48:58.442213+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.433184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.433184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"269'15","reported_seq":461,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.510796+0000","last_change":"2024-03-12T03:52:20.985841+0000","last_active":"2024-03-12T03:52:33.510796+0000","last_peered":"2024-03-12T03:52:33.510796+0000","last_clean":"2024-03-12T03:52:33.510796+0000","last_became_active":"2024-03-12T03:43:01.037348+0000","last_became_peered":"2024-03-12T03:43:01.037348+0000","last_unstale":"2024-03-12T03:52:33.510796+0000","last_undegraded":"2024-03-12T03:52:33.510796+0000","last_fullsized":"2024-03-12T03:52:33.510796+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'15","last_scrub_stamp":"2024-03-12T03:52:20.985668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:52:20.985668+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:21.107290+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":888,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524661+0000","last_change":"2024-03-12T03:51:47.306023+0000","last_active":"2024-03-12T03:52:33.524661+0000","last_peered":"2024-03-12T03:52:33.524661+0000","last_clean":"2024-03-12T03:52:33.524661+0000","last_became_active":"2024-03-12T03:46:25.283602+0000","last_became_peered":"2024-03-12T03:46:25.283602+0000","last_unstale":"2024-03-12T03:52:33.524661+0000","last_undegraded":"2024-03-12T03:52:33.524661+0000","last_fullsized":"2024-03-12T03:52:33.524661+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:47.305919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:51:47.305919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:54.712288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":644,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.527341+0000","last_change":"2024-03-12T03:48:58.460970+0000","last_active":"2024-03-12T03:52:33.527341+0000","last_peered":"2024-03-12T03:52:33.527341+0000","last_clean":"2024-03-12T03:48:53.448497+0000","last_became_active":"2024-03-12T03:48:58.460970+0000","last_became_peered":"2024-03-12T03:48:58.460970+0000","last_unstale":"2024-03-12T03:52:33.527341+0000","last_undegraded":"2024-03-12T03:52:33.527341+0000","last_fullsized":"2024-03-12T03:48:58.444135+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_clean_scrub_stamp":"2024-03-12T03:48:10.625582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":162,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.508506+0000","last_change":"2024-03-12T03:48:58.453783+0000","last_active":"2024-03-12T03:52:33.508506+0000","last_peered":"2024-03-12T03:52:33.508506+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.453783+0000","last_became_peered":"2024-03-12T03:48:58.453783+0000","last_unstale":"2024-03-12T03:52:33.508506+0000","last_undegraded":"2024-03-12T03:52:33.508506+0000","last_fullsized":"2024-03-12T03:48:58.441749+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:13.228117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:48:13.228117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"269'26","reported_seq":668,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.542302+0000","last_change":"2024-03-12T03:51:31.509254+0000","last_active":"2024-03-12T03:52:33.542302+0000","last_peered":"2024-03-12T03:52:33.542302+0000","last_clean":"2024-03-12T03:52:33.542302+0000","last_became_active":"2024-03-12T03:46:25.287095+0000","last_became_peered":"2024-03-12T03:46:25.287095+0000","last_unstale":"2024-03-12T03:52:33.542302+0000","last_undegraded":"2024-03-12T03:52:33.542302+0000","last_fullsized":"2024-03-12T03:52:33.542302+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:51:31.509156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:31.509156+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:41.197038+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":868,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.510627+0000","last_change":"2024-03-12T03:51:42.097053+0000","last_active":"2024-03-12T03:52:33.510627+0000","last_peered":"2024-03-12T03:52:33.510627+0000","last_clean":"2024-03-12T03:52:33.510627+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:52:33.510627+0000","last_undegraded":"2024-03-12T03:52:33.510627+0000","last_fullsized":"2024-03-12T03:52:33.510627+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:42.096972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:51:42.096972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:57.485503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":862,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524182+0000","last_change":"2024-03-12T03:51:39.276331+0000","last_active":"2024-03-12T03:52:33.524182+0000","last_peered":"2024-03-12T03:52:33.524182+0000","last_clean":"2024-03-12T03:52:33.524182+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:52:33.524182+0000","last_undegraded":"2024-03-12T03:52:33.524182+0000","last_fullsized":"2024-03-12T03:52:33.524182+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.276250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.276250+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.276250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:51.293047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":659,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.527905+0000","last_change":"2024-03-12T03:48:58.449798+0000","last_active":"2024-03-12T03:52:33.527905+0000","last_peered":"2024-03-12T03:52:33.527905+0000","last_clean":"2024-03-12T03:48:53.448767+0000","last_became_active":"2024-03-12T03:48:58.449798+0000","last_became_peered":"2024-03-12T03:48:58.449798+0000","last_unstale":"2024-03-12T03:52:33.527905+0000","last_undegraded":"2024-03-12T03:52:33.527905+0000","last_fullsized":"2024-03-12T03:48:58.443857+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"269'10","reported_seq":639,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.530807+0000","last_change":"2024-03-12T03:50:39.504167+0000","last_active":"2024-03-12T03:52:33.530807+0000","last_peered":"2024-03-12T03:52:33.530807+0000","last_clean":"2024-03-12T03:52:33.530807+0000","last_became_active":"2024-03-12T03:46:25.286315+0000","last_became_peered":"2024-03-12T03:46:25.286315+0000","last_unstale":"2024-03-12T03:52:33.530807+0000","last_undegraded":"2024-03-12T03:52:33.530807+0000","last_fullsized":"2024-03-12T03:52:33.530807+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'10","last_scrub_stamp":"2024-03-12T03:50:39.504024+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:50:39.504024+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":738,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.511358+0000","last_change":"2024-03-12T03:48:58.463096+0000","last_active":"2024-03-12T03:52:33.511358+0000","last_peered":"2024-03-12T03:52:33.511358+0000","last_clean":"2024-03-12T03:43:13.332505+0000","last_became_active":"2024-03-12T03:48:58.463096+0000","last_became_peered":"2024-03-12T03:48:58.463096+0000","last_unstale":"2024-03-12T03:52:33.511358+0000","last_undegraded":"2024-03-12T03:52:33.511358+0000","last_fullsized":"2024-03-12T03:48:58.443705+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.851271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.851271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":325,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.536091+0000","last_change":"2024-03-12T03:51:00.809146+0000","last_active":"2024-03-12T03:52:33.536091+0000","last_peered":"2024-03-12T03:52:33.536091+0000","last_clean":"2024-03-12T03:52:33.536091+0000","last_became_active":"2024-03-12T03:45:18.109983+0000","last_became_peered":"2024-03-12T03:45:18.109983+0000","last_unstale":"2024-03-12T03:52:33.536091+0000","last_undegraded":"2024-03-12T03:52:33.536091+0000","last_fullsized":"2024-03-12T03:52:33.536091+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:00.809033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:00.809033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":870,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.525615+0000","last_change":"2024-03-12T03:52:17.430546+0000","last_active":"2024-03-12T03:52:33.525615+0000","last_peered":"2024-03-12T03:52:33.525615+0000","last_clean":"2024-03-12T03:52:33.525615+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:52:33.525615+0000","last_undegraded":"2024-03-12T03:52:33.525615+0000","last_fullsized":"2024-03-12T03:52:33.525615+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:52:17.430447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:52:17.430447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:24.521082+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"461'4435","reported_seq":7507,"reported_epoch":461,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-12T03:52:48.062054+0000","last_change":"2024-03-12T03:52:21.195518+0000","last_active":"2024-03-12T03:52:48.062054+0000","last_peered":"2024-03-12T03:52:48.062054+0000","last_clean":"2024-03-12T03:52:18.831099+0000","last_became_active":"2024-03-12T03:52:21.119326+0000","last_became_peered":"2024-03-12T03:52:21.119326+0000","last_unstale":"2024-03-12T03:52:48.062054+0000","last_undegraded":"2024-03-12T03:52:21.103663+0000","last_fullsized":"2024-03-12T03:52:20.853721+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":452,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170131456,"num_objects":2596,"num_object_clones":0,"num_object_copies":7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2596,"num_objects_misplaced":1766,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2596,"num_write_kb":166144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1002,"num_bytes_recovered":65667072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[1,2],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,2","objects":1713},{"shards":"1,3,7","objects":883}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"269'31","reported_seq":833,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.511598+0000","last_change":"2024-03-12T03:51:47.045902+0000","last_active":"2024-03-12T03:52:33.511598+0000","last_peered":"2024-03-12T03:52:33.511598+0000","last_clean":"2024-03-12T03:52:33.511598+0000","last_became_active":"2024-03-12T03:45:18.107047+0000","last_became_peered":"2024-03-12T03:45:18.107047+0000","last_unstale":"2024-03-12T03:52:33.511598+0000","last_undegraded":"2024-03-12T03:52:33.511598+0000","last_fullsized":"2024-03-12T03:52:33.511598+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:51:47.045769+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:51:47.045769+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:04.653195+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":660,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:33.531017+0000","last_change":"2024-03-12T03:48:58.454467+0000","last_active":"2024-03-12T03:52:33.531017+0000","last_peered":"2024-03-12T03:52:33.531017+0000","last_clean":"2024-03-12T03:48:53.477110+0000","last_became_active":"2024-03-12T03:48:58.454467+0000","last_became_peered":"2024-03-12T03:48:58.454467+0000","last_unstale":"2024-03-12T03:52:33.531017+0000","last_undegraded":"2024-03-12T03:48:58.439427+0000","last_fullsized":"2024-03-12T03:48:58.439320+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:48:00.458460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.458460+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":449,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.525894+0000","last_change":"2024-03-12T03:48:58.461994+0000","last_active":"2024-03-12T03:52:33.525894+0000","last_peered":"2024-03-12T03:52:33.525894+0000","last_clean":"2024-03-12T03:39:31.213817+0000","last_became_active":"2024-03-12T03:48:58.461994+0000","last_became_peered":"2024-03-12T03:48:58.461994+0000","last_unstale":"2024-03-12T03:52:33.525894+0000","last_undegraded":"2024-03-12T03:52:33.525894+0000","last_fullsized":"2024-03-12T03:48:58.446456+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:09.160277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:09.160277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":856,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.531309+0000","last_change":"2024-03-12T03:50:53.919752+0000","last_active":"2024-03-12T03:52:33.531309+0000","last_peered":"2024-03-12T03:52:33.531309+0000","last_clean":"2024-03-12T03:52:33.531309+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:52:33.531309+0000","last_undegraded":"2024-03-12T03:52:33.531309+0000","last_fullsized":"2024-03-12T03:52:33.531309+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:53.919661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.919661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"461'4362","reported_seq":6226,"reported_epoch":461,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-12T03:52:43.716492+0000","last_change":"2024-03-12T03:52:21.104954+0000","last_active":"2024-03-12T03:52:43.716492+0000","last_peered":"2024-03-12T03:52:43.716492+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:52:21.092386+0000","last_became_peered":"2024-03-12T03:52:21.092386+0000","last_unstale":"2024-03-12T03:52:43.716492+0000","last_undegraded":"2024-03-12T03:52:20.963781+0000","last_fullsized":"2024-03-12T03:52:20.855069+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":452,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4362,"log_dups_size":0,"ondisk_log_size":4362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165380096,"num_objects":2524,"num_object_clones":0,"num_object_copies":7572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1305,"num_objects_misplaced":1305,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2524,"num_write_kb":161504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":39387136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[3,2],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"2,3","objects":1219},{"shards":"3,6","objects":1305}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"269'27","reported_seq":655,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:33.525992+0000","last_change":"2024-03-12T03:48:58.462533+0000","last_active":"2024-03-12T03:52:33.525992+0000","last_peered":"2024-03-12T03:52:33.525992+0000","last_clean":"2024-03-12T03:43:13.333148+0000","last_became_active":"2024-03-12T03:48:58.462533+0000","last_became_peered":"2024-03-12T03:48:58.462533+0000","last_unstale":"2024-03-12T03:52:33.525992+0000","last_undegraded":"2024-03-12T03:48:58.446989+0000","last_fullsized":"2024-03-12T03:48:58.446871+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:15.795515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.795515+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"351'2","reported_seq":873,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524407+0000","last_change":"2024-03-12T03:49:14.611362+0000","last_active":"2024-03-12T03:52:33.524407+0000","last_peered":"2024-03-12T03:52:33.524407+0000","last_clean":"2024-03-12T03:52:33.524407+0000","last_became_active":"2024-03-12T03:46:25.283344+0000","last_became_peered":"2024-03-12T03:46:25.283344+0000","last_unstale":"2024-03-12T03:52:33.524407+0000","last_undegraded":"2024-03-12T03:52:33.524407+0000","last_fullsized":"2024-03-12T03:52:33.524407+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'2","last_scrub_stamp":"2024-03-12T03:49:14.611213+0000","last_deep_scrub":"351'2","last_deep_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_clean_scrub_stamp":"2024-03-12T03:49:14.611213+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"441'10","reported_seq":93,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:36.854377+0000","last_change":"2024-03-12T03:51:42.834336+0000","last_active":"2024-03-12T03:52:36.854377+0000","last_peered":"2024-03-12T03:52:36.854377+0000","last_clean":"2024-03-12T03:52:36.854377+0000","last_became_active":"2024-03-12T03:51:42.833920+0000","last_became_peered":"2024-03-12T03:51:42.833920+0000","last_unstale":"2024-03-12T03:52:36.854377+0000","last_undegraded":"2024-03-12T03:52:36.854377+0000","last_fullsized":"2024-03-12T03:52:36.854377+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'9","last_scrub_stamp":"2024-03-12T03:50:46.889538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:46.889538+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":878,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524414+0000","last_change":"2024-03-12T03:51:28.318948+0000","last_active":"2024-03-12T03:52:33.524414+0000","last_peered":"2024-03-12T03:52:33.524414+0000","last_clean":"2024-03-12T03:52:33.524414+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:52:33.524414+0000","last_undegraded":"2024-03-12T03:52:33.524414+0000","last_fullsized":"2024-03-12T03:52:33.524414+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:51:28.318860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:28.318860+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:51.019893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.12","version":"461'4509","reported_seq":8278,"reported_epoch":461,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-12T03:52:47.075777+0000","last_change":"2024-03-12T03:52:21.052825+0000","last_active":"2024-03-12T03:52:47.075777+0000","last_peered":"2024-03-12T03:52:47.075777+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:52:21.034538+0000","last_became_peered":"2024-03-12T03:52:21.034538+0000","last_unstale":"2024-03-12T03:52:47.075777+0000","last_undegraded":"2024-03-12T03:52:20.866369+0000","last_fullsized":"2024-03-12T03:52:20.853352+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":452,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172392448,"num_objects":2631,"num_object_clones":0,"num_object_copies":7893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2631,"num_objects_misplaced":44,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2631,"num_write_kb":168352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1854,"num_bytes_recovered":121503744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":2587}],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"269'24","reported_seq":697,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.522894+0000","last_change":"2024-03-12T03:50:05.585005+0000","last_active":"2024-03-12T03:52:33.522894+0000","last_peered":"2024-03-12T03:52:33.522894+0000","last_clean":"2024-03-12T03:52:33.522894+0000","last_became_active":"2024-03-12T03:46:25.283193+0000","last_became_peered":"2024-03-12T03:46:25.283193+0000","last_unstale":"2024-03-12T03:52:33.522894+0000","last_undegraded":"2024-03-12T03:52:33.522894+0000","last_fullsized":"2024-03-12T03:52:33.522894+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:50:05.584740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:05.584740+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":881,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.537713+0000","last_change":"2024-03-12T03:50:49.933782+0000","last_active":"2024-03-12T03:52:33.537713+0000","last_peered":"2024-03-12T03:52:33.537713+0000","last_clean":"2024-03-12T03:52:33.537713+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:52:33.537713+0000","last_undegraded":"2024-03-12T03:52:33.537713+0000","last_fullsized":"2024-03-12T03:52:33.537713+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:49.933653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:49.933653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":640,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.522662+0000","last_change":"2024-03-12T03:51:42.841557+0000","last_active":"2024-03-12T03:52:33.522662+0000","last_peered":"2024-03-12T03:52:33.522662+0000","last_clean":"2024-03-12T03:52:33.522662+0000","last_became_active":"2024-03-12T03:51:42.841245+0000","last_became_peered":"2024-03-12T03:51:42.841245+0000","last_unstale":"2024-03-12T03:52:33.522662+0000","last_undegraded":"2024-03-12T03:52:33.522662+0000","last_fullsized":"2024-03-12T03:52:33.522662+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:34.600111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:34.600111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:57.735269+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":675,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.526996+0000","last_change":"2024-03-12T03:51:28.687582+0000","last_active":"2024-03-12T03:52:33.526996+0000","last_peered":"2024-03-12T03:52:33.526996+0000","last_clean":"2024-03-12T03:52:33.526996+0000","last_became_active":"2024-03-12T03:46:25.296764+0000","last_became_peered":"2024-03-12T03:46:25.296764+0000","last_unstale":"2024-03-12T03:52:33.526996+0000","last_undegraded":"2024-03-12T03:52:33.526996+0000","last_fullsized":"2024-03-12T03:52:33.526996+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:28.687482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:28.687482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:49.176343+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"461'4476","reported_seq":4547,"reported_epoch":461,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-12T03:52:48.006602+0000","last_change":"2024-03-12T03:52:21.205436+0000","last_active":"2024-03-12T03:52:48.006602+0000","last_peered":"2024-03-12T03:52:48.006602+0000","last_clean":"2024-03-12T03:52:18.820819+0000","last_became_active":"2024-03-12T03:52:21.198549+0000","last_became_peered":"2024-03-12T03:52:21.198549+0000","last_unstale":"2024-03-12T03:52:48.006602+0000","last_undegraded":"2024-03-12T03:52:21.074295+0000","last_fullsized":"2024-03-12T03:52:20.855698+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":452,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4476,"log_dups_size":0,"ondisk_log_size":4476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173080576,"num_objects":2641,"num_object_clones":0,"num_object_copies":7923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3776,"num_objects_misplaced":2641,"num_objects_unfound":0,"num_objects_dirty":2641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2641,"num_write_kb":169024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[1,4],"avail_no_missing":["1","6","7"],"object_location_counts":[{"shards":"1,4","objects":753},{"shards":"1,6,7","objects":1888}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"269'27","reported_seq":648,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.555139+0000","last_change":"2024-03-12T03:51:38.702128+0000","last_active":"2024-03-12T03:52:33.555139+0000","last_peered":"2024-03-12T03:52:33.555139+0000","last_clean":"2024-03-12T03:52:33.555139+0000","last_became_active":"2024-03-12T03:46:25.301700+0000","last_became_peered":"2024-03-12T03:46:25.301700+0000","last_unstale":"2024-03-12T03:52:33.555139+0000","last_undegraded":"2024-03-12T03:52:33.555139+0000","last_fullsized":"2024-03-12T03:52:33.555139+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:51:38.702022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:38.702022+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:56.648527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":51,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.524022+0000","last_change":"2024-03-12T03:51:42.833539+0000","last_active":"2024-03-12T03:52:33.524022+0000","last_peered":"2024-03-12T03:52:33.524022+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:51:42.833539+0000","last_became_peered":"2024-03-12T03:51:42.833539+0000","last_unstale":"2024-03-12T03:52:33.524022+0000","last_undegraded":"2024-03-12T03:52:33.524022+0000","last_fullsized":"2024-03-12T03:51:42.829920+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.797296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.797296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":846,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.524551+0000","last_change":"2024-03-12T03:48:58.458438+0000","last_active":"2024-03-12T03:52:33.524551+0000","last_peered":"2024-03-12T03:52:33.524551+0000","last_clean":"2024-03-12T03:48:53.448127+0000","last_became_active":"2024-03-12T03:48:58.458438+0000","last_became_peered":"2024-03-12T03:48:58.458438+0000","last_unstale":"2024-03-12T03:52:33.524551+0000","last_undegraded":"2024-03-12T03:52:33.524551+0000","last_fullsized":"2024-03-12T03:48:58.440067+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.016030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.016030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":851,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.537513+0000","last_change":"2024-03-12T03:50:51.952305+0000","last_active":"2024-03-12T03:52:33.537513+0000","last_peered":"2024-03-12T03:52:33.537513+0000","last_clean":"2024-03-12T03:52:33.537513+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:52:33.537513+0000","last_undegraded":"2024-03-12T03:52:33.537513+0000","last_fullsized":"2024-03-12T03:52:33.537513+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:51.952189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:51.952189+0000","last_clean_scrub_stamp":"2024-03-12T03:50:51.952189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.14","version":"461'4518","reported_seq":6956,"reported_epoch":461,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-12T03:52:47.060053+0000","last_change":"2024-03-12T03:52:21.254066+0000","last_active":"2024-03-12T03:52:47.060053+0000","last_peered":"2024-03-12T03:52:47.060053+0000","last_clean":"2024-03-12T03:52:18.819587+0000","last_became_active":"2024-03-12T03:52:21.192353+0000","last_became_peered":"2024-03-12T03:52:21.192353+0000","last_unstale":"2024-03-12T03:52:47.060053+0000","last_undegraded":"2024-03-12T03:52:47.060053+0000","last_fullsized":"2024-03-12T03:52:20.851854+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":452,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4518,"log_dups_size":0,"ondisk_log_size":4518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174522368,"num_objects":2663,"num_object_clones":0,"num_object_copies":7989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2176,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2663,"num_write_kb":170432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":51838976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[2,6],"avail_no_missing":["2","6","7"],"object_location_counts":[{"shards":"2,6","objects":1575},{"shards":"2,6,7","objects":1088}],"blocked_by":[],"up_primary":4,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"269'16","reported_seq":697,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.527235+0000","last_change":"2024-03-12T03:51:42.838878+0000","last_active":"2024-03-12T03:52:33.527235+0000","last_peered":"2024-03-12T03:52:33.527235+0000","last_clean":"2024-03-12T03:52:33.527235+0000","last_became_active":"2024-03-12T03:51:42.838579+0000","last_became_peered":"2024-03-12T03:51:42.838579+0000","last_unstale":"2024-03-12T03:52:33.527235+0000","last_undegraded":"2024-03-12T03:52:33.527235+0000","last_fullsized":"2024-03-12T03:52:33.527235+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'16","last_scrub_stamp":"2024-03-12T03:49:51.806651+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:49:51.806651+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":851,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.509626+0000","last_change":"2024-03-12T03:50:02.319955+0000","last_active":"2024-03-12T03:52:33.509626+0000","last_peered":"2024-03-12T03:52:33.509626+0000","last_clean":"2024-03-12T03:52:33.509626+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:52:33.509626+0000","last_undegraded":"2024-03-12T03:52:33.509626+0000","last_fullsized":"2024-03-12T03:52:33.509626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:02.319837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:02.319837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":853,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.523761+0000","last_change":"2024-03-12T03:51:01.307708+0000","last_active":"2024-03-12T03:52:33.523761+0000","last_peered":"2024-03-12T03:52:33.523761+0000","last_clean":"2024-03-12T03:52:33.523761+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:52:33.523761+0000","last_undegraded":"2024-03-12T03:52:33.523761+0000","last_fullsized":"2024-03-12T03:52:33.523761+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:01.307529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:51:01.307529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":857,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.510104+0000","last_change":"2024-03-12T03:51:46.135297+0000","last_active":"2024-03-12T03:52:33.510104+0000","last_peered":"2024-03-12T03:52:33.510104+0000","last_clean":"2024-03-12T03:52:33.510104+0000","last_became_active":"2024-03-12T03:46:25.293777+0000","last_became_peered":"2024-03-12T03:46:25.293777+0000","last_unstale":"2024-03-12T03:52:33.510104+0000","last_undegraded":"2024-03-12T03:52:33.510104+0000","last_fullsized":"2024-03-12T03:52:33.510104+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:46.135156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:46.135156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:53:02.988360+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.15","version":"461'4666","reported_seq":4739,"reported_epoch":461,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-12T03:52:46.715316+0000","last_change":"2024-03-12T03:52:21.199945+0000","last_active":"2024-03-12T03:52:46.715316+0000","last_peered":"2024-03-12T03:52:46.715316+0000","last_clean":"2024-03-12T03:52:18.837629+0000","last_became_active":"2024-03-12T03:52:21.196513+0000","last_became_peered":"2024-03-12T03:52:21.196513+0000","last_unstale":"2024-03-12T03:52:46.715316+0000","last_undegraded":"2024-03-12T03:52:20.911592+0000","last_fullsized":"2024-03-12T03:52:20.853827+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":452,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180584448,"num_objects":2756,"num_object_clones":0,"num_object_copies":8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4050,"num_objects_misplaced":2025,"num_objects_unfound":0,"num_objects_dirty":2756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2756,"num_write_kb":176352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[5,1],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"1,5","objects":731},{"shards":"5,6,7","objects":2025}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"345'33","reported_seq":727,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:33.524936+0000","last_change":"2024-03-12T03:51:42.838076+0000","last_active":"2024-03-12T03:52:33.524936+0000","last_peered":"2024-03-12T03:52:33.524936+0000","last_clean":"2024-03-12T03:47:02.086867+0000","last_became_active":"2024-03-12T03:51:42.838076+0000","last_became_peered":"2024-03-12T03:51:42.838076+0000","last_unstale":"2024-03-12T03:52:33.524936+0000","last_undegraded":"2024-03-12T03:51:42.833048+0000","last_fullsized":"2024-03-12T03:51:42.832909+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'33","last_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_deep_scrub":"345'33","last_deep_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_clean_scrub_stamp":"2024-03-12T03:47:02.110517+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":859,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.512582+0000","last_change":"2024-03-12T03:50:41.172421+0000","last_active":"2024-03-12T03:52:33.512582+0000","last_peered":"2024-03-12T03:52:33.512582+0000","last_clean":"2024-03-12T03:52:33.512582+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:52:33.512582+0000","last_undegraded":"2024-03-12T03:52:33.512582+0000","last_fullsized":"2024-03-12T03:52:33.512582+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:41.172313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:50:41.172313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":838,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.511432+0000","last_change":"2024-03-12T03:48:58.453634+0000","last_active":"2024-03-12T03:52:33.511432+0000","last_peered":"2024-03-12T03:52:33.511432+0000","last_clean":"2024-03-12T03:48:53.472261+0000","last_became_active":"2024-03-12T03:48:58.453634+0000","last_became_peered":"2024-03-12T03:48:58.453634+0000","last_unstale":"2024-03-12T03:52:33.511432+0000","last_undegraded":"2024-03-12T03:52:33.511432+0000","last_fullsized":"2024-03-12T03:48:58.444137+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:02.618599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_clean_scrub_stamp":"2024-03-12T03:46:02.618599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":162,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.530285+0000","last_change":"2024-03-12T03:48:58.463158+0000","last_active":"2024-03-12T03:52:33.530285+0000","last_peered":"2024-03-12T03:52:33.530285+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.463158+0000","last_became_peered":"2024-03-12T03:48:58.463158+0000","last_unstale":"2024-03-12T03:52:33.530285+0000","last_undegraded":"2024-03-12T03:52:33.530285+0000","last_fullsized":"2024-03-12T03:48:58.439250+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_clean_scrub_stamp":"2024-03-12T03:45:58.095196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.16","version":"461'4453","reported_seq":4514,"reported_epoch":461,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-12T03:52:48.006680+0000","last_change":"2024-03-12T03:52:21.133554+0000","last_active":"2024-03-12T03:52:48.006680+0000","last_peered":"2024-03-12T03:52:48.006680+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:52:21.129706+0000","last_became_peered":"2024-03-12T03:52:21.129706+0000","last_unstale":"2024-03-12T03:52:48.006680+0000","last_undegraded":"2024-03-12T03:52:21.118804+0000","last_fullsized":"2024-03-12T03:52:20.855466+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":452,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171376640,"num_objects":2615,"num_object_clones":0,"num_object_copies":7845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3724,"num_objects_misplaced":1862,"num_objects_unfound":0,"num_objects_dirty":2615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":167360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[1,3],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1862},{"shards":"1,3","objects":753}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"269'62","reported_seq":811,"reported_epoch":461,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:52:33.538799+0000","last_change":"2024-03-12T03:48:58.457134+0000","last_active":"2024-03-12T03:52:33.538799+0000","last_peered":"2024-03-12T03:52:33.538799+0000","last_clean":"2024-03-12T03:48:53.477142+0000","last_became_active":"2024-03-12T03:48:58.457134+0000","last_became_peered":"2024-03-12T03:48:58.457134+0000","last_unstale":"2024-03-12T03:52:33.538799+0000","last_undegraded":"2024-03-12T03:48:58.441299+0000","last_fullsized":"2024-03-12T03:48:58.441188+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":692,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.525844+0000","last_change":"2024-03-12T03:50:37.469343+0000","last_active":"2024-03-12T03:52:33.525844+0000","last_peered":"2024-03-12T03:52:33.525844+0000","last_clean":"2024-03-12T03:52:33.525844+0000","last_became_active":"2024-03-12T03:46:25.279131+0000","last_became_peered":"2024-03-12T03:46:25.279131+0000","last_unstale":"2024-03-12T03:52:33.525844+0000","last_undegraded":"2024-03-12T03:52:33.525844+0000","last_fullsized":"2024-03-12T03:52:33.525844+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:37.469265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:50:37.469265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":52,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.530553+0000","last_change":"2024-03-12T03:51:42.832852+0000","last_active":"2024-03-12T03:52:33.530553+0000","last_peered":"2024-03-12T03:52:33.530553+0000","last_clean":"2024-03-12T03:52:33.530553+0000","last_became_active":"2024-03-12T03:51:42.832427+0000","last_became_peered":"2024-03-12T03:51:42.832427+0000","last_unstale":"2024-03-12T03:52:33.530553+0000","last_undegraded":"2024-03-12T03:52:33.530553+0000","last_fullsized":"2024-03-12T03:52:33.530553+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:39.866768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:50:39.866768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":642,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.527702+0000","last_change":"2024-03-12T03:49:55.557456+0000","last_active":"2024-03-12T03:52:33.527702+0000","last_peered":"2024-03-12T03:52:33.527702+0000","last_clean":"2024-03-12T03:52:33.527702+0000","last_became_active":"2024-03-12T03:46:25.281081+0000","last_became_peered":"2024-03-12T03:46:25.281081+0000","last_unstale":"2024-03-12T03:52:33.527702+0000","last_undegraded":"2024-03-12T03:52:33.527702+0000","last_fullsized":"2024-03-12T03:52:33.527702+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:55.557335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:55.557335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"461'4507","reported_seq":4635,"reported_epoch":461,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-12T03:52:48.039017+0000","last_change":"2024-03-12T03:52:47.479305+0000","last_active":"2024-03-12T03:52:48.039017+0000","last_peered":"2024-03-12T03:52:48.039017+0000","last_clean":"2024-03-12T03:52:18.827402+0000","last_became_active":"2024-03-12T03:52:21.110098+0000","last_became_peered":"2024-03-12T03:52:21.110098+0000","last_unstale":"2024-03-12T03:52:48.039017+0000","last_undegraded":"2024-03-12T03:52:48.039017+0000","last_fullsized":"2024-03-12T03:52:20.858048+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":452,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4507,"log_dups_size":0,"ondisk_log_size":4507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173703168,"num_objects":2651,"num_object_clones":0,"num_object_copies":7953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3724,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2651,"num_write_kb":169632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[1,5],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":1862},{"shards":"1,5","objects":789}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"269'26","reported_seq":899,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.536240+0000","last_change":"2024-03-12T03:50:57.888885+0000","last_active":"2024-03-12T03:52:33.536240+0000","last_peered":"2024-03-12T03:52:33.536240+0000","last_clean":"2024-03-12T03:52:33.536240+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:52:33.536240+0000","last_undegraded":"2024-03-12T03:52:33.536240+0000","last_fullsized":"2024-03-12T03:52:33.536240+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:50:57.888633+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:50:57.888633+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":456,"reported_epoch":461,"state":"active+undersized","last_fresh":"2024-03-12T03:52:33.511942+0000","last_change":"2024-03-12T03:48:58.462711+0000","last_active":"2024-03-12T03:52:33.511942+0000","last_peered":"2024-03-12T03:52:33.511942+0000","last_clean":"2024-03-12T03:39:28.536135+0000","last_became_active":"2024-03-12T03:48:58.462711+0000","last_became_peered":"2024-03-12T03:48:58.462711+0000","last_unstale":"2024-03-12T03:52:33.511942+0000","last_undegraded":"2024-03-12T03:52:33.511942+0000","last_fullsized":"2024-03-12T03:48:58.443314+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:12.186146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:48:12.186146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":860,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.524458+0000","last_change":"2024-03-12T03:50:04.535064+0000","last_active":"2024-03-12T03:52:33.524458+0000","last_peered":"2024-03-12T03:52:33.524458+0000","last_clean":"2024-03-12T03:52:33.524458+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:52:33.524458+0000","last_undegraded":"2024-03-12T03:52:33.524458+0000","last_fullsized":"2024-03-12T03:52:33.524458+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_clean_scrub_stamp":"2024-03-12T03:50:04.534937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":53,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-03-12T03:52:33.510999+0000","last_change":"2024-03-12T03:51:42.833194+0000","last_active":"2024-03-12T03:52:33.510999+0000","last_peered":"2024-03-12T03:52:33.510999+0000","last_clean":"2024-03-12T03:52:33.510999+0000","last_became_active":"2024-03-12T03:51:42.832894+0000","last_became_peered":"2024-03-12T03:51:42.832894+0000","last_unstale":"2024-03-12T03:52:33.510999+0000","last_undegraded":"2024-03-12T03:52:33.510999+0000","last_fullsized":"2024-03-12T03:52:33.510999+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:32.662924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_clean_scrub_stamp":"2024-03-12T03:51:32.662924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:52:49.792164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":26,"stat_sum":{"num_bytes":5495914496,"num_objects":83862,"num_object_clones":0,"num_object_copies":251586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38136,"num_objects_misplaced":20328,"num_objects_unfound":0,"num_objects_dirty":83862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72431,"num_write_kb":4635520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4863,"num_bytes_recovered":318701568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15781658624,"data_stored":15781658624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115193,"ondisk_log_size":115193,"up":69,"acting":63,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1156,"num_read_kb":1151,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":19,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":933560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2177,"ondisk_log_size":2177,"up":49,"acting":53,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1393,"num_read_kb":1118,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":2642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":83,"acting":83,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":87,"acting":87,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070702,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3156720,"kb_used_data":2255080,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":90588432,"statfs":{"total":95995035648,"available":92762554368,"internally_reserved":0,"allocated":2309201920,"data_stored":2308603374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":923269578},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332339,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3406616,"kb_used_data":2645200,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":90338536,"statfs":{"total":95995035648,"available":92506660864,"internally_reserved":0,"allocated":2708684800,"data_stored":2708108867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":779680198},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575116,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3792852,"kb_used_data":2845388,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":89952300,"statfs":{"total":95995035648,"available":92111155200,"internally_reserved":0,"allocated":2913677312,"data_stored":2913042037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":970193349},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738595,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3725208,"kb_used_data":3102608,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":90019944,"statfs":{"total":95995035648,"available":92180422656,"internally_reserved":0,"allocated":3177070592,"data_stored":3175997927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411590,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3175320,"kb_used_data":2372240,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":90569832,"statfs":{"total":95995035648,"available":92743507968,"internally_reserved":0,"allocated":2429173760,"data_stored":2428599384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":822344133},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":330,"seq":1417339207757,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2694660,"kb_used_data":2281084,"kb_used_omap":13,"kb_used_meta":413554,"kb_avail":91050492,"statfs":{"total":95995035648,"available":93235703808,"internally_reserved":0,"allocated":2335830016,"data_stored":2335210898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13485,"internal_metadata":423480147},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953726,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2217076,"kb_used_data":1505644,"kb_used_omap":1,"kb_used_meta":711422,"kb_avail":91528076,"statfs":{"total":95995035648,"available":93724749824,"internally_reserved":0,"allocated":1541779456,"data_stored":1541176354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":728496581},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659384,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2296608,"kb_used_data":1394968,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":91448544,"statfs":{"total":95995035648,"available":93643309056,"internally_reserved":0,"allocated":1428447232,"data_stored":1427864074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2940665856,"data_stored":2940665856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2428895232,"data_stored":2428895232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2348482560,"data_stored":2348482560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1565261824,"data_stored":1565261824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1443299328,"data_stored":1443299328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2736717824,"data_stored":2736717824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2318336000,"data_stored":2318336000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:53:04.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool set unique_pool_4 pgp_num 26 2024-03-12T03:53:04.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:03 smithi173 ceph-mon[33544]: pgmap v956: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 5.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 901 op/s; 38356/290049 objects degraded (13.224%); 19293/290049 objects misplaced (6.652%); 12 MiB/s, 189 objects/s recovering 2024-03-12T03:53:04.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:03 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 38356/290049 objects degraded (13.224%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:04.267 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:04.332 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:04.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:04.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:04.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:04.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:04.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:04.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:04.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:04.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:04.736 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:04.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:04.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:04.879 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:04.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:04.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:05.020 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:05.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:05.103 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 75 16 100874 100858 84.0285 52.875 0.0118374 0.0118959 2024-03-12T03:53:05.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:05.161 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:05.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:05.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:05.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:05.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:05.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:05.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:05.312 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:05.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:05.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:05.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:05.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:05.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:05.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:05.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:05.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:05.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:05.707 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:05.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:05.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:05.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:05.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:05.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:05.982 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:05.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:06.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:06.103 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 76 16 101587 101571 83.5093 44.5625 0.00399385 0.011964 2024-03-12T03:53:06.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:06.123 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:06.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:06.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:05 smithi169 ceph-mon[29344]: pgmap v957: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 954 B/s rd, 60 MiB/s wr, 961 op/s; 38949/297012 objects degraded (13.114%); 17936/297012 objects misplaced (6.039%); 14 MiB/s, 219 objects/s recovering 2024-03-12T03:53:06.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:05 smithi169 ceph-mon[39949]: pgmap v957: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 954 B/s rd, 60 MiB/s wr, 961 op/s; 38949/297012 objects degraded (13.114%); 17936/297012 objects misplaced (6.039%); 14 MiB/s, 219 objects/s recovering 2024-03-12T03:53:06.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:06.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:05 smithi173 ceph-mon[33544]: pgmap v957: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 954 B/s rd, 60 MiB/s wr, 961 op/s; 38949/297012 objects degraded (13.114%); 17936/297012 objects misplaced (6.039%); 14 MiB/s, 219 objects/s recovering 2024-03-12T03:53:06.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:06.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:06.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:06.360 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:06.465 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:06.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:06.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:06.603 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:06.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:06.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:06.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:06.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:06.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:06.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:06.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:06.861 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:06.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:06.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:06.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:06.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:07.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:07.103 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 77 16 101977 101961 82.7414 24.375 0.00280325 0.0120759 2024-03-12T03:53:07.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:07.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:07.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:07.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:06 smithi169 ceph-mon[29344]: osdmap e466: 8 total, 7 up, 8 in 2024-03-12T03:53:07.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:06 smithi169 ceph-mon[39949]: osdmap e466: 8 total, 7 up, 8 in 2024-03-12T03:53:07.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:06 smithi173 ceph-mon[33544]: osdmap e466: 8 total, 7 up, 8 in 2024-03-12T03:53:07.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:07.272 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:07.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:07.286 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:53:07.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:07.403 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:07.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:07.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:07.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:07.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:07.638 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:07.679 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:07.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:07.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:07.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:07.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:07.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:07.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:07.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:08.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:08.075 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:08.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:08.103 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 78 16 102728 102712 82.2823 46.9375 0.00737901 0.0121479 2024-03-12T03:53:08.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:07 smithi169 ceph-mon[29344]: pgmap v959: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 896 op/s; 38949/297012 objects degraded (13.114%); 17936/297012 objects misplaced (6.039%); 13 MiB/s, 202 objects/s recovering 2024-03-12T03:53:08.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:07 smithi169 ceph-mon[29344]: osdmap e467: 8 total, 7 up, 8 in 2024-03-12T03:53:08.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:07 smithi169 ceph-mon[39949]: pgmap v959: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 896 op/s; 38949/297012 objects degraded (13.114%); 17936/297012 objects misplaced (6.039%); 13 MiB/s, 202 objects/s recovering 2024-03-12T03:53:08.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:07 smithi169 ceph-mon[39949]: osdmap e467: 8 total, 7 up, 8 in 2024-03-12T03:53:08.176 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:08.214 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:08.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:08.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:08.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:07 smithi173 ceph-mon[33544]: pgmap v959: 176 pgs: 5 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 896 op/s; 38949/297012 objects degraded (13.114%); 17936/297012 objects misplaced (6.039%); 13 MiB/s, 202 objects/s recovering 2024-03-12T03:53:08.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:07 smithi173 ceph-mon[33544]: osdmap e467: 8 total, 7 up, 8 in 2024-03-12T03:53:08.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:08.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:08.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:08.456 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:08.501 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:08.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:08.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:08.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:08.633 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:08.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:08.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:08.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:08.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:08.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:08.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:08.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:08.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:08.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:09.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:09.049 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:09.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:09.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 79 16 103686 103670 81.9981 59.875 0.00599735 0.0121909 2024-03-12T03:53:09.152 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:09.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:08 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 38949/297012 objects degraded (13.114%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:09.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:08 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 38949/297012 objects degraded (13.114%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:09.175 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:09.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:08 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 38949/297012 objects degraded (13.114%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:09.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:09.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:09.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:09.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:09.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:09.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:09.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:09.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:09.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:09.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:09.748 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:09.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:09.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:09.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:09.901 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:09.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:09.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:10.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:10.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:10.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:10.104 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:2024-03-12T03:53:10.103081+0000 min lat: 0.00121451 max lat: 2.38513 avg lat: 0.0122446 2024-03-12T03:53:10.104 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-12T03:53:10.104 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 80 16 104528 104512 81.6308 52.625 0.0108304 0.0122446 2024-03-12T03:53:10.147 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:10.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:10.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:10.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:10.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:09 smithi173 ceph-mon[33544]: pgmap v961: 176 pgs: 6 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 882 op/s; 31520/304731 objects degraded (10.344%); 24254/304731 objects misplaced (7.959%); 15 MiB/s, 236 objects/s recovering 2024-03-12T03:53:10.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:09 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:53:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:09 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:53:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:09 smithi173 ceph-mon[33544]: osdmap e468: 8 total, 7 up, 8 in 2024-03-12T03:53:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:09 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:53:10.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:10.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:10.327 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:10.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:10.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:09 smithi169 ceph-mon[29344]: pgmap v961: 176 pgs: 6 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 882 op/s; 31520/304731 objects degraded (10.344%); 24254/304731 objects misplaced (7.959%); 15 MiB/s, 236 objects/s recovering 2024-03-12T03:53:10.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:09 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:53:10.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:09 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:53:10.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:09 smithi169 ceph-mon[29344]: osdmap e468: 8 total, 7 up, 8 in 2024-03-12T03:53:10.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:09 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:53:10.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:09 smithi169 ceph-mon[39949]: pgmap v961: 176 pgs: 6 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 882 op/s; 31520/304731 objects degraded (10.344%); 24254/304731 objects misplaced (7.959%); 15 MiB/s, 236 objects/s recovering 2024-03-12T03:53:10.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:09 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:53:10.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:09 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:53:10.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:09 smithi169 ceph-mon[39949]: osdmap e468: 8 total, 7 up, 8 in 2024-03-12T03:53:10.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:09 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:53:10.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:10.467 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:10.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:10.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:10.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:10.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:10.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:10.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:10.736 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:10.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:10.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:10.887 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:10.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:10.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:11.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:11.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:11.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:11.085 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:10 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:53:11.085 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:10 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:53:11.085 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:10 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:53:11.085 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:10 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:53:11.085 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:10 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:53:11.085 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:10 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:53:11.104 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 81 16 105526 105510 81.393 62.375 0.00997439 0.0122809 2024-03-12T03:53:11.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:11.141 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:11.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:11.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:10 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:53:11.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:10 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:53:11.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:10 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:53:11.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:11.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:11.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:11.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:11.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:11.420 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:11.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:11.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:11.559 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:11.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:11.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:11.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:11.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:11.792 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:11.826 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:11.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:11.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:11.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:11.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:11.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:12.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:12.083 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:12.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:12.104 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 82 16 106088 106072 80.8288 35.125 0.00408363 0.0122798 2024-03-12T03:53:12.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:11 smithi169 ceph-mon[29344]: pgmap v963: 176 pgs: 6 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 852 B/s rd, 39 MiB/s wr, 625 op/s; 31841/308256 objects degraded (10.329%); 23479/308256 objects misplaced (7.617%); 14 MiB/s, 221 objects/s recovering 2024-03-12T03:53:12.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:11 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/996193294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:53:12.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:11 smithi169 ceph-mon[29344]: osdmap e469: 8 total, 7 up, 8 in 2024-03-12T03:53:12.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:11 smithi169 ceph-mon[39949]: pgmap v963: 176 pgs: 6 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 852 B/s rd, 39 MiB/s wr, 625 op/s; 31841/308256 objects degraded (10.329%); 23479/308256 objects misplaced (7.617%); 14 MiB/s, 221 objects/s recovering 2024-03-12T03:53:12.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:11 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/996193294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:53:12.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:11 smithi169 ceph-mon[39949]: osdmap e469: 8 total, 7 up, 8 in 2024-03-12T03:53:12.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:53:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:53:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:53:12.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:12.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:12.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:12.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:12.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:11 smithi173 ceph-mon[33544]: pgmap v963: 176 pgs: 6 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 852 B/s rd, 39 MiB/s wr, 625 op/s; 31841/308256 objects degraded (10.329%); 23479/308256 objects misplaced (7.617%); 14 MiB/s, 221 objects/s recovering 2024-03-12T03:53:12.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:11 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/996193294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:53:12.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:11 smithi173 ceph-mon[33544]: osdmap e469: 8 total, 7 up, 8 in 2024-03-12T03:53:12.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:12.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:12.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:12.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:12.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:12.482 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:12.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:12.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:12.622 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:12.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:12.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:12.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:12.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:12.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:12.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:12.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:12.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:12.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:13.027 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:13.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:13.104 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 83 16 106826 106810 80.4107 46.125 0.0155303 0.0124304 2024-03-12T03:53:13.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:13.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:13.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:13.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:13.271 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:53:13.272 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:53:13.272 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:53:13.272 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:53:13.272 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:53:13.272 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:53:13.272 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:53:13.272 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:53:13.272 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:53:13.272 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:53:13.272 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:53:13.272 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:53:13.273 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:53:13.273 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:53:13.273 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:53:13.306 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:13.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:13.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:13.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:13.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:13.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:13.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:13.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:13.567 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:13.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:13.668 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:13 smithi169 ceph-mon[29344]: osdmap e470: 8 total, 7 up, 8 in 2024-03-12T03:53:13.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:13 smithi169 ceph-mon[29344]: pgmap v966: 176 pgs: 1 creating+peering, 6 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 959 B/s rd, 53 MiB/s wr, 856 op/s; 32082/314391 objects degraded (10.204%); 22723/314391 objects misplaced (7.228%); 20 MiB/s, 314 objects/s recovering 2024-03-12T03:53:13.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:13 smithi169 ceph-mon[39949]: osdmap e470: 8 total, 7 up, 8 in 2024-03-12T03:53:13.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:13 smithi169 ceph-mon[39949]: pgmap v966: 176 pgs: 1 creating+peering, 6 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 959 B/s rd, 53 MiB/s wr, 856 op/s; 32082/314391 objects degraded (10.204%); 22723/314391 objects misplaced (7.228%); 20 MiB/s, 314 objects/s recovering 2024-03-12T03:53:13.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:13.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:13.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:13.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:53:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:53:13.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:13 smithi173 ceph-mon[33544]: osdmap e470: 8 total, 7 up, 8 in 2024-03-12T03:53:13.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:13 smithi173 ceph-mon[33544]: pgmap v966: 176 pgs: 1 creating+peering, 6 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 959 B/s rd, 53 MiB/s wr, 856 op/s; 32082/314391 objects degraded (10.204%); 22723/314391 objects misplaced (7.228%); 20 MiB/s, 314 objects/s recovering 2024-03-12T03:53:13.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:13.832 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:13.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:13.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:13.934 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:13.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:13.975 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:13.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:14.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:14.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:14.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:14.104 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 84 16 107302 107286 79.8076 29.75 0.00329629 0.012469 2024-03-12T03:53:14.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:14.230 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:14.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:14.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:14.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:14.357 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:14.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:14.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:14.494 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:14.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:14.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:14.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:14.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:14.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:14.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:14 smithi169 ceph-mon[29344]: osdmap e471: 8 total, 7 up, 8 in 2024-03-12T03:53:14.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:14 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:14.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:14 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 32082/314391 objects degraded (10.204%), 26 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:14.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:14 smithi169 ceph-mon[39949]: osdmap e471: 8 total, 7 up, 8 in 2024-03-12T03:53:14.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:14 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:14.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:14 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 32082/314391 objects degraded (10.204%), 26 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:14.730 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:14 smithi173 ceph-mon[33544]: osdmap e471: 8 total, 7 up, 8 in 2024-03-12T03:53:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:14 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:14.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:14 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 32082/314391 objects degraded (10.204%), 26 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:14.767 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:14.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:14.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:14.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:14.906 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:14.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:15.007 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:15.040 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:15.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:15.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 85 16 107752 107736 79.1997 28.125 0.00336391 0.0125567 2024-03-12T03:53:15.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:15.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:15.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:15.271 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:15.309 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:15.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:15.412 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:15.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:15.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:15.599 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:15.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:15.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:15.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:15 smithi169 ceph-mon[29344]: pgmap v968: 176 pgs: 1 creating+peering, 6 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 961 B/s rd, 45 MiB/s wr, 728 op/s; 28996/319875 objects degraded (9.065%); 24746/319875 objects misplaced (7.736%); 15 MiB/s, 244 objects/s recovering 2024-03-12T03:53:15.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:15 smithi169 ceph-mon[29344]: osdmap e472: 8 total, 7 up, 8 in 2024-03-12T03:53:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:15 smithi169 ceph-mon[39949]: pgmap v968: 176 pgs: 1 creating+peering, 6 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 961 B/s rd, 45 MiB/s wr, 728 op/s; 28996/319875 objects degraded (9.065%); 24746/319875 objects misplaced (7.736%); 15 MiB/s, 244 objects/s recovering 2024-03-12T03:53:15.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:15 smithi169 ceph-mon[39949]: osdmap e472: 8 total, 7 up, 8 in 2024-03-12T03:53:15.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:15.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:15.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:15.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:15 smithi173 ceph-mon[33544]: pgmap v968: 176 pgs: 1 creating+peering, 6 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 961 B/s rd, 45 MiB/s wr, 728 op/s; 28996/319875 objects degraded (9.065%); 24746/319875 objects misplaced (7.736%); 15 MiB/s, 244 objects/s recovering 2024-03-12T03:53:15.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:15 smithi173 ceph-mon[33544]: osdmap e472: 8 total, 7 up, 8 in 2024-03-12T03:53:15.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:15.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:15.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:15.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:15.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:16.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:16.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:16.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:16.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:16.104 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 86 16 108453 108437 78.7882 43.8125 0.0171671 0.0126874 2024-03-12T03:53:16.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:16.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:16.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:16.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:16.284 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:16.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:16.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:16.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:16.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:16.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:16.528 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:16.551 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:16.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:16.653 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:16 smithi169 ceph-mon[29344]: osdmap e473: 8 total, 7 up, 8 in 2024-03-12T03:53:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:16 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:53:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:16 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:53:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:16 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:53:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:16 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:53:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:16 smithi169 ceph-mon[39949]: osdmap e473: 8 total, 7 up, 8 in 2024-03-12T03:53:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:16 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:53:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:16 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:53:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:16 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:53:16.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:16 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:53:16.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:16.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:16.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:16 smithi173 ceph-mon[33544]: osdmap e473: 8 total, 7 up, 8 in 2024-03-12T03:53:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:16 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:53:16.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:16 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:53:16.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:16 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:53:16.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:16 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:53:16.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:16.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:16.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:16.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:16.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:16.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:17.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:17.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:17.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:17.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:17.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 87 16 109351 109335 78.5277 56.125 0.020415 0.0127293 2024-03-12T03:53:17.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:17.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:17.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:17.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:17.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:17.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:17.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:17.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:17.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:17.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:17.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:17.647 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:17.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:17.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:17 smithi169 ceph-mon[29344]: pgmap v971: 176 pgs: 1 creating+peering, 6 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 28 MiB/s wr, 455 op/s; 28996/319875 objects degraded (9.065%); 24746/319875 objects misplaced (7.736%); 14 MiB/s, 220 objects/s recovering 2024-03-12T03:53:17.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:17 smithi169 ceph-mon[29344]: osdmap e474: 8 total, 7 up, 8 in 2024-03-12T03:53:17.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:17 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:17.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:17 smithi169 ceph-mon[39949]: pgmap v971: 176 pgs: 1 creating+peering, 6 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 28 MiB/s wr, 455 op/s; 28996/319875 objects degraded (9.065%); 24746/319875 objects misplaced (7.736%); 14 MiB/s, 220 objects/s recovering 2024-03-12T03:53:17.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:17 smithi169 ceph-mon[39949]: osdmap e474: 8 total, 7 up, 8 in 2024-03-12T03:53:17.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:17 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:17 smithi173 ceph-mon[33544]: pgmap v971: 176 pgs: 1 creating+peering, 6 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 107 active+clean; 6.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 28 MiB/s wr, 455 op/s; 28996/319875 objects degraded (9.065%); 24746/319875 objects misplaced (7.736%); 14 MiB/s, 220 objects/s recovering 2024-03-12T03:53:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:17 smithi173 ceph-mon[33544]: osdmap e474: 8 total, 7 up, 8 in 2024-03-12T03:53:17.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:17 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:17.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:17.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:17.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:17.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:17.913 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:17.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:18.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:18.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:18.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:18.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:18.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 88 16 110235 110219 78.2631 55.25 0.0119067 0.0127721 2024-03-12T03:53:18.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:18.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:18.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:18.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:18.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:18.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:18.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:18.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:18.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:18.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:18.596 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:18.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:18.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:18.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:18.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:18.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:18.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:18.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:18.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:18.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:18.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:18.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:19.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:19.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout: 89 16 111063 111047 77.9653 51.75 0.0154894 0.0128213 2024-03-12T03:53:19.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:19.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:19.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:19.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:19.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:19.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:19.356 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:19.398 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:19.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:19.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:19.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:19.536 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:19.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:19.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:19.638 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:19.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:19 smithi169 ceph-mon[29344]: pgmap v973: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1.0 KiB/s rd, 37 MiB/s wr, 585 op/s; 25443/325470 objects degraded (7.817%); 27246/325470 objects misplaced (8.371%); 13 MiB/s, 202 objects/s recovering 2024-03-12T03:53:19.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:19 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 25443/325470 objects degraded (7.817%), 23 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:19.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:19 smithi169 ceph-mon[29344]: osdmap e475: 8 total, 7 up, 8 in 2024-03-12T03:53:19.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:19 smithi169 ceph-mon[39949]: pgmap v973: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1.0 KiB/s rd, 37 MiB/s wr, 585 op/s; 25443/325470 objects degraded (7.817%); 27246/325470 objects misplaced (8.371%); 13 MiB/s, 202 objects/s recovering 2024-03-12T03:53:19.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:19 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 25443/325470 objects degraded (7.817%), 23 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:19.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:19 smithi169 ceph-mon[39949]: osdmap e475: 8 total, 7 up, 8 in 2024-03-12T03:53:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:19.685 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:19.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:19 smithi173 ceph-mon[33544]: pgmap v973: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1.0 KiB/s rd, 37 MiB/s wr, 585 op/s; 25443/325470 objects degraded (7.817%); 27246/325470 objects misplaced (8.371%); 13 MiB/s, 202 objects/s recovering 2024-03-12T03:53:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:19 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 25443/325470 objects degraded (7.817%), 23 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:19.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:19 smithi173 ceph-mon[33544]: osdmap e475: 8 total, 7 up, 8 in 2024-03-12T03:53:19.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:19.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:19.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:19.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:19.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:19.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:19.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:20.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:20.096 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:20.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Total time run: 90.0113 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Total writes made: 111944 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Write size: 65536 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Object size: 65536 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Bandwidth (MB/sec): 77.7291 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev Bandwidth: 46.8184 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max bandwidth (MB/sec): 234.812 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min bandwidth (MB/sec): 0 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Average IOPS: 1243 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev IOPS: 749.103 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max IOPS: 3757 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min IOPS: 0 2024-03-12T03:53:20.105 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Average Latency(s): 0.012861 2024-03-12T03:53:20.106 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Stddev Latency(s): 0.0369495 2024-03-12T03:53:20.106 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Max latency(s): 2.38513 2024-03-12T03:53:20.106 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Min latency(s): 0.00121451 2024-03-12T03:53:20.113 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-03-12T03:53:20.176 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:53:20.177 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:53:20.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:53:20.199 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:53:20.299 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:20.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:20.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:20.434 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:20.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:20.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:20.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:20.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:20.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:20.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:20.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:20.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:20.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:20.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:20.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:20.852 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:20.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:20.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:20.988 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:20.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:21.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:21.137 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:21.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:21.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:21.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:21.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:21.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:21.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:21.419 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:21.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:21.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:21.563 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:21.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:21.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:21.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:21 smithi169 ceph-mon[29344]: pgmap v975: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 853 B/s rd, 35 MiB/s wr, 566 op/s; 25846/330060 objects degraded (7.831%); 25396/330060 objects misplaced (7.694%); 16 MiB/s, 251 objects/s recovering 2024-03-12T03:53:21.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:21 smithi169 ceph-mon[39949]: pgmap v975: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 853 B/s rd, 35 MiB/s wr, 566 op/s; 25846/330060 objects degraded (7.831%); 25396/330060 objects misplaced (7.694%); 16 MiB/s, 251 objects/s recovering 2024-03-12T03:53:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:21.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:21.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:21.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:21 smithi173 ceph-mon[33544]: pgmap v975: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 853 B/s rd, 35 MiB/s wr, 566 op/s; 25846/330060 objects degraded (7.831%); 25396/330060 objects misplaced (7.694%); 16 MiB/s, 251 objects/s recovering 2024-03-12T03:53:21.799 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:21.813 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:53:21.813 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:53:21.845 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:21.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:21.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:21.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:21.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:21.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:22.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:22.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:22.128 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:22.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:22.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:53:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:53:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:53:22.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:53:22.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:22.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:22.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:22.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:22.394 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:22.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:22.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:22.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:22.551 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:22.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:22.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:22.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:22.691 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:22.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:22.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:22.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:22.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:22.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-12T03:53:22.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:22.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:22.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:22.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:23.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:23.120 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:23.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:23.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:23.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:23.284 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:23.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:23.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:23.453 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:23.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:23.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:23.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:23.586 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:23.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:23.625 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:23 smithi169 ceph-mon[29344]: pgmap v976: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.6 KiB/s rd, 45 MiB/s wr, 856 op/s; 25883/331878 objects degraded (7.799%); 24892/331878 objects misplaced (7.500%); 15 MiB/s, 244 objects/s recovering 2024-03-12T03:53:23.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:23 smithi169 ceph-mon[39949]: pgmap v976: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.6 KiB/s rd, 45 MiB/s wr, 856 op/s; 25883/331878 objects degraded (7.799%); 24892/331878 objects misplaced (7.500%); 15 MiB/s, 244 objects/s recovering 2024-03-12T03:53:23.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:23.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:23.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:23.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:23.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:23 smithi173 ceph-mon[33544]: pgmap v976: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.6 KiB/s rd, 45 MiB/s wr, 856 op/s; 25883/331878 objects degraded (7.799%); 24892/331878 objects misplaced (7.500%); 15 MiB/s, 244 objects/s recovering 2024-03-12T03:53:23.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:53:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:53:23.824 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:23.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:23.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:23.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:23.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:23.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:24.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:24.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:24.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:24.240 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:24.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:24.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:24.379 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:24.428 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:24.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:24.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:24.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:24.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:24.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:24.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:24 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:53:24.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:24 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:53:24.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:24 smithi169 ceph-mon[29344]: osdmap e476: 8 total, 7 up, 8 in 2024-03-12T03:53:24.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:24.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:24 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 25883/331878 objects degraded (7.799%), 23 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:24.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:24 smithi169 ceph-mon[29344]: osdmap e477: 8 total, 7 up, 8 in 2024-03-12T03:53:24.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:24 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:53:24.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:24 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:53:24.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:24 smithi169 ceph-mon[39949]: osdmap e476: 8 total, 7 up, 8 in 2024-03-12T03:53:24.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:24.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:24 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 25883/331878 objects degraded (7.799%), 23 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:24.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:24 smithi169 ceph-mon[39949]: osdmap e477: 8 total, 7 up, 8 in 2024-03-12T03:53:24.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:24.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:24.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:24 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:53:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:24 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:53:24.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:24 smithi173 ceph-mon[33544]: osdmap e476: 8 total, 7 up, 8 in 2024-03-12T03:53:24.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:24.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:24 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 25883/331878 objects degraded (7.799%), 23 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:24.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:24 smithi173 ceph-mon[33544]: osdmap e477: 8 total, 7 up, 8 in 2024-03-12T03:53:24.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:24.833 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:24.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:24.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:24.968 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:24.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:24.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:25.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:25.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:25.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:25.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:25.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:25.240 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:25.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:25.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:25.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:25.391 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:25.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:25.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:25.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:25.535 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:25.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:25.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:25.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:25.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:25.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:25.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:25.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:25.903 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:25.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:25 smithi169 ceph-mon[29344]: pgmap v979: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1023 B/s rd, 39 MiB/s wr, 1.06k op/s; 25317/328650 objects degraded (7.703%); 23148/328650 objects misplaced (7.043%); 21 MiB/s, 332 objects/s recovering 2024-03-12T03:53:25.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:25 smithi169 ceph-mon[39949]: pgmap v979: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1023 B/s rd, 39 MiB/s wr, 1.06k op/s; 25317/328650 objects degraded (7.703%); 23148/328650 objects misplaced (7.043%); 21 MiB/s, 332 objects/s recovering 2024-03-12T03:53:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:25.936 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:25.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:25 smithi173 ceph-mon[33544]: pgmap v979: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1023 B/s rd, 39 MiB/s wr, 1.06k op/s; 25317/328650 objects degraded (7.703%); 23148/328650 objects misplaced (7.043%); 21 MiB/s, 332 objects/s recovering 2024-03-12T03:53:26.037 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:26.077 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:26.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:26.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:26.114 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:53:26.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:26.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:26.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:26.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:26.367 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:26.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:26.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:26.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:26.509 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:26.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:26.573 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:53:26.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:26.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:26.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:26.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:26.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:26.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:26.864 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:26 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2517412569' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:53:26.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:26 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2517412569' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:53:26.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:26.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:26.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:26.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:26 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2517412569' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:53:27.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:27.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:27.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:27.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:27.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:27.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:27.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:27.301 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:27.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:27.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:27.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:27.434 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:27.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:27.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:27.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:27.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:27.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:27.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:27 smithi169 ceph-mon[29344]: pgmap v980: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 794 B/s rd, 25 MiB/s wr, 739 op/s; 25317/328650 objects degraded (7.703%); 23148/328650 objects misplaced (7.043%); 10 MiB/s, 167 objects/s recovering 2024-03-12T03:53:27.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:27 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:53:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:27 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2517412569' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:53:27.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:27 smithi169 ceph-mon[29344]: osdmap e478: 8 total, 7 up, 8 in 2024-03-12T03:53:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:27 smithi169 ceph-mon[39949]: pgmap v980: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 794 B/s rd, 25 MiB/s wr, 739 op/s; 25317/328650 objects degraded (7.703%); 23148/328650 objects misplaced (7.043%); 10 MiB/s, 167 objects/s recovering 2024-03-12T03:53:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:27 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:53:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:27 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2517412569' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:53:27.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:27 smithi169 ceph-mon[39949]: osdmap e478: 8 total, 7 up, 8 in 2024-03-12T03:53:27.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:27.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:27.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:27.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:27.831 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:27.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:27.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:27.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:27.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:27.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:27 smithi173 ceph-mon[33544]: pgmap v980: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 794 B/s rd, 25 MiB/s wr, 739 op/s; 25317/328650 objects degraded (7.703%); 23148/328650 objects misplaced (7.043%); 10 MiB/s, 167 objects/s recovering 2024-03-12T03:53:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:27 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:53:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:27 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2517412569' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:53:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:27 smithi173 ceph-mon[33544]: osdmap e478: 8 total, 7 up, 8 in 2024-03-12T03:53:28.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:28.094 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:28.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:28.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:28.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:28.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:28.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:28.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:28.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:28.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:28.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:28.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:28.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:28.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:28.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:28.637 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:28.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:28.739 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:28.765 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:28.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:28.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:28.885 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:28 smithi169 ceph-mon[29344]: osdmap e479: 8 total, 7 up, 8 in 2024-03-12T03:53:28.886 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:28 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 25317/328650 objects degraded (7.703%), 23 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:28.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:28 smithi169 ceph-mon[39949]: osdmap e479: 8 total, 7 up, 8 in 2024-03-12T03:53:28.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:28 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 25317/328650 objects degraded (7.703%), 23 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:28.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:28.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:28.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:28.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:28 smithi173 ceph-mon[33544]: osdmap e479: 8 total, 7 up, 8 in 2024-03-12T03:53:28.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:28 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 25317/328650 objects degraded (7.703%), 23 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:29.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:29.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:29.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:29.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:29.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:29.165 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:29.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:29.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:29.308 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:29.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:29.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:29.433 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:29.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:29.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:29.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:29.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:29.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:29.605 INFO:teuthology.orchestra.run.smithi169.stderr:set pool 11 pgp_num to 26 2024-03-12T03:53:29.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:29.713 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:29.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:29.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:29.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:29.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:29.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:29 smithi173 ceph-mon[33544]: pgmap v983: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.0 KiB/s rd, 3.6 MiB/s wr, 843 op/s; 25017/320220 objects degraded (7.812%); 21999/320220 objects misplaced (6.870%); 5.4 MiB/s, 86 objects/s recovering 2024-03-12T03:53:29.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:29 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2007894895' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-12T03:53:29.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:29 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-12T03:53:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:29 smithi169 ceph-mon[29344]: pgmap v983: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.0 KiB/s rd, 3.6 MiB/s wr, 843 op/s; 25017/320220 objects degraded (7.812%); 21999/320220 objects misplaced (6.870%); 5.4 MiB/s, 86 objects/s recovering 2024-03-12T03:53:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:29 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2007894895' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-12T03:53:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:29 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-12T03:53:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:29 smithi169 ceph-mon[39949]: pgmap v983: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.0 KiB/s rd, 3.6 MiB/s wr, 843 op/s; 25017/320220 objects degraded (7.812%); 21999/320220 objects misplaced (6.870%); 5.4 MiB/s, 86 objects/s recovering 2024-03-12T03:53:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:29 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2007894895' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-12T03:53:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:29 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-03-12T03:53:29.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:29.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:29.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:29.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:30.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:30.116 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:30.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:30.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:30.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:30.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:30.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:30.389 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:30.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:30.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:30.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:30.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:30.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:30.660 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:30.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:30.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:30.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:30.804 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:30.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:30.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:30 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-03-12T03:53:30.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:30 smithi169 ceph-mon[29344]: osdmap e480: 8 total, 7 up, 8 in 2024-03-12T03:53:30.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:30 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-03-12T03:53:30.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:30 smithi169 ceph-mon[39949]: osdmap e480: 8 total, 7 up, 8 in 2024-03-12T03:53:30.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:30.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:30 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-03-12T03:53:30.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:30 smithi173 ceph-mon[33544]: osdmap e480: 8 total, 7 up, 8 in 2024-03-12T03:53:31.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:31.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:31.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:31.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:31.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:31.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:31.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:31.358 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:31.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:31.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:31.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:31.496 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:31.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:31.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:31.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:31.637 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:31.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:31.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:31.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:31.784 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:31.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:31.886 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:31.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:31 smithi169 ceph-mon[29344]: pgmap v985: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.4 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 798 op/s; 24086/314376 objects degraded (7.662%); 20579/314376 objects misplaced (6.546%); 9.8 MiB/s, 157 objects/s recovering 2024-03-12T03:53:31.924 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:53:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:53:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:53:31.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:31 smithi169 ceph-mon[39949]: pgmap v985: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.4 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 798 op/s; 24086/314376 objects degraded (7.662%); 20579/314376 objects misplaced (6.546%); 9.8 MiB/s, 157 objects/s recovering 2024-03-12T03:53:31.926 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:31.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:31.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:31.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:31 smithi173 ceph-mon[33544]: pgmap v985: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 3 active+recovering+undersized+remapped, 34 active+undersized, 109 active+clean; 6.4 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 798 op/s; 24086/314376 objects degraded (7.662%); 20579/314376 objects misplaced (6.546%); 9.8 MiB/s, 157 objects/s recovering 2024-03-12T03:53:32.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:32.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:32.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:32.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:32.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:32.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:32.307 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:32.345 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:32.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:32.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:32.481 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:32.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:32.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:32.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:32.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:32.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:32.747 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:32.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:32.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:32.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:32.883 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:32.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:32.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:33.027 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:33.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:33.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:33.165 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:33.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:33.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:32 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:53:33.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:32 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:33.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:32 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:53:33.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:32 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:32 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:53:33.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:32 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:33.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:33.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:33.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:33.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:33.444 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:33.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:33.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:33.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:33.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:33.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:33.683 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:33.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:33.724 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:33.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:33.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:53:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:53:33.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:33.855 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:33.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:33.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:33.957 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:33.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:33.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:33.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:34.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:34.132 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:34.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:34.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:34.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:33 smithi169 ceph-mon[29344]: pgmap v986: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 6.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 1.35k op/s; 23769/305382 objects degraded (7.783%); 18858/305382 objects misplaced (6.175%); 15 MiB/s, 241 objects/s recovering 2024-03-12T03:53:34.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:33 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 23769/305382 objects degraded (7.783%), 23 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:34.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:33 smithi169 ceph-mon[29344]: osdmap e481: 8 total, 7 up, 8 in 2024-03-12T03:53:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:33 smithi169 ceph-mon[39949]: pgmap v986: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 6.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 1.35k op/s; 23769/305382 objects degraded (7.783%); 18858/305382 objects misplaced (6.175%); 15 MiB/s, 241 objects/s recovering 2024-03-12T03:53:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:33 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 23769/305382 objects degraded (7.783%), 23 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:34.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:33 smithi169 ceph-mon[39949]: osdmap e481: 8 total, 7 up, 8 in 2024-03-12T03:53:34.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:33 smithi173 ceph-mon[33544]: pgmap v986: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 6.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 1.35k op/s; 23769/305382 objects degraded (7.783%); 18858/305382 objects misplaced (6.175%); 15 MiB/s, 241 objects/s recovering 2024-03-12T03:53:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:33 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 23769/305382 objects degraded (7.783%), 23 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:34.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:33 smithi173 ceph-mon[33544]: osdmap e481: 8 total, 7 up, 8 in 2024-03-12T03:53:34.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:34.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:34.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:34.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:34.397 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:34.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:34.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:34.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:34.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:34.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:34.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:34.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:34.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:34.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:34.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:34.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:34.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:34.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:34.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:35.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:35.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:35.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:35.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:35.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:35.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:35.227 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:35.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:35.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:35.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:35.366 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:53:35.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:35.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:35.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:35.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:35.512 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:35.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:35.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:35.665 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:35.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:35.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:35.803 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:35.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:35.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:53:35.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:35.952 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:35.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:36.054 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:36.099 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:36.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:36.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:35 smithi169 ceph-mon[29344]: pgmap v988: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 6.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 1.18k op/s; 22741/298710 objects degraded (7.613%); 17546/298710 objects misplaced (5.874%); 17 MiB/s, 279 objects/s recovering 2024-03-12T03:53:36.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:35 smithi169 ceph-mon[39949]: pgmap v988: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 6.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 1.18k op/s; 22741/298710 objects degraded (7.613%); 17546/298710 objects misplaced (5.874%); 17 MiB/s, 279 objects/s recovering 2024-03-12T03:53:36.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:36.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:36.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:36.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:36.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:35 smithi173 ceph-mon[33544]: pgmap v988: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 6.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 1.18k op/s; 22741/298710 objects degraded (7.613%); 17546/298710 objects misplaced (5.874%); 17 MiB/s, 279 objects/s recovering 2024-03-12T03:53:36.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:36.376 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:36.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:36.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:36.520 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:36.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:36.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:36.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:36.652 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:36.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:36.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:36.794 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:36.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:36.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:36.921 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:36.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:36.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:37.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:37.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:37.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:37.167 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:37.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:36 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:53:37.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:36 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:53:37.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:36 smithi169 ceph-mon[29344]: osdmap e482: 8 total, 7 up, 8 in 2024-03-12T03:53:37.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:36 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:53:37.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:36 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:53:37.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:36 smithi169 ceph-mon[39949]: osdmap e482: 8 total, 7 up, 8 in 2024-03-12T03:53:37.190 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:37.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:36 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:53:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:36 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:53:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:36 smithi173 ceph-mon[33544]: osdmap e482: 8 total, 7 up, 8 in 2024-03-12T03:53:37.292 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:37.339 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:37.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:37.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:37.469 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:37.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:37.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:37.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:37.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:37.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:37.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:37.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:37.754 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:37.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:37.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:37.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:37.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:37.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:37.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:38.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:38.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:38.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:38.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:38.166 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:38.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:38.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:38.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:37 smithi169 ceph-mon[29344]: pgmap v990: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 6.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 886 op/s; 22741/298710 objects degraded (7.613%); 17546/298710 objects misplaced (5.874%); 11 MiB/s, 174 objects/s recovering 2024-03-12T03:53:38.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:37 smithi169 ceph-mon[39949]: pgmap v990: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 6.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 886 op/s; 22741/298710 objects degraded (7.613%); 17546/298710 objects misplaced (5.874%); 11 MiB/s, 174 objects/s recovering 2024-03-12T03:53:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:37 smithi173 ceph-mon[33544]: pgmap v990: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 6.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 886 op/s; 22741/298710 objects degraded (7.613%); 17546/298710 objects misplaced (5.874%); 11 MiB/s, 174 objects/s recovering 2024-03-12T03:53:38.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:38.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:38.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:38.408 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:38.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:38.444 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:38.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:38.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:38.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:38.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:38.686 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:38.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:38.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:38.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:38.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:38.857 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:38.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:38.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:38.995 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:38.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:39.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:39.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:39.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:39.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:38 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 22741/298710 objects degraded (7.613%), 23 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:39.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:38 smithi169 ceph-mon[29344]: osdmap e483: 8 total, 7 up, 8 in 2024-03-12T03:53:39.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:38 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 22741/298710 objects degraded (7.613%), 23 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:39.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:38 smithi169 ceph-mon[39949]: osdmap e483: 8 total, 7 up, 8 in 2024-03-12T03:53:39.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:38 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 22741/298710 objects degraded (7.613%), 23 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:38 smithi173 ceph-mon[33544]: osdmap e483: 8 total, 7 up, 8 in 2024-03-12T03:53:39.276 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:39.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:39.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:39.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:39.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:39.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:39.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:39.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:39.544 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:39.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:39.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:39.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:39.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:39.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:39.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:39.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:39.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:39.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:39.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:40.054 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:40.090 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:40.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:40.191 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:40.227 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:40.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:40.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:40.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:39 smithi173 ceph-mon[33544]: pgmap v991: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 1.17k op/s; 22392/288252 objects degraded (7.768%); 16218/288252 objects misplaced (5.626%); 11 MiB/s, 177 objects/s recovering 2024-03-12T03:53:40.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:40.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:40.368 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:40.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:40.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:39 smithi169 ceph-mon[29344]: pgmap v991: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 1.17k op/s; 22392/288252 objects degraded (7.768%); 16218/288252 objects misplaced (5.626%); 11 MiB/s, 177 objects/s recovering 2024-03-12T03:53:40.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:39 smithi169 ceph-mon[39949]: pgmap v991: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 1.17k op/s; 22392/288252 objects degraded (7.768%); 16218/288252 objects misplaced (5.626%); 11 MiB/s, 177 objects/s recovering 2024-03-12T03:53:40.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:40.500 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:40.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:40.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:40.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:40.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:40.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:40.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:40.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:40.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:40.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:40.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:40.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:41.000 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:41.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:41.043 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:41.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:41.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:41.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:41.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:41.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:41.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:41.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:41.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:41.449 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:41.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:41.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:41.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:41.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:41.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:41.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:41.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:41.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:41.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:41.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:41.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:41.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:41.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:41.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:42.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:42.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:42.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:42.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:42.142 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:42.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:42.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:41 smithi169 ceph-mon[29344]: pgmap v993: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 1.07k op/s; 21395/281763 objects degraded (7.593%); 14784/281763 objects misplaced (5.247%); 11 MiB/s, 178 objects/s recovering 2024-03-12T03:53:42.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:41 smithi169 ceph-mon[39949]: pgmap v993: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 1.07k op/s; 21395/281763 objects degraded (7.593%); 14784/281763 objects misplaced (5.247%); 11 MiB/s, 178 objects/s recovering 2024-03-12T03:53:42.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:53:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:53:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:53:42.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:41 smithi173 ceph-mon[33544]: pgmap v993: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 1.07k op/s; 21395/281763 objects degraded (7.593%); 14784/281763 objects misplaced (5.247%); 11 MiB/s, 178 objects/s recovering 2024-03-12T03:53:42.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:42.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:42.268 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:42.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:42.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:42.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:42.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:42.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:42.547 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:42.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:42.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:42.649 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:42.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:42.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:42.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:42.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:42.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:42.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:42.920 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:42.950 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:42.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:43.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:43.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:43.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:43.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:43.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:43.213 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:43.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:43.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:43.338 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:43.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:43.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:43.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:43.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:43.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:43.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:43.612 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:43.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:43.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:43.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:53:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:53:43.752 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:43.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:43.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:43.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:43.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:43.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:43.987 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:44.017 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:44.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:44.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:44.119 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:44.150 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:44.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:44.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:43 smithi173 ceph-mon[33544]: pgmap v994: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 21079/272451 objects degraded (7.737%); 13499/272451 objects misplaced (4.955%); 9.7 MiB/s, 154 objects/s recovering 2024-03-12T03:53:44.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:43 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 21079/272451 objects degraded (7.737%), 23 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:44.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:43 smithi173 ceph-mon[33544]: osdmap e484: 8 total, 7 up, 8 in 2024-03-12T03:53:44.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:44.281 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:44.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:44.383 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:44.410 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:44.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:44.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:44.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:43 smithi169 ceph-mon[29344]: pgmap v994: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 21079/272451 objects degraded (7.737%); 13499/272451 objects misplaced (4.955%); 9.7 MiB/s, 154 objects/s recovering 2024-03-12T03:53:44.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:43 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 21079/272451 objects degraded (7.737%), 23 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:44.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:43 smithi169 ceph-mon[29344]: osdmap e484: 8 total, 7 up, 8 in 2024-03-12T03:53:44.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:43 smithi169 ceph-mon[39949]: pgmap v994: 176 pgs: 7 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 21079/272451 objects degraded (7.737%); 13499/272451 objects misplaced (4.955%); 9.7 MiB/s, 154 objects/s recovering 2024-03-12T03:53:44.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:43 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 21079/272451 objects degraded (7.737%), 23 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:44.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:43 smithi169 ceph-mon[39949]: osdmap e484: 8 total, 7 up, 8 in 2024-03-12T03:53:44.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:44.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:44.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:44.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:44.680 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:44.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:44.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:44.822 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:44.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:44.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:44.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:44.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:44.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:45.054 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:45.079 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:45.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:45.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:45.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:45.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:45.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:45.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:44 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:53:45.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:44 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:53:45.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:45.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:44 smithi173 ceph-mon[33544]: osdmap e485: 8 total, 7 up, 8 in 2024-03-12T03:53:45.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:45.336 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:45.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:45.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:45.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:44 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:53:45.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:44 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:53:45.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:45.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:44 smithi169 ceph-mon[29344]: osdmap e485: 8 total, 7 up, 8 in 2024-03-12T03:53:45.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:44 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:53:45.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:44 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:53:45.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:45.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:44 smithi169 ceph-mon[39949]: osdmap e485: 8 total, 7 up, 8 in 2024-03-12T03:53:45.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:45.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:45.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:45.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:45.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:45.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:45.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:45.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:45.735 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:45.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:45.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:45.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:45.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:45.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:45.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:46.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:46.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:46.111 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:46.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:46.149 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:46.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:45 smithi173 ceph-mon[33544]: pgmap v996: 176 pgs: 1 active+undersized+degraded+remapped, 7 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 17372/266604 objects degraded (6.516%); 14590/266604 objects misplaced (5.473%); 16 MiB/s, 258 objects/s recovering 2024-03-12T03:53:46.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:45 smithi173 ceph-mon[33544]: osdmap e486: 8 total, 7 up, 8 in 2024-03-12T03:53:46.250 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:46.278 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:46.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:46.380 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:46.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:46.422 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:46.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:46.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:45 smithi169 ceph-mon[29344]: pgmap v996: 176 pgs: 1 active+undersized+degraded+remapped, 7 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 17372/266604 objects degraded (6.516%); 14590/266604 objects misplaced (5.473%); 16 MiB/s, 258 objects/s recovering 2024-03-12T03:53:46.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:45 smithi169 ceph-mon[29344]: osdmap e486: 8 total, 7 up, 8 in 2024-03-12T03:53:46.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:45 smithi169 ceph-mon[39949]: pgmap v996: 176 pgs: 1 active+undersized+degraded+remapped, 7 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 17372/266604 objects degraded (6.516%); 14590/266604 objects misplaced (5.473%); 16 MiB/s, 258 objects/s recovering 2024-03-12T03:53:46.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:45 smithi169 ceph-mon[39949]: osdmap e486: 8 total, 7 up, 8 in 2024-03-12T03:53:46.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:46.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:46.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:46.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:46.694 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:46.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:46.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:46.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:46.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:46.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:46.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:46.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:47.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:47.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:47.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:47.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:47.219 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:46 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:47.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:46 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:47.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:47.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:47.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:47.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:46 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:47.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:47.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:47.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:47.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:47.522 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:47.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:47.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:47.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:47.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:47.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:47.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:47.801 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:47.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:47.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:47.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:47.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:48.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:48.070 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:48.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:48.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:48.172 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:47 smithi169 ceph-mon[29344]: pgmap v999: 176 pgs: 1 active+undersized+degraded+remapped, 7 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 842 op/s; 17372/266604 objects degraded (6.516%); 14590/266604 objects misplaced (5.473%); 11 MiB/s, 182 objects/s recovering 2024-03-12T03:53:48.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:47 smithi169 ceph-mon[39949]: pgmap v999: 176 pgs: 1 active+undersized+degraded+remapped, 7 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 842 op/s; 17372/266604 objects degraded (6.516%); 14590/266604 objects misplaced (5.473%); 11 MiB/s, 182 objects/s recovering 2024-03-12T03:53:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:48.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:47 smithi173 ceph-mon[33544]: pgmap v999: 176 pgs: 1 active+undersized+degraded+remapped, 7 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 2 active+recovering+undersized+remapped, 34 active+undersized, 110 active+clean; 5.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 842 op/s; 17372/266604 objects degraded (6.516%); 14590/266604 objects misplaced (5.473%); 11 MiB/s, 182 objects/s recovering 2024-03-12T03:53:48.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:48.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:48.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:48.454 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:48.494 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:48.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:48.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:48.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:48.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:48.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:48.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:48.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:48.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:48.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:48.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:48.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:48.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:48.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:49.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:49.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:49.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:49.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:49.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:49.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:49.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:48 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:53:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:48 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:53:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:48 smithi173 ceph-mon[33544]: osdmap e487: 8 total, 7 up, 8 in 2024-03-12T03:53:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:48 smithi173 ceph-mon[33544]: pgmap v1001: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 111 active+clean; 5.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 771 op/s; 17107/259257 objects degraded (6.598%); 13267/259257 objects misplaced (5.117%); 6.3 MiB/s, 101 objects/s recovering 2024-03-12T03:53:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:48 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 17372/266604 objects degraded (6.516%), 23 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:48 smithi173 ceph-mon[33544]: osdmap e488: 8 total, 7 up, 8 in 2024-03-12T03:53:49.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:49.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:49.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:49.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:49.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:49.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:49.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:49.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:49.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:48 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:53:49.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:48 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:53:49.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:48 smithi169 ceph-mon[39949]: osdmap e487: 8 total, 7 up, 8 in 2024-03-12T03:53:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:48 smithi169 ceph-mon[39949]: pgmap v1001: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 111 active+clean; 5.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 771 op/s; 17107/259257 objects degraded (6.598%); 13267/259257 objects misplaced (5.117%); 6.3 MiB/s, 101 objects/s recovering 2024-03-12T03:53:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:48 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 17372/266604 objects degraded (6.516%), 23 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:48 smithi169 ceph-mon[39949]: osdmap e488: 8 total, 7 up, 8 in 2024-03-12T03:53:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:48 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:53:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:48 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:53:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:48 smithi169 ceph-mon[29344]: osdmap e487: 8 total, 7 up, 8 in 2024-03-12T03:53:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:48 smithi169 ceph-mon[29344]: pgmap v1001: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 111 active+clean; 5.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 771 op/s; 17107/259257 objects degraded (6.598%); 13267/259257 objects misplaced (5.117%); 6.3 MiB/s, 101 objects/s recovering 2024-03-12T03:53:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:48 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 17372/266604 objects degraded (6.516%), 23 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:49.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:48 smithi169 ceph-mon[29344]: osdmap e488: 8 total, 7 up, 8 in 2024-03-12T03:53:49.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:49.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:49.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:49.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:49.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:49.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:49.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:49.798 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:49.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:49.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:49.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:49.946 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:49.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:49.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:50.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:50.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:50.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:50.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:50.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:50.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:50.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:50.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:50.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:50.448 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:50.487 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:50.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:50.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:50.627 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:50.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:50.628 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:53:50.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:50 smithi169 ceph-mon[39949]: osdmap e489: 8 total, 7 up, 8 in 2024-03-12T03:53:50.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:50 smithi169 ceph-mon[29344]: osdmap e489: 8 total, 7 up, 8 in 2024-03-12T03:53:50.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:50.772 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:50.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:50.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:50.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:50.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:50.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:50.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:50 smithi173 ceph-mon[33544]: osdmap e489: 8 total, 7 up, 8 in 2024-03-12T03:53:51.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:51.043 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:51.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:51.145 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:51.184 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:51.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:51.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:51.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:51.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:51.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:51.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:51.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:51.560 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:51.597 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:51.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:51.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:51.675 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:51 smithi169 ceph-mon[39949]: pgmap v1004: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 111 active+clean; 5.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 869 op/s; 16142/254268 objects degraded (6.348%); 12543/254268 objects misplaced (4.933%); 7.2 MiB/s, 115 objects/s recovering 2024-03-12T03:53:51.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:51 smithi169 ceph-mon[29344]: pgmap v1004: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 111 active+clean; 5.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 869 op/s; 16142/254268 objects degraded (6.348%); 12543/254268 objects misplaced (4.933%); 7.2 MiB/s, 115 objects/s recovering 2024-03-12T03:53:51.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:51.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:51.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:51.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:51.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:51.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:51.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:51 smithi173 ceph-mon[33544]: pgmap v1004: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 1 active+recovering+undersized+remapped, 34 active+undersized, 111 active+clean; 5.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 869 op/s; 16142/254268 objects degraded (6.348%); 12543/254268 objects misplaced (4.933%); 7.2 MiB/s, 115 objects/s recovering 2024-03-12T03:53:52.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:52.011 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:52.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:52.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:52.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:52.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:52.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:53:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:53:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:53:52.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:52.258 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:53:52.258 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:53:52.258 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-12T03:53:52.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-12T03:53:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:52.295 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:52.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:52.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:52.441 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:52.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:52.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:52.575 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:52.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:52.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:52.715 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:52.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:52.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:52.852 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:52.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:52.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:52.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:52.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:52.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:53.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:53.107 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:53.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:53.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:53.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:53.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:53.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:53.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:53.357 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:53.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:53.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:53.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:53.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:53.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:53.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:53.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:53.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:53.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:53.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:53.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:53.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:53:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:53:53.736 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:53 smithi173 ceph-mon[33544]: pgmap v1005: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 4.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.34k op/s; 15761/242424 objects degraded (6.501%); 10819/242424 objects misplaced (4.463%); 11 MiB/s, 172 objects/s recovering 2024-03-12T03:53:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:53 smithi173 ceph-mon[33544]: osdmap e490: 8 total, 7 up, 8 in 2024-03-12T03:53:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:53 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 15761/242424 objects degraded (6.501%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:53.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:53 smithi173 ceph-mon[33544]: osdmap e491: 8 total, 7 up, 8 in 2024-03-12T03:53:53.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:53.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:53.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:53.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:53 smithi169 ceph-mon[29344]: pgmap v1005: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 4.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.34k op/s; 15761/242424 objects degraded (6.501%); 10819/242424 objects misplaced (4.463%); 11 MiB/s, 172 objects/s recovering 2024-03-12T03:53:53.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:53 smithi169 ceph-mon[29344]: osdmap e490: 8 total, 7 up, 8 in 2024-03-12T03:53:53.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:53 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 15761/242424 objects degraded (6.501%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:53.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:53 smithi169 ceph-mon[29344]: osdmap e491: 8 total, 7 up, 8 in 2024-03-12T03:53:53.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:53 smithi169 ceph-mon[39949]: pgmap v1005: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 4.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.34k op/s; 15761/242424 objects degraded (6.501%); 10819/242424 objects misplaced (4.463%); 11 MiB/s, 172 objects/s recovering 2024-03-12T03:53:53.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:53 smithi169 ceph-mon[39949]: osdmap e490: 8 total, 7 up, 8 in 2024-03-12T03:53:53.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:53 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 15761/242424 objects degraded (6.501%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:53.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:53 smithi169 ceph-mon[39949]: osdmap e491: 8 total, 7 up, 8 in 2024-03-12T03:53:53.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:53.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:53.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:53.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:54.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:54.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:54.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:54.185 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:53:54.185 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:53:54.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:54.255 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:54.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:54.357 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:54.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:54.394 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:54.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:54.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:54.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:54.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:54.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:54.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:54.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:54.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:54.765 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:54.781 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:54 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:54.781 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:54 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:54.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:54.808 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:54.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:54.909 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:54.946 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:54.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:54 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:55.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:55.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:55.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:55.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:55.222 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:55.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:55.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:55.361 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:55.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:55.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:55.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:55.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:55.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:55.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:55.637 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:55.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:55.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:55.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:55.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:55.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:55.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:55.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:55.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:55.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:55 smithi169 ceph-mon[39949]: pgmap v1008: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 4.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 892 B/s rd, 0 B/s wr, 1.14k op/s; 15168/237783 objects degraded (6.379%); 10819/237783 objects misplaced (4.550%); 5.3 MiB/s, 85 objects/s recovering 2024-03-12T03:53:55.926 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:55 smithi169 ceph-mon[29344]: pgmap v1008: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 4.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 892 B/s rd, 0 B/s wr, 1.14k op/s; 15168/237783 objects degraded (6.379%); 10819/237783 objects misplaced (4.550%); 5.3 MiB/s, 85 objects/s recovering 2024-03-12T03:53:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:55 smithi173 ceph-mon[33544]: pgmap v1008: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 4.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 892 B/s rd, 0 B/s wr, 1.14k op/s; 15168/237783 objects degraded (6.379%); 10819/237783 objects misplaced (4.550%); 5.3 MiB/s, 85 objects/s recovering 2024-03-12T03:53:56.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:56.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:56.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:56.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:56.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:56.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:56.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:56.312 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:56.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:56.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:56.448 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:56.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:56.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:56.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:56.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:56.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:56.726 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:56.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:56.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-12T03:53:56.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:56.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:56.869 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:56.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:56.970 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:57.032 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:57.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:57.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:57.139 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:53:57.191 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:53:57.203 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:57.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:57.234 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:53:57.275 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:53:57.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:57.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:57.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:57.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:57.336 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:53:57.368 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:53:57.386 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:53:57.393 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:53:57.429 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:53:57.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:53:57.430 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:53:57.430 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:53:57.430 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:53:57.430 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:53:57.430 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:53:57.430 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:53:57.430 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:53:57.436 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:57.468 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:57.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:57.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:57.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:57.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:57.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:57.668 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:57 smithi169 ceph-mon[29344]: pgmap v1009: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 4.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 759 B/s rd, 0 B/s wr, 815 op/s; 15168/237783 objects degraded (6.379%); 10819/237783 objects misplaced (4.550%); 4.5 MiB/s, 72 objects/s recovering 2024-03-12T03:53:57.668 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:57 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3137665752' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:53:57.668 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:57 smithi169 ceph-mon[39949]: pgmap v1009: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 4.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 759 B/s rd, 0 B/s wr, 815 op/s; 15168/237783 objects degraded (6.379%); 10819/237783 objects misplaced (4.550%); 4.5 MiB/s, 72 objects/s recovering 2024-03-12T03:53:57.669 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:57 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3137665752' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:53:57.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:57.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:57.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:57.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:53:57.878 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:57.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:53:57.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:53:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:57 smithi173 ceph-mon[33544]: pgmap v1009: 176 pgs: 8 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 4.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 759 B/s rd, 0 B/s wr, 815 op/s; 15168/237783 objects degraded (6.379%); 10819/237783 objects misplaced (4.550%); 4.5 MiB/s, 72 objects/s recovering 2024-03-12T03:53:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:57 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3137665752' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:53:58.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:58.014 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:58.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:53:58.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:58.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:58.153 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:58.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:58.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:58.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:58.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:58.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:58.431 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:58.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:58.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:58.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:58.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:58.621 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:53:58.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:58.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:58.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:58.821 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:58.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:58.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:58.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:58.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:58 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1748855816' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:53:58.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:58 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 15168/237783 objects degraded (6.379%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:58 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1748855816' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:53:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:58 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 15168/237783 objects degraded (6.379%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:58.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:53:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:58.984 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:58.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:53:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:58 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1748855816' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:53:58.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:58 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 15168/237783 objects degraded (6.379%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-12T03:53:59.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:59.121 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:59.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:59.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:53:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:59.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:59.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:53:59.360 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:59.395 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:59.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:53:59.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:59.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:59.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:59.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:59.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:53:59.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:59.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:59.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:53:59.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:53:59.793 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:53:59.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:59.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:53:59.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:59 smithi173 ceph-mon[33544]: pgmap v1010: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 14373/226062 objects degraded (6.358%); 9226/226062 objects misplaced (4.081%); 7.1 MiB/s, 114 objects/s recovering 2024-03-12T03:53:59.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:59 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:53:59.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:59 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1748855816' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:53:59.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:53:59 smithi173 ceph-mon[33544]: osdmap e492: 8 total, 7 up, 8 in 2024-03-12T03:53:59.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:53:59.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:59 smithi169 ceph-mon[29344]: pgmap v1010: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 14373/226062 objects degraded (6.358%); 9226/226062 objects misplaced (4.081%); 7.1 MiB/s, 114 objects/s recovering 2024-03-12T03:53:59.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:59 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:53:59.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:59 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1748855816' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:53:59.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:53:59 smithi169 ceph-mon[29344]: osdmap e492: 8 total, 7 up, 8 in 2024-03-12T03:53:59.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:59 smithi169 ceph-mon[39949]: pgmap v1010: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 14373/226062 objects degraded (6.358%); 9226/226062 objects misplaced (4.081%); 7.1 MiB/s, 114 objects/s recovering 2024-03-12T03:53:59.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:59 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:53:59.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:59 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1748855816' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:53:59.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:53:59 smithi169 ceph-mon[39949]: osdmap e492: 8 total, 7 up, 8 in 2024-03-12T03:53:59.934 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:53:59.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:53:59.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:00.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:00.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:00.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:00.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:00.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:00.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:00.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:00.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:00.343 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:00.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:00.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:00.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:00.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:00.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:00.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:00.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:00.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:00.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:00.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:00.755 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:00.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:00.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:00.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:00.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:00.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:01.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:01.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:01.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:01.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:01.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:01.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:01.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:01.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:01.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:01.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:01.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:01.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:01.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:01.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:01.581 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:01.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:01.683 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:01.727 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:01.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:01.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:01.835 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:01 smithi169 ceph-mon[29344]: pgmap v1012: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 664 B/s rd, 0 B/s wr, 982 op/s; 13477/219711 objects degraded (6.134%); 9226/219711 objects misplaced (4.199%); 3.4 MiB/s, 55 objects/s recovering 2024-03-12T03:54:01.835 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:54:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:54:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:54:01.836 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:01 smithi169 ceph-mon[39949]: pgmap v1012: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 664 B/s rd, 0 B/s wr, 982 op/s; 13477/219711 objects degraded (6.134%); 9226/219711 objects misplaced (4.199%); 3.4 MiB/s, 55 objects/s recovering 2024-03-12T03:54:01.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:01.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:01.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:01.965 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:01 smithi173 ceph-mon[33544]: pgmap v1012: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 664 B/s rd, 0 B/s wr, 982 op/s; 13477/219711 objects degraded (6.134%); 9226/219711 objects misplaced (4.199%); 3.4 MiB/s, 55 objects/s recovering 2024-03-12T03:54:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:02.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:02.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:02.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:02.149 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:02.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:02.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:02.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:02.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:02.386 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:02.425 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:02.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:02.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:02.566 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:02.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:02.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:02.667 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:02.709 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:02.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:02.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:02.817 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:02 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:02.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:02 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:02.818 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:54:02.820 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:54:02.820 INFO:teuthology.orchestra.run.smithi169.stdout: "epoch": 492, 2024-03-12T03:54:02.820 INFO:teuthology.orchestra.run.smithi169.stdout: "fsid": "677d624a-e020-11ee-95c8-87774f69a715", 2024-03-12T03:54:02.820 INFO:teuthology.orchestra.run.smithi169.stdout: "created": "2024-03-12T03:27:53.319986+0000", 2024-03-12T03:54:02.820 INFO:teuthology.orchestra.run.smithi169.stdout: "modified": "2024-03-12T03:53:58.591888+0000", 2024-03-12T03:54:02.820 INFO:teuthology.orchestra.run.smithi169.stdout: "last_up_change": "2024-03-12T03:48:57.530230+0000", 2024-03-12T03:54:02.820 INFO:teuthology.orchestra.run.smithi169.stdout: "last_in_change": "2024-03-12T03:39:32.320120+0000", 2024-03-12T03:54:02.820 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-12T03:54:02.820 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_num": 5806080, 2024-03-12T03:54:02.821 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_set": [ 2024-03-12T03:54:02.821 INFO:teuthology.orchestra.run.smithi169.stdout: "nodeep-scrub", 2024-03-12T03:54:02.821 INFO:teuthology.orchestra.run.smithi169.stdout: "noscrub", 2024-03-12T03:54:02.821 INFO:teuthology.orchestra.run.smithi169.stdout: "pglog_hardlimit", 2024-03-12T03:54:02.821 INFO:teuthology.orchestra.run.smithi169.stdout: "purged_snapdirs", 2024-03-12T03:54:02.821 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_deletes", 2024-03-12T03:54:02.821 INFO:teuthology.orchestra.run.smithi169.stdout: "sortbitwise" 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_version": 20, 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: "full_ratio": 0.94999998807907104, 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_snapshot": "", 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_max": 11, 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: "max_osd": 8, 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: "require_min_compat_client": "luminous", 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: "min_compat_client": "luminous", 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: "require_osd_release": "squid", 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: "allow_crimson": false, 2024-03-12T03:54:02.822 INFO:teuthology.orchestra.run.smithi169.stdout: "pools": [ 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 1, 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": ".mgr", 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:31:04.863493+0000", 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:54:02.823 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:54:02.824 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:54:02.824 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:54:02.824 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 19, 2024-03-12T03:54:02.824 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 11, 2024-03-12T03:54:02.824 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 11, 2024-03-12T03:54:02.824 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 11, 2024-03-12T03:54:02.824 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 19, 2024-03-12T03:54:02.824 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:54:02.824 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "1.13", 2024-03-12T03:54:02.824 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 428, 2024-03-12T03:54:02.824 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 427, 2024-03-12T03:54:02.824 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 427, 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "83'128", 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "83'128" 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "428", 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "428", 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "426", 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:54:02.825 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:54:02.826 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:54:02.826 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:54:02.826 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:54:02.826 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:54:02.826 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:54:02.826 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:54:02.826 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:54:02.826 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:54:02.826 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:54:02.826 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:54:02.826 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:54:02.826 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:54:02.827 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: "options": { 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_max": 32, 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_min": 1 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr": {} 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 2.2100000381469727, 2024-03-12T03:54:02.828 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 2.2100000381469727, 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1, 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 2.2100000381469727, 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 2.2100000381469727, 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.81114578247070312, 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 0.93000000715255737, 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 2, 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "datapool", 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:18.983144+0000", 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 8193, 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-12T03:54:02.829 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 3, 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 3, 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 3, 2024-03-12T03:54:02.830 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 3, 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 3, 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "70", 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:54:02.831 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 2, 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 70, 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:54:02.832 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:54:02.833 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd": {} 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:54:02.834 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 2.3299999237060547, 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 2.3299999237060547, 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 0.86000001430511475, 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 2, 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 2, 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.81114578247070312, 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 0.93000000715255737, 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 3, 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": ".rgw.root", 2024-03-12T03:54:02.835 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:19.343301+0000", 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:54:02.836 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 32, 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 32, 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 32, 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 32, 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 32, 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "68", 2024-03-12T03:54:02.837 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:54:02.838 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.839 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw": {} 2024-03-12T03:54:02.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 1.9700000286102295, 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 1.9700000286102295, 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1, 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.81114578247070312, 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 0.93000000715255737, 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 4, 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "default.rgw.log", 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:21.864621+0000", 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:54:02.842 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 32, 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 32, 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 32, 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 32, 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 32, 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:54:02.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "70", 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:54:02.844 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:54:02.845 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:54:02.846 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw": {} 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 2.4100000858306885, 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 2.4100000858306885, 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1, 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 2.4100000858306885, 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 2.4100000858306885, 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.81114578247070312, 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 0.93000000715255737, 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 5, 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "default.rgw.control", 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:23.416265+0000", 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:54:02.848 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 32, 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 32, 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 32, 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 32, 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 32, 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:54:02.849 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "72", 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:54:02.850 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:54:02.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:54:02.852 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw": {} 2024-03-12T03:54:02.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.853 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:54:02.853 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:54:02.853 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 1.75, 2024-03-12T03:54:02.853 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 1.75, 2024-03-12T03:54:02.853 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1, 2024-03-12T03:54:02.853 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.75, 2024-03-12T03:54:02.854 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.75, 2024-03-12T03:54:02.854 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.81114578247070312, 2024-03-12T03:54:02.854 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 0.93000000715255737, 2024-03-12T03:54:02.854 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-03-12T03:54:02.854 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 6, 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "default.rgw.meta", 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:33:25.476511+0000", 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:54:02.855 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 32, 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 32, 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 32, 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 32, 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 32, 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:54:02.856 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "74", 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "0", 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:54:02.857 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:54:02.858 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "options": { 2024-03-12T03:54:02.859 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_bias": 4 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: "rgw": {} 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 1.9700000286102295, 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 1.9700000286102295, 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1, 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.81114578247070312, 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 0.93000000715255737, 2024-03-12T03:54:02.860 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: "pool": 11, 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": "unique_pool_4", 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: "create_time": "2024-03-12T03:51:39.115983+0000", 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: "flags_names": "hashpspool", 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: "type": 1, 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 3, 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": 2, 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_rule": 0, 2024-03-12T03:54:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_count": 0, 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_target": 0, 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_barrier": 0, 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "object_hash": 2, 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscale_mode": "off", 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": 26, 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num": 26, 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_placement_num_target": 26, 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_target": 26, 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num_pending": 26, 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "last_pg_merge_meta": { 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "source_pgid": "0.0", 2024-03-12T03:54:02.862 INFO:teuthology.orchestra.run.smithi169.stdout: "ready_epoch": 0, 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_started": 0, 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: "last_epoch_clean": 0, 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: "source_version": "0'0", 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: "target_version": "0'0" 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: "last_change": "480", 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend": "0", 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: "last_force_op_resend_preluminous": "452", 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: "auid": 0, 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_mode": "selfmanaged", 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_seq": 0, 2024-03-12T03:54:02.863 INFO:teuthology.orchestra.run.smithi169.stdout: "snap_epoch": 0, 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_snaps": [], 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps": "[]", 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_bytes": 0, 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "quota_max_objects": 0, 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "tiers": [], 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "tier_of": -1, 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "read_tier": -1, 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "write_tier": -1, 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_mode": "none", 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_bytes": 0, 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "target_max_objects": 0, 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-12T03:54:02.864 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_flush_age": 0, 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "cache_min_evict_age": 0, 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profile": "", 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_params": { 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "none" 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_period": 0, 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_count": 0, 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "use_gmt_hitset": true, 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "min_read_recency_for_promote": 0, 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "min_write_recency_for_promote": 0, 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_grade_decay_rate": 0, 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "hit_set_search_last_n": 0, 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "grade_table": [], 2024-03-12T03:54:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: "stripe_width": 0, 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "expected_num_objects": 0, 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "fast_read": false, 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "options": {}, 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "application_metadata": { 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "rados": {} 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "read_balance": { 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "score_type": "Fair distribution", 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "score_acting": 1.8799999952316284, 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "score_stable": 1.8799999952316284, 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "optimal_score": 1, 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-12T03:54:02.866 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity_weighted": 0.81114578247070312, 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity": 0.93000000715255737, 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 0, 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "c44466b0-3112-44dc-b5f3-d6494b958761", 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 0, 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:54:02.867 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 8, 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 366, 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 384, 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6801", 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 676779916 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6802", 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 676779916 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6804", 2024-03-12T03:54:02.869 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 676779916 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6803", 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 676779916 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.870 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.169:6801/676779916", 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.169:6802/676779916", 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.169:6804/676779916", 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.169:6803/676779916", 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: "exists" 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 1, 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "184c5c97-a992-4be9-bcd3-c08cb58a0e56", 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:54:02.871 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 13, 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 489, 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6805", 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 130042595 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.872 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6806", 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 130042595 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.873 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6808", 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 130042595 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6807", 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 130042595 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.169:6805/130042595", 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.169:6806/130042595", 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.169:6808/130042595", 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.169:6807/130042595", 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 2, 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "beb15e06-1ca4-46a7-b640-efd1991367e8", 2024-03-12T03:54:02.875 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 18, 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 488, 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.876 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6809", 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2751816702 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6810", 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2751816702 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.877 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6812", 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2751816702 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6811", 2024-03-12T03:54:02.878 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2751816702 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.169:6809/2751816702", 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.169:6810/2751816702", 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.169:6812/2751816702", 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.169:6811/2751816702", 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 3, 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "a00b4c50-e064-4c9e-85dc-b0a75ea5bf37", 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 222, 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 264, 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 330, 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 487, 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 265, 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:54:02.880 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6813", 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3056769071 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6814", 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3056769071 2024-03-12T03:54:02.881 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6816", 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3056769071 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6815", 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3056769071 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.169:6813/3056769071", 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.169:6814/3056769071", 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.169:6816/3056769071", 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.169:6815/3056769071", 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:54:02.883 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 4, 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "3a6121ef-6309-41b7-a964-6772c010b3a0", 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 0.489166259765625, 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 32, 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 469, 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:54:02.884 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6800", 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1397212305 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6801", 2024-03-12T03:54:02.885 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1397212305 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6803", 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1397212305 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.886 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6802", 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1397212305 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.173:6800/1397212305", 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.173:6801/1397212305", 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.173:6803/1397212305", 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.173:6802/1397212305", 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:54:02.887 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 5, 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "3c2f6dd9-9784-4a07-8710-55ac3de82964", 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 41, 2024-03-12T03:54:02.888 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 490, 2024-03-12T03:54:02.889 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:54:02.889 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:54:02.889 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:54:02.889 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.889 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.889 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6804", 2024-03-12T03:54:02.889 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3525888417 2024-03-12T03:54:02.889 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.889 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.889 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:54:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:02.891 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:02.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6805", 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3525888417 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6807", 2024-03-12T03:54:02.892 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3525888417 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6806", 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3525888417 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.173:6804/3525888417", 2024-03-12T03:54:02.893 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.173:6805/3525888417", 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.173:6807/3525888417", 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.173:6806/3525888417", 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 6, 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "d7759df7-1978-4d70-94a2-721540804f59", 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:54:02.894 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 0, 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 51, 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 345, 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6808", 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2147192629 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.895 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6809", 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2147192629 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.896 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6811", 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2147192629 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6810", 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 2147192629 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.173:6808/2147192629", 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.173:6809/2147192629", 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.173:6811/2147192629", 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.173:6810/2147192629", 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 7, 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: "uuid": "8843039e-2b17-4230-9cca-5788a2294d8c", 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: "up": 1, 2024-03-12T03:54:02.898 INFO:teuthology.orchestra.run.smithi169.stdout: "in": 1, 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "weight": 1, 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_affinity": 1, 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_begin": 0, 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "last_clean_end": 0, 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "up_from": 59, 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "up_thru": 478, 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "down_at": 0, 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "lost_at": 0, 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addrs": { 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6812", 2024-03-12T03:54:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 4014976876 2024-03-12T03:54:02.900 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.900 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.900 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addrs": { 2024-03-12T03:54:02.900 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.900 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.900 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6813", 2024-03-12T03:54:02.900 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 4014976876 2024-03-12T03:54:02.900 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addrs": { 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6815", 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 4014976876 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addrs": { 2024-03-12T03:54:02.901 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.173:6814", 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 4014976876 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: "public_addr": "172.21.15.173:6812/4014976876", 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_addr": "172.21.15.173:6813/4014976876", 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_back_addr": "172.21.15.173:6815/4014976876", 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: "heartbeat_front_addr": "172.21.15.173:6814/4014976876", 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: "state": [ 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: "exists", 2024-03-12T03:54:02.902 INFO:teuthology.orchestra.run.smithi169.stdout: "up" 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_xinfo": [ 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 0, 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:48:48.703943+0000", 2024-03-12T03:54:02.903 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 383 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 1, 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:53:08.337240+0000", 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 2, 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:54:02.904 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:53:43.248499+0000", 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 3, 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:54:02.905 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:53:47.517570+0000", 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 264 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 4, 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:53:35.382790+0000", 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.906 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 5, 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:53:22.498138+0000", 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 6, 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:54:02.907 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:53:15.919146+0000", 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: "osd": 7, 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: "down_stamp": "0.000000", 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_probability": 0, 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: "laggy_interval": 0, 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: "features": 4540701547738038271, 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: "old_weight": 0, 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: "last_purged_snaps_scrub": "2024-03-12T03:53:15.712103+0000", 2024-03-12T03:54:02.908 INFO:teuthology.orchestra.run.smithi169.stdout: "dead_epoch": 0 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_upmap": [ 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "3.8", 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: 4, 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: 5, 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: 0 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_upmap_items": [ 2024-03-12T03:54:02.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "4.12", 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: "mappings": [ 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: "from": 1, 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: "to": 5 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "4.14", 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: "mappings": [ 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.910 INFO:teuthology.orchestra.run.smithi169.stdout: "from": 1, 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: "to": 2 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_upmap_primaries": [], 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_temp": [ 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "1.e", 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: 3, 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: 6, 2024-03-12T03:54:02.911 INFO:teuthology.orchestra.run.smithi169.stdout: 4 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "1.10", 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: 7, 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: 6, 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: 5 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "1.11", 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:54:02.912 INFO:teuthology.orchestra.run.smithi169.stdout: 5, 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: 2, 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: 3 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "1.12", 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: 7, 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: 4, 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: 1 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.913 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "11.10", 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: 2, 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: 6, 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: 1 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "11.12", 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: 7, 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: 2, 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: 3 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "11.13", 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: 5, 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: 4, 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: 1 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: "pgid": "11.19", 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: "osds": [ 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: 7, 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: 2, 2024-03-12T03:54:02.915 INFO:teuthology.orchestra.run.smithi169.stdout: 3 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: "primary_temp": [], 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: "blocklist": { 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/4041747161": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:6800/2779395456": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/555866474": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/342292643": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/197210445": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/1430250501": "2024-03-13T03:33:59.734991+0000", 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/3918553041": "2024-03-13T03:28:48.518732+0000", 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/1020514981": "2024-03-13T03:28:21.868923+0000", 2024-03-12T03:54:02.916 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/284191239": "2024-03-13T03:28:21.868923+0000", 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/1212459137": "2024-03-13T03:28:48.518732+0000", 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:6800/204892396": "2024-03-13T03:28:21.868923+0000", 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/2751007072": "2024-03-13T03:28:48.518732+0000", 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:0/2349376510": "2024-03-13T03:28:21.868923+0000", 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "172.21.15.169:6800/108891420": "2024-03-13T03:28:48.518732+0000" 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "range_blocklist": {}, 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "erasure_code_profiles": { 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "default": { 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "crush-failure-domain": "osd", 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "k": "2", 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "m": "1", 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "plugin": "jerasure", 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: "technique": "reed_sol_van" 2024-03-12T03:54:02.917 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: "removed_snaps_queue": [], 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: "new_removed_snaps": [], 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: "new_purged_snaps": [], 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_node_flags": {}, 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: "device_class_flags": {}, 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: "stretch_mode": { 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: "stretch_mode_enabled": false, 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: "stretch_bucket_count": 0, 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: "degraded_stretch_mode": 0, 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: "recovering_stretch_mode": 0, 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: "stretch_mode_bucket": 0 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:02.918 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:54:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:02 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:02.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:03.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:03.029 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:03.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:03.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:54:03.177 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:54:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:03.199 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:03.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:03.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:03.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:03.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:03.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:03.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:03.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:03.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:03.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:03.606 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:03.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:03.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:54:03.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:03.744 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:03.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:03.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:03.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:03.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:03.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:03 smithi169 ceph-mon[39949]: pgmap v1013: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.38k op/s; 12986/204561 objects degraded (6.348%); 8284/204561 objects misplaced (4.050%); 3.0 MiB/s, 48 objects/s recovering 2024-03-12T03:54:03.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:03 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1749218382' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-12T03:54:03.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:03 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 12986/204561 objects degraded (6.348%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:03.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:03 smithi169 ceph-mon[39949]: osdmap e493: 8 total, 7 up, 8 in 2024-03-12T03:54:03.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:03 smithi169 ceph-mon[29344]: pgmap v1013: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.38k op/s; 12986/204561 objects degraded (6.348%); 8284/204561 objects misplaced (4.050%); 3.0 MiB/s, 48 objects/s recovering 2024-03-12T03:54:03.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:03 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1749218382' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-12T03:54:03.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:03 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 12986/204561 objects degraded (6.348%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:03.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:03 smithi169 ceph-mon[29344]: osdmap e493: 8 total, 7 up, 8 in 2024-03-12T03:54:03.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:03.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:54:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:54:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:03 smithi173 ceph-mon[33544]: pgmap v1013: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.38k op/s; 12986/204561 objects degraded (6.348%); 8284/204561 objects misplaced (4.050%); 3.0 MiB/s, 48 objects/s recovering 2024-03-12T03:54:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:03 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1749218382' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-12T03:54:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:03 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 12986/204561 objects degraded (6.348%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:03 smithi173 ceph-mon[33544]: osdmap e493: 8 total, 7 up, 8 in 2024-03-12T03:54:04.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:04.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:04.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:04.154 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:04.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:04.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:04.228 INFO:tasks.thrashosds.thrasher:j is {'epoch': 492, 'fsid': '677d624a-e020-11ee-95c8-87774f69a715', 'created': '2024-03-12T03:27:53.319986+0000', 'modified': '2024-03-12T03:53:58.591888+0000', 'last_up_change': '2024-03-12T03:48:57.530230+0000', 'last_in_change': '2024-03-12T03:39:32.320120+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T03:31:04.863493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 19, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 19, 'last_pg_merge_meta': {'source_pgid': '1.13', 'ready_epoch': 428, 'last_epoch_started': 427, 'last_epoch_clean': 427, 'source_version': "83'128", 'target_version': "83'128"}, 'last_change': '428', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '428', 'last_force_op_resend_preluminous': '426', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2100000381469727, 'score_stable': 2.2100000381469727, 'optimal_score': 1, 'raw_score_acting': 2.2100000381469727, 'raw_score_stable': 2.2100000381469727, 'primary_affinity_weighted': 0.8111457824707031, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-12T03:33:18.983144+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8111457824707031, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-12T03:33:19.343301+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8111457824707031, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-12T03:33:21.864621+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4100000858306885, 'score_stable': 2.4100000858306885, 'optimal_score': 1, 'raw_score_acting': 2.4100000858306885, 'raw_score_stable': 2.4100000858306885, 'primary_affinity_weighted': 0.8111457824707031, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-12T03:33:23.416265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8111457824707031, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-12T03:33:25.476511+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8111457824707031, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-12T03:51:39.115983+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '480', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '452', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8111457824707031, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8100000023841858}}], 'osds': [{'osd': 0, 'uuid': 'c44466b0-3112-44dc-b5f3-d6494b958761', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 366, 'down_at': 384, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6801', 'nonce': 676779916}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6802', 'nonce': 676779916}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6804', 'nonce': 676779916}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6803', 'nonce': 676779916}]}, 'public_addr': '172.21.15.169:6801/676779916', 'cluster_addr': '172.21.15.169:6802/676779916', 'heartbeat_back_addr': '172.21.15.169:6804/676779916', 'heartbeat_front_addr': '172.21.15.169:6803/676779916', 'state': ['exists']}, {'osd': 1, 'uuid': '184c5c97-a992-4be9-bcd3-c08cb58a0e56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 489, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6805', 'nonce': 130042595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6806', 'nonce': 130042595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6808', 'nonce': 130042595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6807', 'nonce': 130042595}]}, 'public_addr': '172.21.15.169:6805/130042595', 'cluster_addr': '172.21.15.169:6806/130042595', 'heartbeat_back_addr': '172.21.15.169:6808/130042595', 'heartbeat_front_addr': '172.21.15.169:6807/130042595', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'beb15e06-1ca4-46a7-b640-efd1991367e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 488, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6809', 'nonce': 2751816702}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6810', 'nonce': 2751816702}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6812', 'nonce': 2751816702}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6811', 'nonce': 2751816702}]}, 'public_addr': '172.21.15.169:6809/2751816702', 'cluster_addr': '172.21.15.169:6810/2751816702', 'heartbeat_back_addr': '172.21.15.169:6812/2751816702', 'heartbeat_front_addr': '172.21.15.169:6811/2751816702', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a00b4c50-e064-4c9e-85dc-b0a75ea5bf37', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 222, 'last_clean_end': 264, 'up_from': 330, 'up_thru': 487, 'down_at': 265, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6813', 'nonce': 3056769071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6814', 'nonce': 3056769071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6816', 'nonce': 3056769071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6815', 'nonce': 3056769071}]}, 'public_addr': '172.21.15.169:6813/3056769071', 'cluster_addr': '172.21.15.169:6814/3056769071', 'heartbeat_back_addr': '172.21.15.169:6816/3056769071', 'heartbeat_front_addr': '172.21.15.169:6815/3056769071', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '3a6121ef-6309-41b7-a964-6772c010b3a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.489166259765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 469, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 1397212305}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6801', 'nonce': 1397212305}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6803', 'nonce': 1397212305}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 1397212305}]}, 'public_addr': '172.21.15.173:6800/1397212305', 'cluster_addr': '172.21.15.173:6801/1397212305', 'heartbeat_back_addr': '172.21.15.173:6803/1397212305', 'heartbeat_front_addr': '172.21.15.173:6802/1397212305', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '3c2f6dd9-9784-4a07-8710-55ac3de82964', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 490, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 3525888417}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6805', 'nonce': 3525888417}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6807', 'nonce': 3525888417}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 3525888417}]}, 'public_addr': '172.21.15.173:6804/3525888417', 'cluster_addr': '172.21.15.173:6805/3525888417', 'heartbeat_back_addr': '172.21.15.173:6807/3525888417', 'heartbeat_front_addr': '172.21.15.173:6806/3525888417', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd7759df7-1978-4d70-94a2-721540804f59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 345, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 2147192629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6809', 'nonce': 2147192629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6811', 'nonce': 2147192629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 2147192629}]}, 'public_addr': '172.21.15.173:6808/2147192629', 'cluster_addr': '172.21.15.173:6809/2147192629', 'heartbeat_back_addr': '172.21.15.173:6811/2147192629', 'heartbeat_front_addr': '172.21.15.173:6810/2147192629', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '8843039e-2b17-4230-9cca-5788a2294d8c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 478, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 4014976876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6813', 'nonce': 4014976876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6815', 'nonce': 4014976876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 4014976876}]}, 'public_addr': '172.21.15.173:6812/4014976876', 'cluster_addr': '172.21.15.173:6813/4014976876', 'heartbeat_back_addr': '172.21.15.173:6815/4014976876', 'heartbeat_front_addr': '172.21.15.173:6814/4014976876', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:48:48.703943+0000', 'dead_epoch': 383}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:53:08.337240+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:53:43.248499+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:53:47.517570+0000', 'dead_epoch': 264}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:53:35.382790+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:53:22.498138+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:53:15.919146+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-12T03:53:15.712103+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.8', 'osds': [4, 5, 0]}], 'pg_upmap_items': [{'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.e', 'osds': [3, 6, 4]}, {'pgid': '1.10', 'osds': [7, 6, 5]}, {'pgid': '1.11', 'osds': [5, 2, 3]}, {'pgid': '1.12', 'osds': [7, 4, 1]}, {'pgid': '11.10', 'osds': [2, 6, 1]}, {'pgid': '11.12', 'osds': [7, 2, 3]}, {'pgid': '11.13', 'osds': [5, 4, 1]}, {'pgid': '11.19', 'osds': [7, 2, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.169:0/4041747161': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:6800/2779395456': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/555866474': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/342292643': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/197210445': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/1430250501': '2024-03-13T03:33:59.734991+0000', '172.21.15.169:0/3918553041': '2024-03-13T03:28:48.518732+0000', '172.21.15.169:0/1020514981': '2024-03-13T03:28:21.868923+0000', '172.21.15.169:0/284191239': '2024-03-13T03:28:21.868923+0000', '172.21.15.169:0/1212459137': '2024-03-13T03:28:48.518732+0000', '172.21.15.169:6800/204892396': '2024-03-13T03:28:21.868923+0000', '172.21.15.169:0/2751007072': '2024-03-13T03:28:48.518732+0000', '172.21.15.169:0/2349376510': '2024-03-13T03:28:21.868923+0000', '172.21.15.169:6800/108891420': '2024-03-13T03:28:48.518732+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-12T03:54:04.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:54:04.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:04.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:04.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:04.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:04.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:04.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:04.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:04.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:04.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:04.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:04.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:04.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:04.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:04.801 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:04.849 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:04.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:04.898 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:04.898 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:04.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:04.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:04.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:04.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:05.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:05.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:05.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:05.178 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:54:05.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:05.266 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:05.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:05.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:05.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:05.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:05.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:05.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:05.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:05.546 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:05.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:05.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:05.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:05.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:05.787 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:05.814 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:05 smithi169 ceph-mon[29344]: pgmap v1015: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.68k op/s; 12036/197427 objects degraded (6.096%); 8284/197427 objects misplaced (4.196%); 3.3 MiB/s, 53 objects/s recovering 2024-03-12T03:54:05.815 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:05 smithi169 ceph-mon[39949]: pgmap v1015: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.68k op/s; 12036/197427 objects degraded (6.096%); 8284/197427 objects misplaced (4.196%); 3.3 MiB/s, 53 objects/s recovering 2024-03-12T03:54:05.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:05.835 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:05.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:05.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:05.978 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:05.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:05.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:05 smithi173 ceph-mon[33544]: pgmap v1015: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.68k op/s; 12036/197427 objects degraded (6.096%); 8284/197427 objects misplaced (4.196%); 3.3 MiB/s, 53 objects/s recovering 2024-03-12T03:54:06.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:06.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:06.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:06.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:06.258 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:06.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:06.325 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:54:06.325 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:54:06.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:54:06.325 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:54:06.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.325 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:79301 10.2 10:4768cbda:::benchmark_data_smithi169_162694_object79300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:54:06.325 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:48:57.572720+0000", 2024-03-12T03:54:06.325 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 308.354528518, 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.90356620799999998, 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 79301 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.572720+0000", 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.572720+0000", 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.572722+0000", 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.658e-06 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.572748+0000", 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6435e-05 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.572764+0000", 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5468000000000002e-05 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.572775+0000", 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1268e-05 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.573479+0000", 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00070392100000000004 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.573515+0000", 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5709e-05 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.465055+0000", 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89154066700000001 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.465493+0000", 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00043737099999999998 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.466652+0000", 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0011594039999999999 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.475532+0000", 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088802940000000004 2024-03-12T03:54:06.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.476087+0000", 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00055469700000000005 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.476179+0000", 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1487e-05 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.476286+0000", 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000107829 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:81055 11.16 11:690a7282:::benchmark_data_smithi169_185434_object81054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.823672+0000", 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.103577256999998, 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22090680200000001, 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:06.332 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 81055 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.823672+0000", 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.823672+0000", 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.823673+0000", 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5200000000000001e-06 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.823701+0000", 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7847999999999998e-05 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.823712+0000", 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1436e-05 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.823720+0000", 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3250000000000002e-06 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.829407+0000", 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056871339999999999 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.829560+0000", 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015341500000000001 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,5,7", 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.829726+0000", 2024-03-12T03:54:06.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000165945 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838814+0000", 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090873800000000008 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.839531+0000", 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00071770500000000001 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043203+0000", 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20367144000000001 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043265+0000", 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2575000000000003e-05 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043324+0000", 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9049999999999999e-05 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.044578+0000", 2024-03-12T03:54:06.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001254029 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:81063 11.13 11:c983aafa:::benchmark_data_smithi169_185434_object81062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.832873+0000", 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.094376238999999, 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20785924, 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:06.338 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:06.339 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:06.339 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 81063 2024-03-12T03:54:06.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.339 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832873+0000", 2024-03-12T03:54:06.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832873+0000", 2024-03-12T03:54:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832875+0000", 2024-03-12T03:54:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.103e-06 2024-03-12T03:54:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832897+0000", 2024-03-12T03:54:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2537000000000001e-05 2024-03-12T03:54:06.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832908+0000", 2024-03-12T03:54:06.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1282e-05 2024-03-12T03:54:06.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832916+0000", 2024-03-12T03:54:06.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9030000000000004e-06 2024-03-12T03:54:06.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.833564+0000", 2024-03-12T03:54:06.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00064727199999999995 2024-03-12T03:54:06.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.833737+0000", 2024-03-12T03:54:06.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017373200000000001 2024-03-12T03:54:06.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4,5", 2024-03-12T03:54:06.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.834082+0000", 2024-03-12T03:54:06.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00034505099999999997 2024-03-12T03:54:06.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:06.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038456+0000", 2024-03-12T03:54:06.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20437396499999999 2024-03-12T03:54:06.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038504+0000", 2024-03-12T03:54:06.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7309999999999999e-05 2024-03-12T03:54:06.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038843+0000", 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00033924200000000002 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038903+0000", 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0126e-05 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.040732+0000", 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018287170000000001 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:81065 11.0 11:02432789:::benchmark_data_smithi169_185434_object81064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.832942+0000", 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.094306947000007, 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20864808500000001, 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:06.345 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 81065 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832942+0000", 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832942+0000", 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832942+0000", 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0200000000000002e-07 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832963+0000", 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0863999999999999e-05 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832970+0000", 2024-03-12T03:54:06.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6529999999999997e-06 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832976+0000", 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2310000000000001e-06 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.834700+0000", 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017236350000000001 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.834837+0000", 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000137587 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.835142+0000", 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030461399999999998 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.837882+0000", 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027401249999999999 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037540+0000", 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19965798700000001 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.041470+0000", 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003929493 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.041529+0000", 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9120999999999999e-05 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.041590+0000", 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1173000000000005e-05 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:81066 11.17 11:ec7a2bc3:::benchmark_data_smithi169_185434_object81065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.832992+0000", 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.094256486000006, 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.210340955, 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:06.351 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 81066 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832992+0000", 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832992+0000", 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.832993+0000", 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7700000000000005e-07 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.833012+0000", 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8953e-05 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.833018+0000", 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0469999999999999e-06 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.833023+0000", 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.728e-06 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.834888+0000", 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018649669999999999 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.835033+0000", 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014409199999999999 2024-03-12T03:54:06.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6,7", 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.835392+0000", 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00035952500000000002 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.837790+0000", 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023978620000000002 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037879+0000", 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20008921900000001 2024-03-12T03:54:06.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037924+0000", 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4975999999999998e-05 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.041720+0000", 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0037959360000000002 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.041773+0000", 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3424999999999998e-05 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043333+0000", 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015597479999999999 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:81070 11.16 11:6ebbe24d:::benchmark_data_smithi169_185434_object81069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.833917+0000", 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.093331578000004, 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21513986099999999, 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:06.357 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 81070 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.833917+0000", 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.833917+0000", 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.833918+0000", 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1599999999999999e-06 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.833940+0000", 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2124e-05 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.833951+0000", 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0727000000000001e-05 2024-03-12T03:54:06.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.833959+0000", 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3409999999999997e-06 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.834260+0000", 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030145600000000002 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.834486+0000", 2024-03-12T03:54:06.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000226204 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3,5,7", 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.834655+0000", 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016865399999999999 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043126+0000", 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20847120299999999 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:06.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043223+0000", 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6952000000000003e-05 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.044633+0000", 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014104720000000001 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.047630+0000", 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029961599999999999 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.047734+0000", 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000104036 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.049057+0000", 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001323372 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.15816.0:81071 11.3 e461/452 11:c03fd4b3:::benchmark_data_smithi169_185434_object81070:head v 461'4349, mlcod=461'4347)", 2024-03-12T03:54:06.363 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.836600+0000", 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.090648587000004, 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20194959900000001, 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.836600+0000", 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.836600+0000", 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.836601+0000", 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0100000000000005e-07 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.836647+0000", 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6385999999999999e-05 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.836655+0000", 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2630000000000003e-06 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.836661+0000", 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6969999999999998e-06 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.837876+0000", 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012146749999999999 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.837898+0000", 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1778999999999999e-05 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038514+0000", 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20061595700000001 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038550+0000", 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6241000000000001e-05 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.367 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:81073 11.13 11:cb6aaf77:::benchmark_data_smithi169_185434_object81072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.836667+0000", 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.090582222999998, 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20881195899999999, 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 81073 2024-03-12T03:54:06.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.836667+0000", 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.836667+0000", 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.836668+0000", 2024-03-12T03:54:06.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7010000000000001e-06 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.836693+0000", 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4630999999999998e-05 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.836703+0000", 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8179999999999999e-06 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.836710+0000", 2024-03-12T03:54:06.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9290000000000001e-06 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.837919+0000", 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012093099999999999 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838138+0000", 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00021864399999999999 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4,5", 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838474+0000", 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00033645399999999997 2024-03-12T03:54:06.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038560+0000", 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20008645999999999 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.041089+0000", 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002528495 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043592+0000", 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025032129999999998 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043640+0000", 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7361000000000003e-05 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.045478+0000", 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001838943 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.15816.0:81063 11.13 e461/455 ondisk, result = 0)", 2024-03-12T03:54:06.373 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.838131+0000", 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.089117391000002, 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.2004301, 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838131+0000", 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838131+0000", 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838132+0000", 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.540000000000001e-07 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838139+0000", 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3239999999999999e-06 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838148+0000", 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1900000000000001e-06 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838154+0000", 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1630000000000001e-06 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038488+0000", 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20033364100000001 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038497+0000", 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0000000000000002e-06 2024-03-12T03:54:06.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038561+0000", 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4827999999999997e-05 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.15816.0:81063 11.13 e461/455 ondisk, result = 0)", 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.838509+0000", 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.088739485000005, 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20223486700000001, 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838509+0000", 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838509+0000", 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838510+0000", 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7599999999999997e-07 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838516+0000", 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9739999999999999e-06 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838524+0000", 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0180000000000001e-06 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.838529+0000", 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3619999999999997e-06 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038816+0000", 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20028633100000001 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038825+0000", 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1929999999999999e-06 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.040744+0000", 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001919313 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(11.14 461/455 [PushOp(11:2dd39f35:::benchmark_data_smithi169_185434_object20652:head, version: 444'1326, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:2dd39f35:::benchmark_data_smithi169_185434_object20652:head@444'1326, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.840613+0000", 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.086635521999995, 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19831368299999999, 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840613+0000", 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840613+0000", 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840614+0000", 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9500000000000001e-07 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840817+0000", 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000203024 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037070+0000", 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19625305500000001 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037105+0000", 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5255e-05 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037524+0000", 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00041867500000000003 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037539+0000", 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4898e-05 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.038927+0000", 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013878810000000001 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.15816.0:81070 11.16 e461/455 ondisk, result = 0)", 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.840650+0000", 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.086598690000002, 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20249210300000001, 2024-03-12T03:54:06.384 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840650+0000", 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840650+0000", 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840651+0000", 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3700000000000001e-07 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840655+0000", 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9819999999999999e-06 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840666+0000", 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0101999999999999e-05 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840680+0000", 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4446e-05 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043109+0000", 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20242882800000001 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043118+0000", 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5279999999999992e-06 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043142+0000", 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3779999999999999e-05 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.15816.0:81066 11.17 e461/455 ondisk, result = 0)", 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.840979+0000", 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.086269993000002, 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.196953039, 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840979+0000", 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840979+0000", 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840979+0000", 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2600000000000002e-07 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.840983+0000", 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.67e-06 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037592+0000", 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19660844599999999 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037601+0000", 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8209999999999997e-06 2024-03-12T03:54:06.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037911+0000", 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030949299999999999 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037919+0000", 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7069999999999992e-06 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037932+0000", 2024-03-12T03:54:06.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3276e-05 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.15816.0:81055 11.16 e461/455 ondisk, result = 0)", 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:42.841946+0000", 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.085303166000003, 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20264184800000001, 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.841946+0000", 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.841946+0000", 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.841946+0000", 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3200000000000005e-07 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:42.841949+0000", 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3579999999999999e-06 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037742+0000", 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19579239100000001 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.037747+0000", 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0540000000000002e-06 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043250+0000", 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055034319999999999 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.043259+0000", 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1499999999999999e-06 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.044587+0000", 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001328931 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.395 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:81527 11.13 11:ce1dfcb2:::benchmark_data_smithi169_185434_object81526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:43.571448+0000", 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 82.355800617, 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20269094800000001, 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 81527 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.571448+0000", 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.571448+0000", 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.571450+0000", 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6279999999999999e-06 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.571481+0000", 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1646999999999997e-05 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.571494+0000", 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2632999999999999e-05 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.571503+0000", 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1880000000000008e-06 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.577123+0000", 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056196299999999996 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.577303+0000", 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000180509 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4,5", 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.637208+0000", 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.059904445000000001 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.643907+0000", 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0066996640000000001 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.644090+0000", 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000182425 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.648172+0000", 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040824779999999996 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:06.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.648224+0000", 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1916000000000002e-05 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.774139+0000", 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.125914785 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:81531 11.13 11:c99220b0:::benchmark_data_smithi169_185434_object81530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:43.571724+0000", 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 82.355524720999995, 2024-03-12T03:54:06.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.206523709, 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 81531 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.571724+0000", 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.571724+0000", 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.571725+0000", 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0800000000000005e-07 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.571769+0000", 2024-03-12T03:54:06.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4848000000000002e-05 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.571776+0000", 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1929999999999998e-06 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.571781+0000", 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1660000000000002e-06 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.639047+0000", 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.067266681999999994 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.639203+0000", 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000155697 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4,5", 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.639536+0000", 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00033331399999999997 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:06.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.644104+0000", 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045675760000000003 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.774195+0000", 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13009105200000001 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.774284+0000", 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8583000000000002e-05 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.774326+0000", 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2459000000000001e-05 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:43.778248+0000", 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0039216310000000001 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPushReply(11.17 465/455 [PushReplyOp(11:e9f4fff8:::benchmark_data_smithi169_185434_object8255:head)])", 2024-03-12T03:54:06.407 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:55.015248+0000", 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 70.912001132, 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19730505100000001, 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.015248+0000", 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.015248+0000", 2024-03-12T03:54:06.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.015248+0000", 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0900000000000001e-07 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.015255+0000", 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.652e-06 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.015264+0000", 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5670000000000001e-06 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.015273+0000", 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8140000000000007e-06 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.203761+0000", 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18848794199999999 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.212553+0000", 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087914670000000007 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPushReply(11.17 465/455 [PushReplyOp(11:e9f4fff8:::benchmark_data_smithi169_185434_object8255:head)])", 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:55.019981+0000", 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 70.907268094000003, 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19997830799999999, 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.411 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.019981+0000", 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.019981+0000", 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.019981+0000", 2024-03-12T03:54:06.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1699999999999997e-07 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.019992+0000", 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0121e-05 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.020003+0000", 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1642e-05 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.020019+0000", 2024-03-12T03:54:06.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5787999999999999e-05 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.216345+0000", 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19632644199999999 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.219959+0000", 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036134980000000001 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.15816.0:91933 11.f e465/438 11:f655bd77:::benchmark_data_smithi169_185434_object91932:head v 465'5738, mlcod=465'5734)", 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:52:55.027439+0000", 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 70.899809980000001, 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20737425200000001, 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.027439+0000", 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.027439+0000", 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.027439+0000", 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0499999999999999e-07 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.027481+0000", 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1842999999999999e-05 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.199031+0000", 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17155056799999999 2024-03-12T03:54:06.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.199039+0000", 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6009999999999997e-06 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.201196+0000", 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021569340000000001 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.201215+0000", 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9233e-05 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.234774+0000", 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033559028999999997 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:52:55.234813+0000", 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8739000000000003e-05 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:101408 11.13 11:ce368da2:::benchmark_data_smithi169_185434_object101407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e465)", 2024-03-12T03:54:06.418 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:53:05.723198+0000", 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 60.204050559999999, 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.209548914, 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 101408 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:06.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.723198+0000", 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.723198+0000", 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.723200+0000", 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7099999999999999e-06 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.723224+0000", 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3989e-05 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.723234+0000", 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8570000000000007e-06 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.723241+0000", 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9349999999999997e-06 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.895722+0000", 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17248132599999999 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.895899+0000", 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000176787 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4,5", 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.896928+0000", 2024-03-12T03:54:06.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0010295160000000001 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.930485+0000", 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033556374 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.931078+0000", 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000593427 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.931137+0000", 2024-03-12T03:54:06.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9341e-05 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.931200+0000", 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2757999999999998e-05 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.932747+0000", 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001546894 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:06.424 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:06.425 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:54:06.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:06.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:06.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:06.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:06.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:06.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:06.640 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:06.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:06.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:06.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:06.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:06.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:06.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:06.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:07.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:07.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:07.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:07.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:07.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:07.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:07.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:07.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:07.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:07.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:07.440 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:07.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:07.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-12T03:54:07.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:07.623 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:07.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:07.725 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:07.802 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:07.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:07.815 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:07 smithi169 ceph-mon[29344]: pgmap v1016: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.19k op/s; 12036/197427 objects degraded (6.096%); 8284/197427 objects misplaced (4.196%) 2024-03-12T03:54:07.816 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:07 smithi169 ceph-mon[39949]: pgmap v1016: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.19k op/s; 12036/197427 objects degraded (6.096%); 8284/197427 objects misplaced (4.196%) 2024-03-12T03:54:07.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:07.936 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:07.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:07 smithi173 ceph-mon[33544]: pgmap v1016: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.19k op/s; 12036/197427 objects degraded (6.096%); 8284/197427 objects misplaced (4.196%) 2024-03-12T03:54:08.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:08.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:08.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:08.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:08.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:08.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:08.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:08.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:08.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:08.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:08.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:08.488 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:08.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:08.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:08.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:08.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:08.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:08.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:08.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:08.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:08.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:08.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:08.905 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:08.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:08 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 12036/197427 objects degraded (6.096%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:08.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:08 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 12036/197427 objects degraded (6.096%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:08.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:08 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 12036/197427 objects degraded (6.096%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:09.007 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:09.050 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:09.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:09.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:09.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:09.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:09.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:09.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:09.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:09.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:09.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:09.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:09.459 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:09.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:09.479 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:09.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:09.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:09.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:09.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:09.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:09.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:09.741 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:09 smithi169 ceph-mon[39949]: pgmap v1017: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 3.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.45k op/s; 11573/182199 objects degraded (6.352%); 7372/182199 objects misplaced (4.046%) 2024-03-12T03:54:09.742 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:09 smithi169 ceph-mon[29344]: pgmap v1017: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 3.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.45k op/s; 11573/182199 objects degraded (6.352%); 7372/182199 objects misplaced (4.046%) 2024-03-12T03:54:09.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:09.867 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:09.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:09.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:09.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:09 smithi173 ceph-mon[33544]: pgmap v1017: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 3.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.45k op/s; 11573/182199 objects degraded (6.352%); 7372/182199 objects misplaced (4.046%) 2024-03-12T03:54:10.013 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:10.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:10.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:10.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:10.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:10.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:10.215 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:10.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:10.296 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:10.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:10.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:10.434 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:10.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:10.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:10.592 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:10.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:10.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:10.712 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:54:10.727 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:54:10.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:10.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:10.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:10.874 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:10.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:10.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:10 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:54:10.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:10 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:54:10.976 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:10.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:10 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:54:11.014 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:11.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:11.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:11.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:11.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:11.240 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:11.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:11.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:11.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:11.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:11.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:11.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:11.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:11.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:11.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:11.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:11.680 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:11.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:11.693 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:54:11.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:11.820 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:11.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:11.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:11 smithi169 ceph-mon[29344]: pgmap v1018: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 3.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.48k op/s; 10649/175275 objects degraded (6.076%); 7372/175275 objects misplaced (4.206%) 2024-03-12T03:54:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:11 smithi169 ceph-mon[29344]: from='client.25781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:54:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:11 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4275404369' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:54:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:11 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:54:11.924 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:54:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:54:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:54:11.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:11 smithi169 ceph-mon[39949]: pgmap v1018: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 3.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.48k op/s; 10649/175275 objects degraded (6.076%); 7372/175275 objects misplaced (4.206%) 2024-03-12T03:54:11.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:11 smithi169 ceph-mon[39949]: from='client.25781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:54:11.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:11 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4275404369' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:54:11.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:11 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:54:11.956 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:11.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:11 smithi173 ceph-mon[33544]: pgmap v1018: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 3.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.48k op/s; 10649/175275 objects degraded (6.076%); 7372/175275 objects misplaced (4.206%) 2024-03-12T03:54:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:11 smithi173 ceph-mon[33544]: from='client.25781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:54:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:11 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4275404369' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:54:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:11 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:54:12.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:12.094 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:12.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:12.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:12.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:12.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:12.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:12.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:12.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:12.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:12.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:12.510 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:12.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:12.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:12.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:12.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:12.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:12.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:12.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:12.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:12.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:12.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:12 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:54:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:12 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:54:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:12 smithi173 ceph-mon[33544]: osdmap e494: 8 total, 7 up, 8 in 2024-03-12T03:54:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:12 smithi173 ceph-mon[33544]: 3.1c scrub starts 2024-03-12T03:54:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:12 smithi173 ceph-mon[33544]: 3.1c scrub ok 2024-03-12T03:54:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:12 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:54:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:12 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:54:12.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:12 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:54:13.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:13.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:13.049 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:13.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:13.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:54:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:54:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[29344]: osdmap e494: 8 total, 7 up, 8 in 2024-03-12T03:54:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[29344]: 3.1c scrub starts 2024-03-12T03:54:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[29344]: 3.1c scrub ok 2024-03-12T03:54:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:54:13.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:54:13.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:54:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:54:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:54:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[39949]: osdmap e494: 8 total, 7 up, 8 in 2024-03-12T03:54:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[39949]: 3.1c scrub starts 2024-03-12T03:54:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[39949]: 3.1c scrub ok 2024-03-12T03:54:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:54:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:54:13.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:12 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:54:13.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:13.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:13.286 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:13.327 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:13.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:13.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:13.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:13.464 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:13.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:13.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:13.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:13.596 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:13.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:13.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:13.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:13.738 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:13.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:13.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:13.840 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:13.877 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:13.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:13.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:13.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:13 smithi173 ceph-mon[33544]: pgmap v1020: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 3.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.60k op/s; 10201/160605 objects degraded (6.352%); 6493/160605 objects misplaced (4.043%) 2024-03-12T03:54:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:13 smithi173 ceph-mon[33544]: 11.11 scrub starts 2024-03-12T03:54:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:13 smithi173 ceph-mon[33544]: 3.1d scrub starts 2024-03-12T03:54:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:13 smithi173 ceph-mon[33544]: 3.1d scrub ok 2024-03-12T03:54:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:13 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 10201/160605 objects degraded (6.352%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:13 smithi173 ceph-mon[33544]: osdmap e495: 8 total, 7 up, 8 in 2024-03-12T03:54:13.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:54:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:54:14.014 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:14.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:14.116 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:14.154 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:14.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:14.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:13 smithi169 ceph-mon[29344]: pgmap v1020: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 3.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.60k op/s; 10201/160605 objects degraded (6.352%); 6493/160605 objects misplaced (4.043%) 2024-03-12T03:54:14.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:13 smithi169 ceph-mon[29344]: 11.11 scrub starts 2024-03-12T03:54:14.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:13 smithi169 ceph-mon[29344]: 3.1d scrub starts 2024-03-12T03:54:14.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:13 smithi169 ceph-mon[29344]: 3.1d scrub ok 2024-03-12T03:54:14.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:13 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 10201/160605 objects degraded (6.352%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:14.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:13 smithi169 ceph-mon[29344]: osdmap e495: 8 total, 7 up, 8 in 2024-03-12T03:54:14.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:13 smithi169 ceph-mon[39949]: pgmap v1020: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 3.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.60k op/s; 10201/160605 objects degraded (6.352%); 6493/160605 objects misplaced (4.043%) 2024-03-12T03:54:14.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:13 smithi169 ceph-mon[39949]: 11.11 scrub starts 2024-03-12T03:54:14.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:13 smithi169 ceph-mon[39949]: 3.1d scrub starts 2024-03-12T03:54:14.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:13 smithi169 ceph-mon[39949]: 3.1d scrub ok 2024-03-12T03:54:14.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:13 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 10201/160605 objects degraded (6.352%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:14.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:13 smithi169 ceph-mon[39949]: osdmap e495: 8 total, 7 up, 8 in 2024-03-12T03:54:14.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:14.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:14.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:14.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:14.432 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:14.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:14.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:14.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:14.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:14.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:14.720 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:14.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:14.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:14.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:14.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:14.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:14 smithi173 ceph-mon[33544]: 11.11 scrub ok 2024-03-12T03:54:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:14 smithi173 ceph-mon[33544]: 4.9 scrub starts 2024-03-12T03:54:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:14 smithi173 ceph-mon[33544]: 4.9 scrub ok 2024-03-12T03:54:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:14 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:14 smithi173 ceph-mon[33544]: 11.10 scrub starts 2024-03-12T03:54:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:14 smithi173 ceph-mon[33544]: 11.12 scrub starts 2024-03-12T03:54:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:14 smithi173 ceph-mon[33544]: 11.10 scrub ok 2024-03-12T03:54:14.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:14 smithi173 ceph-mon[33544]: 11.13 scrub starts 2024-03-12T03:54:14.999 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:14.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:15.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:15.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:15.139 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:15.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:15.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[29344]: 11.11 scrub ok 2024-03-12T03:54:15.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[29344]: 4.9 scrub starts 2024-03-12T03:54:15.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[29344]: 4.9 scrub ok 2024-03-12T03:54:15.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:15.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[29344]: 11.10 scrub starts 2024-03-12T03:54:15.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[29344]: 11.12 scrub starts 2024-03-12T03:54:15.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[29344]: 11.10 scrub ok 2024-03-12T03:54:15.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[29344]: 11.13 scrub starts 2024-03-12T03:54:15.167 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[39949]: 11.11 scrub ok 2024-03-12T03:54:15.167 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[39949]: 4.9 scrub starts 2024-03-12T03:54:15.167 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[39949]: 4.9 scrub ok 2024-03-12T03:54:15.167 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:15.167 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[39949]: 11.10 scrub starts 2024-03-12T03:54:15.167 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[39949]: 11.12 scrub starts 2024-03-12T03:54:15.167 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[39949]: 11.10 scrub ok 2024-03-12T03:54:15.167 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:14 smithi169 ceph-mon[39949]: 11.13 scrub starts 2024-03-12T03:54:15.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:15.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:15.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:15.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:15.425 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:15.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:15.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:15.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:15.568 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:15.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:15.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:15.674 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":1018,"stamp":"2024-03-12T03:54:10.283380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3814359699,"num_objects":58425,"num_object_clones":0,"num_object_copies":175275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10649,"num_objects_misplaced":7372,"num_objects_unfound":0,"num_objects_dirty":58425,"num_whiteouts":0,"num_read":3085,"num_read_kb":2809,"num_write":155514,"num_write_kb":6434268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19196,"num_bytes_recovered":1246909196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139681,"ondisk_log_size":139681,"up":465,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":551,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21444080,"kb_used_data":14870064,"kb_used_omap":24,"kb_used_meta":6569511,"kb_avail":728517136,"statfs":{"total":767960285184,"available":746001547264,"internally_reserved":0,"allocated":15226945536,"data_stored":15220543230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24644,"internal_metadata":6727180220},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,6,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-970719232,"num_objects":-14812,"num_object_clones":0,"num_object_copies":-44436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2828,"num_objects_misplaced":-1854,"num_objects_unfound":0,"num_objects_dirty":-14812,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14812,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005662"},"pg_stats":[{"pgid":"11.3","version":"493'6879","reported_seq":7034,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:06.594093+0000","last_change":"2024-03-12T03:52:18.834008+0000","last_active":"2024-03-12T03:54:06.594093+0000","last_peered":"2024-03-12T03:54:06.594093+0000","last_clean":"2024-03-12T03:54:06.594093+0000","last_became_active":"2024-03-12T03:52:17.865581+0000","last_became_peered":"2024-03-12T03:52:17.865581+0000","last_unstale":"2024-03-12T03:54:06.594093+0000","last_undegraded":"2024-03-12T03:54:06.594093+0000","last_fullsized":"2024-03-12T03:54:06.594093+0000","mapping_epoch":438,"log_start":"446'2111","ondisk_log_start":"446'2111","created":438,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'3733","last_scrub_stamp":"2024-03-12T03:52:18.833641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:52:18.833641+0000","objects_scrubbed":1885,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":439,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119537664,"num_objects":1824,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3157,"num_write_kb":99072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1245,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.543746+0000","last_change":"2024-03-12T03:50:53.286652+0000","last_active":"2024-03-12T03:54:03.543746+0000","last_peered":"2024-03-12T03:54:03.543746+0000","last_clean":"2024-03-12T03:54:03.543746+0000","last_became_active":"2024-03-12T03:46:25.289005+0000","last_became_peered":"2024-03-12T03:46:25.289005+0000","last_unstale":"2024-03-12T03:54:03.543746+0000","last_undegraded":"2024-03-12T03:54:03.543746+0000","last_fullsized":"2024-03-12T03:54:03.543746+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:53.286515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.286515+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697726+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"480'35","reported_seq":802,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.556279+0000","last_change":"2024-03-12T03:51:46.582688+0000","last_active":"2024-03-12T03:54:03.556279+0000","last_peered":"2024-03-12T03:54:03.556279+0000","last_clean":"2024-03-12T03:54:03.556279+0000","last_became_active":"2024-03-12T03:51:42.838917+0000","last_became_peered":"2024-03-12T03:51:42.838917+0000","last_unstale":"2024-03-12T03:54:03.556279+0000","last_undegraded":"2024-03-12T03:54:03.556279+0000","last_fullsized":"2024-03-12T03:54:03.556279+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'25","last_scrub_stamp":"2024-03-12T03:51:46.582596+0000","last_deep_scrub":"344'25","last_deep_scrub_stamp":"2024-03-12T03:51:46.582596+0000","last_clean_scrub_stamp":"2024-03-12T03:51:46.582596+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.231158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":689,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.555230+0000","last_change":"2024-03-12T03:48:58.454102+0000","last_active":"2024-03-12T03:54:03.555230+0000","last_peered":"2024-03-12T03:54:03.555230+0000","last_clean":"2024-03-12T03:48:53.448575+0000","last_became_active":"2024-03-12T03:48:58.454102+0000","last_became_peered":"2024-03-12T03:48:58.454102+0000","last_unstale":"2024-03-12T03:54:03.555230+0000","last_undegraded":"2024-03-12T03:54:03.555230+0000","last_fullsized":"2024-03-12T03:48:58.443116+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:07.363439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:07.363439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229391+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"201'10","reported_seq":841,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.550245+0000","last_change":"2024-03-12T03:51:42.650674+0000","last_active":"2024-03-12T03:54:03.550245+0000","last_peered":"2024-03-12T03:54:03.550245+0000","last_clean":"2024-03-12T03:54:03.550245+0000","last_became_active":"2024-03-12T03:39:33.269084+0000","last_became_peered":"2024-03-12T03:39:33.269084+0000","last_unstale":"2024-03-12T03:54:03.550245+0000","last_undegraded":"2024-03-12T03:54:03.550245+0000","last_fullsized":"2024-03-12T03:54:03.550245+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:51:42.650584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:42.650584+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":399,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.553943+0000","last_change":"2024-03-12T03:51:42.833356+0000","last_active":"2024-03-12T03:54:03.553943+0000","last_peered":"2024-03-12T03:54:03.553943+0000","last_clean":"2024-03-12T03:54:03.553943+0000","last_became_active":"2024-03-12T03:51:42.832276+0000","last_became_peered":"2024-03-12T03:51:42.832276+0000","last_unstale":"2024-03-12T03:54:03.553943+0000","last_undegraded":"2024-03-12T03:54:03.553943+0000","last_fullsized":"2024-03-12T03:54:03.553943+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:54.871895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:54.871895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692675+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"493'7144","reported_seq":7269,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:07.219263+0000","last_change":"2024-03-12T03:52:17.842637+0000","last_active":"2024-03-12T03:54:07.219263+0000","last_peered":"2024-03-12T03:54:07.219263+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:52:17.842637+0000","last_became_peered":"2024-03-12T03:52:17.842637+0000","last_unstale":"2024-03-12T03:54:07.219263+0000","last_undegraded":"2024-03-12T03:52:17.830421+0000","last_fullsized":"2024-03-12T03:52:17.830244+0000","mapping_epoch":438,"log_start":"443'272","ondisk_log_start":"443'272","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:11.457972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120946688,"num_objects":1845,"num_object_clones":0,"num_object_copies":5535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5265,"num_write_kb":227552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1845}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"480'45","reported_seq":680,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.550106+0000","last_change":"2024-03-12T03:51:42.832652+0000","last_active":"2024-03-12T03:54:03.550106+0000","last_peered":"2024-03-12T03:54:03.550106+0000","last_clean":"2024-03-12T03:54:03.550106+0000","last_became_active":"2024-03-12T03:51:42.832275+0000","last_became_peered":"2024-03-12T03:51:42.832275+0000","last_unstale":"2024-03-12T03:54:03.550106+0000","last_undegraded":"2024-03-12T03:54:03.550106+0000","last_fullsized":"2024-03-12T03:54:03.550106+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:50:01.871412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:01.871412+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.332791+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":390,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555464+0000","last_change":"2024-03-12T03:50:55.920975+0000","last_active":"2024-03-12T03:54:03.555464+0000","last_peered":"2024-03-12T03:54:03.555464+0000","last_clean":"2024-03-12T03:54:03.555464+0000","last_became_active":"2024-03-12T03:45:18.108377+0000","last_became_peered":"2024-03-12T03:45:18.108377+0000","last_unstale":"2024-03-12T03:54:03.555464+0000","last_undegraded":"2024-03-12T03:54:03.555464+0000","last_fullsized":"2024-03-12T03:54:03.555464+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:55.920704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:50:55.920704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":913,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.555480+0000","last_change":"2024-03-12T03:48:58.457284+0000","last_active":"2024-03-12T03:54:03.555480+0000","last_peered":"2024-03-12T03:54:03.555480+0000","last_clean":"2024-03-12T03:48:53.449180+0000","last_became_active":"2024-03-12T03:48:58.457284+0000","last_became_peered":"2024-03-12T03:48:58.457284+0000","last_unstale":"2024-03-12T03:54:03.555480+0000","last_undegraded":"2024-03-12T03:54:03.555480+0000","last_fullsized":"2024-03-12T03:48:58.441935+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.054936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.054936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692438+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":672,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.564221+0000","last_change":"2024-03-12T03:51:37.847956+0000","last_active":"2024-03-12T03:54:03.564221+0000","last_peered":"2024-03-12T03:54:03.564221+0000","last_clean":"2024-03-12T03:54:03.564221+0000","last_became_active":"2024-03-12T03:46:25.293293+0000","last_became_peered":"2024-03-12T03:46:25.293293+0000","last_unstale":"2024-03-12T03:54:03.564221+0000","last_undegraded":"2024-03-12T03:54:03.564221+0000","last_fullsized":"2024-03-12T03:54:03.564221+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:37.847806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:37.847806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.332910+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"388'129","reported_seq":1086,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.547995+0000","last_change":"2024-03-12T03:50:56.164839+0000","last_active":"2024-03-12T03:54:03.547995+0000","last_peered":"2024-03-12T03:54:03.547995+0000","last_clean":"2024-03-12T03:54:03.547995+0000","last_became_active":"2024-03-12T03:46:25.297913+0000","last_became_peered":"2024-03-12T03:46:25.297913+0000","last_unstale":"2024-03-12T03:54:03.547995+0000","last_undegraded":"2024-03-12T03:54:03.547995+0000","last_fullsized":"2024-03-12T03:54:03.547995+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"388'129","last_scrub_stamp":"2024-03-12T03:50:56.164705+0000","last_deep_scrub":"388'129","last_deep_scrub_stamp":"2024-03-12T03:49:17.168603+0000","last_clean_scrub_stamp":"2024-03-12T03:50:56.164705+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"493'7029","reported_seq":7154,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:08.780137+0000","last_change":"2024-03-12T03:52:17.853813+0000","last_active":"2024-03-12T03:54:08.780137+0000","last_peered":"2024-03-12T03:54:08.780137+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:52:17.853813+0000","last_became_peered":"2024-03-12T03:52:17.853813+0000","last_unstale":"2024-03-12T03:54:08.780137+0000","last_undegraded":"2024-03-12T03:52:17.833553+0000","last_fullsized":"2024-03-12T03:52:17.833423+0000","mapping_epoch":438,"log_start":"448'2560","ondisk_log_start":"448'2560","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:12.275089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115245056,"num_objects":1758,"num_object_clones":0,"num_object_copies":5274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5190,"num_write_kb":222368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":1074,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554516+0000","last_change":"2024-03-12T03:51:30.025307+0000","last_active":"2024-03-12T03:54:03.554516+0000","last_peered":"2024-03-12T03:54:03.554516+0000","last_clean":"2024-03-12T03:54:03.554516+0000","last_became_active":"2024-03-12T03:50:58.842635+0000","last_became_peered":"2024-03-12T03:50:58.842635+0000","last_unstale":"2024-03-12T03:54:03.554516+0000","last_undegraded":"2024-03-12T03:54:03.554516+0000","last_fullsized":"2024-03-12T03:54:03.554516+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:30.025220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:51:30.025220+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"480'39","reported_seq":623,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554613+0000","last_change":"2024-03-12T03:51:42.834522+0000","last_active":"2024-03-12T03:54:03.554613+0000","last_peered":"2024-03-12T03:54:03.554613+0000","last_clean":"2024-03-12T03:54:03.554613+0000","last_became_active":"2024-03-12T03:51:42.834094+0000","last_became_peered":"2024-03-12T03:51:42.834094+0000","last_unstale":"2024-03-12T03:54:03.554613+0000","last_undegraded":"2024-03-12T03:54:03.554613+0000","last_fullsized":"2024-03-12T03:54:03.554613+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:49:09.755469+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:49:09.755469+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692594+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":939,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554996+0000","last_change":"2024-03-12T03:50:47.939438+0000","last_active":"2024-03-12T03:54:03.554996+0000","last_peered":"2024-03-12T03:54:03.554996+0000","last_clean":"2024-03-12T03:54:03.554996+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:54:03.554996+0000","last_undegraded":"2024-03-12T03:54:03.554996+0000","last_fullsized":"2024-03-12T03:54:03.554996+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:47.939203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_clean_scrub_stamp":"2024-03-12T03:50:47.939203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":917,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.551244+0000","last_change":"2024-03-12T03:50:51.214639+0000","last_active":"2024-03-12T03:54:03.551244+0000","last_peered":"2024-03-12T03:54:03.551244+0000","last_clean":"2024-03-12T03:54:03.551244+0000","last_became_active":"2024-03-12T03:33:25.177480+0000","last_became_peered":"2024-03-12T03:33:25.177480+0000","last_unstale":"2024-03-12T03:54:03.551244+0000","last_undegraded":"2024-03-12T03:54:03.551244+0000","last_fullsized":"2024-03-12T03:54:03.551244+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:51.214503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:51.214503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":712,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.556178+0000","last_change":"2024-03-12T03:51:36.531246+0000","last_active":"2024-03-12T03:54:03.556178+0000","last_peered":"2024-03-12T03:54:03.556178+0000","last_clean":"2024-03-12T03:54:03.556178+0000","last_became_active":"2024-03-12T03:46:25.300814+0000","last_became_peered":"2024-03-12T03:46:25.300814+0000","last_unstale":"2024-03-12T03:54:03.556178+0000","last_undegraded":"2024-03-12T03:54:03.556178+0000","last_fullsized":"2024-03-12T03:54:03.556178+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:36.531169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:36.531169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.230214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"493'7019","reported_seq":7176,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:08.436669+0000","last_change":"2024-03-12T03:52:18.576917+0000","last_active":"2024-03-12T03:54:08.436669+0000","last_peered":"2024-03-12T03:54:08.436669+0000","last_clean":"2024-03-12T03:54:08.436669+0000","last_became_active":"2024-03-12T03:52:17.844536+0000","last_became_peered":"2024-03-12T03:52:17.844536+0000","last_unstale":"2024-03-12T03:54:08.436669+0000","last_undegraded":"2024-03-12T03:54:08.436669+0000","last_fullsized":"2024-03-12T03:54:08.436669+0000","mapping_epoch":438,"log_start":"445'1701","ondisk_log_start":"445'1701","created":438,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'3720","last_scrub_stamp":"2024-03-12T03:52:18.576087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:52:18.576087+0000","objects_scrubbed":1876,"log_size":5318,"log_dups_size":0,"ondisk_log_size":5318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:12.590492+0000","scrub_duration":521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114491392,"num_objects":1747,"num_object_clones":0,"num_object_copies":5241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3319,"num_write_kb":102080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":1059,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.546501+0000","last_change":"2024-03-12T03:50:34.205833+0000","last_active":"2024-03-12T03:54:03.546501+0000","last_peered":"2024-03-12T03:54:03.546501+0000","last_clean":"2024-03-12T03:54:03.546501+0000","last_became_active":"2024-03-12T03:39:36.252548+0000","last_became_peered":"2024-03-12T03:39:36.252548+0000","last_unstale":"2024-03-12T03:54:03.546501+0000","last_undegraded":"2024-03-12T03:54:03.546501+0000","last_fullsized":"2024-03-12T03:54:03.546501+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:34.205711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:34.205711+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"480'39","reported_seq":801,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554505+0000","last_change":"2024-03-12T03:50:46.613513+0000","last_active":"2024-03-12T03:54:03.554505+0000","last_peered":"2024-03-12T03:54:03.554505+0000","last_clean":"2024-03-12T03:54:03.554505+0000","last_became_active":"2024-03-12T03:46:25.287373+0000","last_became_peered":"2024-03-12T03:46:25.287373+0000","last_unstale":"2024-03-12T03:54:03.554505+0000","last_undegraded":"2024-03-12T03:54:03.554505+0000","last_fullsized":"2024-03-12T03:54:03.554505+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:50:46.613361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:46.613361+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:12.589381+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":301,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.547312+0000","last_change":"2024-03-12T03:51:42.832487+0000","last_active":"2024-03-12T03:54:03.547312+0000","last_peered":"2024-03-12T03:54:03.547312+0000","last_clean":"2024-03-12T03:48:53.446320+0000","last_became_active":"2024-03-12T03:51:42.832487+0000","last_became_peered":"2024-03-12T03:51:42.832487+0000","last_unstale":"2024-03-12T03:54:03.547312+0000","last_undegraded":"2024-03-12T03:54:03.547312+0000","last_fullsized":"2024-03-12T03:51:42.823170+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:16.876204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:16.876204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"441'10","reported_seq":168,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.548930+0000","last_change":"2024-03-12T03:51:42.832218+0000","last_active":"2024-03-12T03:54:03.548930+0000","last_peered":"2024-03-12T03:54:03.548930+0000","last_clean":"2024-03-12T03:33:28.357470+0000","last_became_active":"2024-03-12T03:51:42.832218+0000","last_became_peered":"2024-03-12T03:51:42.832218+0000","last_unstale":"2024-03-12T03:54:03.548930+0000","last_undegraded":"2024-03-12T03:51:42.824811+0000","last_fullsized":"2024-03-12T03:51:42.824692+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:47:53.964216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:53.964216+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":900,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.548490+0000","last_change":"2024-03-12T03:48:58.448113+0000","last_active":"2024-03-12T03:54:03.548490+0000","last_peered":"2024-03-12T03:54:03.548490+0000","last_clean":"2024-03-12T03:48:53.472090+0000","last_became_active":"2024-03-12T03:48:58.448113+0000","last_became_peered":"2024-03-12T03:48:58.448113+0000","last_unstale":"2024-03-12T03:54:03.548490+0000","last_undegraded":"2024-03-12T03:54:03.548490+0000","last_fullsized":"2024-03-12T03:48:58.441990+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:04.770746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:04.770746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"493'10247","reported_seq":10372,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:06.592048+0000","last_change":"2024-03-12T03:51:40.855404+0000","last_active":"2024-03-12T03:54:06.592048+0000","last_peered":"2024-03-12T03:54:06.592048+0000","last_clean":"2024-03-12T03:54:06.592048+0000","last_became_active":"2024-03-12T03:51:40.855216+0000","last_became_peered":"2024-03-12T03:51:40.855216+0000","last_unstale":"2024-03-12T03:54:06.592048+0000","last_undegraded":"2024-03-12T03:54:06.592048+0000","last_fullsized":"2024-03-12T03:54:06.592048+0000","mapping_epoch":438,"log_start":"465'6200","ondisk_log_start":"465'6200","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240451584,"num_objects":3669,"num_object_clones":0,"num_object_copies":11007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10247,"num_write_kb":445312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":929,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554919+0000","last_change":"2024-03-12T03:51:30.617409+0000","last_active":"2024-03-12T03:54:03.554919+0000","last_peered":"2024-03-12T03:54:03.554919+0000","last_clean":"2024-03-12T03:54:03.554919+0000","last_became_active":"2024-03-12T03:46:25.286985+0000","last_became_peered":"2024-03-12T03:46:25.286985+0000","last_unstale":"2024-03-12T03:54:03.554919+0000","last_undegraded":"2024-03-12T03:54:03.554919+0000","last_fullsized":"2024-03-12T03:54:03.554919+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:30.617326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:30.617326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"480'54","reported_seq":795,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.556603+0000","last_change":"2024-03-12T03:48:58.463377+0000","last_active":"2024-03-12T03:54:03.556603+0000","last_peered":"2024-03-12T03:54:03.556603+0000","last_clean":"2024-03-12T03:48:53.447338+0000","last_became_active":"2024-03-12T03:48:58.463377+0000","last_became_peered":"2024-03-12T03:48:58.463377+0000","last_unstale":"2024-03-12T03:54:03.556603+0000","last_undegraded":"2024-03-12T03:48:58.440183+0000","last_fullsized":"2024-03-12T03:48:58.440082+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:47:13.347128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.347128+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229433+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":86,"num_read_kb":72,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":693,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.554196+0000","last_change":"2024-03-12T03:48:58.464970+0000","last_active":"2024-03-12T03:54:03.554196+0000","last_peered":"2024-03-12T03:54:03.554196+0000","last_clean":"2024-03-12T03:48:53.448054+0000","last_became_active":"2024-03-12T03:48:58.464970+0000","last_became_peered":"2024-03-12T03:48:58.464970+0000","last_unstale":"2024-03-12T03:54:03.554196+0000","last_undegraded":"2024-03-12T03:54:03.554196+0000","last_fullsized":"2024-03-12T03:48:58.441302+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:05.639950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:05.639950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":702,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554226+0000","last_change":"2024-03-12T03:51:42.841507+0000","last_active":"2024-03-12T03:54:03.554226+0000","last_peered":"2024-03-12T03:54:03.554226+0000","last_clean":"2024-03-12T03:54:03.554226+0000","last_became_active":"2024-03-12T03:51:42.840902+0000","last_became_peered":"2024-03-12T03:51:42.840902+0000","last_unstale":"2024-03-12T03:54:03.554226+0000","last_undegraded":"2024-03-12T03:54:03.554226+0000","last_fullsized":"2024-03-12T03:54:03.554226+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:14.836179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_clean_scrub_stamp":"2024-03-12T03:49:14.836179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229536+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":1074,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.557266+0000","last_change":"2024-03-12T03:50:48.924955+0000","last_active":"2024-03-12T03:54:03.557266+0000","last_peered":"2024-03-12T03:54:03.557266+0000","last_clean":"2024-03-12T03:54:03.557266+0000","last_became_active":"2024-03-12T03:44:19.373715+0000","last_became_peered":"2024-03-12T03:44:19.373715+0000","last_unstale":"2024-03-12T03:54:03.557266+0000","last_undegraded":"2024-03-12T03:54:03.557266+0000","last_fullsized":"2024-03-12T03:54:03.557266+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:48.924831+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:15.895638+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.924831+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"493'10187","reported_seq":10312,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:06.388625+0000","last_change":"2024-03-12T03:51:40.862482+0000","last_active":"2024-03-12T03:54:06.388625+0000","last_peered":"2024-03-12T03:54:06.388625+0000","last_clean":"2024-03-12T03:54:06.388625+0000","last_became_active":"2024-03-12T03:51:40.861867+0000","last_became_peered":"2024-03-12T03:51:40.861867+0000","last_unstale":"2024-03-12T03:54:06.388625+0000","last_undegraded":"2024-03-12T03:54:06.388625+0000","last_fullsized":"2024-03-12T03:54:06.388625+0000","mapping_epoch":438,"log_start":"465'6100","ondisk_log_start":"465'6100","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:09.550891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239534080,"num_objects":3655,"num_object_clones":0,"num_object_copies":10965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10187,"num_write_kb":442944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"332'12","reported_seq":948,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.556550+0000","last_change":"2024-03-12T03:50:32.360621+0000","last_active":"2024-03-12T03:54:03.556550+0000","last_peered":"2024-03-12T03:54:03.556550+0000","last_clean":"2024-03-12T03:54:03.556550+0000","last_became_active":"2024-03-12T03:46:25.288674+0000","last_became_peered":"2024-03-12T03:46:25.288674+0000","last_unstale":"2024-03-12T03:54:03.556550+0000","last_undegraded":"2024-03-12T03:54:03.556550+0000","last_fullsized":"2024-03-12T03:54:03.556550+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'12","last_scrub_stamp":"2024-03-12T03:50:32.360502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:32.360502+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229671+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"480'56","reported_seq":645,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.546884+0000","last_change":"2024-03-12T03:51:42.832600+0000","last_active":"2024-03-12T03:54:03.546884+0000","last_peered":"2024-03-12T03:54:03.546884+0000","last_clean":"2024-03-12T03:54:03.546884+0000","last_became_active":"2024-03-12T03:51:42.832229+0000","last_became_peered":"2024-03-12T03:51:42.832229+0000","last_unstale":"2024-03-12T03:54:03.546884+0000","last_undegraded":"2024-03-12T03:54:03.546884+0000","last_fullsized":"2024-03-12T03:54:03.546884+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:50:35.839859+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.839859+0000","objects_scrubbed":8,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":63,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"320'2","reported_seq":784,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.554058+0000","last_change":"2024-03-12T03:48:58.457502+0000","last_active":"2024-03-12T03:54:03.554058+0000","last_peered":"2024-03-12T03:54:03.554058+0000","last_clean":"2024-03-12T03:45:56.084290+0000","last_became_active":"2024-03-12T03:48:58.457502+0000","last_became_peered":"2024-03-12T03:48:58.457502+0000","last_unstale":"2024-03-12T03:54:03.554058+0000","last_undegraded":"2024-03-12T03:48:58.439950+0000","last_fullsized":"2024-03-12T03:48:58.439792+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'2","last_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_deep_scrub":"320'2","last_deep_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_clean_scrub_stamp":"2024-03-12T03:45:56.089247+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692505+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":935,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.543617+0000","last_change":"2024-03-12T03:51:31.400543+0000","last_active":"2024-03-12T03:54:03.543617+0000","last_peered":"2024-03-12T03:54:03.543617+0000","last_clean":"2024-03-12T03:54:03.543617+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:54:03.543617+0000","last_undegraded":"2024-03-12T03:54:03.543617+0000","last_fullsized":"2024-03-12T03:54:03.543617+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:31.400418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:49:54.471078+0000","last_clean_scrub_stamp":"2024-03-12T03:51:31.400418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.698038+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":894,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554020+0000","last_change":"2024-03-12T03:51:37.519315+0000","last_active":"2024-03-12T03:54:03.554020+0000","last_peered":"2024-03-12T03:54:03.554020+0000","last_clean":"2024-03-12T03:54:03.554020+0000","last_became_active":"2024-03-12T03:51:07.938356+0000","last_became_peered":"2024-03-12T03:51:07.938356+0000","last_unstale":"2024-03-12T03:54:03.554020+0000","last_undegraded":"2024-03-12T03:54:03.554020+0000","last_fullsized":"2024-03-12T03:54:03.554020+0000","mapping_epoch":330,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:37.519232+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_clean_scrub_stamp":"2024-03-12T03:51:37.519232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.231188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"493'10362","reported_seq":10488,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:07.237026+0000","last_change":"2024-03-12T03:51:40.889015+0000","last_active":"2024-03-12T03:54:07.237026+0000","last_peered":"2024-03-12T03:54:07.237026+0000","last_clean":"2024-03-12T03:54:07.237026+0000","last_became_active":"2024-03-12T03:51:40.888684+0000","last_became_peered":"2024-03-12T03:51:40.888684+0000","last_unstale":"2024-03-12T03:54:07.237026+0000","last_undegraded":"2024-03-12T03:54:07.237026+0000","last_fullsized":"2024-03-12T03:54:07.237026+0000","mapping_epoch":438,"log_start":"461'5100","ondisk_log_start":"461'5100","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":5262,"log_dups_size":0,"ondisk_log_size":5262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246349848,"num_objects":3760,"num_object_clones":0,"num_object_copies":11280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10362,"num_write_kb":451841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"385'9","reported_seq":313,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.548222+0000","last_change":"2024-03-12T03:48:58.451360+0000","last_active":"2024-03-12T03:54:03.548222+0000","last_peered":"2024-03-12T03:54:03.548222+0000","last_clean":"2024-03-12T03:33:28.360558+0000","last_became_active":"2024-03-12T03:48:58.451360+0000","last_became_peered":"2024-03-12T03:48:58.451360+0000","last_unstale":"2024-03-12T03:54:03.548222+0000","last_undegraded":"2024-03-12T03:48:58.442329+0000","last_fullsized":"2024-03-12T03:48:58.441949+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:48:14.138048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:14.138048+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"480'36","reported_seq":987,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.553577+0000","last_change":"2024-03-12T03:50:35.597476+0000","last_active":"2024-03-12T03:54:03.553577+0000","last_peered":"2024-03-12T03:54:03.553577+0000","last_clean":"2024-03-12T03:54:03.553577+0000","last_became_active":"2024-03-12T03:33:23.163315+0000","last_became_peered":"2024-03-12T03:33:23.163315+0000","last_unstale":"2024-03-12T03:54:03.553577+0000","last_undegraded":"2024-03-12T03:54:03.553577+0000","last_fullsized":"2024-03-12T03:54:03.553577+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:50:35.597225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.597225+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":947,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.548436+0000","last_change":"2024-03-12T03:50:48.222807+0000","last_active":"2024-03-12T03:54:03.548436+0000","last_peered":"2024-03-12T03:54:03.548436+0000","last_clean":"2024-03-12T03:54:03.548436+0000","last_became_active":"2024-03-12T03:46:25.294348+0000","last_became_peered":"2024-03-12T03:46:25.294348+0000","last_unstale":"2024-03-12T03:54:03.548436+0000","last_undegraded":"2024-03-12T03:54:03.548436+0000","last_fullsized":"2024-03-12T03:54:03.548436+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.222697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.222697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":804,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.556308+0000","last_change":"2024-03-12T03:48:58.455443+0000","last_active":"2024-03-12T03:54:03.556308+0000","last_peered":"2024-03-12T03:54:03.556308+0000","last_clean":"2024-03-12T03:43:13.330724+0000","last_became_active":"2024-03-12T03:48:58.455443+0000","last_became_peered":"2024-03-12T03:48:58.455443+0000","last_unstale":"2024-03-12T03:54:03.556308+0000","last_undegraded":"2024-03-12T03:54:03.556308+0000","last_fullsized":"2024-03-12T03:48:58.439578+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:13.892760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.892760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229404+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":671,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555674+0000","last_change":"2024-03-12T03:50:54.587747+0000","last_active":"2024-03-12T03:54:03.555674+0000","last_peered":"2024-03-12T03:54:03.555674+0000","last_clean":"2024-03-12T03:54:03.555674+0000","last_became_active":"2024-03-12T03:46:25.287360+0000","last_became_peered":"2024-03-12T03:46:25.287360+0000","last_unstale":"2024-03-12T03:54:03.555674+0000","last_undegraded":"2024-03-12T03:54:03.555674+0000","last_fullsized":"2024-03-12T03:54:03.555674+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:54.587604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:54.587604+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.230050+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"493'10511","reported_seq":10636,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:08.436006+0000","last_change":"2024-03-12T03:51:40.859739+0000","last_active":"2024-03-12T03:54:08.436006+0000","last_peered":"2024-03-12T03:54:08.436006+0000","last_clean":"2024-03-12T03:54:08.436006+0000","last_became_active":"2024-03-12T03:51:40.859434+0000","last_became_peered":"2024-03-12T03:51:40.859434+0000","last_unstale":"2024-03-12T03:54:08.436006+0000","last_undegraded":"2024-03-12T03:54:08.436006+0000","last_fullsized":"2024-03-12T03:54:08.436006+0000","mapping_epoch":438,"log_start":"465'6100","ondisk_log_start":"465'6100","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4411,"log_dups_size":0,"ondisk_log_size":4411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:12.590421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234815488,"num_objects":3583,"num_object_clones":0,"num_object_copies":10749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10511,"num_write_kb":451008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":229,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.548005+0000","last_change":"2024-03-12T03:48:58.449166+0000","last_active":"2024-03-12T03:54:03.548005+0000","last_peered":"2024-03-12T03:54:03.548005+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.449166+0000","last_became_peered":"2024-03-12T03:48:58.449166+0000","last_unstale":"2024-03-12T03:54:03.548005+0000","last_undegraded":"2024-03-12T03:54:03.548005+0000","last_fullsized":"2024-03-12T03:48:58.441301+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.179416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.179416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"480'45","reported_seq":742,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.549995+0000","last_change":"2024-03-12T03:48:58.454600+0000","last_active":"2024-03-12T03:54:03.549995+0000","last_peered":"2024-03-12T03:54:03.549995+0000","last_clean":"2024-03-12T03:43:28.848207+0000","last_became_active":"2024-03-12T03:48:58.454600+0000","last_became_peered":"2024-03-12T03:48:58.454600+0000","last_unstale":"2024-03-12T03:54:03.549995+0000","last_undegraded":"2024-03-12T03:48:58.443785+0000","last_fullsized":"2024-03-12T03:48:58.443677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-03-12T03:45:59.055318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.055318+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":77,"num_read_kb":62,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":946,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555397+0000","last_change":"2024-03-12T03:49:51.465759+0000","last_active":"2024-03-12T03:54:03.555397+0000","last_peered":"2024-03-12T03:54:03.555397+0000","last_clean":"2024-03-12T03:54:03.555397+0000","last_became_active":"2024-03-12T03:39:33.291879+0000","last_became_peered":"2024-03-12T03:39:33.291879+0000","last_unstale":"2024-03-12T03:54:03.555397+0000","last_undegraded":"2024-03-12T03:54:03.555397+0000","last_fullsized":"2024-03-12T03:54:03.555397+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:51.465633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:49:51.465633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":935,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.552915+0000","last_change":"2024-03-12T03:51:36.030269+0000","last_active":"2024-03-12T03:54:03.552915+0000","last_peered":"2024-03-12T03:54:03.552915+0000","last_clean":"2024-03-12T03:54:03.552915+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:54:03.552915+0000","last_undegraded":"2024-03-12T03:54:03.552915+0000","last_fullsized":"2024-03-12T03:54:03.552915+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:36.030166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:36.030166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.693009+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":533,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.561051+0000","last_change":"2024-03-12T03:48:58.454186+0000","last_active":"2024-03-12T03:54:03.561051+0000","last_peered":"2024-03-12T03:54:03.561051+0000","last_clean":"2024-03-12T03:34:01.860694+0000","last_became_active":"2024-03-12T03:48:58.454186+0000","last_became_peered":"2024-03-12T03:48:58.454186+0000","last_unstale":"2024-03-12T03:54:03.561051+0000","last_undegraded":"2024-03-12T03:54:03.561051+0000","last_fullsized":"2024-03-12T03:48:58.443407+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:04.857986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:04.857986+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:06.194921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"493'7194","reported_seq":7319,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:08.916598+0000","last_change":"2024-03-12T03:52:17.854261+0000","last_active":"2024-03-12T03:54:08.916598+0000","last_peered":"2024-03-12T03:54:08.916598+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:52:17.854261+0000","last_became_peered":"2024-03-12T03:52:17.854261+0000","last_unstale":"2024-03-12T03:54:08.916598+0000","last_undegraded":"2024-03-12T03:52:17.834203+0000","last_fullsized":"2024-03-12T03:52:17.834063+0000","mapping_epoch":438,"log_start":"448'2788","ondisk_log_start":"448'2788","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4406,"log_dups_size":0,"ondisk_log_size":4406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:12.274218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116391936,"num_objects":1776,"num_object_clones":0,"num_object_copies":5328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5284,"num_write_kb":225920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":904,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.551329+0000","last_change":"2024-03-12T03:48:58.467253+0000","last_active":"2024-03-12T03:54:03.551329+0000","last_peered":"2024-03-12T03:54:03.551329+0000","last_clean":"2024-03-12T03:48:53.472859+0000","last_became_active":"2024-03-12T03:48:58.467253+0000","last_became_peered":"2024-03-12T03:48:58.467253+0000","last_unstale":"2024-03-12T03:54:03.551329+0000","last_undegraded":"2024-03-12T03:54:03.551329+0000","last_fullsized":"2024-03-12T03:48:58.444374+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:00.577693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:46:00.577693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"480'36","reported_seq":766,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.554795+0000","last_change":"2024-03-12T03:48:58.449415+0000","last_active":"2024-03-12T03:54:03.554795+0000","last_peered":"2024-03-12T03:54:03.554795+0000","last_clean":"2024-03-12T03:43:28.788554+0000","last_became_active":"2024-03-12T03:48:58.449415+0000","last_became_peered":"2024-03-12T03:48:58.449415+0000","last_unstale":"2024-03-12T03:54:03.554795+0000","last_undegraded":"2024-03-12T03:48:58.441112+0000","last_fullsized":"2024-03-12T03:48:58.441001+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:48:17.191968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:17.191968+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692518+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":2173,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:05.091227+0000","last_change":"2024-03-12T03:48:58.462253+0000","last_active":"2024-03-12T03:54:05.091227+0000","last_peered":"2024-03-12T03:54:05.091227+0000","last_clean":"2024-03-12T03:48:56.580904+0000","last_became_active":"2024-03-12T03:48:58.462253+0000","last_became_peered":"2024-03-12T03:48:58.462253+0000","last_unstale":"2024-03-12T03:54:05.091227+0000","last_undegraded":"2024-03-12T03:48:58.445825+0000","last_fullsized":"2024-03-12T03:48:58.445711+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:47:13.591375+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.591375+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1236,"num_read_kb":1231,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":804,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.555030+0000","last_change":"2024-03-12T03:48:58.464609+0000","last_active":"2024-03-12T03:54:03.555030+0000","last_peered":"2024-03-12T03:54:03.555030+0000","last_clean":"2024-03-12T03:43:13.332262+0000","last_became_active":"2024-03-12T03:48:58.464609+0000","last_became_peered":"2024-03-12T03:48:58.464609+0000","last_unstale":"2024-03-12T03:54:03.555030+0000","last_undegraded":"2024-03-12T03:54:03.555030+0000","last_fullsized":"2024-03-12T03:48:58.443632+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:58.050985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.050985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":932,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555040+0000","last_change":"2024-03-12T03:49:13.598817+0000","last_active":"2024-03-12T03:54:03.555040+0000","last_peered":"2024-03-12T03:54:03.555040+0000","last_clean":"2024-03-12T03:54:03.555040+0000","last_became_active":"2024-03-12T03:46:25.302256+0000","last_became_peered":"2024-03-12T03:46:25.302256+0000","last_unstale":"2024-03-12T03:54:03.555040+0000","last_undegraded":"2024-03-12T03:54:03.555040+0000","last_fullsized":"2024-03-12T03:54:03.555040+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:13.598692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:13.598692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1066,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554744+0000","last_change":"2024-03-12T03:51:11.966626+0000","last_active":"2024-03-12T03:54:03.554744+0000","last_peered":"2024-03-12T03:54:03.554744+0000","last_clean":"2024-03-12T03:54:03.554744+0000","last_became_active":"2024-03-12T03:51:11.965655+0000","last_became_peered":"2024-03-12T03:51:11.965655+0000","last_unstale":"2024-03-12T03:54:03.554744+0000","last_undegraded":"2024-03-12T03:54:03.554744+0000","last_fullsized":"2024-03-12T03:54:03.554744+0000","mapping_epoch":154,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:49:50.467976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:49:50.467976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692763+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"493'10619","reported_seq":10744,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:08.899304+0000","last_change":"2024-03-12T03:51:40.888952+0000","last_active":"2024-03-12T03:54:08.899304+0000","last_peered":"2024-03-12T03:54:08.899304+0000","last_clean":"2024-03-12T03:54:08.899304+0000","last_became_active":"2024-03-12T03:51:40.888646+0000","last_became_peered":"2024-03-12T03:51:40.888646+0000","last_unstale":"2024-03-12T03:54:08.899304+0000","last_undegraded":"2024-03-12T03:54:08.899304+0000","last_fullsized":"2024-03-12T03:54:08.899304+0000","mapping_epoch":438,"log_start":"470'6700","ondisk_log_start":"470'6700","created":438,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:13.248676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227737600,"num_objects":3475,"num_object_clones":0,"num_object_copies":10425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10619,"num_write_kb":451008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":931,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.545583+0000","last_change":"2024-03-12T03:50:48.205459+0000","last_active":"2024-03-12T03:54:03.545583+0000","last_peered":"2024-03-12T03:54:03.545583+0000","last_clean":"2024-03-12T03:54:03.545583+0000","last_became_active":"2024-03-12T03:39:33.265521+0000","last_became_peered":"2024-03-12T03:39:33.265521+0000","last_unstale":"2024-03-12T03:54:03.545583+0000","last_undegraded":"2024-03-12T03:54:03.545583+0000","last_fullsized":"2024-03-12T03:54:03.545583+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.205352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.205352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697697+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"480'41","reported_seq":1135,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.557453+0000","last_change":"2024-03-12T03:52:19.658496+0000","last_active":"2024-03-12T03:54:03.557453+0000","last_peered":"2024-03-12T03:54:03.557453+0000","last_clean":"2024-03-12T03:54:03.557453+0000","last_became_active":"2024-03-12T03:45:18.109048+0000","last_became_peered":"2024-03-12T03:45:18.109048+0000","last_unstale":"2024-03-12T03:54:03.557453+0000","last_undegraded":"2024-03-12T03:54:03.557453+0000","last_fullsized":"2024-03-12T03:54:03.557453+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'29","last_scrub_stamp":"2024-03-12T03:52:19.658412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:52:19.658412+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":697,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554135+0000","last_change":"2024-03-12T03:50:33.413216+0000","last_active":"2024-03-12T03:54:03.554135+0000","last_peered":"2024-03-12T03:54:03.554135+0000","last_clean":"2024-03-12T03:54:03.554135+0000","last_became_active":"2024-03-12T03:46:25.292535+0000","last_became_peered":"2024-03-12T03:46:25.292535+0000","last_unstale":"2024-03-12T03:54:03.554135+0000","last_undegraded":"2024-03-12T03:54:03.554135+0000","last_fullsized":"2024-03-12T03:54:03.554135+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:33.413098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:33.413098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229911+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1202,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.543047+0000","last_change":"2024-03-12T03:48:58.452024+0000","last_active":"2024-03-12T03:54:03.543047+0000","last_peered":"2024-03-12T03:54:03.543047+0000","last_clean":"2024-03-12T03:48:53.448868+0000","last_became_active":"2024-03-12T03:48:58.452024+0000","last_became_peered":"2024-03-12T03:48:58.452024+0000","last_unstale":"2024-03-12T03:54:03.543047+0000","last_undegraded":"2024-03-12T03:54:03.543047+0000","last_fullsized":"2024-03-12T03:48:58.444796+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_clean_scrub_stamp":"2024-03-12T03:48:11.435684+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697463+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":923,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.548786+0000","last_change":"2024-03-12T03:51:39.169999+0000","last_active":"2024-03-12T03:54:03.548786+0000","last_peered":"2024-03-12T03:54:03.548786+0000","last_clean":"2024-03-12T03:54:03.548786+0000","last_became_active":"2024-03-12T03:46:25.295668+0000","last_became_peered":"2024-03-12T03:46:25.295668+0000","last_unstale":"2024-03-12T03:54:03.548786+0000","last_undegraded":"2024-03-12T03:54:03.548786+0000","last_fullsized":"2024-03-12T03:54:03.548786+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.169871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.169871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"493'7088","reported_seq":7246,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:08.887212+0000","last_change":"2024-03-12T03:52:19.002532+0000","last_active":"2024-03-12T03:54:08.887212+0000","last_peered":"2024-03-12T03:54:08.887212+0000","last_clean":"2024-03-12T03:54:08.887212+0000","last_became_active":"2024-03-12T03:52:17.842035+0000","last_became_peered":"2024-03-12T03:52:17.842035+0000","last_unstale":"2024-03-12T03:54:08.887212+0000","last_undegraded":"2024-03-12T03:54:08.887212+0000","last_fullsized":"2024-03-12T03:54:08.887212+0000","mapping_epoch":438,"log_start":"448'2622","ondisk_log_start":"448'2622","created":438,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'3776","last_scrub_stamp":"2024-03-12T03:52:19.002413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:52:19.002413+0000","objects_scrubbed":1905,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:13.250564+0000","scrub_duration":316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115343360,"num_objects":1760,"num_object_clones":0,"num_object_copies":5280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":101376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":662,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.550872+0000","last_change":"2024-03-12T03:50:59.748930+0000","last_active":"2024-03-12T03:54:03.550872+0000","last_peered":"2024-03-12T03:54:03.550872+0000","last_clean":"2024-03-12T03:54:03.550872+0000","last_became_active":"2024-03-12T03:39:33.267729+0000","last_became_peered":"2024-03-12T03:39:33.267729+0000","last_unstale":"2024-03-12T03:54:03.550872+0000","last_undegraded":"2024-03-12T03:54:03.550872+0000","last_fullsized":"2024-03-12T03:54:03.550872+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:59.748792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.748792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.332859+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"480'27","reported_seq":968,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.554399+0000","last_change":"2024-03-12T03:48:58.451116+0000","last_active":"2024-03-12T03:54:03.554399+0000","last_peered":"2024-03-12T03:54:03.554399+0000","last_clean":"2024-03-12T03:48:53.448806+0000","last_became_active":"2024-03-12T03:48:58.451116+0000","last_became_peered":"2024-03-12T03:48:58.451116+0000","last_unstale":"2024-03-12T03:54:03.554399+0000","last_undegraded":"2024-03-12T03:48:58.440926+0000","last_fullsized":"2024-03-12T03:48:58.440811+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'23","last_scrub_stamp":"2024-03-12T03:45:59.998395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.998395+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692479+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":651,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.550550+0000","last_change":"2024-03-12T03:48:58.469230+0000","last_active":"2024-03-12T03:54:03.550550+0000","last_peered":"2024-03-12T03:54:03.550550+0000","last_clean":"2024-03-12T03:48:53.446422+0000","last_became_active":"2024-03-12T03:48:58.469230+0000","last_became_peered":"2024-03-12T03:48:58.469230+0000","last_unstale":"2024-03-12T03:54:03.550550+0000","last_undegraded":"2024-03-12T03:54:03.550550+0000","last_fullsized":"2024-03-12T03:48:58.446209+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:06.903506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.903506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.332724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":947,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.559210+0000","last_change":"2024-03-12T03:50:41.599042+0000","last_active":"2024-03-12T03:54:03.559210+0000","last_peered":"2024-03-12T03:54:03.559210+0000","last_clean":"2024-03-12T03:54:03.559210+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:54:03.559210+0000","last_undegraded":"2024-03-12T03:54:03.559210+0000","last_fullsized":"2024-03-12T03:54:03.559210+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:41.598934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:50:41.598934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692813+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":935,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.545247+0000","last_change":"2024-03-12T03:50:44.293342+0000","last_active":"2024-03-12T03:54:03.545247+0000","last_peered":"2024-03-12T03:54:03.545247+0000","last_clean":"2024-03-12T03:54:03.545247+0000","last_became_active":"2024-03-12T03:46:25.284168+0000","last_became_peered":"2024-03-12T03:46:25.284168+0000","last_unstale":"2024-03-12T03:54:03.545247+0000","last_undegraded":"2024-03-12T03:54:03.545247+0000","last_fullsized":"2024-03-12T03:54:03.545247+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:50:44.293155+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:49:18.522103+0000","last_clean_scrub_stamp":"2024-03-12T03:50:44.293155+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697623+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1217,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.544219+0000","last_change":"2024-03-12T03:48:58.455161+0000","last_active":"2024-03-12T03:54:03.544219+0000","last_peered":"2024-03-12T03:54:03.544219+0000","last_clean":"2024-03-12T03:48:53.449188+0000","last_became_active":"2024-03-12T03:48:58.455161+0000","last_became_peered":"2024-03-12T03:48:58.455161+0000","last_unstale":"2024-03-12T03:54:03.544219+0000","last_undegraded":"2024-03-12T03:54:03.544219+0000","last_fullsized":"2024-03-12T03:48:58.444402+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:01.456424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:01.456424+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"493'10550","reported_seq":10672,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:08.906286+0000","last_change":"2024-03-12T03:51:50.089118+0000","last_active":"2024-03-12T03:54:08.906286+0000","last_peered":"2024-03-12T03:54:08.906286+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:51:40.861779+0000","last_became_peered":"2024-03-12T03:51:40.861779+0000","last_unstale":"2024-03-12T03:54:08.906286+0000","last_undegraded":"2024-03-12T03:51:50.086761+0000","last_fullsized":"2024-03-12T03:51:40.858211+0000","mapping_epoch":438,"log_start":"469'6600","ondisk_log_start":"469'6600","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":3950,"log_dups_size":0,"ondisk_log_size":3950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:13.250467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227672064,"num_objects":3474,"num_object_clones":0,"num_object_copies":10422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10550,"num_write_kb":448768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":3474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":227,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.553737+0000","last_change":"2024-03-12T03:48:58.458916+0000","last_active":"2024-03-12T03:54:03.553737+0000","last_peered":"2024-03-12T03:54:03.553737+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.458916+0000","last_became_peered":"2024-03-12T03:48:58.458916+0000","last_unstale":"2024-03-12T03:54:03.553737+0000","last_undegraded":"2024-03-12T03:54:03.553737+0000","last_fullsized":"2024-03-12T03:48:58.440354+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:06.864708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.864708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"480'66","reported_seq":359,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554982+0000","last_change":"2024-03-12T03:51:38.936343+0000","last_active":"2024-03-12T03:54:03.554982+0000","last_peered":"2024-03-12T03:54:03.554982+0000","last_clean":"2024-03-12T03:54:03.554982+0000","last_became_active":"2024-03-12T03:48:48.462111+0000","last_became_peered":"2024-03-12T03:48:48.462111+0000","last_unstale":"2024-03-12T03:54:03.554982+0000","last_undegraded":"2024-03-12T03:54:03.554982+0000","last_fullsized":"2024-03-12T03:54:03.554982+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'54","last_scrub_stamp":"2024-03-12T03:51:38.936190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:38.936190+0000","objects_scrubbed":11,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":76,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":922,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.555549+0000","last_change":"2024-03-12T03:48:58.453277+0000","last_active":"2024-03-12T03:54:03.555549+0000","last_peered":"2024-03-12T03:54:03.555549+0000","last_clean":"2024-03-12T03:48:53.447092+0000","last_became_active":"2024-03-12T03:48:58.453277+0000","last_became_peered":"2024-03-12T03:48:58.453277+0000","last_unstale":"2024-03-12T03:54:03.555549+0000","last_undegraded":"2024-03-12T03:54:03.555549+0000","last_fullsized":"2024-03-12T03:48:58.443592+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.648824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.648824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":936,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.547902+0000","last_change":"2024-03-12T03:50:58.107026+0000","last_active":"2024-03-12T03:54:03.547902+0000","last_peered":"2024-03-12T03:54:03.547902+0000","last_clean":"2024-03-12T03:54:03.547902+0000","last_became_active":"2024-03-12T03:33:20.092873+0000","last_became_peered":"2024-03-12T03:33:20.092873+0000","last_unstale":"2024-03-12T03:54:03.547902+0000","last_undegraded":"2024-03-12T03:54:03.547902+0000","last_fullsized":"2024-03-12T03:54:03.547902+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:50:58.106746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:50:58.106746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":737,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.561313+0000","last_change":"2024-03-12T03:50:40.511469+0000","last_active":"2024-03-12T03:54:03.561313+0000","last_peered":"2024-03-12T03:54:03.561313+0000","last_clean":"2024-03-12T03:54:03.561313+0000","last_became_active":"2024-03-12T03:46:25.284416+0000","last_became_peered":"2024-03-12T03:46:25.284416+0000","last_unstale":"2024-03-12T03:54:03.561313+0000","last_undegraded":"2024-03-12T03:54:03.561313+0000","last_fullsized":"2024-03-12T03:54:03.561313+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:40.511391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:40.511391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229938+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":1059,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.546703+0000","last_change":"2024-03-12T03:48:58.460115+0000","last_active":"2024-03-12T03:54:03.546703+0000","last_peered":"2024-03-12T03:54:03.546703+0000","last_clean":"2024-03-12T03:48:53.471868+0000","last_became_active":"2024-03-12T03:48:58.460115+0000","last_became_peered":"2024-03-12T03:48:58.460115+0000","last_unstale":"2024-03-12T03:54:03.546703+0000","last_undegraded":"2024-03-12T03:54:03.546703+0000","last_fullsized":"2024-03-12T03:48:58.445534+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:06.690873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.690873+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"493'7098","reported_seq":7259,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:06.562641+0000","last_change":"2024-03-12T03:52:20.532953+0000","last_active":"2024-03-12T03:54:06.562641+0000","last_peered":"2024-03-12T03:54:06.562641+0000","last_clean":"2024-03-12T03:54:06.562641+0000","last_became_active":"2024-03-12T03:52:17.865616+0000","last_became_peered":"2024-03-12T03:52:17.865616+0000","last_unstale":"2024-03-12T03:54:06.562641+0000","last_undegraded":"2024-03-12T03:54:06.562641+0000","last_fullsized":"2024-03-12T03:54:06.562641+0000","mapping_epoch":438,"log_start":"448'2529","ondisk_log_start":"448'2529","created":438,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'3769","last_scrub_stamp":"2024-03-12T03:52:20.532779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:52:20.532779+0000","objects_scrubbed":1944,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128253952,"num_objects":1957,"num_object_clones":0,"num_object_copies":5871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3329,"num_write_kb":106944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":480,"reported_epoch":493,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:03.560851+0000","last_change":"2024-03-12T03:51:00.633662+0000","last_active":"2024-03-12T03:54:03.560851+0000","last_peered":"2024-03-12T03:54:03.560851+0000","last_clean":"2024-03-12T03:54:03.560851+0000","last_became_active":"2024-03-12T03:50:58.843997+0000","last_became_peered":"2024-03-12T03:50:58.843997+0000","last_unstale":"2024-03-12T03:54:03.560851+0000","last_undegraded":"2024-03-12T03:54:03.560851+0000","last_fullsized":"2024-03-12T03:54:03.560851+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:00.633453+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.924742+0000","last_clean_scrub_stamp":"2024-03-12T03:51:00.633453+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.230086+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"480'32","reported_seq":715,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.554848+0000","last_change":"2024-03-12T03:48:58.462328+0000","last_active":"2024-03-12T03:54:03.554848+0000","last_peered":"2024-03-12T03:54:03.554848+0000","last_clean":"2024-03-12T03:43:28.842466+0000","last_became_active":"2024-03-12T03:48:58.462328+0000","last_became_peered":"2024-03-12T03:48:58.462328+0000","last_unstale":"2024-03-12T03:54:03.554848+0000","last_undegraded":"2024-03-12T03:48:58.442501+0000","last_fullsized":"2024-03-12T03:48:58.442398+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:15.774076+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.774076+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692452+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":935,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.548511+0000","last_change":"2024-03-12T03:51:28.049718+0000","last_active":"2024-03-12T03:54:03.548511+0000","last_peered":"2024-03-12T03:54:03.548511+0000","last_clean":"2024-03-12T03:54:03.548511+0000","last_became_active":"2024-03-12T03:46:25.303784+0000","last_became_peered":"2024-03-12T03:46:25.303784+0000","last_unstale":"2024-03-12T03:54:03.548511+0000","last_undegraded":"2024-03-12T03:54:03.548511+0000","last_fullsized":"2024-03-12T03:54:03.548511+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:28.049638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:51:28.049638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":664,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.550188+0000","last_change":"2024-03-12T03:51:33.662176+0000","last_active":"2024-03-12T03:54:03.550188+0000","last_peered":"2024-03-12T03:54:03.550188+0000","last_clean":"2024-03-12T03:54:03.550188+0000","last_became_active":"2024-03-12T03:46:25.284556+0000","last_became_peered":"2024-03-12T03:46:25.284556+0000","last_unstale":"2024-03-12T03:54:03.550188+0000","last_undegraded":"2024-03-12T03:54:03.550188+0000","last_fullsized":"2024-03-12T03:54:03.550188+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:33.662099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:33.662099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.332885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":509,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.543896+0000","last_change":"2024-03-12T03:48:58.464502+0000","last_active":"2024-03-12T03:54:03.543896+0000","last_peered":"2024-03-12T03:54:03.543896+0000","last_clean":"2024-03-12T03:39:28.535672+0000","last_became_active":"2024-03-12T03:48:58.464502+0000","last_became_peered":"2024-03-12T03:48:58.464502+0000","last_unstale":"2024-03-12T03:54:03.543896+0000","last_undegraded":"2024-03-12T03:54:03.543896+0000","last_fullsized":"2024-03-12T03:48:58.446034+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:55.192996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:55.192996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697373+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"493'7046","reported_seq":7203,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:06.498887+0000","last_change":"2024-03-12T03:52:19.581816+0000","last_active":"2024-03-12T03:54:06.498887+0000","last_peered":"2024-03-12T03:54:06.498887+0000","last_clean":"2024-03-12T03:54:06.498887+0000","last_became_active":"2024-03-12T03:52:17.874643+0000","last_became_peered":"2024-03-12T03:52:17.874643+0000","last_unstale":"2024-03-12T03:54:06.498887+0000","last_undegraded":"2024-03-12T03:54:06.498887+0000","last_fullsized":"2024-03-12T03:54:06.498887+0000","mapping_epoch":438,"log_start":"446'2212","ondisk_log_start":"446'2212","created":438,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'3868","last_scrub_stamp":"2024-03-12T03:52:19.581698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:52:19.581698+0000","objects_scrubbed":1911,"log_size":4834,"log_dups_size":0,"ondisk_log_size":4834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118816768,"num_objects":1813,"num_object_clones":0,"num_object_copies":5439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3178,"num_write_kb":98560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":809,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554027+0000","last_change":"2024-03-12T03:50:59.586938+0000","last_active":"2024-03-12T03:54:03.554027+0000","last_peered":"2024-03-12T03:54:03.554027+0000","last_clean":"2024-03-12T03:54:03.554027+0000","last_became_active":"2024-03-12T03:50:58.840022+0000","last_became_peered":"2024-03-12T03:50:58.840022+0000","last_unstale":"2024-03-12T03:54:03.554027+0000","last_undegraded":"2024-03-12T03:54:03.554027+0000","last_fullsized":"2024-03-12T03:54:03.554027+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.586800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.586800+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.230121+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"480'67","reported_seq":942,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.553770+0000","last_change":"2024-03-12T03:49:53.484809+0000","last_active":"2024-03-12T03:54:03.553770+0000","last_peered":"2024-03-12T03:54:03.553770+0000","last_clean":"2024-03-12T03:54:03.553770+0000","last_became_active":"2024-03-12T03:45:18.106503+0000","last_became_peered":"2024-03-12T03:45:18.106503+0000","last_unstale":"2024-03-12T03:54:03.553770+0000","last_undegraded":"2024-03-12T03:54:03.553770+0000","last_fullsized":"2024-03-12T03:54:03.553770+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'51","last_scrub_stamp":"2024-03-12T03:49:53.484646+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:49:53.484646+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692733+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":76,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":935,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.545082+0000","last_change":"2024-03-12T03:51:34.290875+0000","last_active":"2024-03-12T03:54:03.545082+0000","last_peered":"2024-03-12T03:54:03.545082+0000","last_clean":"2024-03-12T03:54:03.545082+0000","last_became_active":"2024-03-12T03:33:21.108192+0000","last_became_peered":"2024-03-12T03:33:21.108192+0000","last_unstale":"2024-03-12T03:54:03.545082+0000","last_undegraded":"2024-03-12T03:54:03.545082+0000","last_fullsized":"2024-03-12T03:54:03.545082+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:34.290763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:51:34.290763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":641,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.549940+0000","last_change":"2024-03-12T03:48:58.459230+0000","last_active":"2024-03-12T03:54:03.549940+0000","last_peered":"2024-03-12T03:54:03.549940+0000","last_clean":"2024-03-12T03:48:53.446740+0000","last_became_active":"2024-03-12T03:48:58.459230+0000","last_became_peered":"2024-03-12T03:48:58.459230+0000","last_unstale":"2024-03-12T03:54:03.549940+0000","last_undegraded":"2024-03-12T03:54:03.549940+0000","last_fullsized":"2024-03-12T03:48:58.447475+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.954731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.954731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.332712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":945,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.542509+0000","last_change":"2024-03-12T03:50:42.276895+0000","last_active":"2024-03-12T03:54:03.542509+0000","last_peered":"2024-03-12T03:54:03.542509+0000","last_clean":"2024-03-12T03:54:03.542509+0000","last_became_active":"2024-03-12T03:46:25.292122+0000","last_became_peered":"2024-03-12T03:46:25.292122+0000","last_unstale":"2024-03-12T03:54:03.542509+0000","last_undegraded":"2024-03-12T03:54:03.542509+0000","last_fullsized":"2024-03-12T03:54:03.542509+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:42.276562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:42.276562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697672+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"493'6911","reported_seq":7036,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:07.192054+0000","last_change":"2024-03-12T03:52:17.847287+0000","last_active":"2024-03-12T03:54:07.192054+0000","last_peered":"2024-03-12T03:54:07.192054+0000","last_clean":"2024-03-12T03:51:39.806188+0000","last_became_active":"2024-03-12T03:52:17.847287+0000","last_became_peered":"2024-03-12T03:52:17.847287+0000","last_unstale":"2024-03-12T03:54:07.192054+0000","last_undegraded":"2024-03-12T03:52:17.831115+0000","last_fullsized":"2024-03-12T03:52:17.830980+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":438,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":6911,"log_dups_size":0,"ondisk_log_size":6911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:11.454326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112918528,"num_objects":1723,"num_object_clones":0,"num_object_copies":5169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5073,"num_write_kb":217472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1723}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":881,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.553903+0000","last_change":"2024-03-12T03:51:27.598751+0000","last_active":"2024-03-12T03:54:03.553903+0000","last_peered":"2024-03-12T03:54:03.553903+0000","last_clean":"2024-03-12T03:54:03.553903+0000","last_became_active":"2024-03-12T03:50:58.836091+0000","last_became_peered":"2024-03-12T03:50:58.836091+0000","last_unstale":"2024-03-12T03:54:03.553903+0000","last_undegraded":"2024-03-12T03:54:03.553903+0000","last_fullsized":"2024-03-12T03:54:03.553903+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:27.598607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:51:27.598607+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.230154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":657,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.552223+0000","last_change":"2024-03-12T03:48:58.461069+0000","last_active":"2024-03-12T03:54:03.552223+0000","last_peered":"2024-03-12T03:54:03.552223+0000","last_clean":"2024-03-12T03:48:53.445898+0000","last_became_active":"2024-03-12T03:48:58.461069+0000","last_became_peered":"2024-03-12T03:48:58.461069+0000","last_unstale":"2024-03-12T03:54:03.552223+0000","last_undegraded":"2024-03-12T03:54:03.552223+0000","last_fullsized":"2024-03-12T03:48:58.444677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.981014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.981014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.332736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"480'36","reported_seq":639,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.552579+0000","last_change":"2024-03-12T03:51:42.837070+0000","last_active":"2024-03-12T03:54:03.552579+0000","last_peered":"2024-03-12T03:54:03.552579+0000","last_clean":"2024-03-12T03:54:03.552579+0000","last_became_active":"2024-03-12T03:51:42.836618+0000","last_became_peered":"2024-03-12T03:51:42.836618+0000","last_unstale":"2024-03-12T03:54:03.552579+0000","last_undegraded":"2024-03-12T03:54:03.552579+0000","last_fullsized":"2024-03-12T03:54:03.552579+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'30","last_scrub_stamp":"2024-03-12T03:49:16.863622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:49:16.863622+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":935,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.544786+0000","last_change":"2024-03-12T03:50:39.308944+0000","last_active":"2024-03-12T03:54:03.544786+0000","last_peered":"2024-03-12T03:54:03.544786+0000","last_clean":"2024-03-12T03:54:03.544786+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:54:03.544786+0000","last_undegraded":"2024-03-12T03:54:03.544786+0000","last_fullsized":"2024-03-12T03:54:03.544786+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:39.308832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:50:39.308832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697594+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":660,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555529+0000","last_change":"2024-03-12T03:51:41.674084+0000","last_active":"2024-03-12T03:54:03.555529+0000","last_peered":"2024-03-12T03:54:03.555529+0000","last_clean":"2024-03-12T03:54:03.555529+0000","last_became_active":"2024-03-12T03:46:25.280784+0000","last_became_peered":"2024-03-12T03:46:25.280784+0000","last_unstale":"2024-03-12T03:54:03.555529+0000","last_undegraded":"2024-03-12T03:54:03.555529+0000","last_fullsized":"2024-03-12T03:54:03.555529+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:41.673933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:41.673933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.233062+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"493'6995","reported_seq":7153,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:06.723519+0000","last_change":"2024-03-12T03:52:18.432284+0000","last_active":"2024-03-12T03:54:06.723519+0000","last_peered":"2024-03-12T03:54:06.723519+0000","last_clean":"2024-03-12T03:54:06.723519+0000","last_became_active":"2024-03-12T03:52:17.849018+0000","last_became_peered":"2024-03-12T03:52:17.849018+0000","last_unstale":"2024-03-12T03:54:06.723519+0000","last_undegraded":"2024-03-12T03:54:06.723519+0000","last_fullsized":"2024-03-12T03:54:06.723519+0000","mapping_epoch":438,"log_start":"444'1058","ondisk_log_start":"444'1058","created":438,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'3756","last_scrub_stamp":"2024-03-12T03:52:18.431993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:52:18.431993+0000","objects_scrubbed":1910,"log_size":5937,"log_dups_size":0,"ondisk_log_size":5937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:11.133537+0000","scrub_duration":472,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120913920,"num_objects":1845,"num_object_clones":0,"num_object_copies":5535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3257,"num_write_kb":102144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":814,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.557211+0000","last_change":"2024-03-12T03:50:58.872833+0000","last_active":"2024-03-12T03:54:03.557211+0000","last_peered":"2024-03-12T03:54:03.557211+0000","last_clean":"2024-03-12T03:54:03.557211+0000","last_became_active":"2024-03-12T03:50:58.850740+0000","last_became_peered":"2024-03-12T03:50:58.850740+0000","last_unstale":"2024-03-12T03:54:03.557211+0000","last_undegraded":"2024-03-12T03:54:03.557211+0000","last_fullsized":"2024-03-12T03:54:03.557211+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:58.872716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:58.872716+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":1177,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.544686+0000","last_change":"2024-03-12T03:50:37.369892+0000","last_active":"2024-03-12T03:54:03.544686+0000","last_peered":"2024-03-12T03:54:03.544686+0000","last_clean":"2024-03-12T03:54:03.544686+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:54:03.544686+0000","last_undegraded":"2024-03-12T03:54:03.544686+0000","last_fullsized":"2024-03-12T03:54:03.544686+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:50:37.369758+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:50:37.369758+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697514+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"480'60","reported_seq":1023,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.547530+0000","last_change":"2024-03-12T03:52:18.945880+0000","last_active":"2024-03-12T03:54:03.547530+0000","last_peered":"2024-03-12T03:54:03.547530+0000","last_clean":"2024-03-12T03:54:03.547530+0000","last_became_active":"2024-03-12T03:33:23.552474+0000","last_became_peered":"2024-03-12T03:33:23.552474+0000","last_unstale":"2024-03-12T03:54:03.547530+0000","last_undegraded":"2024-03-12T03:54:03.547530+0000","last_fullsized":"2024-03-12T03:54:03.547530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'48","last_scrub_stamp":"2024-03-12T03:52:18.945763+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:52:18.945763+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":926,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.544314+0000","last_change":"2024-03-12T03:48:58.456357+0000","last_active":"2024-03-12T03:54:03.544314+0000","last_peered":"2024-03-12T03:54:03.544314+0000","last_clean":"2024-03-12T03:48:53.450296+0000","last_became_active":"2024-03-12T03:48:58.456357+0000","last_became_peered":"2024-03-12T03:48:58.456357+0000","last_unstale":"2024-03-12T03:54:03.544314+0000","last_undegraded":"2024-03-12T03:48:58.445880+0000","last_fullsized":"2024-03-12T03:48:58.445769+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:47:05.588582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:05.588582+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697401+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":900,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.547598+0000","last_change":"2024-03-12T03:48:58.452652+0000","last_active":"2024-03-12T03:54:03.547598+0000","last_peered":"2024-03-12T03:54:03.547598+0000","last_clean":"2024-03-12T03:48:53.472055+0000","last_became_active":"2024-03-12T03:48:58.452652+0000","last_became_peered":"2024-03-12T03:48:58.452652+0000","last_unstale":"2024-03-12T03:54:03.547598+0000","last_undegraded":"2024-03-12T03:54:03.547598+0000","last_fullsized":"2024-03-12T03:48:58.441674+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:59.567890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.567890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.18","version":"493'6983","reported_seq":12907,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:07.184468+0000","last_change":"2024-03-12T03:53:49.549311+0000","last_active":"2024-03-12T03:54:07.184468+0000","last_peered":"2024-03-12T03:54:07.184468+0000","last_clean":"2024-03-12T03:54:07.184468+0000","last_became_active":"2024-03-12T03:53:49.541966+0000","last_became_peered":"2024-03-12T03:53:49.541966+0000","last_unstale":"2024-03-12T03:54:07.184468+0000","last_undegraded":"2024-03-12T03:54:07.184468+0000","last_fullsized":"2024-03-12T03:54:07.184468+0000","mapping_epoch":488,"log_start":"443'873","ondisk_log_start":"443'873","created":452,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":6110,"log_dups_size":0,"ondisk_log_size":6110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:10.468062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118358016,"num_objects":1806,"num_object_clones":0,"num_object_copies":5418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5118,"num_write_kb":221568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1927,"num_bytes_recovered":126287872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":473,"reported_epoch":493,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:03.543519+0000","last_change":"2024-03-12T03:50:59.305188+0000","last_active":"2024-03-12T03:54:03.543519+0000","last_peered":"2024-03-12T03:54:03.543519+0000","last_clean":"2024-03-12T03:54:03.543519+0000","last_became_active":"2024-03-12T03:50:58.836425+0000","last_became_peered":"2024-03-12T03:50:58.836425+0000","last_unstale":"2024-03-12T03:54:03.543519+0000","last_undegraded":"2024-03-12T03:54:03.543519+0000","last_fullsized":"2024-03-12T03:54:03.543519+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.304950+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:50:06.384808+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.304950+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697758+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,1],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"480'24","reported_seq":732,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555239+0000","last_change":"2024-03-12T03:50:48.504074+0000","last_active":"2024-03-12T03:54:03.555239+0000","last_peered":"2024-03-12T03:54:03.555239+0000","last_clean":"2024-03-12T03:54:03.555239+0000","last_became_active":"2024-03-12T03:46:25.288573+0000","last_became_peered":"2024-03-12T03:46:25.288573+0000","last_unstale":"2024-03-12T03:54:03.555239+0000","last_undegraded":"2024-03-12T03:54:03.555239+0000","last_fullsized":"2024-03-12T03:54:03.555239+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-03-12T03:50:48.503336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.503336+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229973+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":229,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.547673+0000","last_change":"2024-03-12T03:48:58.452359+0000","last_active":"2024-03-12T03:54:03.547673+0000","last_peered":"2024-03-12T03:54:03.547673+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:48:58.452359+0000","last_became_peered":"2024-03-12T03:48:58.452359+0000","last_unstale":"2024-03-12T03:54:03.547673+0000","last_undegraded":"2024-03-12T03:54:03.547673+0000","last_fullsized":"2024-03-12T03:48:58.441273+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:10.400135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.400135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":926,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.548730+0000","last_change":"2024-03-12T03:50:42.134792+0000","last_active":"2024-03-12T03:54:03.548730+0000","last_peered":"2024-03-12T03:54:03.548730+0000","last_clean":"2024-03-12T03:54:03.548730+0000","last_became_active":"2024-03-12T03:46:25.293521+0000","last_became_peered":"2024-03-12T03:46:25.293521+0000","last_unstale":"2024-03-12T03:54:03.548730+0000","last_undegraded":"2024-03-12T03:54:03.548730+0000","last_fullsized":"2024-03-12T03:54:03.548730+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:42.134482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:42.134482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":922,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.544658+0000","last_change":"2024-03-12T03:51:38.361542+0000","last_active":"2024-03-12T03:54:03.544658+0000","last_peered":"2024-03-12T03:54:03.544658+0000","last_clean":"2024-03-12T03:54:03.544658+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:54:03.544658+0000","last_undegraded":"2024-03-12T03:54:03.544658+0000","last_fullsized":"2024-03-12T03:54:03.544658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:38.361430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:38.361430+0000","last_clean_scrub_stamp":"2024-03-12T03:51:38.361430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697877+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.19","version":"493'7211","reported_seq":13221,"reported_epoch":493,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:06.586185+0000","last_change":"2024-03-12T03:53:15.280488+0000","last_active":"2024-03-12T03:54:06.586185+0000","last_peered":"2024-03-12T03:54:06.586185+0000","last_clean":"2024-03-12T03:54:06.586185+0000","last_became_active":"2024-03-12T03:53:15.276255+0000","last_became_peered":"2024-03-12T03:53:15.276255+0000","last_unstale":"2024-03-12T03:54:06.586185+0000","last_undegraded":"2024-03-12T03:54:06.586185+0000","last_fullsized":"2024-03-12T03:54:06.586185+0000","mapping_epoch":472,"log_start":"448'2603","ondisk_log_start":"448'2603","created":452,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.673881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119996416,"num_objects":1831,"num_object_clones":0,"num_object_copies":5493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1831,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5301,"num_write_kb":228224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1959,"num_bytes_recovered":128385024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,3],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":1831}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"480'33","reported_seq":858,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.563694+0000","last_change":"2024-03-12T03:48:58.462615+0000","last_active":"2024-03-12T03:54:03.563694+0000","last_peered":"2024-03-12T03:54:03.563694+0000","last_clean":"2024-03-12T03:43:13.333279+0000","last_became_active":"2024-03-12T03:48:58.462615+0000","last_became_peered":"2024-03-12T03:48:58.462615+0000","last_unstale":"2024-03-12T03:54:03.563694+0000","last_undegraded":"2024-03-12T03:48:58.443892+0000","last_fullsized":"2024-03-12T03:48:58.443761+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:48:06.104297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.104297+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229489+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":926,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.544441+0000","last_change":"2024-03-12T03:50:35.416894+0000","last_active":"2024-03-12T03:54:03.544441+0000","last_peered":"2024-03-12T03:54:03.544441+0000","last_clean":"2024-03-12T03:54:03.544441+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:54:03.544441+0000","last_undegraded":"2024-03-12T03:54:03.544441+0000","last_fullsized":"2024-03-12T03:54:03.544441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:35.416780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:35.416780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697570+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":694,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555389+0000","last_change":"2024-03-12T03:52:19.693182+0000","last_active":"2024-03-12T03:54:03.555389+0000","last_peered":"2024-03-12T03:54:03.555389+0000","last_clean":"2024-03-12T03:54:03.555389+0000","last_became_active":"2024-03-12T03:46:25.286866+0000","last_became_peered":"2024-03-12T03:46:25.286866+0000","last_unstale":"2024-03-12T03:54:03.555389+0000","last_undegraded":"2024-03-12T03:54:03.555389+0000","last_fullsized":"2024-03-12T03:54:03.555389+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:52:19.693090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:52:19.693090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.233106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":654,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.550999+0000","last_change":"2024-03-12T03:50:37.807983+0000","last_active":"2024-03-12T03:54:03.550999+0000","last_peered":"2024-03-12T03:54:03.550999+0000","last_clean":"2024-03-12T03:54:03.550999+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:54:03.550999+0000","last_undegraded":"2024-03-12T03:54:03.550999+0000","last_fullsized":"2024-03-12T03:54:03.550999+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:37.807855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:37.807855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.332816+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":943,"reported_epoch":493,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:03.542249+0000","last_change":"2024-03-12T03:50:57.227680+0000","last_active":"2024-03-12T03:54:03.542249+0000","last_peered":"2024-03-12T03:54:03.542249+0000","last_clean":"2024-03-12T03:54:03.542249+0000","last_became_active":"2024-03-12T03:50:56.823948+0000","last_became_peered":"2024-03-12T03:50:56.823948+0000","last_unstale":"2024-03-12T03:54:03.542249+0000","last_undegraded":"2024-03-12T03:54:03.542249+0000","last_fullsized":"2024-03-12T03:54:03.542249+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:57.227429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:57.227429+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697789+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"480'27","reported_seq":993,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.550700+0000","last_change":"2024-03-12T03:51:01.099499+0000","last_active":"2024-03-12T03:54:03.550700+0000","last_peered":"2024-03-12T03:54:03.550700+0000","last_clean":"2024-03-12T03:54:03.550700+0000","last_became_active":"2024-03-12T03:46:25.298007+0000","last_became_peered":"2024-03-12T03:46:25.298007+0000","last_unstale":"2024-03-12T03:54:03.550700+0000","last_undegraded":"2024-03-12T03:54:03.550700+0000","last_fullsized":"2024-03-12T03:54:03.550700+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'21","last_scrub_stamp":"2024-03-12T03:51:01.099350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:01.099350+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":803,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.543893+0000","last_change":"2024-03-12T03:48:58.462149+0000","last_active":"2024-03-12T03:54:03.543893+0000","last_peered":"2024-03-12T03:54:03.543893+0000","last_clean":"2024-03-12T03:43:13.331412+0000","last_became_active":"2024-03-12T03:48:58.462149+0000","last_became_peered":"2024-03-12T03:48:58.462149+0000","last_unstale":"2024-03-12T03:54:03.543893+0000","last_undegraded":"2024-03-12T03:54:03.543893+0000","last_fullsized":"2024-03-12T03:48:58.443909+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697426+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"332'14","reported_seq":966,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.556502+0000","last_change":"2024-03-12T03:50:53.568427+0000","last_active":"2024-03-12T03:54:03.556502+0000","last_peered":"2024-03-12T03:54:03.556502+0000","last_clean":"2024-03-12T03:54:03.556502+0000","last_became_active":"2024-03-12T03:46:25.300463+0000","last_became_peered":"2024-03-12T03:46:25.300463+0000","last_unstale":"2024-03-12T03:54:03.556502+0000","last_undegraded":"2024-03-12T03:54:03.556502+0000","last_fullsized":"2024-03-12T03:54:03.556502+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'14","last_scrub_stamp":"2024-03-12T03:50:53.568307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.568307+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.230016+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":407,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.563304+0000","last_change":"2024-03-12T03:51:42.834189+0000","last_active":"2024-03-12T03:54:03.563304+0000","last_peered":"2024-03-12T03:54:03.563304+0000","last_clean":"2024-03-12T03:54:03.563304+0000","last_became_active":"2024-03-12T03:51:42.833867+0000","last_became_peered":"2024-03-12T03:51:42.833867+0000","last_unstale":"2024-03-12T03:54:03.563304+0000","last_undegraded":"2024-03-12T03:54:03.563304+0000","last_fullsized":"2024-03-12T03:54:03.563304+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:04.042226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:04.042226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":874,"reported_epoch":493,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:03.546445+0000","last_change":"2024-03-12T03:50:59.098952+0000","last_active":"2024-03-12T03:54:03.546445+0000","last_peered":"2024-03-12T03:54:03.546445+0000","last_clean":"2024-03-12T03:54:03.546445+0000","last_became_active":"2024-03-12T03:50:58.849455+0000","last_became_peered":"2024-03-12T03:50:58.849455+0000","last_unstale":"2024-03-12T03:54:03.546445+0000","last_undegraded":"2024-03-12T03:54:03.546445+0000","last_fullsized":"2024-03-12T03:54:03.546445+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.098619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.098619+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,3],"avail_no_missing":["5","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"480'49","reported_seq":808,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.556081+0000","last_change":"2024-03-12T03:51:40.548114+0000","last_active":"2024-03-12T03:54:03.556081+0000","last_peered":"2024-03-12T03:54:03.556081+0000","last_clean":"2024-03-12T03:54:03.556081+0000","last_became_active":"2024-03-12T03:46:25.299798+0000","last_became_peered":"2024-03-12T03:46:25.299798+0000","last_unstale":"2024-03-12T03:54:03.556081+0000","last_undegraded":"2024-03-12T03:54:03.556081+0000","last_fullsized":"2024-03-12T03:54:03.556081+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'39","last_scrub_stamp":"2024-03-12T03:51:40.547967+0000","last_deep_scrub":"403'39","last_deep_scrub_stamp":"2024-03-12T03:50:06.535772+0000","last_clean_scrub_stamp":"2024-03-12T03:51:40.547967+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.231129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":961,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.543709+0000","last_change":"2024-03-12T03:51:32.285686+0000","last_active":"2024-03-12T03:54:03.543709+0000","last_peered":"2024-03-12T03:54:03.543709+0000","last_clean":"2024-03-12T03:54:03.543709+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:54:03.543709+0000","last_undegraded":"2024-03-12T03:54:03.543709+0000","last_fullsized":"2024-03-12T03:54:03.543709+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:51:32.285604+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:51:32.285604+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"385'9","reported_seq":1161,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.548729+0000","last_change":"2024-03-12T03:48:58.448464+0000","last_active":"2024-03-12T03:54:03.548729+0000","last_peered":"2024-03-12T03:54:03.548729+0000","last_clean":"2024-03-12T03:48:56.813639+0000","last_became_active":"2024-03-12T03:48:58.448464+0000","last_became_peered":"2024-03-12T03:48:58.448464+0000","last_unstale":"2024-03-12T03:54:03.548729+0000","last_undegraded":"2024-03-12T03:48:58.442644+0000","last_fullsized":"2024-03-12T03:48:58.442315+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:53.458170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.458170+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":791,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.544543+0000","last_change":"2024-03-12T03:48:58.463189+0000","last_active":"2024-03-12T03:54:03.544543+0000","last_peered":"2024-03-12T03:54:03.544543+0000","last_clean":"2024-03-12T03:43:13.332853+0000","last_became_active":"2024-03-12T03:48:58.463189+0000","last_became_peered":"2024-03-12T03:48:58.463189+0000","last_unstale":"2024-03-12T03:54:03.544543+0000","last_undegraded":"2024-03-12T03:54:03.544543+0000","last_fullsized":"2024-03-12T03:48:58.445060+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.079569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.079569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697438+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"480'27","reported_seq":973,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555990+0000","last_change":"2024-03-12T03:51:01.797152+0000","last_active":"2024-03-12T03:54:03.555990+0000","last_peered":"2024-03-12T03:54:03.555990+0000","last_clean":"2024-03-12T03:54:03.555990+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:54:03.555990+0000","last_undegraded":"2024-03-12T03:54:03.555990+0000","last_fullsized":"2024-03-12T03:54:03.555990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'19","last_scrub_stamp":"2024-03-12T03:51:01.797018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:01.797018+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":125,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.544979+0000","last_change":"2024-03-12T03:51:46.344521+0000","last_active":"2024-03-12T03:54:03.544979+0000","last_peered":"2024-03-12T03:54:03.544979+0000","last_clean":"2024-03-12T03:54:03.544979+0000","last_became_active":"2024-03-12T03:51:42.833922+0000","last_became_peered":"2024-03-12T03:51:42.833922+0000","last_unstale":"2024-03-12T03:54:03.544979+0000","last_undegraded":"2024-03-12T03:54:03.544979+0000","last_fullsized":"2024-03-12T03:54:03.544979+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:46.344443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_clean_scrub_stamp":"2024-03-12T03:51:46.344443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":935,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.553489+0000","last_change":"2024-03-12T03:50:48.723164+0000","last_active":"2024-03-12T03:54:03.553489+0000","last_peered":"2024-03-12T03:54:03.553489+0000","last_clean":"2024-03-12T03:54:03.553489+0000","last_became_active":"2024-03-12T03:46:25.288050+0000","last_became_peered":"2024-03-12T03:46:25.288050+0000","last_unstale":"2024-03-12T03:54:03.553489+0000","last_undegraded":"2024-03-12T03:54:03.553489+0000","last_fullsized":"2024-03-12T03:54:03.553489+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.723073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.723073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692906+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":697,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.562310+0000","last_change":"2024-03-12T03:48:58.456788+0000","last_active":"2024-03-12T03:54:03.562310+0000","last_peered":"2024-03-12T03:54:03.562310+0000","last_clean":"2024-03-12T03:48:53.447742+0000","last_became_active":"2024-03-12T03:48:58.456788+0000","last_became_peered":"2024-03-12T03:48:58.456788+0000","last_unstale":"2024-03-12T03:54:03.562310+0000","last_undegraded":"2024-03-12T03:54:03.562310+0000","last_fullsized":"2024-03-12T03:48:58.442213+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.433184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.433184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229446+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"480'21","reported_seq":534,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.546442+0000","last_change":"2024-03-12T03:52:20.985841+0000","last_active":"2024-03-12T03:54:03.546442+0000","last_peered":"2024-03-12T03:54:03.546442+0000","last_clean":"2024-03-12T03:54:03.546442+0000","last_became_active":"2024-03-12T03:43:01.037348+0000","last_became_peered":"2024-03-12T03:43:01.037348+0000","last_unstale":"2024-03-12T03:54:03.546442+0000","last_undegraded":"2024-03-12T03:54:03.546442+0000","last_fullsized":"2024-03-12T03:54:03.546442+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'15","last_scrub_stamp":"2024-03-12T03:52:20.985668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:52:20.985668+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":952,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.544756+0000","last_change":"2024-03-12T03:51:47.306023+0000","last_active":"2024-03-12T03:54:03.544756+0000","last_peered":"2024-03-12T03:54:03.544756+0000","last_clean":"2024-03-12T03:54:03.544756+0000","last_became_active":"2024-03-12T03:46:25.283602+0000","last_became_peered":"2024-03-12T03:46:25.283602+0000","last_unstale":"2024-03-12T03:54:03.544756+0000","last_undegraded":"2024-03-12T03:54:03.544756+0000","last_fullsized":"2024-03-12T03:54:03.544756+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:47.305919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:51:47.305919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.698011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":709,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.554187+0000","last_change":"2024-03-12T03:48:58.460970+0000","last_active":"2024-03-12T03:54:03.554187+0000","last_peered":"2024-03-12T03:54:03.554187+0000","last_clean":"2024-03-12T03:48:53.448497+0000","last_became_active":"2024-03-12T03:48:58.460970+0000","last_became_peered":"2024-03-12T03:48:58.460970+0000","last_unstale":"2024-03-12T03:54:03.554187+0000","last_undegraded":"2024-03-12T03:54:03.554187+0000","last_fullsized":"2024-03-12T03:48:58.444135+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_clean_scrub_stamp":"2024-03-12T03:48:10.625582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":227,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.547830+0000","last_change":"2024-03-12T03:48:58.453783+0000","last_active":"2024-03-12T03:54:03.547830+0000","last_peered":"2024-03-12T03:54:03.547830+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.453783+0000","last_became_peered":"2024-03-12T03:48:58.453783+0000","last_unstale":"2024-03-12T03:54:03.547830+0000","last_undegraded":"2024-03-12T03:54:03.547830+0000","last_fullsized":"2024-03-12T03:48:58.441749+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:13.228117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:48:13.228117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"480'34","reported_seq":745,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.562589+0000","last_change":"2024-03-12T03:51:31.509254+0000","last_active":"2024-03-12T03:54:03.562589+0000","last_peered":"2024-03-12T03:54:03.562589+0000","last_clean":"2024-03-12T03:54:03.562589+0000","last_became_active":"2024-03-12T03:46:25.287095+0000","last_became_peered":"2024-03-12T03:46:25.287095+0000","last_unstale":"2024-03-12T03:54:03.562589+0000","last_undegraded":"2024-03-12T03:54:03.562589+0000","last_fullsized":"2024-03-12T03:54:03.562589+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:51:31.509156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:31.509156+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.230186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":932,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.548847+0000","last_change":"2024-03-12T03:51:42.097053+0000","last_active":"2024-03-12T03:54:03.548847+0000","last_peered":"2024-03-12T03:54:03.548847+0000","last_clean":"2024-03-12T03:54:03.548847+0000","last_became_active":"2024-03-12T03:33:21.107073+0000","last_became_peered":"2024-03-12T03:33:21.107073+0000","last_unstale":"2024-03-12T03:54:03.548847+0000","last_undegraded":"2024-03-12T03:54:03.548847+0000","last_fullsized":"2024-03-12T03:54:03.548847+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:42.096972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:51:42.096972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":926,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.543349+0000","last_change":"2024-03-12T03:51:39.276331+0000","last_active":"2024-03-12T03:54:03.543349+0000","last_peered":"2024-03-12T03:54:03.543349+0000","last_clean":"2024-03-12T03:54:03.543349+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:54:03.543349+0000","last_undegraded":"2024-03-12T03:54:03.543349+0000","last_fullsized":"2024-03-12T03:54:03.543349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.276250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.276250+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.276250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":724,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.555631+0000","last_change":"2024-03-12T03:48:58.449798+0000","last_active":"2024-03-12T03:54:03.555631+0000","last_peered":"2024-03-12T03:54:03.555631+0000","last_clean":"2024-03-12T03:48:53.448767+0000","last_became_active":"2024-03-12T03:48:58.449798+0000","last_became_peered":"2024-03-12T03:48:58.449798+0000","last_unstale":"2024-03-12T03:54:03.555631+0000","last_undegraded":"2024-03-12T03:54:03.555631+0000","last_fullsized":"2024-03-12T03:48:58.443857+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"480'14","reported_seq":710,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555658+0000","last_change":"2024-03-12T03:50:39.504167+0000","last_active":"2024-03-12T03:54:03.555658+0000","last_peered":"2024-03-12T03:54:03.555658+0000","last_clean":"2024-03-12T03:54:03.555658+0000","last_became_active":"2024-03-12T03:46:25.286315+0000","last_became_peered":"2024-03-12T03:46:25.286315+0000","last_unstale":"2024-03-12T03:54:03.555658+0000","last_undegraded":"2024-03-12T03:54:03.555658+0000","last_fullsized":"2024-03-12T03:54:03.555658+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'10","last_scrub_stamp":"2024-03-12T03:50:39.504024+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:50:39.504024+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229882+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":803,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.550906+0000","last_change":"2024-03-12T03:48:58.463096+0000","last_active":"2024-03-12T03:54:03.550906+0000","last_peered":"2024-03-12T03:54:03.550906+0000","last_clean":"2024-03-12T03:43:13.332505+0000","last_became_active":"2024-03-12T03:48:58.463096+0000","last_became_peered":"2024-03-12T03:48:58.463096+0000","last_unstale":"2024-03-12T03:54:03.550906+0000","last_undegraded":"2024-03-12T03:54:03.550906+0000","last_fullsized":"2024-03-12T03:48:58.443705+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.851271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.851271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":389,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554901+0000","last_change":"2024-03-12T03:51:00.809146+0000","last_active":"2024-03-12T03:54:03.554901+0000","last_peered":"2024-03-12T03:54:03.554901+0000","last_clean":"2024-03-12T03:54:03.554901+0000","last_became_active":"2024-03-12T03:45:18.109983+0000","last_became_peered":"2024-03-12T03:45:18.109983+0000","last_unstale":"2024-03-12T03:54:03.554901+0000","last_undegraded":"2024-03-12T03:54:03.554901+0000","last_fullsized":"2024-03-12T03:54:03.554901+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:00.809033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:00.809033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":934,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.544417+0000","last_change":"2024-03-12T03:52:17.430546+0000","last_active":"2024-03-12T03:54:03.544417+0000","last_peered":"2024-03-12T03:54:03.544417+0000","last_clean":"2024-03-12T03:54:03.544417+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:54:03.544417+0000","last_undegraded":"2024-03-12T03:54:03.544417+0000","last_fullsized":"2024-03-12T03:54:03.544417+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:52:17.430447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:52:17.430447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.698130+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"493'6906","reported_seq":12698,"reported_epoch":493,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:07.212943+0000","last_change":"2024-03-12T03:53:06.930239+0000","last_active":"2024-03-12T03:54:07.212943+0000","last_peered":"2024-03-12T03:54:07.212943+0000","last_clean":"2024-03-12T03:54:07.212943+0000","last_became_active":"2024-03-12T03:53:06.924057+0000","last_became_peered":"2024-03-12T03:53:06.924057+0000","last_unstale":"2024-03-12T03:54:07.212943+0000","last_undegraded":"2024-03-12T03:54:07.212943+0000","last_fullsized":"2024-03-12T03:54:07.212943+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":452,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":6906,"log_dups_size":0,"ondisk_log_size":6906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:11.457956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118292480,"num_objects":1805,"num_object_clones":0,"num_object_copies":5415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1805,"num_objects_unfound":0,"num_objects_dirty":1805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5067,"num_write_kb":219904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1885,"num_bytes_recovered":123535360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6,1],"avail_no_missing":["2","1","6"],"object_location_counts":[{"shards":"1,2,6","objects":1805}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"480'39","reported_seq":910,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.551146+0000","last_change":"2024-03-12T03:51:47.045902+0000","last_active":"2024-03-12T03:54:03.551146+0000","last_peered":"2024-03-12T03:54:03.551146+0000","last_clean":"2024-03-12T03:54:03.551146+0000","last_became_active":"2024-03-12T03:45:18.107047+0000","last_became_peered":"2024-03-12T03:45:18.107047+0000","last_unstale":"2024-03-12T03:54:03.551146+0000","last_undegraded":"2024-03-12T03:54:03.551146+0000","last_fullsized":"2024-03-12T03:54:03.551146+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:51:47.045769+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:51:47.045769+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":725,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.556085+0000","last_change":"2024-03-12T03:48:58.454467+0000","last_active":"2024-03-12T03:54:03.556085+0000","last_peered":"2024-03-12T03:54:03.556085+0000","last_clean":"2024-03-12T03:48:53.477110+0000","last_became_active":"2024-03-12T03:48:58.454467+0000","last_became_peered":"2024-03-12T03:48:58.454467+0000","last_unstale":"2024-03-12T03:54:03.556085+0000","last_undegraded":"2024-03-12T03:48:58.439427+0000","last_fullsized":"2024-03-12T03:48:58.439320+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:48:00.458460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.458460+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":514,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.544898+0000","last_change":"2024-03-12T03:48:58.461994+0000","last_active":"2024-03-12T03:54:03.544898+0000","last_peered":"2024-03-12T03:54:03.544898+0000","last_clean":"2024-03-12T03:39:31.213817+0000","last_became_active":"2024-03-12T03:48:58.461994+0000","last_became_peered":"2024-03-12T03:48:58.461994+0000","last_unstale":"2024-03-12T03:54:03.544898+0000","last_undegraded":"2024-03-12T03:54:03.544898+0000","last_fullsized":"2024-03-12T03:48:58.446456+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:09.160277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:09.160277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697450+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":921,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.553998+0000","last_change":"2024-03-12T03:50:53.919752+0000","last_active":"2024-03-12T03:54:03.553998+0000","last_peered":"2024-03-12T03:54:03.553998+0000","last_clean":"2024-03-12T03:54:03.553998+0000","last_became_active":"2024-03-12T03:33:27.184171+0000","last_became_peered":"2024-03-12T03:33:27.184171+0000","last_unstale":"2024-03-12T03:54:03.553998+0000","last_undegraded":"2024-03-12T03:54:03.553998+0000","last_fullsized":"2024-03-12T03:54:03.553998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:53.919661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.919661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:12.589430+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"493'7024","reported_seq":12873,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:07.230065+0000","last_change":"2024-03-12T03:53:14.276944+0000","last_active":"2024-03-12T03:54:07.230065+0000","last_peered":"2024-03-12T03:54:07.230065+0000","last_clean":"2024-03-12T03:54:07.230065+0000","last_became_active":"2024-03-12T03:53:14.271050+0000","last_became_peered":"2024-03-12T03:53:14.271050+0000","last_unstale":"2024-03-12T03:54:07.230065+0000","last_undegraded":"2024-03-12T03:54:07.230065+0000","last_fullsized":"2024-03-12T03:54:07.230065+0000","mapping_epoch":471,"log_start":"443'69","ondisk_log_start":"443'69","created":452,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":6955,"log_dups_size":0,"ondisk_log_size":6955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:11.457914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121602048,"num_objects":1856,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5186,"num_write_kb":225312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1906,"num_bytes_recovered":124911616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"480'37","reported_seq":735,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.545222+0000","last_change":"2024-03-12T03:48:58.462533+0000","last_active":"2024-03-12T03:54:03.545222+0000","last_peered":"2024-03-12T03:54:03.545222+0000","last_clean":"2024-03-12T03:43:13.333148+0000","last_became_active":"2024-03-12T03:48:58.462533+0000","last_became_peered":"2024-03-12T03:48:58.462533+0000","last_unstale":"2024-03-12T03:54:03.545222+0000","last_undegraded":"2024-03-12T03:48:58.446989+0000","last_fullsized":"2024-03-12T03:48:58.446871+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:15.795515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.795515+0000","objects_scrubbed":6,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"351'2","reported_seq":937,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.552712+0000","last_change":"2024-03-12T03:49:14.611362+0000","last_active":"2024-03-12T03:54:03.552712+0000","last_peered":"2024-03-12T03:54:03.552712+0000","last_clean":"2024-03-12T03:54:03.552712+0000","last_became_active":"2024-03-12T03:46:25.283344+0000","last_became_peered":"2024-03-12T03:46:25.283344+0000","last_unstale":"2024-03-12T03:54:03.552712+0000","last_undegraded":"2024-03-12T03:54:03.552712+0000","last_fullsized":"2024-03-12T03:54:03.552712+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'2","last_scrub_stamp":"2024-03-12T03:49:14.611213+0000","last_deep_scrub":"351'2","last_deep_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_clean_scrub_stamp":"2024-03-12T03:49:14.611213+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692651+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"441'10","reported_seq":174,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.544885+0000","last_change":"2024-03-12T03:51:42.834336+0000","last_active":"2024-03-12T03:54:03.544885+0000","last_peered":"2024-03-12T03:54:03.544885+0000","last_clean":"2024-03-12T03:54:03.544885+0000","last_became_active":"2024-03-12T03:51:42.833920+0000","last_became_peered":"2024-03-12T03:51:42.833920+0000","last_unstale":"2024-03-12T03:54:03.544885+0000","last_undegraded":"2024-03-12T03:54:03.544885+0000","last_fullsized":"2024-03-12T03:54:03.544885+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'9","last_scrub_stamp":"2024-03-12T03:50:46.889538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:46.889538+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":943,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.545472+0000","last_change":"2024-03-12T03:51:28.318948+0000","last_active":"2024-03-12T03:54:03.545472+0000","last_peered":"2024-03-12T03:54:03.545472+0000","last_clean":"2024-03-12T03:54:03.545472+0000","last_became_active":"2024-03-12T03:39:33.266406+0000","last_became_peered":"2024-03-12T03:39:33.266406+0000","last_unstale":"2024-03-12T03:54:03.545472+0000","last_undegraded":"2024-03-12T03:54:03.545472+0000","last_fullsized":"2024-03-12T03:54:03.545472+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:51:28.318860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:28.318860+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.12","version":"493'6975","reported_seq":10906,"reported_epoch":493,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:06.557501+0000","last_change":"2024-03-12T03:52:48.544295+0000","last_active":"2024-03-12T03:54:06.557501+0000","last_peered":"2024-03-12T03:54:06.557501+0000","last_clean":"2024-03-12T03:54:06.557501+0000","last_became_active":"2024-03-12T03:52:48.542027+0000","last_became_peered":"2024-03-12T03:52:48.542027+0000","last_unstale":"2024-03-12T03:54:06.557501+0000","last_undegraded":"2024-03-12T03:54:06.557501+0000","last_fullsized":"2024-03-12T03:54:06.557501+0000","mapping_epoch":462,"log_start":"446'2161","ondisk_log_start":"446'2161","created":452,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.673840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123240448,"num_objects":1881,"num_object_clones":0,"num_object_copies":5643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1881,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5097,"num_write_kb":223264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1898,"num_bytes_recovered":124387328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,3],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":1881}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"480'30","reported_seq":771,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554208+0000","last_change":"2024-03-12T03:50:05.585005+0000","last_active":"2024-03-12T03:54:03.554208+0000","last_peered":"2024-03-12T03:54:03.554208+0000","last_clean":"2024-03-12T03:54:03.554208+0000","last_became_active":"2024-03-12T03:46:25.283193+0000","last_became_peered":"2024-03-12T03:46:25.283193+0000","last_unstale":"2024-03-12T03:54:03.554208+0000","last_undegraded":"2024-03-12T03:54:03.554208+0000","last_fullsized":"2024-03-12T03:54:03.554208+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:50:05.584740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:50:05.584740+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229599+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":946,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555564+0000","last_change":"2024-03-12T03:50:49.933782+0000","last_active":"2024-03-12T03:54:03.555564+0000","last_peered":"2024-03-12T03:54:03.555564+0000","last_clean":"2024-03-12T03:54:03.555564+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:54:03.555564+0000","last_undegraded":"2024-03-12T03:54:03.555564+0000","last_fullsized":"2024-03-12T03:54:03.555564+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:49.933653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:49.933653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":704,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.556594+0000","last_change":"2024-03-12T03:51:42.841557+0000","last_active":"2024-03-12T03:54:03.556594+0000","last_peered":"2024-03-12T03:54:03.556594+0000","last_clean":"2024-03-12T03:54:03.556594+0000","last_became_active":"2024-03-12T03:51:42.841245+0000","last_became_peered":"2024-03-12T03:51:42.841245+0000","last_unstale":"2024-03-12T03:54:03.556594+0000","last_undegraded":"2024-03-12T03:54:03.556594+0000","last_fullsized":"2024-03-12T03:54:03.556594+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:34.600111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:34.600111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.231099+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":739,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555587+0000","last_change":"2024-03-12T03:51:28.687582+0000","last_active":"2024-03-12T03:54:03.555587+0000","last_peered":"2024-03-12T03:54:03.555587+0000","last_clean":"2024-03-12T03:54:03.555587+0000","last_became_active":"2024-03-12T03:46:25.296764+0000","last_became_peered":"2024-03-12T03:46:25.296764+0000","last_unstale":"2024-03-12T03:54:03.555587+0000","last_undegraded":"2024-03-12T03:54:03.555587+0000","last_fullsized":"2024-03-12T03:54:03.555587+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:28.687482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:28.687482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.230242+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"493'6946","reported_seq":12755,"reported_epoch":493,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:06.723927+0000","last_change":"2024-03-12T03:53:44.552856+0000","last_active":"2024-03-12T03:54:06.723927+0000","last_peered":"2024-03-12T03:54:06.723927+0000","last_clean":"2024-03-12T03:54:06.723927+0000","last_became_active":"2024-03-12T03:53:44.545413+0000","last_became_peered":"2024-03-12T03:53:44.545413+0000","last_unstale":"2024-03-12T03:54:06.723927+0000","last_undegraded":"2024-03-12T03:54:06.723927+0000","last_fullsized":"2024-03-12T03:54:06.723927+0000","mapping_epoch":484,"log_start":"444'990","ondisk_log_start":"444'990","created":452,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":5956,"log_dups_size":0,"ondisk_log_size":5956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.144640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121569280,"num_objects":1855,"num_object_clones":0,"num_object_copies":5565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1855,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5111,"num_write_kb":222912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1890,"num_bytes_recovered":123863040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[{"shards":"1,4,5","objects":1855}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"480'39","reported_seq":730,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.550886+0000","last_change":"2024-03-12T03:51:38.702128+0000","last_active":"2024-03-12T03:54:03.550886+0000","last_peered":"2024-03-12T03:54:03.550886+0000","last_clean":"2024-03-12T03:54:03.550886+0000","last_became_active":"2024-03-12T03:46:25.301700+0000","last_became_peered":"2024-03-12T03:46:25.301700+0000","last_unstale":"2024-03-12T03:54:03.550886+0000","last_undegraded":"2024-03-12T03:54:03.550886+0000","last_fullsized":"2024-03-12T03:54:03.550886+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:51:38.702022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:38.702022+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.332968+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":115,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.543862+0000","last_change":"2024-03-12T03:51:42.833539+0000","last_active":"2024-03-12T03:54:03.543862+0000","last_peered":"2024-03-12T03:54:03.543862+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:51:42.833539+0000","last_became_peered":"2024-03-12T03:51:42.833539+0000","last_unstale":"2024-03-12T03:54:03.543862+0000","last_undegraded":"2024-03-12T03:54:03.543862+0000","last_fullsized":"2024-03-12T03:51:42.829920+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.797296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.797296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":911,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.554071+0000","last_change":"2024-03-12T03:48:58.458438+0000","last_active":"2024-03-12T03:54:03.554071+0000","last_peered":"2024-03-12T03:54:03.554071+0000","last_clean":"2024-03-12T03:48:53.448127+0000","last_became_active":"2024-03-12T03:48:58.458438+0000","last_became_peered":"2024-03-12T03:48:58.458438+0000","last_unstale":"2024-03-12T03:54:03.554071+0000","last_undegraded":"2024-03-12T03:54:03.554071+0000","last_fullsized":"2024-03-12T03:48:58.440067+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.016030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.016030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692492+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":916,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555719+0000","last_change":"2024-03-12T03:50:51.952305+0000","last_active":"2024-03-12T03:54:03.555719+0000","last_peered":"2024-03-12T03:54:03.555719+0000","last_clean":"2024-03-12T03:54:03.555719+0000","last_became_active":"2024-03-12T03:33:27.380605+0000","last_became_peered":"2024-03-12T03:33:27.380605+0000","last_unstale":"2024-03-12T03:54:03.555719+0000","last_undegraded":"2024-03-12T03:54:03.555719+0000","last_fullsized":"2024-03-12T03:54:03.555719+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:51.952189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:51.952189+0000","last_clean_scrub_stamp":"2024-03-12T03:50:51.952189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.14","version":"493'6960","reported_seq":12736,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:06.405910+0000","last_change":"2024-03-12T03:53:12.252674+0000","last_active":"2024-03-12T03:54:06.405910+0000","last_peered":"2024-03-12T03:54:06.405910+0000","last_clean":"2024-03-12T03:54:06.405910+0000","last_became_active":"2024-03-12T03:53:12.247588+0000","last_became_peered":"2024-03-12T03:53:12.247588+0000","last_unstale":"2024-03-12T03:54:06.405910+0000","last_undegraded":"2024-03-12T03:54:06.405910+0000","last_fullsized":"2024-03-12T03:54:06.405910+0000","mapping_epoch":469,"log_start":"446'2132","ondisk_log_start":"446'2132","created":452,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4828,"log_dups_size":0,"ondisk_log_size":4828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:08.576371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128253952,"num_objects":1957,"num_object_clones":0,"num_object_copies":5871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":225984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1879,"num_bytes_recovered":123142144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"480'18","reported_seq":764,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554071+0000","last_change":"2024-03-12T03:51:42.838878+0000","last_active":"2024-03-12T03:54:03.554071+0000","last_peered":"2024-03-12T03:54:03.554071+0000","last_clean":"2024-03-12T03:54:03.554071+0000","last_became_active":"2024-03-12T03:51:42.838579+0000","last_became_peered":"2024-03-12T03:51:42.838579+0000","last_unstale":"2024-03-12T03:54:03.554071+0000","last_undegraded":"2024-03-12T03:54:03.554071+0000","last_fullsized":"2024-03-12T03:54:03.554071+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'16","last_scrub_stamp":"2024-03-12T03:49:51.806651+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:49:51.806651+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229568+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":916,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.548792+0000","last_change":"2024-03-12T03:50:02.319955+0000","last_active":"2024-03-12T03:54:03.548792+0000","last_peered":"2024-03-12T03:54:03.548792+0000","last_clean":"2024-03-12T03:54:03.548792+0000","last_became_active":"2024-03-12T03:33:21.098481+0000","last_became_peered":"2024-03-12T03:33:21.098481+0000","last_unstale":"2024-03-12T03:54:03.548792+0000","last_undegraded":"2024-03-12T03:54:03.548792+0000","last_fullsized":"2024-03-12T03:54:03.548792+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:02.319837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:50:02.319837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":918,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.544131+0000","last_change":"2024-03-12T03:51:01.307708+0000","last_active":"2024-03-12T03:54:03.544131+0000","last_peered":"2024-03-12T03:54:03.544131+0000","last_clean":"2024-03-12T03:54:03.544131+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:54:03.544131+0000","last_undegraded":"2024-03-12T03:54:03.544131+0000","last_fullsized":"2024-03-12T03:54:03.544131+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:01.307529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:51:01.307529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.697648+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":922,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.547004+0000","last_change":"2024-03-12T03:51:46.135297+0000","last_active":"2024-03-12T03:54:03.547004+0000","last_peered":"2024-03-12T03:54:03.547004+0000","last_clean":"2024-03-12T03:54:03.547004+0000","last_became_active":"2024-03-12T03:46:25.293777+0000","last_became_peered":"2024-03-12T03:46:25.293777+0000","last_unstale":"2024-03-12T03:54:03.547004+0000","last_undegraded":"2024-03-12T03:54:03.547004+0000","last_fullsized":"2024-03-12T03:54:03.547004+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:46.135156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:46.135156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.15","version":"493'7287","reported_seq":15122,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:07.205581+0000","last_change":"2024-03-12T03:53:45.553650+0000","last_active":"2024-03-12T03:54:07.205581+0000","last_peered":"2024-03-12T03:54:07.205581+0000","last_clean":"2024-03-12T03:54:07.205581+0000","last_became_active":"2024-03-12T03:53:45.543253+0000","last_became_peered":"2024-03-12T03:53:45.543253+0000","last_unstale":"2024-03-12T03:54:07.205581+0000","last_undegraded":"2024-03-12T03:54:07.205581+0000","last_fullsized":"2024-03-12T03:54:07.205581+0000","mapping_epoch":485,"log_start":"443'483","ondisk_log_start":"443'483","created":452,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":6804,"log_dups_size":0,"ondisk_log_size":6804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:11.458001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122716160,"num_objects":1873,"num_object_clones":0,"num_object_copies":5619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5377,"num_write_kb":231968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1924,"num_bytes_recovered":126091264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"480'39","reported_seq":800,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.559287+0000","last_change":"2024-03-12T03:51:42.838076+0000","last_active":"2024-03-12T03:54:03.559287+0000","last_peered":"2024-03-12T03:54:03.559287+0000","last_clean":"2024-03-12T03:47:02.086867+0000","last_became_active":"2024-03-12T03:51:42.838076+0000","last_became_peered":"2024-03-12T03:51:42.838076+0000","last_unstale":"2024-03-12T03:54:03.559287+0000","last_undegraded":"2024-03-12T03:51:42.833048+0000","last_fullsized":"2024-03-12T03:51:42.832909+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'33","last_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_deep_scrub":"345'33","last_deep_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_clean_scrub_stamp":"2024-03-12T03:47:02.110517+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229419+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":924,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.548602+0000","last_change":"2024-03-12T03:50:41.172421+0000","last_active":"2024-03-12T03:54:03.548602+0000","last_peered":"2024-03-12T03:54:03.548602+0000","last_clean":"2024-03-12T03:54:03.548602+0000","last_became_active":"2024-03-12T03:33:21.102339+0000","last_became_peered":"2024-03-12T03:33:21.102339+0000","last_unstale":"2024-03-12T03:54:03.548602+0000","last_undegraded":"2024-03-12T03:54:03.548602+0000","last_fullsized":"2024-03-12T03:54:03.548602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:41.172313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:50:41.172313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":903,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.550241+0000","last_change":"2024-03-12T03:48:58.453634+0000","last_active":"2024-03-12T03:54:03.550241+0000","last_peered":"2024-03-12T03:54:03.550241+0000","last_clean":"2024-03-12T03:48:53.472261+0000","last_became_active":"2024-03-12T03:48:58.453634+0000","last_became_peered":"2024-03-12T03:48:58.453634+0000","last_unstale":"2024-03-12T03:54:03.550241+0000","last_undegraded":"2024-03-12T03:54:03.550241+0000","last_fullsized":"2024-03-12T03:48:58.444137+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:02.618599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_clean_scrub_stamp":"2024-03-12T03:46:02.618599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":227,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.552598+0000","last_change":"2024-03-12T03:48:58.463158+0000","last_active":"2024-03-12T03:54:03.552598+0000","last_peered":"2024-03-12T03:54:03.552598+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.463158+0000","last_became_peered":"2024-03-12T03:48:58.463158+0000","last_unstale":"2024-03-12T03:54:03.552598+0000","last_undegraded":"2024-03-12T03:54:03.552598+0000","last_fullsized":"2024-03-12T03:48:58.439250+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_clean_scrub_stamp":"2024-03-12T03:45:58.095196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.16","version":"493'6884","reported_seq":14475,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:06.760500+0000","last_change":"2024-03-12T03:53:54.138955+0000","last_active":"2024-03-12T03:54:06.760500+0000","last_peered":"2024-03-12T03:54:06.760500+0000","last_clean":"2024-03-12T03:54:06.760500+0000","last_became_active":"2024-03-12T03:53:54.086265+0000","last_became_peered":"2024-03-12T03:53:54.086265+0000","last_unstale":"2024-03-12T03:54:06.760500+0000","last_undegraded":"2024-03-12T03:54:06.760500+0000","last_fullsized":"2024-03-12T03:54:06.760500+0000","mapping_epoch":490,"log_start":"443'769","ondisk_log_start":"443'769","created":452,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":6115,"log_dups_size":0,"ondisk_log_size":6115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:10.164726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119799808,"num_objects":1828,"num_object_clones":0,"num_object_copies":5484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5046,"num_write_kb":219968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1863,"num_bytes_recovered":122028032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"480'74","reported_seq":894,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:03.559979+0000","last_change":"2024-03-12T03:48:58.457134+0000","last_active":"2024-03-12T03:54:03.559979+0000","last_peered":"2024-03-12T03:54:03.559979+0000","last_clean":"2024-03-12T03:48:53.477142+0000","last_became_active":"2024-03-12T03:48:58.457134+0000","last_became_peered":"2024-03-12T03:48:58.457134+0000","last_unstale":"2024-03-12T03:54:03.559979+0000","last_undegraded":"2024-03-12T03:48:58.441299+0000","last_fullsized":"2024-03-12T03:48:58.441188+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229378+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":97,"num_read_kb":84,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":756,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.563948+0000","last_change":"2024-03-12T03:50:37.469343+0000","last_active":"2024-03-12T03:54:03.563948+0000","last_peered":"2024-03-12T03:54:03.563948+0000","last_clean":"2024-03-12T03:54:03.563948+0000","last_became_active":"2024-03-12T03:46:25.279131+0000","last_became_peered":"2024-03-12T03:46:25.279131+0000","last_unstale":"2024-03-12T03:54:03.563948+0000","last_undegraded":"2024-03-12T03:54:03.563948+0000","last_fullsized":"2024-03-12T03:54:03.563948+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:37.469265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:50:37.469265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229846+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":117,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.558809+0000","last_change":"2024-03-12T03:51:42.832852+0000","last_active":"2024-03-12T03:54:03.558809+0000","last_peered":"2024-03-12T03:54:03.558809+0000","last_clean":"2024-03-12T03:54:03.558809+0000","last_became_active":"2024-03-12T03:51:42.832427+0000","last_became_peered":"2024-03-12T03:51:42.832427+0000","last_unstale":"2024-03-12T03:54:03.558809+0000","last_undegraded":"2024-03-12T03:54:03.558809+0000","last_fullsized":"2024-03-12T03:54:03.558809+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:39.866768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:50:39.866768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:12.589455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":707,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.555413+0000","last_change":"2024-03-12T03:49:55.557456+0000","last_active":"2024-03-12T03:54:03.555413+0000","last_peered":"2024-03-12T03:54:03.555413+0000","last_clean":"2024-03-12T03:54:03.555413+0000","last_became_active":"2024-03-12T03:46:25.281081+0000","last_became_peered":"2024-03-12T03:46:25.281081+0000","last_unstale":"2024-03-12T03:54:03.555413+0000","last_undegraded":"2024-03-12T03:54:03.555413+0000","last_fullsized":"2024-03-12T03:54:03.555413+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:49:55.557335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:49:55.557335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.229640+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"493'6913","reported_seq":12700,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:06.579108+0000","last_change":"2024-03-12T03:53:27.578683+0000","last_active":"2024-03-12T03:54:06.579108+0000","last_peered":"2024-03-12T03:54:06.579108+0000","last_clean":"2024-03-12T03:54:06.579108+0000","last_became_active":"2024-03-12T03:53:27.573027+0000","last_became_peered":"2024-03-12T03:53:27.573027+0000","last_unstale":"2024-03-12T03:54:06.579108+0000","last_undegraded":"2024-03-12T03:54:06.579108+0000","last_fullsized":"2024-03-12T03:54:06.579108+0000","mapping_epoch":478,"log_start":"446'2085","ondisk_log_start":"446'2085","created":452,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.806188+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.806188+0000","objects_scrubbed":0,"log_size":4828,"log_dups_size":0,"ondisk_log_size":4828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:07.673862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120487936,"num_objects":1839,"num_object_clones":0,"num_object_copies":5517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5057,"num_write_kb":220640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1882,"num_bytes_recovered":123338752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"480'32","reported_seq":973,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.554766+0000","last_change":"2024-03-12T03:50:57.888885+0000","last_active":"2024-03-12T03:54:03.554766+0000","last_peered":"2024-03-12T03:54:03.554766+0000","last_clean":"2024-03-12T03:54:03.554766+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:54:03.554766+0000","last_undegraded":"2024-03-12T03:54:03.554766+0000","last_fullsized":"2024-03-12T03:54:03.554766+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:50:57.888633+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:50:57.888633+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":521,"reported_epoch":493,"state":"active+undersized","last_fresh":"2024-03-12T03:54:03.548985+0000","last_change":"2024-03-12T03:48:58.462711+0000","last_active":"2024-03-12T03:54:03.548985+0000","last_peered":"2024-03-12T03:54:03.548985+0000","last_clean":"2024-03-12T03:39:28.536135+0000","last_became_active":"2024-03-12T03:48:58.462711+0000","last_became_peered":"2024-03-12T03:48:58.462711+0000","last_unstale":"2024-03-12T03:54:03.548985+0000","last_undegraded":"2024-03-12T03:54:03.548985+0000","last_fullsized":"2024-03-12T03:48:58.443314+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:12.186146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:48:12.186146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":925,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.552817+0000","last_change":"2024-03-12T03:50:04.535064+0000","last_active":"2024-03-12T03:54:03.552817+0000","last_peered":"2024-03-12T03:54:03.552817+0000","last_clean":"2024-03-12T03:54:03.552817+0000","last_became_active":"2024-03-12T03:33:25.179267+0000","last_became_peered":"2024-03-12T03:33:25.179267+0000","last_unstale":"2024-03-12T03:54:03.552817+0000","last_undegraded":"2024-03-12T03:54:03.552817+0000","last_fullsized":"2024-03-12T03:54:03.552817+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_clean_scrub_stamp":"2024-03-12T03:50:04.534937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:05.692788+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":117,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-03-12T03:54:03.547488+0000","last_change":"2024-03-12T03:51:42.833194+0000","last_active":"2024-03-12T03:54:03.547488+0000","last_peered":"2024-03-12T03:54:03.547488+0000","last_clean":"2024-03-12T03:54:03.547488+0000","last_became_active":"2024-03-12T03:51:42.832894+0000","last_became_peered":"2024-03-12T03:51:42.832894+0000","last_unstale":"2024-03-12T03:54:03.547488+0000","last_undegraded":"2024-03-12T03:54:03.547488+0000","last_fullsized":"2024-03-12T03:54:03.547488+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:32.662924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_clean_scrub_stamp":"2024-03-12T03:51:32.662924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":26,"stat_sum":{"num_bytes":3813736472,"num_objects":58195,"num_object_clones":0,"num_object_copies":174585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10576,"num_objects_misplaced":7372,"num_objects_unfound":0,"num_objects_dirty":58195,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154306,"num_write_kb":6434113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19013,"num_bytes_recovered":1245970432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11782991872,"data_stored":11782979656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136150,"ondisk_log_size":136150,"up":69,"acting":73,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1236,"num_read_kb":1231,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":19,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":933560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2177,"ondisk_log_size":2177,"up":49,"acting":53,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1822,"num_read_kb":1547,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":2642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},"log_size":1251,"ondisk_log_size":1251,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":83,"acting":83,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":87,"acting":87,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070718,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2735920,"kb_used_data":1766248,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":91009232,"statfs":{"total":95995035648,"available":93193453568,"internally_reserved":0,"allocated":1808637952,"data_stored":1807878275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":992934346},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332355,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2764700,"kb_used_data":1931284,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":90980452,"statfs":{"total":95995035648,"available":93163982848,"internally_reserved":0,"allocated":1977634816,"data_stored":1976893680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":849344966},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575132,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2878868,"kb_used_data":1846604,"kb_used_omap":1,"kb_used_meta":1032126,"kb_avail":90866284,"statfs":{"total":95995035648,"available":93047074816,"internally_reserved":0,"allocated":1890922496,"data_stored":1890126090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1056897477},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738595,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3725208,"kb_used_data":3102608,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":90019944,"statfs":{"total":95995035648,"available":92180422656,"internally_reserved":0,"allocated":3177070592,"data_stored":3175997927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411606,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2948636,"kb_used_data":2010836,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":90796516,"statfs":{"total":95995035648,"available":92975632384,"internally_reserved":0,"allocated":2059096064,"data_stored":2058356485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":330,"seq":1417339207774,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2117636,"kb_used_data":1626300,"kb_used_omap":13,"kb_used_meta":491314,"kb_avail":91627516,"statfs":{"total":95995035648,"available":93826576384,"internally_reserved":0,"allocated":1665331200,"data_stored":1664550951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13485,"internal_metadata":503106387},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953742,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2148792,"kb_used_data":1399792,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":91596360,"statfs":{"total":95995035648,"available":93794672640,"internally_reserved":0,"allocated":1433387008,"data_stored":1432618703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659400,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2124320,"kb_used_data":1186392,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":91620832,"statfs":{"total":95995035648,"available":93819731968,"internally_reserved":0,"allocated":1214865408,"data_stored":1214121119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1843527680,"data_stored":1843527680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2014580736,"data_stored":2014576664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1630601216,"data_stored":1630601216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1404768256,"data_stored":1404764184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1192820736,"data_stored":1192820736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1934036992,"data_stored":1934032920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1762656256,"data_stored":1762656256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:54:15.683 INFO:tasks.thrashosds.thrasher:Setting 3.14 to [3, 6, 1] 2024-03-12T03:54:15.683 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.14', '3', '6', '1'] 2024-03-12T03:54:15.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pg-upmap 3.14 3 6 1 2024-03-12T03:54:15.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:15.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:15.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:15.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:15.856 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:15.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[29344]: 11.12 scrub ok 2024-03-12T03:54:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[29344]: pgmap v1022: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 3.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 9375/154206 objects degraded (6.080%); 6493/154206 objects misplaced (4.211%) 2024-03-12T03:54:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[29344]: 11.13 scrub ok 2024-03-12T03:54:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[29344]: 11.15 scrub starts 2024-03-12T03:54:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[29344]: 11.14 scrub starts 2024-03-12T03:54:15.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[29344]: 11.17 scrub starts 2024-03-12T03:54:15.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:54:15.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:54:15.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[29344]: 11.14 scrub ok 2024-03-12T03:54:15.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[29344]: 4.1 scrub starts 2024-03-12T03:54:15.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[29344]: 4.1 scrub ok 2024-03-12T03:54:15.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[39949]: 11.12 scrub ok 2024-03-12T03:54:15.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[39949]: pgmap v1022: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 3.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 9375/154206 objects degraded (6.080%); 6493/154206 objects misplaced (4.211%) 2024-03-12T03:54:15.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[39949]: 11.13 scrub ok 2024-03-12T03:54:15.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[39949]: 11.15 scrub starts 2024-03-12T03:54:15.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[39949]: 11.14 scrub starts 2024-03-12T03:54:15.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[39949]: 11.17 scrub starts 2024-03-12T03:54:15.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:54:15.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:54:15.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[39949]: 11.14 scrub ok 2024-03-12T03:54:15.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[39949]: 4.1 scrub starts 2024-03-12T03:54:15.927 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:15 smithi169 ceph-mon[39949]: 4.1 scrub ok 2024-03-12T03:54:15.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:15 smithi173 ceph-mon[33544]: 11.12 scrub ok 2024-03-12T03:54:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:15 smithi173 ceph-mon[33544]: pgmap v1022: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 3.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 9375/154206 objects degraded (6.080%); 6493/154206 objects misplaced (4.211%) 2024-03-12T03:54:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:15 smithi173 ceph-mon[33544]: 11.13 scrub ok 2024-03-12T03:54:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:15 smithi173 ceph-mon[33544]: 11.15 scrub starts 2024-03-12T03:54:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:15 smithi173 ceph-mon[33544]: 11.14 scrub starts 2024-03-12T03:54:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:15 smithi173 ceph-mon[33544]: 11.17 scrub starts 2024-03-12T03:54:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:15 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:54:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:15 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:54:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:15 smithi173 ceph-mon[33544]: 11.14 scrub ok 2024-03-12T03:54:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:15 smithi173 ceph-mon[33544]: 4.1 scrub starts 2024-03-12T03:54:15.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:15 smithi173 ceph-mon[33544]: 4.1 scrub ok 2024-03-12T03:54:15.988 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:15.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:16.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:16.156 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:16.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:16.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:16.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:16.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:16.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:16.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:16.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:16.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:16.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:16.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:16.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:16.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:16.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:16.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:16.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:16.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:16.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:16.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:16.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:16 smithi173 ceph-mon[33544]: 11.17 scrub ok 2024-03-12T03:54:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:16 smithi173 ceph-mon[33544]: 11.15 scrub ok 2024-03-12T03:54:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:16 smithi173 ceph-mon[33544]: 6.17 scrub starts 2024-03-12T03:54:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:16 smithi173 ceph-mon[33544]: 4.e scrub starts 2024-03-12T03:54:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:16 smithi173 ceph-mon[33544]: 4.e scrub ok 2024-03-12T03:54:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:16 smithi173 ceph-mon[33544]: 11.19 scrub starts 2024-03-12T03:54:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:16 smithi173 ceph-mon[33544]: osdmap e496: 8 total, 7 up, 8 in 2024-03-12T03:54:16.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:16 smithi173 ceph-mon[33544]: 6.17 scrub ok 2024-03-12T03:54:16.994 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:16.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:17.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:17.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:17.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:17.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[29344]: 11.17 scrub ok 2024-03-12T03:54:17.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[29344]: 11.15 scrub ok 2024-03-12T03:54:17.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[29344]: 6.17 scrub starts 2024-03-12T03:54:17.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[29344]: 4.e scrub starts 2024-03-12T03:54:17.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[29344]: 4.e scrub ok 2024-03-12T03:54:17.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[29344]: 11.19 scrub starts 2024-03-12T03:54:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[29344]: osdmap e496: 8 total, 7 up, 8 in 2024-03-12T03:54:17.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[29344]: 6.17 scrub ok 2024-03-12T03:54:17.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[39949]: 11.17 scrub ok 2024-03-12T03:54:17.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[39949]: 11.15 scrub ok 2024-03-12T03:54:17.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[39949]: 6.17 scrub starts 2024-03-12T03:54:17.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[39949]: 4.e scrub starts 2024-03-12T03:54:17.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[39949]: 4.e scrub ok 2024-03-12T03:54:17.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[39949]: 11.19 scrub starts 2024-03-12T03:54:17.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[39949]: osdmap e496: 8 total, 7 up, 8 in 2024-03-12T03:54:17.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:16 smithi169 ceph-mon[39949]: 6.17 scrub ok 2024-03-12T03:54:17.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:17.261 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:17.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:17.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:17.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:17.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:17.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:17.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:17.530 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:17.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:17.632 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:17.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:17.674 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:17.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:17.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:17.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:17.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:17.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:17.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:17.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:17.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:17 smithi173 ceph-mon[33544]: 11.19 scrub ok 2024-03-12T03:54:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:17 smithi173 ceph-mon[33544]: pgmap v1024: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 3.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.17k op/s; 9375/154206 objects degraded (6.080%); 6493/154206 objects misplaced (4.211%) 2024-03-12T03:54:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:17 smithi173 ceph-mon[33544]: 11.18 scrub starts 2024-03-12T03:54:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:17 smithi173 ceph-mon[33544]: 1.5 scrub starts 2024-03-12T03:54:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:17 smithi173 ceph-mon[33544]: 1.5 scrub ok 2024-03-12T03:54:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:17 smithi173 ceph-mon[33544]: 11.18 scrub ok 2024-03-12T03:54:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:17 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:17 smithi173 ceph-mon[33544]: 11.16 scrub starts 2024-03-12T03:54:18.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:18.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:18.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:18.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[29344]: 11.19 scrub ok 2024-03-12T03:54:18.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[29344]: pgmap v1024: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 3.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.17k op/s; 9375/154206 objects degraded (6.080%); 6493/154206 objects misplaced (4.211%) 2024-03-12T03:54:18.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[29344]: 11.18 scrub starts 2024-03-12T03:54:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[29344]: 1.5 scrub starts 2024-03-12T03:54:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[29344]: 1.5 scrub ok 2024-03-12T03:54:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[29344]: 11.18 scrub ok 2024-03-12T03:54:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[29344]: 11.16 scrub starts 2024-03-12T03:54:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[39949]: 11.19 scrub ok 2024-03-12T03:54:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[39949]: pgmap v1024: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 3.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.17k op/s; 9375/154206 objects degraded (6.080%); 6493/154206 objects misplaced (4.211%) 2024-03-12T03:54:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[39949]: 11.18 scrub starts 2024-03-12T03:54:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[39949]: 1.5 scrub starts 2024-03-12T03:54:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[39949]: 1.5 scrub ok 2024-03-12T03:54:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[39949]: 11.18 scrub ok 2024-03-12T03:54:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:18.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:17 smithi169 ceph-mon[39949]: 11.16 scrub starts 2024-03-12T03:54:18.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:18.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:18.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:18.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:18.359 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:18.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:18.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:18.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:18.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:18.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:18.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:18.640 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:18.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:18.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:18.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:18.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:18.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:18.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:18.874 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:18.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:18.914 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:18.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:19.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:19.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:19.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:19.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:19.121 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:18 smithi169 ceph-mon[29344]: 11.16 scrub ok 2024-03-12T03:54:19.121 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:18 smithi169 ceph-mon[29344]: 5.9 scrub starts 2024-03-12T03:54:19.121 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:18 smithi169 ceph-mon[29344]: 5.9 scrub ok 2024-03-12T03:54:19.121 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:18 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 9375/154206 objects degraded (6.080%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:19.121 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:18 smithi169 ceph-mon[29344]: osdmap e497: 8 total, 7 up, 8 in 2024-03-12T03:54:19.122 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:18 smithi169 ceph-mon[39949]: 11.16 scrub ok 2024-03-12T03:54:19.122 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:18 smithi169 ceph-mon[39949]: 5.9 scrub starts 2024-03-12T03:54:19.122 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:18 smithi169 ceph-mon[39949]: 5.9 scrub ok 2024-03-12T03:54:19.122 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:18 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 9375/154206 objects degraded (6.080%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:19.122 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:18 smithi169 ceph-mon[39949]: osdmap e497: 8 total, 7 up, 8 in 2024-03-12T03:54:19.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:19.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:19.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:19.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:19.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:18 smithi173 ceph-mon[33544]: 11.16 scrub ok 2024-03-12T03:54:19.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:18 smithi173 ceph-mon[33544]: 5.9 scrub starts 2024-03-12T03:54:19.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:18 smithi173 ceph-mon[33544]: 5.9 scrub ok 2024-03-12T03:54:19.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:18 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 9375/154206 objects degraded (6.080%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:19.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:18 smithi173 ceph-mon[33544]: osdmap e497: 8 total, 7 up, 8 in 2024-03-12T03:54:19.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:19.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:19.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:19.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:19.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:19.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:19.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:19.472 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:54:19.511 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:54:19.550 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:54:19.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:19.590 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:54:19.595 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:19.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:19.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:19.628 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:54:19.670 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:54:19.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:19.713 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:54:19.714 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:54:19.714 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:54:19.714 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:54:19.714 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:54:19.714 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:54:19.715 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:54:19.715 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:54:19.715 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:54:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:19.740 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:19.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:19.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:19.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:19.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:19.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:19.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:20.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:20.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:20.110 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:20.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:20.163 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:20.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:20.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:20.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:19 smithi169 ceph-mon[29344]: pgmap v1025: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 2.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 775 B/s rd, 0 op/s; 8963/139986 objects degraded (6.403%); 5614/139986 objects misplaced (4.010%) 2024-03-12T03:54:20.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:19 smithi169 ceph-mon[29344]: 4.d scrub starts 2024-03-12T03:54:20.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:19 smithi169 ceph-mon[29344]: 4.d scrub ok 2024-03-12T03:54:20.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:19 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2048883786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:54:20.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:19 smithi169 ceph-mon[39949]: pgmap v1025: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 2.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 775 B/s rd, 0 op/s; 8963/139986 objects degraded (6.403%); 5614/139986 objects misplaced (4.010%) 2024-03-12T03:54:20.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:19 smithi169 ceph-mon[39949]: 4.d scrub starts 2024-03-12T03:54:20.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:19 smithi169 ceph-mon[39949]: 4.d scrub ok 2024-03-12T03:54:20.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:19 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2048883786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:54:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:19 smithi173 ceph-mon[33544]: pgmap v1025: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 2.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 775 B/s rd, 0 op/s; 8963/139986 objects degraded (6.403%); 5614/139986 objects misplaced (4.010%) 2024-03-12T03:54:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:19 smithi173 ceph-mon[33544]: 4.d scrub starts 2024-03-12T03:54:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:19 smithi173 ceph-mon[33544]: 4.d scrub ok 2024-03-12T03:54:20.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:19 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2048883786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:54:20.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:20.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:20.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:20.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:20.437 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:20.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:20.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:20.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:20.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:20.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:20.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:20.704 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:54:20.704 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:54:20.724 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:20.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:20.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:20.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:20.861 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:20.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:20.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:20 smithi169 ceph-mon[39949]: 6.4 scrub starts 2024-03-12T03:54:20.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:20 smithi169 ceph-mon[39949]: 6.4 scrub ok 2024-03-12T03:54:20.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:20 smithi169 ceph-mon[39949]: 6.14 scrub starts 2024-03-12T03:54:20.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:20 smithi169 ceph-mon[39949]: 6.14 scrub ok 2024-03-12T03:54:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:20 smithi169 ceph-mon[39949]: 5.1f scrub starts 2024-03-12T03:54:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:20 smithi169 ceph-mon[39949]: 5.1f scrub ok 2024-03-12T03:54:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:20 smithi169 ceph-mon[29344]: 6.4 scrub starts 2024-03-12T03:54:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:20 smithi169 ceph-mon[29344]: 6.4 scrub ok 2024-03-12T03:54:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:20 smithi169 ceph-mon[29344]: 6.14 scrub starts 2024-03-12T03:54:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:20 smithi169 ceph-mon[29344]: 6.14 scrub ok 2024-03-12T03:54:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:20 smithi169 ceph-mon[29344]: 5.1f scrub starts 2024-03-12T03:54:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:20 smithi169 ceph-mon[29344]: 5.1f scrub ok 2024-03-12T03:54:20.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:20.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:20.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:21.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:21.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:21.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:21.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:20 smithi173 ceph-mon[33544]: 6.4 scrub starts 2024-03-12T03:54:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:20 smithi173 ceph-mon[33544]: 6.4 scrub ok 2024-03-12T03:54:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:20 smithi173 ceph-mon[33544]: 6.14 scrub starts 2024-03-12T03:54:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:20 smithi173 ceph-mon[33544]: 6.14 scrub ok 2024-03-12T03:54:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:20 smithi173 ceph-mon[33544]: 5.1f scrub starts 2024-03-12T03:54:21.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:20 smithi173 ceph-mon[33544]: 5.1f scrub ok 2024-03-12T03:54:21.264 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:21.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:21.366 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:21.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:21.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:21.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:21.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:21.530 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:21.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:21.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:21.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:21.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:21.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:21.806 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:21.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:21.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:21.910 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:54:21.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:21.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:21.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:21.938 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:54:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:54:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:54:21.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:21 smithi169 ceph-mon[29344]: pgmap v1027: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 2.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 759 B/s rd, 0 op/s; 8125/133590 objects degraded (6.082%); 5614/133590 objects misplaced (4.202%) 2024-03-12T03:54:21.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:21 smithi169 ceph-mon[29344]: 1.d scrub starts 2024-03-12T03:54:21.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:21 smithi169 ceph-mon[29344]: 1.d scrub ok 2024-03-12T03:54:21.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:21 smithi169 ceph-mon[39949]: pgmap v1027: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 2.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 759 B/s rd, 0 op/s; 8125/133590 objects degraded (6.082%); 5614/133590 objects misplaced (4.202%) 2024-03-12T03:54:21.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:21 smithi169 ceph-mon[39949]: 1.d scrub starts 2024-03-12T03:54:21.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:21 smithi169 ceph-mon[39949]: 1.d scrub ok 2024-03-12T03:54:22.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:22.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:22.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:22.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:22.198 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:22.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:22.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:21 smithi173 ceph-mon[33544]: pgmap v1027: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 2.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 759 B/s rd, 0 op/s; 8125/133590 objects degraded (6.082%); 5614/133590 objects misplaced (4.202%) 2024-03-12T03:54:22.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:21 smithi173 ceph-mon[33544]: 1.d scrub starts 2024-03-12T03:54:22.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:21 smithi173 ceph-mon[33544]: 1.d scrub ok 2024-03-12T03:54:22.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:22.339 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:22.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:22.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:22.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:22.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:22.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:22.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:22.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:22.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:22.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:54:22.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:22.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:22.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:22.844 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:22.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:22 smithi169 ceph-mon[29344]: 4.13 scrub starts 2024-03-12T03:54:22.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:22 smithi169 ceph-mon[29344]: 4.13 scrub ok 2024-03-12T03:54:22.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:22 smithi169 ceph-mon[29344]: 3.11 scrub starts 2024-03-12T03:54:22.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:22 smithi169 ceph-mon[29344]: 3.11 scrub ok 2024-03-12T03:54:22.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:22 smithi169 ceph-mon[29344]: 6.2 scrub starts 2024-03-12T03:54:22.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:22 smithi169 ceph-mon[29344]: 6.2 scrub ok 2024-03-12T03:54:22.853 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:22 smithi169 ceph-mon[39949]: 4.13 scrub starts 2024-03-12T03:54:22.853 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:22 smithi169 ceph-mon[39949]: 4.13 scrub ok 2024-03-12T03:54:22.853 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:22 smithi169 ceph-mon[39949]: 3.11 scrub starts 2024-03-12T03:54:22.854 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:22 smithi169 ceph-mon[39949]: 3.11 scrub ok 2024-03-12T03:54:22.854 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:22 smithi169 ceph-mon[39949]: 6.2 scrub starts 2024-03-12T03:54:22.854 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:22 smithi169 ceph-mon[39949]: 6.2 scrub ok 2024-03-12T03:54:22.904 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:22.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:22.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:23.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:23.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:23.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:23.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:23.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:23.166 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:23.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:23.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:23.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:22 smithi173 ceph-mon[33544]: 4.13 scrub starts 2024-03-12T03:54:23.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:22 smithi173 ceph-mon[33544]: 4.13 scrub ok 2024-03-12T03:54:23.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:22 smithi173 ceph-mon[33544]: 3.11 scrub starts 2024-03-12T03:54:23.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:22 smithi173 ceph-mon[33544]: 3.11 scrub ok 2024-03-12T03:54:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:22 smithi173 ceph-mon[33544]: 6.2 scrub starts 2024-03-12T03:54:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:22 smithi173 ceph-mon[33544]: 6.2 scrub ok 2024-03-12T03:54:23.267 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:23.304 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:23.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:23.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:23.440 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:23.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:23.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:23.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:23.579 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:23.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:23.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:23.718 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:23.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:23.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:23.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:54:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:54:23.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:23.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:23.855 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:23.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:23 smithi169 ceph-mon[29344]: pgmap v1028: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 2.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 7667/118821 objects degraded (6.453%); 4751/118821 objects misplaced (3.998%) 2024-03-12T03:54:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:23 smithi169 ceph-mon[29344]: 5.6 scrub starts 2024-03-12T03:54:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:23 smithi169 ceph-mon[29344]: 5.6 scrub ok 2024-03-12T03:54:23.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:23 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 7667/118821 objects degraded (6.453%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:23 smithi169 ceph-mon[39949]: pgmap v1028: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 2.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 7667/118821 objects degraded (6.453%); 4751/118821 objects misplaced (3.998%) 2024-03-12T03:54:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:23 smithi169 ceph-mon[39949]: 5.6 scrub starts 2024-03-12T03:54:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:23 smithi169 ceph-mon[39949]: 5.6 scrub ok 2024-03-12T03:54:23.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:23 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 7667/118821 objects degraded (6.453%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:23.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:23.995 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:23.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:24.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:24.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:24.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:24.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:23 smithi173 ceph-mon[33544]: pgmap v1028: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 2.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 7667/118821 objects degraded (6.453%); 4751/118821 objects misplaced (3.998%) 2024-03-12T03:54:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:23 smithi173 ceph-mon[33544]: 5.6 scrub starts 2024-03-12T03:54:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:23 smithi173 ceph-mon[33544]: 5.6 scrub ok 2024-03-12T03:54:24.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:23 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 7667/118821 objects degraded (6.453%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:24.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:24.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:24.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:24.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:24.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:24.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:24.546 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:24.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:24.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:24.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:24.674 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:24.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:24.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:24.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:24.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:24.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:24.917 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:24.961 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:24.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:25.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:25.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:25.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:25.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[39949]: 3.19 scrub starts 2024-03-12T03:54:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[39949]: 3.19 scrub ok 2024-03-12T03:54:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[39949]: 3.9 scrub starts 2024-03-12T03:54:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[39949]: 3.9 scrub ok 2024-03-12T03:54:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[39949]: 3.e scrub starts 2024-03-12T03:54:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[39949]: 3.e scrub ok 2024-03-12T03:54:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[39949]: 4.1b scrub starts 2024-03-12T03:54:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[39949]: 4.1b scrub ok 2024-03-12T03:54:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[29344]: 3.19 scrub starts 2024-03-12T03:54:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[29344]: 3.19 scrub ok 2024-03-12T03:54:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[29344]: 3.9 scrub starts 2024-03-12T03:54:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[29344]: 3.9 scrub ok 2024-03-12T03:54:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[29344]: 3.e scrub starts 2024-03-12T03:54:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[29344]: 3.e scrub ok 2024-03-12T03:54:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[29344]: 4.1b scrub starts 2024-03-12T03:54:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:24 smithi169 ceph-mon[29344]: 4.1b scrub ok 2024-03-12T03:54:25.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:25.223 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:25.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:24 smithi173 ceph-mon[33544]: 3.19 scrub starts 2024-03-12T03:54:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:24 smithi173 ceph-mon[33544]: 3.19 scrub ok 2024-03-12T03:54:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:24 smithi173 ceph-mon[33544]: 3.9 scrub starts 2024-03-12T03:54:25.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:24 smithi173 ceph-mon[33544]: 3.9 scrub ok 2024-03-12T03:54:25.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:24 smithi173 ceph-mon[33544]: 3.e scrub starts 2024-03-12T03:54:25.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:24 smithi173 ceph-mon[33544]: 3.e scrub ok 2024-03-12T03:54:25.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:24 smithi173 ceph-mon[33544]: 4.1b scrub starts 2024-03-12T03:54:25.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:24 smithi173 ceph-mon[33544]: 4.1b scrub ok 2024-03-12T03:54:25.241 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-03-12T03:54:25.241 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:54:25.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:25.362 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:25.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:25.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:25.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:25.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:25.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:25.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:25.635 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:25.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:25.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:25.791 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:25.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:25.892 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:25.946 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:25.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:26.034 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:25 smithi169 ceph-mon[29344]: pgmap v1029: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 2.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 6767/111981 objects degraded (6.043%); 4751/111981 objects misplaced (4.243%) 2024-03-12T03:54:26.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:25 smithi169 ceph-mon[39949]: pgmap v1029: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 2.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 6767/111981 objects degraded (6.043%); 4751/111981 objects misplaced (4.243%) 2024-03-12T03:54:26.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:26.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:26.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:26.187 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:26.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:26.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:26.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:25 smithi173 ceph-mon[33544]: pgmap v1029: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 2.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 6767/111981 objects degraded (6.043%); 4751/111981 objects misplaced (4.243%) 2024-03-12T03:54:26.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:26.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:26.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:26.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:26.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:26.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:26.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:26.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-12T03:54:26.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:26.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:26.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:26.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:26.771 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:26.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:26.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:26.913 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:26.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:27.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:27.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:27.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:27.118 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:26 smithi169 ceph-mon[29344]: 3.1a scrub starts 2024-03-12T03:54:27.119 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:26 smithi169 ceph-mon[29344]: 3.1a scrub ok 2024-03-12T03:54:27.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:26 smithi169 ceph-mon[39949]: 3.1a scrub starts 2024-03-12T03:54:27.119 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:26 smithi169 ceph-mon[39949]: 3.1a scrub ok 2024-03-12T03:54:27.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:27.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:27.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:27.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:26 smithi173 ceph-mon[33544]: 3.1a scrub starts 2024-03-12T03:54:27.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:26 smithi173 ceph-mon[33544]: 3.1a scrub ok 2024-03-12T03:54:27.285 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:27.334 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:27.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:27.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:27.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:27.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:27.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:27.521 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:27.531 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:27.537 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:27.562 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:27.633 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:27.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:27.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:27.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:27.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:27.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:27.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:27.871 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:27.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:27.909 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:27.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:27.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[29344]: pgmap v1030: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 2.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 6767/111981 objects degraded (6.043%); 4751/111981 objects misplaced (4.243%) 2024-03-12T03:54:27.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[29344]: 6.e scrub starts 2024-03-12T03:54:27.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:54:27.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:54:27.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[29344]: 5.a scrub starts 2024-03-12T03:54:27.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[29344]: 5.a scrub ok 2024-03-12T03:54:27.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[29344]: 6.e scrub ok 2024-03-12T03:54:27.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[29344]: osdmap e498: 8 total, 7 up, 8 in 2024-03-12T03:54:27.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[29344]: 5.16 scrub starts 2024-03-12T03:54:27.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[29344]: 5.16 scrub ok 2024-03-12T03:54:27.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[39949]: pgmap v1030: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 2.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 6767/111981 objects degraded (6.043%); 4751/111981 objects misplaced (4.243%) 2024-03-12T03:54:27.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[39949]: 6.e scrub starts 2024-03-12T03:54:27.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:54:27.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:54:27.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[39949]: 5.a scrub starts 2024-03-12T03:54:27.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[39949]: 5.a scrub ok 2024-03-12T03:54:27.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[39949]: 6.e scrub ok 2024-03-12T03:54:27.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[39949]: osdmap e498: 8 total, 7 up, 8 in 2024-03-12T03:54:27.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[39949]: 5.16 scrub starts 2024-03-12T03:54:27.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:27 smithi169 ceph-mon[39949]: 5.16 scrub ok 2024-03-12T03:54:28.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:28.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:28.048 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:28.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:28.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:28.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:28.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:28.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:27 smithi173 ceph-mon[33544]: pgmap v1030: 176 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 2.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 6767/111981 objects degraded (6.043%); 4751/111981 objects misplaced (4.243%) 2024-03-12T03:54:28.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:27 smithi173 ceph-mon[33544]: 6.e scrub starts 2024-03-12T03:54:28.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:27 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:54:28.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:27 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:54:28.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:27 smithi173 ceph-mon[33544]: 5.a scrub starts 2024-03-12T03:54:28.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:27 smithi173 ceph-mon[33544]: 5.a scrub ok 2024-03-12T03:54:28.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:27 smithi173 ceph-mon[33544]: 6.e scrub ok 2024-03-12T03:54:28.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:27 smithi173 ceph-mon[33544]: osdmap e498: 8 total, 7 up, 8 in 2024-03-12T03:54:28.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:27 smithi173 ceph-mon[33544]: 5.16 scrub starts 2024-03-12T03:54:28.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:27 smithi173 ceph-mon[33544]: 5.16 scrub ok 2024-03-12T03:54:28.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:28.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:28.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:28.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:28.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:28.467 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:28.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:28.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:28.570 INFO:teuthology.orchestra.run.smithi169.stderr:set 3.14 pg_upmap mapping to [3,6,1] 2024-03-12T03:54:28.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:28.601 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:28.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:28.702 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:28.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:28.742 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:28.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:28.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:28.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:28.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:28.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:29.011 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:29.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:29.095 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[29344]: 6.f scrub starts 2024-03-12T03:54:29.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[29344]: 6.5 scrub starts 2024-03-12T03:54:29.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[29344]: 6.5 scrub ok 2024-03-12T03:54:29.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[29344]: 6.f scrub ok 2024-03-12T03:54:29.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/853741230' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.14", "id": [3, 6, 1]}]: dispatch 2024-03-12T03:54:29.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.14", "id": [3, 6, 1]}]: dispatch 2024-03-12T03:54:29.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[29344]: 1.a scrub starts 2024-03-12T03:54:29.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[29344]: 1.a scrub ok 2024-03-12T03:54:29.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 6767/111981 objects degraded (6.043%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:29.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.14", "id": [3, 6, 1]}]': finished 2024-03-12T03:54:29.096 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[29344]: osdmap e499: 8 total, 7 up, 8 in 2024-03-12T03:54:29.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[39949]: 6.f scrub starts 2024-03-12T03:54:29.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[39949]: 6.5 scrub starts 2024-03-12T03:54:29.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[39949]: 6.5 scrub ok 2024-03-12T03:54:29.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[39949]: 6.f scrub ok 2024-03-12T03:54:29.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/853741230' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.14", "id": [3, 6, 1]}]: dispatch 2024-03-12T03:54:29.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.14", "id": [3, 6, 1]}]: dispatch 2024-03-12T03:54:29.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[39949]: 1.a scrub starts 2024-03-12T03:54:29.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[39949]: 1.a scrub ok 2024-03-12T03:54:29.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 6767/111981 objects degraded (6.043%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:29.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.14", "id": [3, 6, 1]}]': finished 2024-03-12T03:54:29.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:28 smithi169 ceph-mon[39949]: osdmap e499: 8 total, 7 up, 8 in 2024-03-12T03:54:29.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:28 smithi173 ceph-mon[33544]: 6.f scrub starts 2024-03-12T03:54:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:28 smithi173 ceph-mon[33544]: 6.5 scrub starts 2024-03-12T03:54:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:28 smithi173 ceph-mon[33544]: 6.5 scrub ok 2024-03-12T03:54:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:28 smithi173 ceph-mon[33544]: 6.f scrub ok 2024-03-12T03:54:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:28 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/853741230' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.14", "id": [3, 6, 1]}]: dispatch 2024-03-12T03:54:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:28 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.14", "id": [3, 6, 1]}]: dispatch 2024-03-12T03:54:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:28 smithi173 ceph-mon[33544]: 1.a scrub starts 2024-03-12T03:54:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:28 smithi173 ceph-mon[33544]: 1.a scrub ok 2024-03-12T03:54:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:28 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 6767/111981 objects degraded (6.043%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:28 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.14", "id": [3, 6, 1]}]': finished 2024-03-12T03:54:29.138 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:28 smithi173 ceph-mon[33544]: osdmap e499: 8 total, 7 up, 8 in 2024-03-12T03:54:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:29.154 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:29.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:29.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:29.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:29.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:29.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:29.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:29.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:29.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:29.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:29.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:29.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:29.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:29.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:29.705 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:29.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:29.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:29.844 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:29.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:29.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:29.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:29.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:29.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:30.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:30.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:30.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[29344]: pgmap v1032: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 2.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 815 op/s; 6370/97770 objects degraded (6.515%); 3881/97770 objects misplaced (3.970%) 2024-03-12T03:54:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[29344]: 3.4 scrub starts 2024-03-12T03:54:30.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[29344]: 3.4 scrub ok 2024-03-12T03:54:30.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[29344]: 11.d scrub starts 2024-03-12T03:54:30.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[29344]: 5.1a scrub starts 2024-03-12T03:54:30.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[29344]: 5.1a scrub ok 2024-03-12T03:54:30.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[29344]: 4.15 scrub starts 2024-03-12T03:54:30.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[29344]: 4.15 scrub ok 2024-03-12T03:54:30.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[29344]: osdmap e500: 8 total, 7 up, 8 in 2024-03-12T03:54:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[39949]: pgmap v1032: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 2.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 815 op/s; 6370/97770 objects degraded (6.515%); 3881/97770 objects misplaced (3.970%) 2024-03-12T03:54:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[39949]: 3.4 scrub starts 2024-03-12T03:54:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[39949]: 3.4 scrub ok 2024-03-12T03:54:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[39949]: 11.d scrub starts 2024-03-12T03:54:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[39949]: 5.1a scrub starts 2024-03-12T03:54:30.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[39949]: 5.1a scrub ok 2024-03-12T03:54:30.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[39949]: 4.15 scrub starts 2024-03-12T03:54:30.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[39949]: 4.15 scrub ok 2024-03-12T03:54:30.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:29 smithi169 ceph-mon[39949]: osdmap e500: 8 total, 7 up, 8 in 2024-03-12T03:54:30.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:29 smithi173 ceph-mon[33544]: pgmap v1032: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 2.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 815 op/s; 6370/97770 objects degraded (6.515%); 3881/97770 objects misplaced (3.970%) 2024-03-12T03:54:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:29 smithi173 ceph-mon[33544]: 3.4 scrub starts 2024-03-12T03:54:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:29 smithi173 ceph-mon[33544]: 3.4 scrub ok 2024-03-12T03:54:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:29 smithi173 ceph-mon[33544]: 11.d scrub starts 2024-03-12T03:54:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:29 smithi173 ceph-mon[33544]: 5.1a scrub starts 2024-03-12T03:54:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:29 smithi173 ceph-mon[33544]: 5.1a scrub ok 2024-03-12T03:54:30.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:29 smithi173 ceph-mon[33544]: 4.15 scrub starts 2024-03-12T03:54:30.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:29 smithi173 ceph-mon[33544]: 4.15 scrub ok 2024-03-12T03:54:30.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:29 smithi173 ceph-mon[33544]: osdmap e500: 8 total, 7 up, 8 in 2024-03-12T03:54:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:30.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:30.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:30.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:30.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:30.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:30.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:30.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:30.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:30.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:30.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:30.661 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:30.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:30.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:30.796 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:30.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:30.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:30.934 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:30.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:31.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:31.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:31.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:31.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:31.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:30 smithi169 ceph-mon[29344]: 11.d scrub ok 2024-03-12T03:54:31.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:30 smithi169 ceph-mon[29344]: 4.a scrub starts 2024-03-12T03:54:31.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:30 smithi169 ceph-mon[29344]: 4.a scrub ok 2024-03-12T03:54:31.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:30 smithi169 ceph-mon[39949]: 11.d scrub ok 2024-03-12T03:54:31.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:30 smithi169 ceph-mon[39949]: 4.a scrub starts 2024-03-12T03:54:31.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:30 smithi169 ceph-mon[39949]: 4.a scrub ok 2024-03-12T03:54:31.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:31.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:31.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:31.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:30 smithi173 ceph-mon[33544]: 11.d scrub ok 2024-03-12T03:54:31.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:30 smithi173 ceph-mon[33544]: 4.a scrub starts 2024-03-12T03:54:31.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:30 smithi173 ceph-mon[33544]: 4.a scrub ok 2024-03-12T03:54:31.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:31.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:31.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:31.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:31.485 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:31.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:31.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:31.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:31.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:31.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:31.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:31.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:31.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:31.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:31.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:31.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:31.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:32.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:32.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:32.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:32.028 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:31 smithi169 ceph-mon[29344]: pgmap v1035: 176 pgs: 1 peering, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 1.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 5477/91251 objects degraded (6.002%); 3881/91251 objects misplaced (4.253%) 2024-03-12T03:54:32.028 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:31 smithi169 ceph-mon[29344]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-12T03:54:32.028 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:31 smithi169 ceph-mon[29344]: 4.1d scrub starts 2024-03-12T03:54:32.028 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:31 smithi169 ceph-mon[29344]: 4.1d scrub ok 2024-03-12T03:54:32.028 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:31 smithi169 ceph-mon[39949]: pgmap v1035: 176 pgs: 1 peering, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 1.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 5477/91251 objects degraded (6.002%); 3881/91251 objects misplaced (4.253%) 2024-03-12T03:54:32.028 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:31 smithi169 ceph-mon[39949]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-12T03:54:32.028 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:31 smithi169 ceph-mon[39949]: 4.1d scrub starts 2024-03-12T03:54:32.028 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:31 smithi169 ceph-mon[39949]: 4.1d scrub ok 2024-03-12T03:54:32.029 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:54:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:54:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:54:32.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:32.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:32.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:32.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:31 smithi173 ceph-mon[33544]: pgmap v1035: 176 pgs: 1 peering, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 1.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 5477/91251 objects degraded (6.002%); 3881/91251 objects misplaced (4.253%) 2024-03-12T03:54:32.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:31 smithi173 ceph-mon[33544]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-12T03:54:32.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:31 smithi173 ceph-mon[33544]: 4.1d scrub starts 2024-03-12T03:54:32.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:31 smithi173 ceph-mon[33544]: 4.1d scrub ok 2024-03-12T03:54:32.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:32.304 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:32.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:32.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:32.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:32.448 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:32.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:32.549 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:32.589 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:32.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:32.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:32.735 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:32.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:32.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:32.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:32.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:32.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:32.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:33.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:33.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:33.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:33.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:33.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:33.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:32 smithi173 ceph-mon[33544]: 3.a scrub starts 2024-03-12T03:54:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:32 smithi173 ceph-mon[33544]: 3.a scrub ok 2024-03-12T03:54:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:32 smithi173 ceph-mon[33544]: 4.1c scrub starts 2024-03-12T03:54:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:32 smithi173 ceph-mon[33544]: 4.1c scrub ok 2024-03-12T03:54:33.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:32 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:54:33.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:32 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:33.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:33.279 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:33.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:33.350 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:32 smithi169 ceph-mon[29344]: 3.a scrub starts 2024-03-12T03:54:33.350 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:32 smithi169 ceph-mon[29344]: 3.a scrub ok 2024-03-12T03:54:33.350 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:32 smithi169 ceph-mon[29344]: 4.1c scrub starts 2024-03-12T03:54:33.350 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:32 smithi169 ceph-mon[29344]: 4.1c scrub ok 2024-03-12T03:54:33.350 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:32 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:54:33.351 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:32 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:33.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:32 smithi169 ceph-mon[39949]: 3.a scrub starts 2024-03-12T03:54:33.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:32 smithi169 ceph-mon[39949]: 3.a scrub ok 2024-03-12T03:54:33.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:32 smithi169 ceph-mon[39949]: 4.1c scrub starts 2024-03-12T03:54:33.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:32 smithi169 ceph-mon[39949]: 4.1c scrub ok 2024-03-12T03:54:33.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:32 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:54:33.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:32 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:33.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:33.410 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:33.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:33.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:33.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:33.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:33.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:33.568 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:54:33.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:33.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:33.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:33.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:33.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:33.820 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:33.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:33.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:33.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:33.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:33.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:33.971 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:54:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:54:34.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:34.097 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:34.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:34.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:34.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:33 smithi173 ceph-mon[33544]: pgmap v1036: 176 pgs: 1 peering, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 1.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.68k op/s; 5477/81681 objects degraded (6.705%); 3170/81681 objects misplaced (3.881%) 2024-03-12T03:54:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:33 smithi173 ceph-mon[33544]: 1.3 scrub starts 2024-03-12T03:54:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:33 smithi173 ceph-mon[33544]: 1.3 scrub ok 2024-03-12T03:54:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:33 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3829736660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:54:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:33 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:54:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:33 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 5477/81681 objects degraded (6.705%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:33 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:54:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:33 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:54:34.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:33 smithi173 ceph-mon[33544]: osdmap e501: 8 total, 7 up, 8 in 2024-03-12T03:54:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:34.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:34.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:34.378 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:34.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:34.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[29344]: pgmap v1036: 176 pgs: 1 peering, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 1.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.68k op/s; 5477/81681 objects degraded (6.705%); 3170/81681 objects misplaced (3.881%) 2024-03-12T03:54:34.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[29344]: 1.3 scrub starts 2024-03-12T03:54:34.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[29344]: 1.3 scrub ok 2024-03-12T03:54:34.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3829736660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:54:34.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:54:34.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 5477/81681 objects degraded (6.705%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:34.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:54:34.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:54:34.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[29344]: osdmap e501: 8 total, 7 up, 8 in 2024-03-12T03:54:34.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[39949]: pgmap v1036: 176 pgs: 1 peering, 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 112 active+clean; 1.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.68k op/s; 5477/81681 objects degraded (6.705%); 3170/81681 objects misplaced (3.881%) 2024-03-12T03:54:34.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[39949]: 1.3 scrub starts 2024-03-12T03:54:34.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[39949]: 1.3 scrub ok 2024-03-12T03:54:34.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3829736660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:54:34.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:54:34.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 5477/81681 objects degraded (6.705%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:34.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:54:34.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:54:34.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:33 smithi169 ceph-mon[39949]: osdmap e501: 8 total, 7 up, 8 in 2024-03-12T03:54:34.480 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:34.515 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:34.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:34.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:34.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:34.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:34.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:34.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:34.784 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:34.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:34.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:34.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:34.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:34.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:34.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:35.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:35.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:35.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:35.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:35.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:35.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:35.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:34 smithi173 ceph-mon[33544]: 5.19 scrub starts 2024-03-12T03:54:35.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:34 smithi173 ceph-mon[33544]: 5.19 scrub ok 2024-03-12T03:54:35.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:35.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:34 smithi173 ceph-mon[33544]: 6.8 scrub starts 2024-03-12T03:54:35.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:34 smithi173 ceph-mon[33544]: 6.8 scrub ok 2024-03-12T03:54:35.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:34 smithi173 ceph-mon[33544]: 6.1b scrub starts 2024-03-12T03:54:35.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:34 smithi173 ceph-mon[33544]: 6.1b scrub ok 2024-03-12T03:54:35.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:35.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:35.319 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:35.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:35.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:35.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[29344]: 5.19 scrub starts 2024-03-12T03:54:35.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[29344]: 5.19 scrub ok 2024-03-12T03:54:35.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:35.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[29344]: 6.8 scrub starts 2024-03-12T03:54:35.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[29344]: 6.8 scrub ok 2024-03-12T03:54:35.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[29344]: 6.1b scrub starts 2024-03-12T03:54:35.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[29344]: 6.1b scrub ok 2024-03-12T03:54:35.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[39949]: 5.19 scrub starts 2024-03-12T03:54:35.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[39949]: 5.19 scrub ok 2024-03-12T03:54:35.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:35.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[39949]: 6.8 scrub starts 2024-03-12T03:54:35.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[39949]: 6.8 scrub ok 2024-03-12T03:54:35.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[39949]: 6.1b scrub starts 2024-03-12T03:54:35.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:34 smithi169 ceph-mon[39949]: 6.1b scrub ok 2024-03-12T03:54:35.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:35.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:35.560 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:35.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:35.610 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:35.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:35.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:35.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:35.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:35.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:35.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:35.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:35.888 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:35.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:35.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:36.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:36.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:36.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:36.167 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:36.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:36.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:36.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:35 smithi173 ceph-mon[33544]: pgmap v1038: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 1.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.57k op/s; 4139/69414 objects degraded (5.963%); 2971/69414 objects misplaced (4.280%) 2024-03-12T03:54:36.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:35 smithi173 ceph-mon[33544]: 3.0 scrub starts 2024-03-12T03:54:36.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:35 smithi173 ceph-mon[33544]: 3.0 scrub ok 2024-03-12T03:54:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:35 smithi173 ceph-mon[33544]: 6.1e scrub starts 2024-03-12T03:54:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:35 smithi173 ceph-mon[33544]: 6.1e scrub ok 2024-03-12T03:54:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:35 smithi173 ceph-mon[33544]: 5.d scrub starts 2024-03-12T03:54:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:35 smithi173 ceph-mon[33544]: 5.d scrub ok 2024-03-12T03:54:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:35 smithi173 ceph-mon[33544]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-12T03:54:36.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[29344]: pgmap v1038: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 1.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.57k op/s; 4139/69414 objects degraded (5.963%); 2971/69414 objects misplaced (4.280%) 2024-03-12T03:54:36.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[29344]: 3.0 scrub starts 2024-03-12T03:54:36.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[29344]: 3.0 scrub ok 2024-03-12T03:54:36.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[29344]: 6.1e scrub starts 2024-03-12T03:54:36.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[29344]: 6.1e scrub ok 2024-03-12T03:54:36.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[29344]: 5.d scrub starts 2024-03-12T03:54:36.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[29344]: 5.d scrub ok 2024-03-12T03:54:36.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[29344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-12T03:54:36.248 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[39949]: pgmap v1038: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 1.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.57k op/s; 4139/69414 objects degraded (5.963%); 2971/69414 objects misplaced (4.280%) 2024-03-12T03:54:36.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[39949]: 3.0 scrub starts 2024-03-12T03:54:36.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[39949]: 3.0 scrub ok 2024-03-12T03:54:36.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[39949]: 6.1e scrub starts 2024-03-12T03:54:36.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[39949]: 6.1e scrub ok 2024-03-12T03:54:36.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[39949]: 5.d scrub starts 2024-03-12T03:54:36.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[39949]: 5.d scrub ok 2024-03-12T03:54:36.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:35 smithi169 ceph-mon[39949]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-12T03:54:36.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:36.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:36.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:36.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:36.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:36.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:36.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:36.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:36.584 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:36.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:36.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:36.665 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:54:36.665 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:54:36.666 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-12T03:54:36.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd out 5 2024-03-12T03:54:36.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:36.720 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:36.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:36.821 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:36.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:36.870 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:36.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:36.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:37.032 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:37.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:37.041 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[29344]: 4.2 scrub starts 2024-03-12T03:54:37.041 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[29344]: 4.2 scrub ok 2024-03-12T03:54:37.041 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[29344]: 4.18 scrub starts 2024-03-12T03:54:37.042 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[29344]: 4.18 scrub ok 2024-03-12T03:54:37.042 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[29344]: 5.0 deep-scrub starts 2024-03-12T03:54:37.042 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[29344]: 5.0 deep-scrub ok 2024-03-12T03:54:37.042 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[29344]: 5.f scrub starts 2024-03-12T03:54:37.042 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[29344]: 5.f scrub ok 2024-03-12T03:54:37.042 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[29344]: pgmap v1039: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 1.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 661 B/s rd, 0 B/s wr, 1.14k op/s; 4139/69414 objects degraded (5.963%); 2971/69414 objects misplaced (4.280%) 2024-03-12T03:54:37.042 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2597638484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:54:37.042 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[39949]: 4.2 scrub starts 2024-03-12T03:54:37.042 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[39949]: 4.2 scrub ok 2024-03-12T03:54:37.042 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[39949]: 4.18 scrub starts 2024-03-12T03:54:37.043 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[39949]: 4.18 scrub ok 2024-03-12T03:54:37.043 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[39949]: 5.0 deep-scrub starts 2024-03-12T03:54:37.043 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[39949]: 5.0 deep-scrub ok 2024-03-12T03:54:37.043 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[39949]: 5.f scrub starts 2024-03-12T03:54:37.043 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[39949]: 5.f scrub ok 2024-03-12T03:54:37.043 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[39949]: pgmap v1039: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 1.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 661 B/s rd, 0 B/s wr, 1.14k op/s; 4139/69414 objects degraded (5.963%); 2971/69414 objects misplaced (4.280%) 2024-03-12T03:54:37.043 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:36 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2597638484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:54:37.043 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:54:37.087 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:54:37.131 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:54:37.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:37.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:37.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:37.179 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:54:37.199 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:54:37.200 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:54:37.200 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-03-12T03:54:37.200 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-03-12T03:54:37.200 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:54:37.217 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:54:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:36 smithi173 ceph-mon[33544]: 4.2 scrub starts 2024-03-12T03:54:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:36 smithi173 ceph-mon[33544]: 4.2 scrub ok 2024-03-12T03:54:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:36 smithi173 ceph-mon[33544]: 4.18 scrub starts 2024-03-12T03:54:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:36 smithi173 ceph-mon[33544]: 4.18 scrub ok 2024-03-12T03:54:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:36 smithi173 ceph-mon[33544]: 5.0 deep-scrub starts 2024-03-12T03:54:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:36 smithi173 ceph-mon[33544]: 5.0 deep-scrub ok 2024-03-12T03:54:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:36 smithi173 ceph-mon[33544]: 5.f scrub starts 2024-03-12T03:54:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:36 smithi173 ceph-mon[33544]: 5.f scrub ok 2024-03-12T03:54:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:36 smithi173 ceph-mon[33544]: pgmap v1039: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 1.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 661 B/s rd, 0 B/s wr, 1.14k op/s; 4139/69414 objects degraded (5.963%); 2971/69414 objects misplaced (4.280%) 2024-03-12T03:54:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:36 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2597638484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:54:37.257 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:54:37.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:37.295 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:54:37.295 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:54:37.295 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:54:37.295 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:54:37.295 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:54:37.295 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:54:37.295 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:54:37.295 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:54:37.295 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:54:37.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:37.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:37.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:37.443 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:37.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:37.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:37.580 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:37.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:37.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:37.692 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:37.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:37.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:37.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:37.813 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:37.852 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:37.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:37.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:37.991 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:37.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:38.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-12T03:54:38.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:38.114 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:37 smithi169 ceph-mon[29344]: 4.f scrub starts 2024-03-12T03:54:38.114 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:37 smithi169 ceph-mon[29344]: 4.f scrub ok 2024-03-12T03:54:38.114 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:37 smithi169 ceph-mon[29344]: 1.8 scrub starts 2024-03-12T03:54:38.114 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:38 smithi169 ceph-mon[29344]: 1.8 scrub ok 2024-03-12T03:54:38.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:38 smithi169 ceph-mon[39949]: 4.f scrub starts 2024-03-12T03:54:38.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:38 smithi169 ceph-mon[39949]: 4.f scrub ok 2024-03-12T03:54:38.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:38 smithi169 ceph-mon[39949]: 1.8 scrub starts 2024-03-12T03:54:38.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:38 smithi169 ceph-mon[39949]: 1.8 scrub ok 2024-03-12T03:54:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:38.159 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:38.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:38.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:38.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:38.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:38.294 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:54:38.295 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:54:38.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:38.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:38.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:37 smithi173 ceph-mon[33544]: 4.f scrub starts 2024-03-12T03:54:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:37 smithi173 ceph-mon[33544]: 4.f scrub ok 2024-03-12T03:54:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:38 smithi173 ceph-mon[33544]: 1.8 scrub starts 2024-03-12T03:54:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:38 smithi173 ceph-mon[33544]: 1.8 scrub ok 2024-03-12T03:54:38.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:38.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:38.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:38.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:38.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:38.718 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:38.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:38.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:38.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:38.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:38.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:38.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:38.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:38.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:39.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-12T03:54:39.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:39.155 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:39.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:39.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:39.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:39.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:39.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:39.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:39 smithi169 ceph-mon[29344]: pgmap v1040: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 1.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s; 3676/54519 objects degraded (6.743%); 2114/54519 objects misplaced (3.878%) 2024-03-12T03:54:39.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:39 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 4139/69414 objects degraded (5.963%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:39.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:39 smithi169 ceph-mon[39949]: pgmap v1040: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 1.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s; 3676/54519 objects degraded (6.743%); 2114/54519 objects misplaced (3.878%) 2024-03-12T03:54:39.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:39 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 4139/69414 objects degraded (5.963%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:39.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:39.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:39.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:39.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:39 smithi173 ceph-mon[33544]: pgmap v1040: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 1.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s; 3676/54519 objects degraded (6.743%); 2114/54519 objects misplaced (3.878%) 2024-03-12T03:54:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:39 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 4139/69414 objects degraded (5.963%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:39.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:39.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:39.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:39.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:39.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:39.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:39.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:39.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:39.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:39.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:39.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:39.981 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:39.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:40.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:40.102 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:40.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:40.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:40.168 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:40.214 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:40.266 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:40.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:40.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-12T03:54:40.321 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:40 smithi169 ceph-mon[29344]: 5.4 deep-scrub starts 2024-03-12T03:54:40.321 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:40 smithi169 ceph-mon[29344]: 5.4 deep-scrub ok 2024-03-12T03:54:40.321 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:40 smithi169 ceph-mon[29344]: 3.1e scrub starts 2024-03-12T03:54:40.321 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:40 smithi169 ceph-mon[29344]: 3.1e scrub ok 2024-03-12T03:54:40.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:40 smithi169 ceph-mon[39949]: 5.4 deep-scrub starts 2024-03-12T03:54:40.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:40 smithi169 ceph-mon[39949]: 5.4 deep-scrub ok 2024-03-12T03:54:40.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:40 smithi169 ceph-mon[39949]: 3.1e scrub starts 2024-03-12T03:54:40.322 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:40 smithi169 ceph-mon[39949]: 3.1e scrub ok 2024-03-12T03:54:40.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:40.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:40.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:40 smithi173 ceph-mon[33544]: 5.4 deep-scrub starts 2024-03-12T03:54:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:40 smithi173 ceph-mon[33544]: 5.4 deep-scrub ok 2024-03-12T03:54:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:40 smithi173 ceph-mon[33544]: 3.1e scrub starts 2024-03-12T03:54:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:40 smithi173 ceph-mon[33544]: 3.1e scrub ok 2024-03-12T03:54:40.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:40.553 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:40.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:40.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:40.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:40.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:40.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:40.784 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:40.824 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:40.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:40.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:40.950 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:40.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:41.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:41.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:41.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:41.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:41.187 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:41.223 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:41.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:41.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:41.364 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:41.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:41.377 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[29344]: 5.11 scrub starts 2024-03-12T03:54:41.377 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[29344]: 5.11 scrub ok 2024-03-12T03:54:41.377 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[29344]: 4.10 scrub starts 2024-03-12T03:54:41.377 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[29344]: 4.10 scrub ok 2024-03-12T03:54:41.377 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[29344]: 3.5 deep-scrub starts 2024-03-12T03:54:41.377 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[29344]: 3.5 deep-scrub ok 2024-03-12T03:54:41.377 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[29344]: pgmap v1041: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 982 MiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 2797/47793 objects degraded (5.852%); 2114/47793 objects misplaced (4.423%) 2024-03-12T03:54:41.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[39949]: 5.11 scrub starts 2024-03-12T03:54:41.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[39949]: 5.11 scrub ok 2024-03-12T03:54:41.378 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[39949]: 4.10 scrub starts 2024-03-12T03:54:41.378 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[39949]: 4.10 scrub ok 2024-03-12T03:54:41.378 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[39949]: 3.5 deep-scrub starts 2024-03-12T03:54:41.378 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[39949]: 3.5 deep-scrub ok 2024-03-12T03:54:41.378 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-mon[39949]: pgmap v1041: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 982 MiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 2797/47793 objects degraded (5.852%); 2114/47793 objects misplaced (4.423%) 2024-03-12T03:54:41.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:41 smithi173 ceph-mon[33544]: 5.11 scrub starts 2024-03-12T03:54:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:41 smithi173 ceph-mon[33544]: 5.11 scrub ok 2024-03-12T03:54:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:41 smithi173 ceph-mon[33544]: 4.10 scrub starts 2024-03-12T03:54:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:41 smithi173 ceph-mon[33544]: 4.10 scrub ok 2024-03-12T03:54:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:41 smithi173 ceph-mon[33544]: 3.5 deep-scrub starts 2024-03-12T03:54:41.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:41 smithi173 ceph-mon[33544]: 3.5 deep-scrub ok 2024-03-12T03:54:41.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:41 smithi173 ceph-mon[33544]: pgmap v1041: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 982 MiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 2797/47793 objects degraded (5.852%); 2114/47793 objects misplaced (4.423%) 2024-03-12T03:54:41.491 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:41.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:41.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:41.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:41.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:41.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:41.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:41.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:41.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:41.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:41.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:41.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:42.003 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:42.041 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:42.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:42.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:42.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:42 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:54:42.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:42 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:54:42.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:42 smithi169 ceph-mon[29344]: osdmap e502: 8 total, 7 up, 8 in 2024-03-12T03:54:42.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:42 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:54:42.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:42 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:54:42.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:42 smithi169 ceph-mon[39949]: osdmap e502: 8 total, 7 up, 8 in 2024-03-12T03:54:42.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:54:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:54:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:54:42.179 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:42.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:42.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:42.319 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:42.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:42.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:42.468 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:42.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:42.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:42 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:54:42.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:42 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:54:42.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:42 smithi173 ceph-mon[33544]: osdmap e502: 8 total, 7 up, 8 in 2024-03-12T03:54:42.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:42.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:42.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:42.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:42.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:42.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:42.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:42.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:42.868 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:42.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:42.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:43.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:43.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:43.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:43.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:43.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:43.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:43.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:43.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:43.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:43.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:43.353 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:43 smithi169 ceph-mon[29344]: 3.2 scrub starts 2024-03-12T03:54:43.353 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:43 smithi169 ceph-mon[29344]: 3.2 scrub ok 2024-03-12T03:54:43.354 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:43 smithi169 ceph-mon[29344]: 1.11 scrub starts 2024-03-12T03:54:43.354 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:43 smithi169 ceph-mon[29344]: 1.11 scrub ok 2024-03-12T03:54:43.354 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:43 smithi169 ceph-mon[29344]: pgmap v1043: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 794 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.35k op/s; 2797/38769 objects degraded (7.215%); 1431/38769 objects misplaced (3.691%) 2024-03-12T03:54:43.354 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:43 smithi169 ceph-mon[39949]: 3.2 scrub starts 2024-03-12T03:54:43.354 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:43 smithi169 ceph-mon[39949]: 3.2 scrub ok 2024-03-12T03:54:43.354 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:43 smithi169 ceph-mon[39949]: 1.11 scrub starts 2024-03-12T03:54:43.354 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:43 smithi169 ceph-mon[39949]: 1.11 scrub ok 2024-03-12T03:54:43.354 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:43 smithi169 ceph-mon[39949]: pgmap v1043: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 794 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.35k op/s; 2797/38769 objects degraded (7.215%); 1431/38769 objects misplaced (3.691%) 2024-03-12T03:54:43.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:43.406 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:43.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:43 smithi173 ceph-mon[33544]: 3.2 scrub starts 2024-03-12T03:54:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:43 smithi173 ceph-mon[33544]: 3.2 scrub ok 2024-03-12T03:54:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:43 smithi173 ceph-mon[33544]: 1.11 scrub starts 2024-03-12T03:54:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:43 smithi173 ceph-mon[33544]: 1.11 scrub ok 2024-03-12T03:54:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:43 smithi173 ceph-mon[33544]: pgmap v1043: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 794 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.35k op/s; 2797/38769 objects degraded (7.215%); 1431/38769 objects misplaced (3.691%) 2024-03-12T03:54:43.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:43.549 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:43.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:43.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:43.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:43.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:43.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:43.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:43.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:43.923 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:43.960 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:43.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:43.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:54:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:54:44.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:44.098 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:44.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:44.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:44.232 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:44.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:44.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:44.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:44.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:44.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:44.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:44 smithi169 ceph-mon[39949]: 11.e deep-scrub starts 2024-03-12T03:54:44.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:44.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:44 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 2797/38769 objects degraded (7.215%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:44.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:44 smithi169 ceph-mon[39949]: osdmap e503: 8 total, 7 up, 8 in 2024-03-12T03:54:44.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:44 smithi169 ceph-mon[29344]: 11.e deep-scrub starts 2024-03-12T03:54:44.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:44.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:44 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 2797/38769 objects degraded (7.215%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:44.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:44 smithi169 ceph-mon[29344]: osdmap e503: 8 total, 7 up, 8 in 2024-03-12T03:54:44.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:44.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:44 smithi173 ceph-mon[33544]: 11.e deep-scrub starts 2024-03-12T03:54:44.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:44.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:44 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 2797/38769 objects degraded (7.215%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:44.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:44 smithi173 ceph-mon[33544]: osdmap e503: 8 total, 7 up, 8 in 2024-03-12T03:54:44.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:44.509 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:44.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:44.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:44.643 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:44.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:44.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:44.745 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:44.782 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:44.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:44.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:44.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:44.932 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:44.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:45.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:45.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:45.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:45.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:45.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:45.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[29344]: 5.1 scrub starts 2024-03-12T03:54:45.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[29344]: 5.1 scrub ok 2024-03-12T03:54:45.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[29344]: 1.9 scrub starts 2024-03-12T03:54:45.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[29344]: 1.9 scrub ok 2024-03-12T03:54:45.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[29344]: 6.6 scrub starts 2024-03-12T03:54:45.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[29344]: 6.6 scrub ok 2024-03-12T03:54:45.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[29344]: pgmap v1045: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 555 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s; 1552/27321 objects degraded (5.681%); 1203/27321 objects misplaced (4.403%) 2024-03-12T03:54:45.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[39949]: 5.1 scrub starts 2024-03-12T03:54:45.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[39949]: 5.1 scrub ok 2024-03-12T03:54:45.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[39949]: 1.9 scrub starts 2024-03-12T03:54:45.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[39949]: 1.9 scrub ok 2024-03-12T03:54:45.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[39949]: 6.6 scrub starts 2024-03-12T03:54:45.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[39949]: 6.6 scrub ok 2024-03-12T03:54:45.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-mon[39949]: pgmap v1045: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 555 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s; 1552/27321 objects degraded (5.681%); 1203/27321 objects misplaced (4.403%) 2024-03-12T03:54:45.205 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:45.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:45.307 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:45.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:45.351 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:45.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:45.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:45 smithi173 ceph-mon[33544]: 5.1 scrub starts 2024-03-12T03:54:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:45 smithi173 ceph-mon[33544]: 5.1 scrub ok 2024-03-12T03:54:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:45 smithi173 ceph-mon[33544]: 1.9 scrub starts 2024-03-12T03:54:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:45 smithi173 ceph-mon[33544]: 1.9 scrub ok 2024-03-12T03:54:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:45 smithi173 ceph-mon[33544]: 6.6 scrub starts 2024-03-12T03:54:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:45 smithi173 ceph-mon[33544]: 6.6 scrub ok 2024-03-12T03:54:45.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:45 smithi173 ceph-mon[33544]: pgmap v1045: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 555 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s; 1552/27321 objects degraded (5.681%); 1203/27321 objects misplaced (4.403%) 2024-03-12T03:54:45.497 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:45.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:45.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:45.642 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:45.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:45.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:45 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a[29340]: 2024-03-12T03:54:45.428+0000 7f7050ca9700 -1 mon.a@0(leader).osd e503 definitely_dead 0 2024-03-12T03:54:45.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:45.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:45.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:45.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:45.923 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:45.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:45.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:46.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:46.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:46.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:46.103 INFO:teuthology.orchestra.run.smithi169.stderr:marked out osd.5. 2024-03-12T03:54:46.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:46.184 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:46.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:46.285 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:46.310 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:46.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:46.358 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[29344]: 4.5 scrub starts 2024-03-12T03:54:46.358 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[29344]: 4.5 scrub ok 2024-03-12T03:54:46.358 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[29344]: 4.17 scrub starts 2024-03-12T03:54:46.358 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[29344]: 4.17 scrub ok 2024-03-12T03:54:46.359 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[29344]: 11.e deep-scrub ok 2024-03-12T03:54:46.359 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[29344]: 2.2 scrub starts 2024-03-12T03:54:46.359 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[29344]: 2.2 scrub ok 2024-03-12T03:54:46.359 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2729133932' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-12T03:54:46.359 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-12T03:54:46.359 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[29344]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-12T03:54:46.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[39949]: 4.5 scrub starts 2024-03-12T03:54:46.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[39949]: 4.5 scrub ok 2024-03-12T03:54:46.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[39949]: 4.17 scrub starts 2024-03-12T03:54:46.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[39949]: 4.17 scrub ok 2024-03-12T03:54:46.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[39949]: 11.e deep-scrub ok 2024-03-12T03:54:46.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[39949]: 2.2 scrub starts 2024-03-12T03:54:46.360 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[39949]: 2.2 scrub ok 2024-03-12T03:54:46.360 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2729133932' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-12T03:54:46.360 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-12T03:54:46.360 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:46 smithi169 ceph-mon[39949]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-12T03:54:46.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:46.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:46.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:46.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:46 smithi173 ceph-mon[33544]: 4.5 scrub starts 2024-03-12T03:54:46.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:46 smithi173 ceph-mon[33544]: 4.5 scrub ok 2024-03-12T03:54:46.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:46 smithi173 ceph-mon[33544]: 4.17 scrub starts 2024-03-12T03:54:46.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:46 smithi173 ceph-mon[33544]: 4.17 scrub ok 2024-03-12T03:54:46.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:46 smithi173 ceph-mon[33544]: 11.e deep-scrub ok 2024-03-12T03:54:46.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:46 smithi173 ceph-mon[33544]: 2.2 scrub starts 2024-03-12T03:54:46.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:46 smithi173 ceph-mon[33544]: 2.2 scrub ok 2024-03-12T03:54:46.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2729133932' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-12T03:54:46.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:46 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-12T03:54:46.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:46 smithi173 ceph-mon[33544]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-12T03:54:46.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:46.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:46.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:46.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:46.689 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:46.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:46.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:46.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:46.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:46.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:46.942 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:46.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:47.044 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:47.071 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:47.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:47.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:47.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:47.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:47.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:47.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:47.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:47.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:47.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:47.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:47.367 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[29344]: 3.c scrub starts 2024-03-12T03:54:47.367 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[29344]: 3.c scrub ok 2024-03-12T03:54:47.367 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-12T03:54:47.367 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[29344]: osdmap e504: 8 total, 7 up, 7 in 2024-03-12T03:54:47.367 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[29344]: pgmap v1047: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 555 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s; 1552/27321 objects degraded (5.681%); 1203/27321 objects misplaced (4.403%) 2024-03-12T03:54:47.367 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[29344]: 5.18 scrub starts 2024-03-12T03:54:47.367 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[29344]: 5.18 scrub ok 2024-03-12T03:54:47.367 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:54:47.367 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:54:47.367 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:54:47.368 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:47.368 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[29344]: osdmap e505: 8 total, 7 up, 7 in 2024-03-12T03:54:47.368 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[39949]: 3.c scrub starts 2024-03-12T03:54:47.368 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[39949]: 3.c scrub ok 2024-03-12T03:54:47.368 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-12T03:54:47.368 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[39949]: osdmap e504: 8 total, 7 up, 7 in 2024-03-12T03:54:47.368 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[39949]: pgmap v1047: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 555 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s; 1552/27321 objects degraded (5.681%); 1203/27321 objects misplaced (4.403%) 2024-03-12T03:54:47.368 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[39949]: 5.18 scrub starts 2024-03-12T03:54:47.368 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[39949]: 5.18 scrub ok 2024-03-12T03:54:47.368 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:54:47.368 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:54:47.369 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:54:47.369 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:47.369 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:47 smithi169 ceph-mon[39949]: osdmap e505: 8 total, 7 up, 7 in 2024-03-12T03:54:47.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:47.450 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:47.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:47.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:47.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:47 smithi173 ceph-mon[33544]: 3.c scrub starts 2024-03-12T03:54:47.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:47 smithi173 ceph-mon[33544]: 3.c scrub ok 2024-03-12T03:54:47.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:47 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-12T03:54:47.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:47 smithi173 ceph-mon[33544]: osdmap e504: 8 total, 7 up, 7 in 2024-03-12T03:54:47.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:47 smithi173 ceph-mon[33544]: pgmap v1047: 176 pgs: 8 active+clean+remapped, 21 active+undersized+degraded, 34 active+undersized, 113 active+clean; 555 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s; 1552/27321 objects degraded (5.681%); 1203/27321 objects misplaced (4.403%) 2024-03-12T03:54:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:47 smithi173 ceph-mon[33544]: 5.18 scrub starts 2024-03-12T03:54:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:47 smithi173 ceph-mon[33544]: 5.18 scrub ok 2024-03-12T03:54:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:47 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:54:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:47 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:54:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:47 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:54:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:47 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:47 smithi173 ceph-mon[33544]: osdmap e505: 8 total, 7 up, 7 in 2024-03-12T03:54:47.551 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:47.575 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:47.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:47.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:47.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:47.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:47.803 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:47.827 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:47.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:47.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:47.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:47.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:48.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:48.079 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is set 2024-03-12T03:54:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:48.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:48.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:48.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:48.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:48.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:48.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:48.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:48.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:48.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[29344]: 1.12 scrub starts 2024-03-12T03:54:48.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[29344]: 1.12 scrub ok 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1661596124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[29344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[29344]: osdmap e506: 8 total, 7 up, 7 in 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[29344]: 11.16 starting backfill to osd.1 from (0'0,0'0] MAX to 503'8547 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[29344]: 11.17 starting backfill to osd.1 from (0'0,0'0] MAX to 503'8601 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[29344]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 503'8728 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[39949]: 1.12 scrub starts 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[39949]: 1.12 scrub ok 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1661596124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:54:48.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[39949]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:54:48.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:54:48.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[39949]: osdmap e506: 8 total, 7 up, 7 in 2024-03-12T03:54:48.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[39949]: 11.16 starting backfill to osd.1 from (0'0,0'0] MAX to 503'8547 2024-03-12T03:54:48.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[39949]: 11.17 starting backfill to osd.1 from (0'0,0'0] MAX to 503'8601 2024-03-12T03:54:48.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:48 smithi169 ceph-mon[39949]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 503'8728 2024-03-12T03:54:48.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:48.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:48 smithi173 ceph-mon[33544]: 1.12 scrub starts 2024-03-12T03:54:48.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:48 smithi173 ceph-mon[33544]: 1.12 scrub ok 2024-03-12T03:54:48.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:48 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1661596124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:54:48.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:48 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-12T03:54:48.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:48 smithi173 ceph-mon[33544]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:54:48.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:48 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-12T03:54:48.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:48 smithi173 ceph-mon[33544]: osdmap e506: 8 total, 7 up, 7 in 2024-03-12T03:54:48.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:48 smithi173 ceph-mon[33544]: 11.16 starting backfill to osd.1 from (0'0,0'0] MAX to 503'8547 2024-03-12T03:54:48.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:48 smithi173 ceph-mon[33544]: 11.17 starting backfill to osd.1 from (0'0,0'0] MAX to 503'8601 2024-03-12T03:54:48.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:48 smithi173 ceph-mon[33544]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 503'8728 2024-03-12T03:54:48.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:48.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:48.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:48.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:48.621 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:48.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:48.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:48.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:48.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:48.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:48.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:48.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:48.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:48.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:48.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:49.034 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:49.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:49.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:49.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:49.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:49.271 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:49.315 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:49.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:49.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:49.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:49 smithi169 ceph-mon[29344]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 503'13816 2024-03-12T03:54:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:49 smithi169 ceph-mon[29344]: 11.13 starting backfill to osd.6 from (0'0,453'3722] MIN to 503'8670 2024-03-12T03:54:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:49 smithi169 ceph-mon[29344]: 1.4 starting backfill to osd.6 from (0'0,0'0] MAX to 83'128 2024-03-12T03:54:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:49 smithi169 ceph-mon[29344]: pgmap v1050: 176 pgs: 4 unknown, 11 peering, 12 remapped+peering, 6 active+clean+remapped, 19 active+undersized+degraded, 28 active+undersized, 96 active+clean; 330 MiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 215 B/s rd, 0 B/s wr, 1.06k op/s; 1246/16518 objects degraded (7.543%); 370/16518 objects misplaced (2.240%) 2024-03-12T03:54:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:49 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 1552/27321 objects degraded (5.681%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:49.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:49 smithi169 ceph-mon[29344]: osdmap e507: 8 total, 7 up, 7 in 2024-03-12T03:54:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:49 smithi169 ceph-mon[39949]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 503'13816 2024-03-12T03:54:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:49 smithi169 ceph-mon[39949]: 11.13 starting backfill to osd.6 from (0'0,453'3722] MIN to 503'8670 2024-03-12T03:54:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:49 smithi169 ceph-mon[39949]: 1.4 starting backfill to osd.6 from (0'0,0'0] MAX to 83'128 2024-03-12T03:54:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:49 smithi169 ceph-mon[39949]: pgmap v1050: 176 pgs: 4 unknown, 11 peering, 12 remapped+peering, 6 active+clean+remapped, 19 active+undersized+degraded, 28 active+undersized, 96 active+clean; 330 MiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 215 B/s rd, 0 B/s wr, 1.06k op/s; 1246/16518 objects degraded (7.543%); 370/16518 objects misplaced (2.240%) 2024-03-12T03:54:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:49 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 1552/27321 objects degraded (5.681%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:49.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:49 smithi169 ceph-mon[39949]: osdmap e507: 8 total, 7 up, 7 in 2024-03-12T03:54:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:49.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:49.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:49.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:49 smithi173 ceph-mon[33544]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 503'13816 2024-03-12T03:54:49.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:49 smithi173 ceph-mon[33544]: 11.13 starting backfill to osd.6 from (0'0,453'3722] MIN to 503'8670 2024-03-12T03:54:49.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:49 smithi173 ceph-mon[33544]: 1.4 starting backfill to osd.6 from (0'0,0'0] MAX to 83'128 2024-03-12T03:54:49.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:49 smithi173 ceph-mon[33544]: pgmap v1050: 176 pgs: 4 unknown, 11 peering, 12 remapped+peering, 6 active+clean+remapped, 19 active+undersized+degraded, 28 active+undersized, 96 active+clean; 330 MiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 215 B/s rd, 0 B/s wr, 1.06k op/s; 1246/16518 objects degraded (7.543%); 370/16518 objects misplaced (2.240%) 2024-03-12T03:54:49.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:49 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 1552/27321 objects degraded (5.681%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:49.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:49 smithi173 ceph-mon[33544]: osdmap e507: 8 total, 7 up, 7 in 2024-03-12T03:54:49.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:49.603 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:49.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:49.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:49.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:49.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:49.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:49.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:49.857 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:49.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:49.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:49.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:49.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:50.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:50.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:50.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:50.115 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:54:49 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a[63957]: logger=cleanup t=2024-03-12T03:54:49.739076703Z level=info msg="Completed cleanup jobs" duration=21.100641ms 2024-03-12T03:54:50.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:50.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:50.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:50.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:50.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:50.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:50.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:50 smithi169 ceph-mon[29344]: Health check failed: Reduced data availability: 8 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-03-12T03:54:50.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:50 smithi169 ceph-mon[29344]: osdmap e508: 8 total, 7 up, 7 in 2024-03-12T03:54:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:50 smithi169 ceph-mon[39949]: Health check failed: Reduced data availability: 8 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-03-12T03:54:50.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:50 smithi169 ceph-mon[39949]: osdmap e508: 8 total, 7 up, 7 in 2024-03-12T03:54:50.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:50.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:50 smithi173 ceph-mon[33544]: Health check failed: Reduced data availability: 8 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-03-12T03:54:50.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:50 smithi173 ceph-mon[33544]: osdmap e508: 8 total, 7 up, 7 in 2024-03-12T03:54:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:50.487 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:50.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:50.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:50.622 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:50.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:50.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:50.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:50.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:50.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:50.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:50.897 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:50.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:50.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:51.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:51.060 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:51.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:51.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:51.194 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:51.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:51.296 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:51.350 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:51.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:51.451 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:51.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:51.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:51.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:51.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:51.630 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:51.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:51.656 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:51.698 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:54:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:54:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:54:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:54:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:79287 10.5 10:a690a0cc:::benchmark_data_smithi169_162694_object79286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-12T03:54:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:48:57.545816+0000", 2024-03-12T03:54:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 354.142717248, 2024-03-12T03:54:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.93226661300000002, 2024-03-12T03:54:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 79287 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.545816+0000", 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.545816+0000", 2024-03-12T03:54:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.545817+0000", 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.39e-06 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.545853+0000", 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5645000000000002e-05 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.545867+0000", 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.466e-05 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.545876+0000", 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8820000000000007e-06 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.546023+0000", 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014697399999999999 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.546162+0000", 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000139178 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.546318+0000", 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000155372 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.552200+0000", 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058821300000000002 2024-03-12T03:54:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.561037+0000", 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088368700000000001 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.570037+0000", 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.009000526 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.570050+0000", 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2323e-05 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.477986+0000", 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.90793648599999999 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.478082+0000", 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6176999999999998e-05 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:79288 10.5 10:a6aed8ce:::benchmark_data_smithi169_162694_object79287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:48:57.545905+0000", 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 354.142627914, 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.932512762, 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 79288 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.545905+0000", 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.545905+0000", 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.545906+0000", 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0800000000000004e-07 2024-03-12T03:54:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.545926+0000", 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0324999999999999e-05 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.545934+0000", 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6070000000000002e-06 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.545940+0000", 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5379999999999999e-06 2024-03-12T03:54:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.549217+0000", 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032764389999999999 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.549364+0000", 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014738800000000001 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.549504+0000", 2024-03-12T03:54:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000139746 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.554077+0000", 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045732400000000001 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.570092+0000", 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016015171000000002 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.570100+0000", 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2290000000000003e-06 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.478346+0000", 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.908246043 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.478418+0000", 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1427999999999994e-05 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:79298 10.5 10:a5dc4a4e:::benchmark_data_smithi169_162694_object79297:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:48:57.554235+0000", 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 354.13429774299999, 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.93728345899999999, 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 79298 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.554235+0000", 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.554235+0000", 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.554237+0000", 2024-03-12T03:54:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5579999999999999e-06 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.554263+0000", 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6553000000000001e-05 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.554279+0000", 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5594e-05 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.554288+0000", 2024-03-12T03:54:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4509999999999998e-06 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.554369+0000", 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0197000000000005e-05 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.554646+0000", 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00027754100000000002 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,3", 2024-03-12T03:54:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.554802+0000", 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015608500000000001 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.565794+0000", 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010991712000000001 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.570767+0000", 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049733479999999998 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.570777+0000", 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.825e-06 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.478452+0000", 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.90767457900000004 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for degraded object", 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.481593+0000", 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0031416830000000001 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.491426+0000", 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.009832627 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.491519+0000", 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2706000000000003e-05 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:79300 10.8 10:133201be:::benchmark_data_smithi169_162694_object79299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:48:57.571046+0000", 2024-03-12T03:54:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 354.11748731300003, 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.91800410899999996, 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 79300 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.571046+0000", 2024-03-12T03:54:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.571046+0000", 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.571047+0000", 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.575e-06 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.571073+0000", 2024-03-12T03:54:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5619e-05 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.571082+0000", 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5990000000000008e-06 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.571090+0000", 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6109999999999996e-06 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.572035+0000", 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00094537199999999999 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.572044+0000", 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1950000000000003e-06 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.469918+0000", 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89787405600000003 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.472449+0000", 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002530844 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.472603+0000", 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000154487 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.479642+0000", 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0070390230000000002 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.488926+0000", 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0092841020000000007 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.488982+0000", 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5427000000000003e-05 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.489050+0000", 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8199000000000002e-05 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25463.0:79302 10.8 10:1a6cca97:::benchmark_data_smithi169_162694_object79301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:54:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:48:57.575228+0000", 2024-03-12T03:54:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 354.11330527000001, 2024-03-12T03:54:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.91391460999999996, 2024-03-12T03:54:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25463", 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 79302 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.575228+0000", 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.575228+0000", 2024-03-12T03:54:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.575229+0000", 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6300000000000001e-06 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.575249+0000", 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9236999999999999e-05 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.575259+0000", 2024-03-12T03:54:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0251999999999999e-05 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.575269+0000", 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7780000000000002e-06 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.575373+0000", 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103949 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:57.575381+0000", 2024-03-12T03:54:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3969999999999994e-06 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.475897+0000", 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.90051562500000004 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.476057+0000", 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016002900000000001 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.476235+0000", 2024-03-12T03:54:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017809799999999999 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.480280+0000", 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040450269999999997 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.489064+0000", 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087843049999999992 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.489090+0000", 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5473000000000002e-05 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:48:58.489142+0000", 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2809999999999998e-05 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:101445 11.10 11:08d991c4:::benchmark_data_smithi169_185434_object101444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e466)", 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:53:05.954250+0000", 2024-03-12T03:54:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 105.734283268, 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.98236237199999998, 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 101445 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.954250+0000", 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.954250+0000", 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.954252+0000", 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9290000000000001e-06 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.954275+0000", 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3726999999999998e-05 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.954291+0000", 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5109000000000001e-05 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.954299+0000", 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8109999999999992e-06 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.954737+0000", 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000437334 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.954748+0000", 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1766000000000001e-05 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.927087+0000", 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97233841700000001 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.927246+0000", 2024-03-12T03:54:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000158985 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,6", 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.927420+0000", 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017382899999999999 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.930479+0000", 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0030592369999999998 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.930493+0000", 2024-03-12T03:54:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4217e-05 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.936507+0000", 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006013661 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.936547+0000", 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9904000000000003e-05 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.936612+0000", 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5445999999999996e-05 2024-03-12T03:54:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:101460 11.10 11:09b90254:::benchmark_data_smithi169_185434_object101459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e466)", 2024-03-12T03:54:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:53:05.969501+0000", 2024-03-12T03:54:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 105.719031903, 2024-03-12T03:54:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96747004700000006, 2024-03-12T03:54:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 101460 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.969501+0000", 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.969501+0000", 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.969503+0000", 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4619999999999999e-06 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.969525+0000", 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2223000000000001e-05 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.969540+0000", 2024-03-12T03:54:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4803999999999999e-05 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.969548+0000", 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1990000000000006e-06 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.969610+0000", 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2225999999999997e-05 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.969621+0000", 2024-03-12T03:54:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0803e-05 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.930519+0000", 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96089848700000002 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.930649+0000", 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012936100000000001 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,6", 2024-03-12T03:54:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.930805+0000", 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015662299999999999 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.936365+0000", 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055600600000000003 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.936410+0000", 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4591e-05 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.936731+0000", 2024-03-12T03:54:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00032112800000000001 2024-03-12T03:54:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.936845+0000", 2024-03-12T03:54:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000114047 2024-03-12T03:54:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.936971+0000", 2024-03-12T03:54:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000126033 2024-03-12T03:54:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:101481 11.10 11:0d1b05e7:::benchmark_data_smithi169_185434_object101480:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e466)", 2024-03-12T03:54:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:53:05.989812+0000", 2024-03-12T03:54:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 105.698721253, 2024-03-12T03:54:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.956553388, 2024-03-12T03:54:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 101481 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.989812+0000", 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.989812+0000", 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.989813+0000", 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.192e-06 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.989841+0000", 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8229999999999999e-05 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.989851+0000", 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8309999999999996e-06 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.989859+0000", 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6190000000000002e-06 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.989924+0000", 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5085000000000005e-05 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.989934+0000", 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0013999999999999e-05 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.932385+0000", 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.94245166800000002 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.932536+0000", 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000150641 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,6", 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.932698+0000", 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000162375 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.936423+0000", 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0037248049999999999 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.936465+0000", 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1557999999999998e-05 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946210+0000", 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0097455930000000003 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946266+0000", 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6113000000000003e-05 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946365+0000", 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8664000000000003e-05 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:101483 11.10 11:0edb0324:::benchmark_data_smithi169_185434_object101482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e466)", 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:53:05.992575+0000", 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 105.69595826600001, 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.95401771499999999, 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 101483 2024-03-12T03:54:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.992575+0000", 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.992575+0000", 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.992576+0000", 2024-03-12T03:54:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2800000000000005e-07 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.992597+0000", 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1512999999999999e-05 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.992609+0000", 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1336e-05 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.992617+0000", 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3620000000000006e-06 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.992763+0000", 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014584600000000001 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:05.992773+0000", 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0730999999999999e-05 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.934360+0000", 2024-03-12T03:54:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.941586068 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.934491+0000", 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000131058 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,6", 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.934644+0000", 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000153709 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.936428+0000", 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017837829999999999 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.936676+0000", 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000247996 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946457+0000", 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0097813339999999992 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946504+0000", 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6093000000000001e-05 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946592+0000", 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8957999999999998e-05 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:101512 11.10 11:0b4827cb:::benchmark_data_smithi169_185434_object101511:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e466)", 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:53:06.021078+0000", 2024-03-12T03:54:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 105.667454947, 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.925720082, 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 101512 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.021078+0000", 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.021078+0000", 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.021080+0000", 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7999999999999999e-06 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.021103+0000", 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2710000000000001e-05 2024-03-12T03:54:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.021115+0000", 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2428e-05 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.021125+0000", 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9119999999999993e-06 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.021193+0000", 2024-03-12T03:54:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8559999999999994e-05 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.021205+0000", 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1027e-05 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.937017+0000", 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.91581201899999998 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.937163+0000", 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014606499999999999 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,6", 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.937337+0000", 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017450300000000001 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946406+0000", 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090693709999999997 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946418+0000", 2024-03-12T03:54:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.166e-05 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946665+0000", 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000246829 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946715+0000", 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0429000000000001e-05 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946798+0000", 2024-03-12T03:54:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2769000000000007e-05 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:101569 11.10 11:0bf41b26:::benchmark_data_smithi169_185434_object101568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e466)", 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:53:06.083246+0000", 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 105.605287082, 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.86389339899999995, 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 101569 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.083246+0000", 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.083246+0000", 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.083247+0000", 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3379999999999999e-06 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.083272+0000", 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4525e-05 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.083284+0000", 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2359e-05 2024-03-12T03:54:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.083292+0000", 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8390000000000007e-06 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.083354+0000", 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2145999999999998e-05 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.083365+0000", 2024-03-12T03:54:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0838e-05 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.938762+0000", 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.85539733799999995 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.938907+0000", 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014491999999999999 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,6", 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.939041+0000", 2024-03-12T03:54:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013415399999999999 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946424+0000", 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0073821720000000002 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946629+0000", 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000205441 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.947016+0000", 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038748800000000001 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.947060+0000", 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3331999999999998e-05 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.947139+0000", 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9509000000000001e-05 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:101647 11.10 11:0c2ce655:::benchmark_data_smithi169_185434_object101646:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e466)", 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:53:06.250424+0000", 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 105.43810931900001, 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.70169462699999996, 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 101647 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.250424+0000", 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.250424+0000", 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.250426+0000", 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3860000000000001e-06 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.250446+0000", 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0267000000000001e-05 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.250455+0000", 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1370000000000008e-06 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.250463+0000", 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1450000000000002e-06 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.250521+0000", 2024-03-12T03:54:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8319000000000002e-05 2024-03-12T03:54:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.250532+0000", 2024-03-12T03:54:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1165999999999999e-05 2024-03-12T03:54:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.940771+0000", 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.69023907900000003 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.940903+0000", 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013195000000000001 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,6", 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.941057+0000", 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015391800000000001 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946428+0000", 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053713709999999998 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.946982+0000", 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00055371299999999995 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.951971+0000", 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049887810000000003 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.952026+0000", 2024-03-12T03:54:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4928e-05 2024-03-12T03:54:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.952118+0000", 2024-03-12T03:54:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2466999999999994e-05 2024-03-12T03:54:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:101670 11.10 11:0ac67815:::benchmark_data_smithi169_185434_object101669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e466)", 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:53:06.271495+0000", 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 105.417037943, 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.68744376100000004, 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 101670 2024-03-12T03:54:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.271495+0000", 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.271495+0000", 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.271496+0000", 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.223e-06 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.271520+0000", 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3519e-05 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.271532+0000", 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2670999999999999e-05 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.271540+0000", 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7870000000000002e-06 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.271612+0000", 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1462000000000001e-05 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.271622+0000", 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0681e-05 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.947165+0000", 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.67554252800000003 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.947296+0000", 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013115300000000001 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,6", 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.947521+0000", 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00022534700000000001 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.952424+0000", 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004902661 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.952464+0000", 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9832000000000001e-05 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.958677+0000", 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062134579999999998 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.958730+0000", 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2518000000000001e-05 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.958939+0000", 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020892099999999999 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:101718 11.10 11:0c9f1d20:::benchmark_data_smithi169_185434_object101717:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e466)", 2024-03-12T03:54:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:53:06.327563+0000", 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 105.36096966700001, 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63918967500000001, 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 101718 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.327563+0000", 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.327563+0000", 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.327564+0000", 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7399999999999991e-07 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.327585+0000", 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1036e-05 2024-03-12T03:54:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.327610+0000", 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4189000000000002e-05 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.327644+0000", 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4894999999999997e-05 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.327776+0000", 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013143200000000001 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.327786+0000", 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9939999999999995e-06 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.949209+0000", 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.62142311400000005 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.949339+0000", 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000130314 2024-03-12T03:54:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,6", 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.949539+0000", 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000200037 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.952474+0000", 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029348450000000002 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.952506+0000", 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1387000000000003e-05 2024-03-12T03:54:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.966585+0000", 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014079533 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.966637+0000", 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2269999999999999e-05 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.966753+0000", 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115655 2024-03-12T03:54:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:101727 11.10 11:0d30b243:::benchmark_data_smithi169_185434_object101726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e466)", 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:53:06.341459+0000", 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 105.347074138, 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63091026900000002, 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 101727 2024-03-12T03:54:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.341459+0000", 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.341459+0000", 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.341460+0000", 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2589999999999999e-06 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.341480+0000", 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9898000000000002e-05 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.341494+0000", 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3920999999999999e-05 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.341501+0000", 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3679999999999996e-06 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.341563+0000", 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1507999999999995e-05 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.341573+0000", 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8700000000000004e-06 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.952533+0000", 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.61096074600000005 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.952669+0000", 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013580299999999999 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,6", 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.952808+0000", 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013892700000000001 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.958989+0000", 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061811499999999998 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.966840+0000", 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078508350000000005 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.972203+0000", 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005362688 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.972259+0000", 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6542999999999997e-05 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:53:06.972369+0000", 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109753 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24359.0:4764 5.d 5:b4812045:::notify.4:head [watch ping cookie 94067641429248 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-03-12T03:54:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:46.880154+0000", 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.8083794309999996, 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2483559900000001, 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24359", 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4764 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880154+0000", 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880154+0000", 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880155+0000", 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7600000000000001e-06 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880163+0000", 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5789999999999998e-06 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880183+0000", 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0369000000000001e-05 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880195+0000", 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1942e-05 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880328+0000", 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000132946 2024-03-12T03:54:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880351+0000", 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3127999999999999e-05 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.082603+0000", 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20225183199999999 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.082614+0000", 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0372e-05 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.128190+0000", 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.045576294 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.128364+0000", 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000174418 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.128510+0000", 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014535000000000001 2024-03-12T03:54:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24359.0:4773 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94067641429248 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:46.880767+0000", 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.8077655230000005, 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2543622210000001, 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24359", 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:54:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4773 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880767+0000", 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880767+0000", 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880768+0000", 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2399999999999996e-07 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880775+0000", 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1349999999999997e-06 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880789+0000", 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.395e-05 2024-03-12T03:54:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880798+0000", 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1819999999999995e-06 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880869+0000", 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1614999999999998e-05 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880881+0000", 2024-03-12T03:54:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2111e-05 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.082641+0000", 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20175936 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.082649+0000", 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4339999999999992e-06 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.128540+0000", 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0458909839999999 2024-03-12T03:54:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.128608+0000", 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8040999999999994e-05 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.135130+0000", 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0065214849999999996 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24359.0:4775 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94067641428096 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:46.880816+0000", 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.8077171950000004, 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2534485659999999, 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24359", 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4775 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880816+0000", 2024-03-12T03:54:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880816+0000", 2024-03-12T03:54:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880816+0000", 2024-03-12T03:54:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4600000000000005e-07 2024-03-12T03:54:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880820+0000", 2024-03-12T03:54:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7069999999999998e-06 2024-03-12T03:54:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880828+0000", 2024-03-12T03:54:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4420000000000001e-06 2024-03-12T03:54:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880841+0000", 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3269e-05 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880893+0000", 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.215e-05 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880899+0000", 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9660000000000001e-06 2024-03-12T03:54:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.080147+0000", 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19924840799999999 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.080158+0000", 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0501e-05 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.117807+0000", 2024-03-12T03:54:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.037649233 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for missing object", 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.117977+0000", 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017037700000000001 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.123859+0000", 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058811849999999997 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.123967+0000", 2024-03-12T03:54:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000107989 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.124356+0000", 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038987899999999999 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.124426+0000", 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9438999999999997e-05 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.134264+0000", 2024-03-12T03:54:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0098384749999999993 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24359.0:4778 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94067641428096 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e505)", 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:47.083016+0000", 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.6055166930000002, 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.051398115, 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24359", 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:54:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4778 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083016+0000", 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083016+0000", 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083018+0000", 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6360000000000001e-06 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083022+0000", 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6169999999999998e-06 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083030+0000", 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2879999999999993e-06 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083039+0000", 2024-03-12T03:54:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3649999999999993e-06 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083181+0000", 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000141617 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083189+0000", 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3969999999999994e-06 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.118274+0000", 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.035084895 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for missing object", 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.118324+0000", 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9870999999999997e-05 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.124005+0000", 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056813799999999998 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.124098+0000", 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2417000000000006e-05 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.124768+0000", 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00067063099999999998 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.124854+0000", 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5866999999999996e-05 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.134414+0000", 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0095601339999999996 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24359.0:4779 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94067641429248 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e505)", 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:47.083057+0000", 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.6054761610000003, 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0522011840000001, 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:54:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24359", 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4779 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083057+0000", 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083057+0000", 2024-03-12T03:54:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083057+0000", 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6899999999999998e-07 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083064+0000", 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4409999999999998e-06 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083070+0000", 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1299999999999998e-06 2024-03-12T03:54:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.083076+0000", 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5490000000000003e-06 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.085559+0000", 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024826610000000002 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.085570+0000", 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1099e-05 2024-03-12T03:54:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.129002+0000", 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.043432216 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.129105+0000", 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000102768 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.135258+0000", 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006152851 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:54:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:54:51.830 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:54:51.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:51 smithi169 ceph-mon[29344]: pgmap v1053: 176 pgs: 1 active+remapped+backfill_toofull, 4 unknown, 13 peering, 12 remapped+peering, 21 active+clean+remapped, 16 active+undersized+degraded, 21 active+undersized, 88 active+clean; 246 MiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.24k op/s; 758/12471 objects degraded (6.078%); 804/12471 objects misplaced (6.447%); 80 KiB/s, 3 objects/s recovering 2024-03-12T03:54:51.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:51 smithi169 ceph-mon[29344]: osdmap e509: 8 total, 7 up, 7 in 2024-03-12T03:54:51.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:51 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:54:51.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:51 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:54:51.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:51 smithi169 ceph-mon[29344]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:54:51.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:51 smithi169 ceph-mon[39949]: pgmap v1053: 176 pgs: 1 active+remapped+backfill_toofull, 4 unknown, 13 peering, 12 remapped+peering, 21 active+clean+remapped, 16 active+undersized+degraded, 21 active+undersized, 88 active+clean; 246 MiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.24k op/s; 758/12471 objects degraded (6.078%); 804/12471 objects misplaced (6.447%); 80 KiB/s, 3 objects/s recovering 2024-03-12T03:54:51.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:51 smithi169 ceph-mon[39949]: osdmap e509: 8 total, 7 up, 7 in 2024-03-12T03:54:51.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:51 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:54:51.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:51 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:54:51.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:51 smithi169 ceph-mon[39949]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:54:51.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:51.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:51.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:51.910 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Removed 111944 objects 2024-03-12T03:54:51.910 INFO:tasks.radosbench.radosbench.0.smithi169.stdout:Clean up completed and total clean up time :91.7973 2024-03-12T03:54:51.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:51.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:51.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:51 smithi173 ceph-mon[33544]: pgmap v1053: 176 pgs: 1 active+remapped+backfill_toofull, 4 unknown, 13 peering, 12 remapped+peering, 21 active+clean+remapped, 16 active+undersized+degraded, 21 active+undersized, 88 active+clean; 246 MiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.24k op/s; 758/12471 objects degraded (6.078%); 804/12471 objects misplaced (6.447%); 80 KiB/s, 3 objects/s recovering 2024-03-12T03:54:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:51 smithi173 ceph-mon[33544]: osdmap e509: 8 total, 7 up, 7 in 2024-03-12T03:54:51.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:51 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:54:51.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:51 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:54:51.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:51 smithi173 ceph-mon[33544]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:54:52.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:52.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:52.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:52.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:54:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:54:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:54:52.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:52.211 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:52.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:52.312 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:52.336 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:52.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:52.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:52.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:52.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:52.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:52.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:52.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:52.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:52.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:52.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:52.714 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:52.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:52.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:52.841 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:52.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:52.875 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:52 smithi169 ceph-mon[29344]: osdmap e510: 8 total, 7 up, 7 in 2024-03-12T03:54:52.875 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:52 smithi169 ceph-mon[39949]: osdmap e510: 8 total, 7 up, 7 in 2024-03-12T03:54:52.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:52.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:52.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:52.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:52 smithi173 ceph-mon[33544]: osdmap e510: 8 total, 7 up, 7 in 2024-03-12T03:54:52.987 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-12T03:54:52.987 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-12T03:54:53.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-12T03:54:53.067 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:53.101 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:53.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:53.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:53.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:53.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:53.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:53.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:53.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:53.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:53.354 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:53.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:53.454 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:53.483 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:53.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:53.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:53.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:53.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:53.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:53.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:53.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:53.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:53.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:53.860 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:53.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:53.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:53 smithi169 ceph-mon[29344]: pgmap v1056: 176 pgs: 2 active+remapped+backfill_toofull, 2 peering, 11 remapped+peering, 26 active+clean+remapped, 16 active+undersized+degraded, 21 active+undersized, 98 active+clean; 151 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 717 op/s; 758/7911 objects degraded (9.582%); 624/7911 objects misplaced (7.888%); 5.5 MiB/s, 96 objects/s recovering 2024-03-12T03:54:53.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:53 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:54:53.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:53 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:54:53.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:53 smithi169 ceph-mon[29344]: osdmap e511: 8 total, 7 up, 7 in 2024-03-12T03:54:53.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:53 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 758/7911 objects degraded (9.582%), 16 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:53.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:53 smithi169 ceph-mon[39949]: pgmap v1056: 176 pgs: 2 active+remapped+backfill_toofull, 2 peering, 11 remapped+peering, 26 active+clean+remapped, 16 active+undersized+degraded, 21 active+undersized, 98 active+clean; 151 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 717 op/s; 758/7911 objects degraded (9.582%); 624/7911 objects misplaced (7.888%); 5.5 MiB/s, 96 objects/s recovering 2024-03-12T03:54:53.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:53 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:54:53.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:53 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:54:53.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:53 smithi169 ceph-mon[39949]: osdmap e511: 8 total, 7 up, 7 in 2024-03-12T03:54:53.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:53 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 758/7911 objects degraded (9.582%), 16 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:53.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:54:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:54:53.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:53 smithi173 ceph-mon[33544]: pgmap v1056: 176 pgs: 2 active+remapped+backfill_toofull, 2 peering, 11 remapped+peering, 26 active+clean+remapped, 16 active+undersized+degraded, 21 active+undersized, 98 active+clean; 151 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 717 op/s; 758/7911 objects degraded (9.582%); 624/7911 objects misplaced (7.888%); 5.5 MiB/s, 96 objects/s recovering 2024-03-12T03:54:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:53 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:54:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:53 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:54:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:53 smithi173 ceph-mon[33544]: osdmap e511: 8 total, 7 up, 7 in 2024-03-12T03:54:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:53 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 758/7911 objects degraded (9.582%), 16 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:53.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:54.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:54.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:54.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-12T03:54:54.106 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:54.119 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:54.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:54.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:54.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:54.247 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 6, 7, 2] out_osds: [5] dead_osds: [0] live_osds: [1, 2, 4, 5, 6, 7, 3] 2024-03-12T03:54:54.247 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-12T03:54:54.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:54:54.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:54.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:54.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:54.291 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:54.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:54.388 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:54:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:54.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:54.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:54.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:54.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:54.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:54.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:54.659 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:54.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:54.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:54.782 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:54 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:54.787 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:54.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:54.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:54 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:54.809 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:54 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:54.888 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:54.914 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:54.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:54.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:54.931 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:54:54.950 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:54:54.970 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:54:54.990 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:54:55.011 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:54:55.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:55.031 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:54:55.038 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:55.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:55.052 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:54:55.053 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:54:55.053 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:54:55.053 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:54:55.053 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:54:55.053 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:54:55.053 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:54:55.053 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:54:55.053 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:54:55.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:55.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:55.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:55.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:55.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:55.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:55.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:55.394 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:55.419 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:55.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:55.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:55.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:55.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:55.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:55.671 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:55.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:55.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:55.773 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:55.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:55.797 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:55.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:55.859 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:55 smithi169 ceph-mon[39949]: pgmap v1058: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 15 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 795 op/s; 126/1377 objects degraded (9.150%); 85/1377 objects misplaced (6.173%); 4.7 MiB/s, 79 objects/s recovering 2024-03-12T03:54:55.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:55 smithi169 ceph-mon[39949]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2024-03-12T03:54:55.860 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:55 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1301912112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:54:55.860 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:55 smithi169 ceph-mon[29344]: pgmap v1058: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 15 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 795 op/s; 126/1377 objects degraded (9.150%); 85/1377 objects misplaced (6.173%); 4.7 MiB/s, 79 objects/s recovering 2024-03-12T03:54:55.860 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:55 smithi169 ceph-mon[29344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2024-03-12T03:54:55.860 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:55 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1301912112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:54:55.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:55.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:55.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:55.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:55.924 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:55 smithi173 ceph-mon[33544]: pgmap v1058: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 15 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 795 op/s; 126/1377 objects degraded (9.150%); 85/1377 objects misplaced (6.173%); 4.7 MiB/s, 79 objects/s recovering 2024-03-12T03:54:55.924 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:55 smithi173 ceph-mon[33544]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2024-03-12T03:54:55.924 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:55 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1301912112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:54:56.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:56.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:56.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:56.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:56.175 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:56.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:56.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:56.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:56.302 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:56.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:56.316 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:54:56.316 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:54:56.403 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:56.427 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:56.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:56.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:56.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:56.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:56.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:56.611 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:54:56.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:54:56.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:56.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:56.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:56.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:56.807 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:56.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:56.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:56.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:56.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:57.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:57.059 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:57.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:57.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-12T03:54:57.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:54:57.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:57.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:57.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:54:57.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:57.322 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:57.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:57.373 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:54:57.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:57.448 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:57.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:57.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:57.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:57.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:57.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:57.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:57 smithi169 ceph-mon[29344]: pgmap v1059: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 15 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 628 op/s; 126/1377 objects degraded (9.150%); 85/1377 objects misplaced (6.173%); 3.7 MiB/s, 62 objects/s recovering 2024-03-12T03:54:57.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:57 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:54:57.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:57 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:54:57.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:57 smithi169 ceph-mon[39949]: pgmap v1059: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 15 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 628 op/s; 126/1377 objects degraded (9.150%); 85/1377 objects misplaced (6.173%); 3.7 MiB/s, 62 objects/s recovering 2024-03-12T03:54:57.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:57 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:54:57.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:57 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:54:57.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:57.700 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:57.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:57.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:57 smithi173 ceph-mon[33544]: pgmap v1059: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 15 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 628 op/s; 126/1377 objects degraded (9.150%); 85/1377 objects misplaced (6.173%); 3.7 MiB/s, 62 objects/s recovering 2024-03-12T03:54:57.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:57 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:54:57.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:57 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:54:57.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:57.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:57.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:57.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:57.953 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:57.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:58.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:58.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:58.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:58.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:58.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:58.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:58.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:58.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:58.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:58.330 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:58.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:58.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:58.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:58.455 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:58.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:58.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:58.562 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is set 2024-03-12T03:54:58.584 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:58.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:58.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-12T03:54:58.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:58.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:58.712 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:58.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:58.813 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:54:58.837 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:58.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:58.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:54:58.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:58 smithi173 ceph-mon[33544]: osdmap e512: 8 total, 7 up, 7 in 2024-03-12T03:54:58.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:58 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2531801435' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:54:58.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:58 smithi173 ceph-mon[33544]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:54:58.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:58 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 126/1377 objects degraded (9.150%), 13 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:58.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:58 smithi173 ceph-mon[33544]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:54:58.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:58 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2531801435' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:54:58.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:58 smithi173 ceph-mon[33544]: osdmap e513: 8 total, 7 up, 7 in 2024-03-12T03:54:58.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[29344]: osdmap e512: 8 total, 7 up, 7 in 2024-03-12T03:54:58.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2531801435' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:54:58.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[29344]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:54:58.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 126/1377 objects degraded (9.150%), 13 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:58.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[29344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:54:58.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2531801435' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:54:58.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[29344]: osdmap e513: 8 total, 7 up, 7 in 2024-03-12T03:54:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[39949]: osdmap e512: 8 total, 7 up, 7 in 2024-03-12T03:54:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2531801435' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:54:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[39949]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:54:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 126/1377 objects degraded (9.150%), 13 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-03-12T03:54:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[39949]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:54:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2531801435' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-12T03:54:58.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:58 smithi169 ceph-mon[39949]: osdmap e513: 8 total, 7 up, 7 in 2024-03-12T03:54:58.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:58.945 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:54:58.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:58.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:59.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:54:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:59.088 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:59.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:54:59.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:59.216 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:59.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:59.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:59.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:59.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:59.365 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:54:59.365 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:54:59.443 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:59.468 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:59.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:59.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:54:59.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:59.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:59.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:54:59.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:54:59 smithi169 ceph-mon[29344]: pgmap v1061: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 5.1 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 516 op/s; 126/903 objects degraded (13.953%); 40/903 objects misplaced (4.430%); 2.1 MiB/s, 36 objects/s recovering 2024-03-12T03:54:59.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:54:59 smithi169 ceph-mon[39949]: pgmap v1061: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 5.1 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 516 op/s; 126/903 objects degraded (13.953%); 40/903 objects misplaced (4.430%); 2.1 MiB/s, 36 objects/s recovering 2024-03-12T03:54:59.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:54:59.719 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:54:59.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:59.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:54:59.821 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:54:59.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:59.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:54:59.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:54:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:54:59.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:54:59.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:54:59.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:54:59 smithi173 ceph-mon[33544]: pgmap v1061: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 5.1 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 516 op/s; 126/903 objects degraded (13.953%); 40/903 objects misplaced (4.430%); 2.1 MiB/s, 36 objects/s recovering 2024-03-12T03:55:00.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:55:00.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:00.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:00.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:55:00.141 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:00.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-12T03:55:00.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:55:00.226 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:00.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:00.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:55:00.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:55:00.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:00.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:00.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:55:00.392 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:55:00.454 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:55:00.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:00.479 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:00.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:55:00.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:55:00.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:00.605 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:00.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:55:00.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:55:00.730 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:00.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:00.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:55:00.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:55:00.855 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:00.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:00.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:55:00.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:55:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:00.981 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:00.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:55:01.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:55:01.106 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:01.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:55:01.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:55:01.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:01.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:01.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:55:01.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:55:01.357 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:01.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:55:01.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:55:01.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:01.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:01.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:55:01.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:55:01.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:01.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:01.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:55:01.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:55:01.736 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:01.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:01.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:55:01.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:55:01.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:01 smithi173 ceph-mon[33544]: pgmap v1063: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 5.1 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 662 B/s rd, 0 B/s wr, 176 op/s; 126/903 objects degraded (13.953%); 40/903 objects misplaced (4.430%) 2024-03-12T03:55:01.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:01.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:55:01.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:01 smithi169 ceph-mon[29344]: pgmap v1063: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 5.1 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 662 B/s rd, 0 B/s wr, 176 op/s; 126/903 objects degraded (13.953%); 40/903 objects misplaced (4.430%) 2024-03-12T03:55:01.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:01 smithi169 ceph-mon[39949]: pgmap v1063: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 5.1 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 662 B/s rd, 0 B/s wr, 176 op/s; 126/903 objects degraded (13.953%); 40/903 objects misplaced (4.430%) 2024-03-12T03:55:01.924 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:55:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:55:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:55:01.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:55:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:01.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:01.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:55:02.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:55:02.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:55:02.105 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-03-12T03:55:02.105 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-12T03:55:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:02.115 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:02.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:55:02.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:55:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:02.242 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:02.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:55:02.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:55:02.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:02.369 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:02.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:55:02.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:55:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:02.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:02.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:55:02.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:55:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:02.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:02.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:55:02.624 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'unique_pool_4' removed 2024-03-12T03:55:02.642 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:02 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3859304846' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:55:02.643 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:02 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:55:02.643 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:02 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:02.643 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:02 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:02.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:02 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3859304846' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:55:02.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:02 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:55:02.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:02 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:02.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:02 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:02.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:55:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:02.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:02.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:55:02.793 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:02 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3859304846' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:55:02.793 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:02 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:55:02.793 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:02 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:02.793 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:02 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:02.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:55:02.873 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:02.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:55:02.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-12T03:55:02.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:55:02.998 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:02.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:02.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:55:03.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3 2024-03-12T03:55:03.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:03.125 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:03.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.3' 2024-03-12T03:55:03.136 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:55:03.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:55:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:03.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:03.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:55:03.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:55:03.377 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:03.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:55:03.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:55:03.504 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:03.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:55:03.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1 2024-03-12T03:55:03.630 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:03.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.1' 2024-03-12T03:55:03.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:55:03.756 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:03.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:55:03.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4 2024-03-12T03:55:03.882 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:03.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:03.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.4' 2024-03-12T03:55:03.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:03 smithi169 ceph-mon[29344]: pgmap v1064: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 5.1 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 20 op/s; 126/903 objects degraded (13.953%); 40/903 objects misplaced (4.430%) 2024-03-12T03:55:03.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:03 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:55:03.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:03 smithi169 ceph-mon[29344]: osdmap e514: 8 total, 7 up, 7 in 2024-03-12T03:55:03.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:03 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 126/903 objects degraded (13.953%), 13 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-03-12T03:55:03.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:03 smithi169 ceph-mon[29344]: osdmap e515: 8 total, 7 up, 7 in 2024-03-12T03:55:03.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:03 smithi169 ceph-mon[39949]: pgmap v1064: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 5.1 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 20 op/s; 126/903 objects degraded (13.953%); 40/903 objects misplaced (4.430%) 2024-03-12T03:55:03.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:03 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:55:03.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:03 smithi169 ceph-mon[39949]: osdmap e514: 8 total, 7 up, 7 in 2024-03-12T03:55:03.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:03 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 126/903 objects degraded (13.953%), 13 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-03-12T03:55:03.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:03 smithi169 ceph-mon[39949]: osdmap e515: 8 total, 7 up, 7 in 2024-03-12T03:55:03.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:55:03.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:55:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:55:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:03 smithi173 ceph-mon[33544]: pgmap v1064: 176 pgs: 2 active+remapped+backfill_toofull, 35 active+clean+remapped, 13 active+undersized+degraded, 24 active+undersized, 102 active+clean; 5.1 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 20 op/s; 126/903 objects degraded (13.953%); 40/903 objects misplaced (4.430%) 2024-03-12T03:55:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:03 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:55:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:03 smithi173 ceph-mon[33544]: osdmap e514: 8 total, 7 up, 7 in 2024-03-12T03:55:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:03 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 126/903 objects degraded (13.953%), 13 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-03-12T03:55:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:03 smithi173 ceph-mon[33544]: osdmap e515: 8 total, 7 up, 7 in 2024-03-12T03:55:04.011 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:04.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:04.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:55:04.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:55:04.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:04.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:55:04.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5 2024-03-12T03:55:04.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:04.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.5' 2024-03-12T03:55:04.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2 2024-03-12T03:55:04.388 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: docker: command not found 2024-03-12T03:55:04.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:04.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.2' 2024-03-12T03:55:04.443 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:55:04.446 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:55:04.446 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1: {} 2024-03-12T03:55:04.463 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2: {} 2024-03-12T03:55:04.489 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3: {} 2024-03-12T03:55:04.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:55:04.510 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4: {} 2024-03-12T03:55:04.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:04.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:55:04.529 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5: {} 2024-03-12T03:55:04.549 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6: {} 2024-03-12T03:55:04.568 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7: {} 2024-03-12T03:55:04.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-12T03:55:04.569 INFO:teuthology.orchestra.run.smithi169.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-12T03:55:04.569 INFO:teuthology.orchestra.run.smithi169.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-12T03:55:04.569 INFO:teuthology.orchestra.run.smithi169.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-12T03:55:04.569 INFO:teuthology.orchestra.run.smithi169.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-12T03:55:04.569 INFO:teuthology.orchestra.run.smithi169.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-12T03:55:04.569 INFO:teuthology.orchestra.run.smithi169.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-12T03:55:04.569 INFO:teuthology.orchestra.run.smithi169.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-12T03:55:04.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:55:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:04.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:04.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:55:04.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7 2024-03-12T03:55:04.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:04.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:04.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.7' 2024-03-12T03:55:04.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6 2024-03-12T03:55:04.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-03-12T03:55:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:55:04.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd.6' 2024-03-12T03:55:04.900 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:55:04.900 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:55:04.900 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-03-12T03:55:04.901 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:55:04.901 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:04.901 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:04 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4268647186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:55:04.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:04.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:04 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4268647186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:55:04.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:04.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:04 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4268647186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-12T03:55:04.959 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-12T03:55:04.971 INFO:tasks.thrashosds:joining thrashosds 2024-03-12T03:55:05.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-12T03:55:05.804 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:55:05.834 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:05 smithi169 ceph-mon[29344]: pgmap v1067: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:05.835 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:05 smithi169 ceph-mon[29344]: from='client.15945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:55:05.835 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:05 smithi169 ceph-mon[29344]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-12T03:55:05.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:05 smithi169 ceph-mon[39949]: pgmap v1067: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:05.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:05 smithi169 ceph-mon[39949]: from='client.15945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:55:05.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:05 smithi169 ceph-mon[39949]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-12T03:55:05.863 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":1067,"stamp":"2024-03-12T03:55:04.308563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":623227,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":55,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":3133,"num_read_kb":2857,"num_write":1208,"num_write_kb":155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":1530492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3540,"ondisk_log_size":3540,"up":389,"acting":418,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":527,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":9056196,"kb_used_data":3142504,"kb_used_omap":22,"kb_used_meta":5913577,"kb_avail":647159868,"statfs":{"total":671965249536,"available":662691704832,"internally_reserved":0,"allocated":3217924096,"data_stored":3211682340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23049,"internal_metadata":6055503351},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4685824,"num_objects":-71,"num_object_clones":0,"num_object_copies":-213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-71,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-71,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-157452,"num_write_kb":-4194369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-350,"num_bytes_recovered":-22937600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002230"},"pg_stats":[{"pgid":"1.9","version":"83'128","reported_seq":1293,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.898559+0000","last_change":"2024-03-12T03:54:43.679158+0000","last_active":"2024-03-12T03:54:58.898559+0000","last_peered":"2024-03-12T03:54:58.898559+0000","last_clean":"2024-03-12T03:54:58.898559+0000","last_became_active":"2024-03-12T03:46:25.289005+0000","last_became_peered":"2024-03-12T03:46:25.289005+0000","last_unstale":"2024-03-12T03:54:58.898559+0000","last_undegraded":"2024-03-12T03:54:58.898559+0000","last_fullsized":"2024-03-12T03:54:58.898559+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:54:43.678917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:54:43.678917+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:49.350568+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"480'35","reported_seq":844,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619072+0000","last_change":"2024-03-12T03:51:46.582688+0000","last_active":"2024-03-12T03:55:02.619072+0000","last_peered":"2024-03-12T03:55:02.619072+0000","last_clean":"2024-03-12T03:55:02.619072+0000","last_became_active":"2024-03-12T03:51:42.838917+0000","last_became_peered":"2024-03-12T03:51:42.838917+0000","last_unstale":"2024-03-12T03:55:02.619072+0000","last_undegraded":"2024-03-12T03:55:02.619072+0000","last_fullsized":"2024-03-12T03:55:02.619072+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'25","last_scrub_stamp":"2024-03-12T03:51:46.582596+0000","last_deep_scrub":"344'25","last_deep_scrub_stamp":"2024-03-12T03:51:46.582596+0000","last_clean_scrub_stamp":"2024-03-12T03:51:46.582596+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":731,"reported_epoch":514,"state":"active+undersized","last_fresh":"2024-03-12T03:55:02.621111+0000","last_change":"2024-03-12T03:48:58.454102+0000","last_active":"2024-03-12T03:55:02.621111+0000","last_peered":"2024-03-12T03:55:02.621111+0000","last_clean":"2024-03-12T03:48:53.448575+0000","last_became_active":"2024-03-12T03:48:58.454102+0000","last_became_peered":"2024-03-12T03:48:58.454102+0000","last_unstale":"2024-03-12T03:55:02.621111+0000","last_undegraded":"2024-03-12T03:55:02.621111+0000","last_fullsized":"2024-03-12T03:48:58.443116+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:07.363439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:07.363439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"506'12","reported_seq":906,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.818291+0000","last_change":"2024-03-12T03:54:48.127948+0000","last_active":"2024-03-12T03:54:58.818291+0000","last_peered":"2024-03-12T03:54:58.818291+0000","last_clean":"2024-03-12T03:54:58.818291+0000","last_became_active":"2024-03-12T03:54:48.117302+0000","last_became_peered":"2024-03-12T03:54:48.117302+0000","last_unstale":"2024-03-12T03:54:58.818291+0000","last_undegraded":"2024-03-12T03:54:58.818291+0000","last_fullsized":"2024-03-12T03:54:58.818291+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'10","last_scrub_stamp":"2024-03-12T03:54:34.569905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:54:34.569905+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:59.564313+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7,5],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":449,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.234128+0000","last_change":"2024-03-12T03:54:26.474267+0000","last_active":"2024-03-12T03:55:03.234128+0000","last_peered":"2024-03-12T03:55:03.234128+0000","last_clean":"2024-03-12T03:55:03.234128+0000","last_became_active":"2024-03-12T03:51:42.832276+0000","last_became_peered":"2024-03-12T03:51:42.832276+0000","last_unstale":"2024-03-12T03:55:03.234128+0000","last_undegraded":"2024-03-12T03:55:03.234128+0000","last_fullsized":"2024-03-12T03:55:03.234128+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:26.474168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:54:26.474168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:53.937568+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"480'45","reported_seq":728,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.818123+0000","last_change":"2024-03-12T03:54:18.511358+0000","last_active":"2024-03-12T03:54:58.818123+0000","last_peered":"2024-03-12T03:54:58.818123+0000","last_clean":"2024-03-12T03:54:58.818123+0000","last_became_active":"2024-03-12T03:51:42.832275+0000","last_became_peered":"2024-03-12T03:51:42.832275+0000","last_unstale":"2024-03-12T03:54:58.818123+0000","last_undegraded":"2024-03-12T03:54:58.818123+0000","last_fullsized":"2024-03-12T03:54:58.818123+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'45","last_scrub_stamp":"2024-03-12T03:54:18.511135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:54:18.511135+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:46.406031+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":438,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.926192+0000","last_change":"2024-03-12T03:54:31.473211+0000","last_active":"2024-03-12T03:54:58.926192+0000","last_peered":"2024-03-12T03:54:58.926192+0000","last_clean":"2024-03-12T03:54:58.926192+0000","last_became_active":"2024-03-12T03:45:18.108377+0000","last_became_peered":"2024-03-12T03:45:18.108377+0000","last_unstale":"2024-03-12T03:54:58.926192+0000","last_undegraded":"2024-03-12T03:54:58.926192+0000","last_fullsized":"2024-03-12T03:54:58.926192+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:31.473101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:54:31.473101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:55.922322+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":955,"reported_epoch":514,"state":"active+undersized","last_fresh":"2024-03-12T03:55:03.233981+0000","last_change":"2024-03-12T03:48:58.457284+0000","last_active":"2024-03-12T03:55:03.233981+0000","last_peered":"2024-03-12T03:55:03.233981+0000","last_clean":"2024-03-12T03:48:53.449180+0000","last_became_active":"2024-03-12T03:48:58.457284+0000","last_became_peered":"2024-03-12T03:48:58.457284+0000","last_unstale":"2024-03-12T03:55:03.233981+0000","last_undegraded":"2024-03-12T03:55:03.233981+0000","last_fullsized":"2024-03-12T03:48:58.441935+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:04.054936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:34.824210+0000","last_clean_scrub_stamp":"2024-03-12T03:45:04.054936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261470+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":720,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.818288+0000","last_change":"2024-03-12T03:54:27.794668+0000","last_active":"2024-03-12T03:54:58.818288+0000","last_peered":"2024-03-12T03:54:58.818288+0000","last_clean":"2024-03-12T03:54:58.818288+0000","last_became_active":"2024-03-12T03:46:25.293293+0000","last_became_peered":"2024-03-12T03:46:25.293293+0000","last_unstale":"2024-03-12T03:54:58.818288+0000","last_undegraded":"2024-03-12T03:54:58.818288+0000","last_fullsized":"2024-03-12T03:54:58.818288+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:27.794446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:54:27.794446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:46.102409+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"388'129","reported_seq":889,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.817186+0000","last_change":"2024-03-12T03:54:49.580359+0000","last_active":"2024-03-12T03:54:58.817186+0000","last_peered":"2024-03-12T03:54:58.817186+0000","last_clean":"2024-03-12T03:54:58.817186+0000","last_became_active":"2024-03-12T03:54:49.579887+0000","last_became_peered":"2024-03-12T03:54:49.579887+0000","last_unstale":"2024-03-12T03:54:58.817186+0000","last_undegraded":"2024-03-12T03:54:58.817186+0000","last_fullsized":"2024-03-12T03:54:58.817186+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":4,"last_scrub":"388'129","last_scrub_stamp":"2024-03-12T03:54:37.218528+0000","last_deep_scrub":"388'129","last_deep_scrub_stamp":"2024-03-12T03:49:17.168603+0000","last_clean_scrub_stamp":"2024-03-12T03:54:37.218528+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:58.073601+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":1116,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.232613+0000","last_change":"2024-03-12T03:51:30.025307+0000","last_active":"2024-03-12T03:55:03.232613+0000","last_peered":"2024-03-12T03:55:03.232613+0000","last_clean":"2024-03-12T03:55:03.232613+0000","last_became_active":"2024-03-12T03:50:58.842635+0000","last_became_peered":"2024-03-12T03:50:58.842635+0000","last_unstale":"2024-03-12T03:55:03.232613+0000","last_undegraded":"2024-03-12T03:55:03.232613+0000","last_fullsized":"2024-03-12T03:55:03.232613+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:30.025220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:51:30.025220+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"480'39","reported_seq":673,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.234921+0000","last_change":"2024-03-12T03:54:15.543952+0000","last_active":"2024-03-12T03:55:03.234921+0000","last_peered":"2024-03-12T03:55:03.234921+0000","last_clean":"2024-03-12T03:55:03.234921+0000","last_became_active":"2024-03-12T03:51:42.834094+0000","last_became_peered":"2024-03-12T03:51:42.834094+0000","last_unstale":"2024-03-12T03:55:03.234921+0000","last_undegraded":"2024-03-12T03:55:03.234921+0000","last_fullsized":"2024-03-12T03:55:03.234921+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:54:15.543802+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:54:15.543802+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:23.965340+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":987,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.926023+0000","last_change":"2024-03-12T03:54:23.546326+0000","last_active":"2024-03-12T03:54:58.926023+0000","last_peered":"2024-03-12T03:54:58.926023+0000","last_clean":"2024-03-12T03:54:58.926023+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:54:58.926023+0000","last_undegraded":"2024-03-12T03:54:58.926023+0000","last_fullsized":"2024-03-12T03:54:58.926023+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:23.546216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_clean_scrub_stamp":"2024-03-12T03:54:23.546216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:44.133568+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":28,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.927041+0000","last_change":"2024-03-12T03:54:47.088397+0000","last_active":"2024-03-12T03:54:58.927041+0000","last_peered":"2024-03-12T03:54:58.927041+0000","last_clean":"2024-03-12T03:54:58.927041+0000","last_became_active":"2024-03-12T03:54:47.088011+0000","last_became_peered":"2024-03-12T03:54:47.088011+0000","last_unstale":"2024-03-12T03:54:58.927041+0000","last_undegraded":"2024-03-12T03:54:58.927041+0000","last_fullsized":"2024-03-12T03:54:58.927041+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:36.238867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:54:36.238867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:53.700079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":760,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619041+0000","last_change":"2024-03-12T03:54:47.088817+0000","last_active":"2024-03-12T03:55:02.619041+0000","last_peered":"2024-03-12T03:55:02.619041+0000","last_clean":"2024-03-12T03:55:02.619041+0000","last_became_active":"2024-03-12T03:54:47.088474+0000","last_became_peered":"2024-03-12T03:54:47.088474+0000","last_unstale":"2024-03-12T03:55:02.619041+0000","last_undegraded":"2024-03-12T03:55:02.619041+0000","last_fullsized":"2024-03-12T03:55:02.619041+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:36.531169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:36.531169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":444,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.817548+0000","last_change":"2024-03-12T03:54:49.577961+0000","last_active":"2024-03-12T03:54:58.817548+0000","last_peered":"2024-03-12T03:54:58.817548+0000","last_clean":"2024-03-12T03:54:58.817548+0000","last_became_active":"2024-03-12T03:54:49.577309+0000","last_became_peered":"2024-03-12T03:54:49.577309+0000","last_unstale":"2024-03-12T03:54:58.817548+0000","last_undegraded":"2024-03-12T03:54:58.817548+0000","last_fullsized":"2024-03-12T03:54:58.817548+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:54:28.287230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:54:28.287230+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:47.690506+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"480'39","reported_seq":851,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.235584+0000","last_change":"2024-03-12T03:54:36.470898+0000","last_active":"2024-03-12T03:55:03.235584+0000","last_peered":"2024-03-12T03:55:03.235584+0000","last_clean":"2024-03-12T03:55:03.235584+0000","last_became_active":"2024-03-12T03:46:25.287373+0000","last_became_peered":"2024-03-12T03:46:25.287373+0000","last_unstale":"2024-03-12T03:55:03.235584+0000","last_undegraded":"2024-03-12T03:55:03.235584+0000","last_fullsized":"2024-03-12T03:55:03.235584+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:54:36.470756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:54:36.470756+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:49.895847+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.617808+0000","last_change":"2024-03-12T03:54:47.091092+0000","last_active":"2024-03-12T03:55:02.617808+0000","last_peered":"2024-03-12T03:55:02.617808+0000","last_clean":"2024-03-12T03:55:02.617808+0000","last_became_active":"2024-03-12T03:54:47.090620+0000","last_became_peered":"2024-03-12T03:54:47.090620+0000","last_unstale":"2024-03-12T03:55:02.617808+0000","last_undegraded":"2024-03-12T03:55:02.617808+0000","last_fullsized":"2024-03-12T03:55:02.617808+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:16.876204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:16.876204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"506'12","reported_seq":35,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.817932+0000","last_change":"2024-03-12T03:54:48.127662+0000","last_active":"2024-03-12T03:54:58.817932+0000","last_peered":"2024-03-12T03:54:58.817932+0000","last_clean":"2024-03-12T03:54:58.817932+0000","last_became_active":"2024-03-12T03:54:48.117473+0000","last_became_peered":"2024-03-12T03:54:48.117473+0000","last_unstale":"2024-03-12T03:54:58.817932+0000","last_undegraded":"2024-03-12T03:54:58.817932+0000","last_fullsized":"2024-03-12T03:54:58.817932+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:47:53.964216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:47:53.964216+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:05.885525+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":704,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:03.232183+0000","last_change":"2024-03-12T03:54:48.129019+0000","last_active":"2024-03-12T03:55:03.232183+0000","last_peered":"2024-03-12T03:55:03.232183+0000","last_clean":"2024-03-12T03:55:03.232183+0000","last_became_active":"2024-03-12T03:54:48.128137+0000","last_became_peered":"2024-03-12T03:54:48.128137+0000","last_unstale":"2024-03-12T03:55:03.232183+0000","last_undegraded":"2024-03-12T03:55:03.232183+0000","last_fullsized":"2024-03-12T03:55:03.232183+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:04.770746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:48:04.770746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261673+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":978,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.925991+0000","last_change":"2024-03-12T03:54:43.586702+0000","last_active":"2024-03-12T03:54:58.925991+0000","last_peered":"2024-03-12T03:54:58.925991+0000","last_clean":"2024-03-12T03:54:58.925991+0000","last_became_active":"2024-03-12T03:46:25.286985+0000","last_became_peered":"2024-03-12T03:46:25.286985+0000","last_unstale":"2024-03-12T03:54:58.925991+0000","last_undegraded":"2024-03-12T03:54:58.925991+0000","last_fullsized":"2024-03-12T03:54:58.925991+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:43.586591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:54:43.586591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:56:06.930880+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"480'54","reported_seq":838,"reported_epoch":514,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:55:02.621268+0000","last_change":"2024-03-12T03:48:58.463377+0000","last_active":"2024-03-12T03:55:02.621268+0000","last_peered":"2024-03-12T03:55:02.621268+0000","last_clean":"2024-03-12T03:48:53.447338+0000","last_became_active":"2024-03-12T03:48:58.463377+0000","last_became_peered":"2024-03-12T03:48:58.463377+0000","last_unstale":"2024-03-12T03:55:02.621268+0000","last_undegraded":"2024-03-12T03:48:58.440183+0000","last_fullsized":"2024-03-12T03:48:58.440082+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'42","last_scrub_stamp":"2024-03-12T03:47:13.347128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.347128+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":86,"num_read_kb":72,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":735,"reported_epoch":514,"state":"active+undersized","last_fresh":"2024-03-12T03:55:02.618664+0000","last_change":"2024-03-12T03:48:58.464970+0000","last_active":"2024-03-12T03:55:02.618664+0000","last_peered":"2024-03-12T03:55:02.618664+0000","last_clean":"2024-03-12T03:48:53.448054+0000","last_became_active":"2024-03-12T03:48:58.464970+0000","last_became_peered":"2024-03-12T03:48:58.464970+0000","last_unstale":"2024-03-12T03:55:02.618664+0000","last_undegraded":"2024-03-12T03:55:02.618664+0000","last_fullsized":"2024-03-12T03:48:58.441302+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:05.639950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:05.639950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":752,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619811+0000","last_change":"2024-03-12T03:54:22.219746+0000","last_active":"2024-03-12T03:55:02.619811+0000","last_peered":"2024-03-12T03:55:02.619811+0000","last_clean":"2024-03-12T03:55:02.619811+0000","last_became_active":"2024-03-12T03:51:42.840902+0000","last_became_peered":"2024-03-12T03:51:42.840902+0000","last_unstale":"2024-03-12T03:55:02.619811+0000","last_undegraded":"2024-03-12T03:55:02.619811+0000","last_fullsized":"2024-03-12T03:55:02.619811+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:22.219500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_clean_scrub_stamp":"2024-03-12T03:54:22.219500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:41.321129+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":1123,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.926483+0000","last_change":"2024-03-12T03:54:16.518007+0000","last_active":"2024-03-12T03:54:58.926483+0000","last_peered":"2024-03-12T03:54:58.926483+0000","last_clean":"2024-03-12T03:54:58.926483+0000","last_became_active":"2024-03-12T03:44:19.373715+0000","last_became_peered":"2024-03-12T03:44:19.373715+0000","last_unstale":"2024-03-12T03:54:58.926483+0000","last_undegraded":"2024-03-12T03:54:58.926483+0000","last_fullsized":"2024-03-12T03:54:58.926483+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:54:16.515858+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:15.895638+0000","last_clean_scrub_stamp":"2024-03-12T03:54:16.515858+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:24.717191+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"501'13","reported_seq":1011,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.621039+0000","last_change":"2024-03-12T03:54:36.019170+0000","last_active":"2024-03-12T03:55:02.621039+0000","last_peered":"2024-03-12T03:55:02.621039+0000","last_clean":"2024-03-12T03:55:02.621039+0000","last_became_active":"2024-03-12T03:46:25.288674+0000","last_became_peered":"2024-03-12T03:46:25.288674+0000","last_unstale":"2024-03-12T03:55:02.621039+0000","last_undegraded":"2024-03-12T03:55:02.621039+0000","last_fullsized":"2024-03-12T03:55:02.621039+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-03-12T03:54:36.018930+0000","last_deep_scrub":"501'13","last_deep_scrub_stamp":"2024-03-12T03:54:36.018930+0000","last_clean_scrub_stamp":"2024-03-12T03:54:36.018930+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:56:00.696424+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"480'56","reported_seq":643,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.926204+0000","last_change":"2024-03-12T03:54:48.082635+0000","last_active":"2024-03-12T03:54:58.926204+0000","last_peered":"2024-03-12T03:54:58.926204+0000","last_clean":"2024-03-12T03:54:58.926204+0000","last_became_active":"2024-03-12T03:54:48.082119+0000","last_became_peered":"2024-03-12T03:54:48.082119+0000","last_unstale":"2024-03-12T03:54:58.926204+0000","last_undegraded":"2024-03-12T03:54:58.926204+0000","last_fullsized":"2024-03-12T03:54:58.926204+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'56","last_scrub_stamp":"2024-03-12T03:54:15.097616+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:54:15.097616+0000","objects_scrubbed":8,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:21.157932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":62,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,5],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"320'2","reported_seq":826,"reported_epoch":514,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:55:03.234224+0000","last_change":"2024-03-12T03:48:58.457502+0000","last_active":"2024-03-12T03:55:03.234224+0000","last_peered":"2024-03-12T03:55:03.234224+0000","last_clean":"2024-03-12T03:45:56.084290+0000","last_became_active":"2024-03-12T03:48:58.457502+0000","last_became_peered":"2024-03-12T03:48:58.457502+0000","last_unstale":"2024-03-12T03:55:03.234224+0000","last_undegraded":"2024-03-12T03:48:58.439950+0000","last_fullsized":"2024-03-12T03:48:58.439792+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'2","last_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_deep_scrub":"320'2","last_deep_scrub_stamp":"2024-03-12T03:45:56.089247+0000","last_clean_scrub_stamp":"2024-03-12T03:45:56.089247+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261642+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":976,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.898920+0000","last_change":"2024-03-12T03:51:31.400543+0000","last_active":"2024-03-12T03:54:58.898920+0000","last_peered":"2024-03-12T03:54:58.898920+0000","last_clean":"2024-03-12T03:54:58.898920+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:54:58.898920+0000","last_undegraded":"2024-03-12T03:54:58.898920+0000","last_fullsized":"2024-03-12T03:54:58.898920+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:31.400418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:49:54.471078+0000","last_clean_scrub_stamp":"2024-03-12T03:51:31.400418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836235+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":952,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.618412+0000","last_change":"2024-03-12T03:54:49.575670+0000","last_active":"2024-03-12T03:55:02.618412+0000","last_peered":"2024-03-12T03:55:02.618412+0000","last_clean":"2024-03-12T03:55:02.618412+0000","last_became_active":"2024-03-12T03:54:49.571823+0000","last_became_peered":"2024-03-12T03:54:49.571823+0000","last_unstale":"2024-03-12T03:55:02.618412+0000","last_undegraded":"2024-03-12T03:55:02.618412+0000","last_fullsized":"2024-03-12T03:55:02.618412+0000","mapping_epoch":507,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:37.519232+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_clean_scrub_stamp":"2024-03-12T03:51:37.519232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"506'11","reported_seq":37,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:02.619847+0000","last_change":"2024-03-12T03:54:48.139190+0000","last_active":"2024-03-12T03:55:02.619847+0000","last_peered":"2024-03-12T03:55:02.619847+0000","last_clean":"2024-03-12T03:55:02.619847+0000","last_became_active":"2024-03-12T03:54:48.132047+0000","last_became_peered":"2024-03-12T03:54:48.132047+0000","last_unstale":"2024-03-12T03:55:02.619847+0000","last_undegraded":"2024-03-12T03:55:02.619847+0000","last_fullsized":"2024-03-12T03:55:02.619847+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:48:14.138048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:14.138048+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"480'36","reported_seq":1043,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:03.232782+0000","last_change":"2024-03-12T03:54:48.117408+0000","last_active":"2024-03-12T03:55:03.232782+0000","last_peered":"2024-03-12T03:55:03.232782+0000","last_clean":"2024-03-12T03:55:03.232782+0000","last_became_active":"2024-03-12T03:54:48.116942+0000","last_became_peered":"2024-03-12T03:54:48.116942+0000","last_unstale":"2024-03-12T03:55:03.232782+0000","last_undegraded":"2024-03-12T03:55:03.232782+0000","last_fullsized":"2024-03-12T03:55:03.232782+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'36","last_scrub_stamp":"2024-03-12T03:54:35.415429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:54:35.415429+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:40.307376+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":618,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.618180+0000","last_change":"2024-03-12T03:54:47.084722+0000","last_active":"2024-03-12T03:55:02.618180+0000","last_peered":"2024-03-12T03:55:02.618180+0000","last_clean":"2024-03-12T03:55:02.618180+0000","last_became_active":"2024-03-12T03:54:47.084279+0000","last_became_peered":"2024-03-12T03:54:47.084279+0000","last_unstale":"2024-03-12T03:55:02.618180+0000","last_undegraded":"2024-03-12T03:55:02.618180+0000","last_fullsized":"2024-03-12T03:55:02.618180+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:40.122207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:54:40.122207+0000","last_clean_scrub_stamp":"2024-03-12T03:54:40.122207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:50.577716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":847,"reported_epoch":514,"state":"active+undersized","last_fresh":"2024-03-12T03:55:02.620845+0000","last_change":"2024-03-12T03:48:58.455443+0000","last_active":"2024-03-12T03:55:02.620845+0000","last_peered":"2024-03-12T03:55:02.620845+0000","last_clean":"2024-03-12T03:43:13.330724+0000","last_became_active":"2024-03-12T03:48:58.455443+0000","last_became_peered":"2024-03-12T03:48:58.455443+0000","last_unstale":"2024-03-12T03:55:02.620845+0000","last_undegraded":"2024-03-12T03:55:02.620845+0000","last_fullsized":"2024-03-12T03:48:58.439578+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:13.892760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.892760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":713,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619705+0000","last_change":"2024-03-12T03:50:54.587747+0000","last_active":"2024-03-12T03:55:02.619705+0000","last_peered":"2024-03-12T03:55:02.619705+0000","last_clean":"2024-03-12T03:55:02.619705+0000","last_became_active":"2024-03-12T03:46:25.287360+0000","last_became_peered":"2024-03-12T03:46:25.287360+0000","last_unstale":"2024-03-12T03:55:02.619705+0000","last_undegraded":"2024-03-12T03:55:02.619705+0000","last_fullsized":"2024-03-12T03:55:02.619705+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:54.587604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:54.587604+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":26,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.817106+0000","last_change":"2024-03-12T03:54:48.116043+0000","last_active":"2024-03-12T03:54:58.817106+0000","last_peered":"2024-03-12T03:54:58.817106+0000","last_clean":"2024-03-12T03:54:58.817106+0000","last_became_active":"2024-03-12T03:54:48.114901+0000","last_became_peered":"2024-03-12T03:54:48.114901+0000","last_unstale":"2024-03-12T03:54:58.817106+0000","last_undegraded":"2024-03-12T03:54:58.817106+0000","last_fullsized":"2024-03-12T03:54:58.817106+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.179416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.179416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.715062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6,5],"avail_no_missing":["2","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"480'45","reported_seq":761,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.899238+0000","last_change":"2024-03-12T03:54:48.161448+0000","last_active":"2024-03-12T03:54:58.899238+0000","last_peered":"2024-03-12T03:54:58.899238+0000","last_clean":"2024-03-12T03:54:58.899238+0000","last_became_active":"2024-03-12T03:54:48.116231+0000","last_became_peered":"2024-03-12T03:54:48.116231+0000","last_unstale":"2024-03-12T03:54:58.899238+0000","last_undegraded":"2024-03-12T03:54:58.899238+0000","last_fullsized":"2024-03-12T03:54:58.899238+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-03-12T03:45:59.055318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.055318+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":61,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,5],"avail_no_missing":["7","1","5"],"object_location_counts":[{"shards":"1,5,7","objects":9}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1002,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:03.233382+0000","last_change":"2024-03-12T03:54:48.120349+0000","last_active":"2024-03-12T03:55:03.233382+0000","last_peered":"2024-03-12T03:55:03.233382+0000","last_clean":"2024-03-12T03:55:03.233382+0000","last_became_active":"2024-03-12T03:54:48.119891+0000","last_became_peered":"2024-03-12T03:54:48.119891+0000","last_unstale":"2024-03-12T03:55:03.233382+0000","last_undegraded":"2024-03-12T03:55:03.233382+0000","last_fullsized":"2024-03-12T03:55:03.233382+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:28.364732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:54:28.364732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:51.279612+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":977,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.231659+0000","last_change":"2024-03-12T03:51:36.030269+0000","last_active":"2024-03-12T03:55:03.231659+0000","last_peered":"2024-03-12T03:55:03.231659+0000","last_clean":"2024-03-12T03:55:03.231659+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:55:03.231659+0000","last_undegraded":"2024-03-12T03:55:03.231659+0000","last_fullsized":"2024-03-12T03:55:03.231659+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:36.030166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:36.030166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261864+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":575,"reported_epoch":514,"state":"active+undersized","last_fresh":"2024-03-12T03:55:02.618776+0000","last_change":"2024-03-12T03:48:58.454186+0000","last_active":"2024-03-12T03:55:02.618776+0000","last_peered":"2024-03-12T03:55:02.618776+0000","last_clean":"2024-03-12T03:34:01.860694+0000","last_became_active":"2024-03-12T03:48:58.454186+0000","last_became_peered":"2024-03-12T03:48:58.454186+0000","last_unstale":"2024-03-12T03:55:02.618776+0000","last_undegraded":"2024-03-12T03:55:02.618776+0000","last_fullsized":"2024-03-12T03:48:58.443407+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:47:04.857986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:47:04.857986+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.014266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":709,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:03.235711+0000","last_change":"2024-03-12T03:54:48.207020+0000","last_active":"2024-03-12T03:55:03.235711+0000","last_peered":"2024-03-12T03:55:03.235711+0000","last_clean":"2024-03-12T03:55:03.235711+0000","last_became_active":"2024-03-12T03:54:48.200954+0000","last_became_peered":"2024-03-12T03:54:48.200954+0000","last_unstale":"2024-03-12T03:55:03.235711+0000","last_undegraded":"2024-03-12T03:55:03.235711+0000","last_fullsized":"2024-03-12T03:55:03.235711+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:00.577693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:46:00.577693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261574+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"480'36","reported_seq":808,"reported_epoch":514,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:55:03.235547+0000","last_change":"2024-03-12T03:48:58.449415+0000","last_active":"2024-03-12T03:55:03.235547+0000","last_peered":"2024-03-12T03:55:03.235547+0000","last_clean":"2024-03-12T03:43:28.788554+0000","last_became_active":"2024-03-12T03:48:58.449415+0000","last_became_peered":"2024-03-12T03:48:58.449415+0000","last_unstale":"2024-03-12T03:55:03.235547+0000","last_undegraded":"2024-03-12T03:48:58.441112+0000","last_fullsized":"2024-03-12T03:48:58.441001+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'24","last_scrub_stamp":"2024-03-12T03:48:17.191968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:17.191968+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261723+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":2268,"reported_epoch":513,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:55:00.184353+0000","last_change":"2024-03-12T03:48:58.462253+0000","last_active":"2024-03-12T03:55:00.184353+0000","last_peered":"2024-03-12T03:55:00.184353+0000","last_clean":"2024-03-12T03:48:56.580904+0000","last_became_active":"2024-03-12T03:48:58.462253+0000","last_became_peered":"2024-03-12T03:48:58.462253+0000","last_unstale":"2024-03-12T03:55:00.184353+0000","last_undegraded":"2024-03-12T03:48:58.445825+0000","last_fullsized":"2024-03-12T03:48:58.445711+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:47:13.591375+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:47:13.591375+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:05.915126+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1291,"num_read_kb":1286,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":844,"reported_epoch":513,"state":"active+undersized","last_fresh":"2024-03-12T03:54:58.926104+0000","last_change":"2024-03-12T03:48:58.464609+0000","last_active":"2024-03-12T03:54:58.926104+0000","last_peered":"2024-03-12T03:54:58.926104+0000","last_clean":"2024-03-12T03:43:13.332262+0000","last_became_active":"2024-03-12T03:48:58.464609+0000","last_became_peered":"2024-03-12T03:48:58.464609+0000","last_unstale":"2024-03-12T03:54:58.926104+0000","last_undegraded":"2024-03-12T03:54:58.926104+0000","last_fullsized":"2024-03-12T03:48:58.443632+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:58.050985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:47:58.050985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:03.391083+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":989,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.231158+0000","last_change":"2024-03-12T03:54:47.095532+0000","last_active":"2024-03-12T03:55:03.231158+0000","last_peered":"2024-03-12T03:55:03.231158+0000","last_clean":"2024-03-12T03:55:03.231158+0000","last_became_active":"2024-03-12T03:54:47.095172+0000","last_became_peered":"2024-03-12T03:54:47.095172+0000","last_unstale":"2024-03-12T03:55:03.231158+0000","last_undegraded":"2024-03-12T03:55:03.231158+0000","last_fullsized":"2024-03-12T03:55:03.231158+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:19.408606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:54:19.408606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:32.239972+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1117,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.232800+0000","last_change":"2024-03-12T03:54:32.447180+0000","last_active":"2024-03-12T03:55:03.232800+0000","last_peered":"2024-03-12T03:55:03.232800+0000","last_clean":"2024-03-12T03:55:03.232800+0000","last_became_active":"2024-03-12T03:51:11.965655+0000","last_became_peered":"2024-03-12T03:51:11.965655+0000","last_unstale":"2024-03-12T03:55:03.232800+0000","last_undegraded":"2024-03-12T03:55:03.232800+0000","last_fullsized":"2024-03-12T03:55:03.232800+0000","mapping_epoch":154,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:54:32.447065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:54:32.447065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:52.443067+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":985,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.899275+0000","last_change":"2024-03-12T03:54:48.117967+0000","last_active":"2024-03-12T03:54:58.899275+0000","last_peered":"2024-03-12T03:54:58.899275+0000","last_clean":"2024-03-12T03:54:58.899275+0000","last_became_active":"2024-03-12T03:54:48.117571+0000","last_became_peered":"2024-03-12T03:54:48.117571+0000","last_unstale":"2024-03-12T03:54:58.899275+0000","last_undegraded":"2024-03-12T03:54:58.899275+0000","last_fullsized":"2024-03-12T03:54:58.899275+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:38.655865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:54:38.655865+0000","last_clean_scrub_stamp":"2024-03-12T03:54:38.655865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:56:07.542021+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,5],"avail_no_missing":["7","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"480'41","reported_seq":1194,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.926781+0000","last_change":"2024-03-12T03:54:44.535745+0000","last_active":"2024-03-12T03:54:58.926781+0000","last_peered":"2024-03-12T03:54:58.926781+0000","last_clean":"2024-03-12T03:54:58.926781+0000","last_became_active":"2024-03-12T03:45:18.109048+0000","last_became_peered":"2024-03-12T03:45:18.109048+0000","last_unstale":"2024-03-12T03:54:58.926781+0000","last_undegraded":"2024-03-12T03:54:58.926781+0000","last_fullsized":"2024-03-12T03:54:58.926781+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'41","last_scrub_stamp":"2024-03-12T03:54:44.535615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:54:44.535615+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:56:09.445316+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":754,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619691+0000","last_change":"2024-03-12T03:54:47.090546+0000","last_active":"2024-03-12T03:55:02.619691+0000","last_peered":"2024-03-12T03:55:02.619691+0000","last_clean":"2024-03-12T03:55:02.619691+0000","last_became_active":"2024-03-12T03:54:47.089381+0000","last_became_peered":"2024-03-12T03:54:47.089381+0000","last_unstale":"2024-03-12T03:55:02.619691+0000","last_undegraded":"2024-03-12T03:55:02.619691+0000","last_fullsized":"2024-03-12T03:55:02.619691+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:42.026768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:54:42.026768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:53.174834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1242,"reported_epoch":513,"state":"active+undersized","last_fresh":"2024-03-12T03:54:58.898759+0000","last_change":"2024-03-12T03:48:58.452024+0000","last_active":"2024-03-12T03:54:58.898759+0000","last_peered":"2024-03-12T03:54:58.898759+0000","last_clean":"2024-03-12T03:48:53.448868+0000","last_became_active":"2024-03-12T03:48:58.452024+0000","last_became_peered":"2024-03-12T03:48:58.452024+0000","last_unstale":"2024-03-12T03:54:58.898759+0000","last_undegraded":"2024-03-12T03:54:58.898759+0000","last_fullsized":"2024-03-12T03:48:58.444796+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.435684+0000","last_clean_scrub_stamp":"2024-03-12T03:48:11.435684+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836086+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":593,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619236+0000","last_change":"2024-03-12T03:54:47.100024+0000","last_active":"2024-03-12T03:55:02.619236+0000","last_peered":"2024-03-12T03:55:02.619236+0000","last_clean":"2024-03-12T03:55:02.619236+0000","last_became_active":"2024-03-12T03:54:47.099410+0000","last_became_peered":"2024-03-12T03:54:47.099410+0000","last_unstale":"2024-03-12T03:55:02.619236+0000","last_undegraded":"2024-03-12T03:55:02.619236+0000","last_fullsized":"2024-03-12T03:55:02.619236+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.169871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.169871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":717,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.818196+0000","last_change":"2024-03-12T03:54:47.095328+0000","last_active":"2024-03-12T03:54:58.818196+0000","last_peered":"2024-03-12T03:54:58.818196+0000","last_clean":"2024-03-12T03:54:58.818196+0000","last_became_active":"2024-03-12T03:54:47.094680+0000","last_became_peered":"2024-03-12T03:54:47.094680+0000","last_unstale":"2024-03-12T03:54:58.818196+0000","last_undegraded":"2024-03-12T03:54:58.818196+0000","last_fullsized":"2024-03-12T03:54:58.818196+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:22.410302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:54:22.410302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:49.471051+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"480'27","reported_seq":1029,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:03.235045+0000","last_change":"2024-03-12T03:54:48.365812+0000","last_active":"2024-03-12T03:55:03.235045+0000","last_peered":"2024-03-12T03:55:03.235045+0000","last_clean":"2024-03-12T03:55:03.235045+0000","last_became_active":"2024-03-12T03:54:48.201476+0000","last_became_peered":"2024-03-12T03:54:48.201476+0000","last_unstale":"2024-03-12T03:55:03.235045+0000","last_undegraded":"2024-03-12T03:55:03.235045+0000","last_fullsized":"2024-03-12T03:55:03.235045+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'23","last_scrub_stamp":"2024-03-12T03:45:59.998395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.998395+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261625+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,5],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":692,"reported_epoch":513,"state":"active+undersized","last_fresh":"2024-03-12T03:54:58.818362+0000","last_change":"2024-03-12T03:48:58.469230+0000","last_active":"2024-03-12T03:54:58.818362+0000","last_peered":"2024-03-12T03:54:58.818362+0000","last_clean":"2024-03-12T03:48:53.446422+0000","last_became_active":"2024-03-12T03:48:58.469230+0000","last_became_peered":"2024-03-12T03:48:58.469230+0000","last_unstale":"2024-03-12T03:54:58.818362+0000","last_undegraded":"2024-03-12T03:54:58.818362+0000","last_fullsized":"2024-03-12T03:48:58.446209+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:06.903506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.903506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":997,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.231867+0000","last_change":"2024-03-12T03:54:34.426057+0000","last_active":"2024-03-12T03:55:03.231867+0000","last_peered":"2024-03-12T03:55:03.231867+0000","last_clean":"2024-03-12T03:55:03.231867+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:55:03.231867+0000","last_undegraded":"2024-03-12T03:55:03.231867+0000","last_fullsized":"2024-03-12T03:55:03.231867+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:34.425929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:54:34.425929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:40.127874+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":984,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.900008+0000","last_change":"2024-03-12T03:54:27.639783+0000","last_active":"2024-03-12T03:54:58.900008+0000","last_peered":"2024-03-12T03:54:58.900008+0000","last_clean":"2024-03-12T03:54:58.900008+0000","last_became_active":"2024-03-12T03:46:25.284168+0000","last_became_peered":"2024-03-12T03:46:25.284168+0000","last_unstale":"2024-03-12T03:54:58.900008+0000","last_undegraded":"2024-03-12T03:54:58.900008+0000","last_fullsized":"2024-03-12T03:54:58.900008+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:54:27.639634+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:49:18.522103+0000","last_clean_scrub_stamp":"2024-03-12T03:54:27.639634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:42.320944+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1258,"reported_epoch":513,"state":"active+undersized","last_fresh":"2024-03-12T03:54:58.899094+0000","last_change":"2024-03-12T03:48:58.455161+0000","last_active":"2024-03-12T03:54:58.899094+0000","last_peered":"2024-03-12T03:54:58.899094+0000","last_clean":"2024-03-12T03:48:53.449188+0000","last_became_active":"2024-03-12T03:48:58.455161+0000","last_became_peered":"2024-03-12T03:48:58.455161+0000","last_unstale":"2024-03-12T03:54:58.899094+0000","last_undegraded":"2024-03-12T03:54:58.899094+0000","last_fullsized":"2024-03-12T03:48:58.444402+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:01.456424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:01.456424+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":269,"reported_epoch":514,"state":"active+undersized","last_fresh":"2024-03-12T03:55:03.232856+0000","last_change":"2024-03-12T03:48:58.458916+0000","last_active":"2024-03-12T03:55:03.232856+0000","last_peered":"2024-03-12T03:55:03.232856+0000","last_clean":"2024-03-12T03:33:24.155889+0000","last_became_active":"2024-03-12T03:48:58.458916+0000","last_became_peered":"2024-03-12T03:48:58.458916+0000","last_unstale":"2024-03-12T03:55:03.232856+0000","last_undegraded":"2024-03-12T03:55:03.232856+0000","last_fullsized":"2024-03-12T03:48:58.440354+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:06.864708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:06.864708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"480'66","reported_seq":412,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.231981+0000","last_change":"2024-03-12T03:51:38.936343+0000","last_active":"2024-03-12T03:55:03.231981+0000","last_peered":"2024-03-12T03:55:03.231981+0000","last_clean":"2024-03-12T03:55:03.231981+0000","last_became_active":"2024-03-12T03:48:48.462111+0000","last_became_peered":"2024-03-12T03:48:48.462111+0000","last_unstale":"2024-03-12T03:55:03.231981+0000","last_undegraded":"2024-03-12T03:55:03.231981+0000","last_fullsized":"2024-03-12T03:55:03.231981+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'54","last_scrub_stamp":"2024-03-12T03:51:38.936190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:38.936190+0000","objects_scrubbed":11,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":76,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":963,"reported_epoch":513,"state":"active+undersized","last_fresh":"2024-03-12T03:54:58.926237+0000","last_change":"2024-03-12T03:48:58.453277+0000","last_active":"2024-03-12T03:54:58.926237+0000","last_peered":"2024-03-12T03:54:58.926237+0000","last_clean":"2024-03-12T03:48:53.447092+0000","last_became_active":"2024-03-12T03:48:58.453277+0000","last_became_peered":"2024-03-12T03:48:58.453277+0000","last_unstale":"2024-03-12T03:54:58.926237+0000","last_undegraded":"2024-03-12T03:54:58.926237+0000","last_fullsized":"2024-03-12T03:48:58.443592+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.648824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.648824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:03.391072+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":55,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:03.231975+0000","last_change":"2024-03-12T03:54:48.112844+0000","last_active":"2024-03-12T03:55:03.231975+0000","last_peered":"2024-03-12T03:55:03.231975+0000","last_clean":"2024-03-12T03:55:03.231975+0000","last_became_active":"2024-03-12T03:54:48.112369+0000","last_became_peered":"2024-03-12T03:54:48.112369+0000","last_unstale":"2024-03-12T03:55:03.231975+0000","last_undegraded":"2024-03-12T03:55:03.231975+0000","last_fullsized":"2024-03-12T03:55:03.231975+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-03-12T03:54:45.107716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:54:45.107716+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:52.536263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6,5],"avail_no_missing":["1","5","6"],"object_location_counts":[{"shards":"1,5,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":787,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.618999+0000","last_change":"2024-03-12T03:54:44.025976+0000","last_active":"2024-03-12T03:55:02.618999+0000","last_peered":"2024-03-12T03:55:02.618999+0000","last_clean":"2024-03-12T03:55:02.618999+0000","last_became_active":"2024-03-12T03:46:25.284416+0000","last_became_peered":"2024-03-12T03:46:25.284416+0000","last_unstale":"2024-03-12T03:55:02.618999+0000","last_undegraded":"2024-03-12T03:55:02.618999+0000","last_fullsized":"2024-03-12T03:55:02.618999+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:44.025902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:54:44.025902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:56:07.616294+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":444,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.818149+0000","last_change":"2024-03-12T03:54:49.573425+0000","last_active":"2024-03-12T03:54:58.818149+0000","last_peered":"2024-03-12T03:54:58.818149+0000","last_clean":"2024-03-12T03:54:58.818149+0000","last_became_active":"2024-03-12T03:54:49.572278+0000","last_became_peered":"2024-03-12T03:54:49.572278+0000","last_unstale":"2024-03-12T03:54:58.818149+0000","last_undegraded":"2024-03-12T03:54:58.818149+0000","last_fullsized":"2024-03-12T03:54:58.818149+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:48:06.690873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.690873+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:59.710978+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4,5],"avail_no_missing":["2","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":522,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:02.618513+0000","last_change":"2024-03-12T03:51:00.633662+0000","last_active":"2024-03-12T03:55:02.618513+0000","last_peered":"2024-03-12T03:55:02.618513+0000","last_clean":"2024-03-12T03:55:02.618513+0000","last_became_active":"2024-03-12T03:50:58.843997+0000","last_became_peered":"2024-03-12T03:50:58.843997+0000","last_unstale":"2024-03-12T03:55:02.618513+0000","last_undegraded":"2024-03-12T03:55:02.618513+0000","last_fullsized":"2024-03-12T03:55:02.618513+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:00.633453+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:11.924742+0000","last_clean_scrub_stamp":"2024-03-12T03:51:00.633453+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"480'32","reported_seq":757,"reported_epoch":514,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:55:03.232956+0000","last_change":"2024-03-12T03:48:58.462328+0000","last_active":"2024-03-12T03:55:03.232956+0000","last_peered":"2024-03-12T03:55:03.232956+0000","last_clean":"2024-03-12T03:43:28.842466+0000","last_became_active":"2024-03-12T03:48:58.462328+0000","last_became_peered":"2024-03-12T03:48:58.462328+0000","last_unstale":"2024-03-12T03:55:03.232956+0000","last_undegraded":"2024-03-12T03:48:58.442501+0000","last_fullsized":"2024-03-12T03:48:58.442398+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:45:15.774076+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:45:15.774076+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261486+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":605,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.618800+0000","last_change":"2024-03-12T03:54:47.091727+0000","last_active":"2024-03-12T03:55:02.618800+0000","last_peered":"2024-03-12T03:55:02.618800+0000","last_clean":"2024-03-12T03:55:02.618800+0000","last_became_active":"2024-03-12T03:54:47.091210+0000","last_became_peered":"2024-03-12T03:54:47.091210+0000","last_unstale":"2024-03-12T03:55:02.618800+0000","last_undegraded":"2024-03-12T03:55:02.618800+0000","last_fullsized":"2024-03-12T03:55:02.618800+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:46.059573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:54:46.059573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:55.592565+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":713,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.817691+0000","last_change":"2024-03-12T03:54:26.449171+0000","last_active":"2024-03-12T03:54:58.817691+0000","last_peered":"2024-03-12T03:54:58.817691+0000","last_clean":"2024-03-12T03:54:58.817691+0000","last_became_active":"2024-03-12T03:46:25.284556+0000","last_became_peered":"2024-03-12T03:46:25.284556+0000","last_unstale":"2024-03-12T03:54:58.817691+0000","last_undegraded":"2024-03-12T03:54:58.817691+0000","last_fullsized":"2024-03-12T03:54:58.817691+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:26.448564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:54:26.448564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:39.898677+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":549,"reported_epoch":513,"state":"active+undersized","last_fresh":"2024-03-12T03:54:58.898607+0000","last_change":"2024-03-12T03:48:58.464502+0000","last_active":"2024-03-12T03:54:58.898607+0000","last_peered":"2024-03-12T03:54:58.898607+0000","last_clean":"2024-03-12T03:39:28.535672+0000","last_became_active":"2024-03-12T03:48:58.464502+0000","last_became_peered":"2024-03-12T03:48:58.464502+0000","last_unstale":"2024-03-12T03:54:58.898607+0000","last_undegraded":"2024-03-12T03:54:58.898607+0000","last_fullsized":"2024-03-12T03:48:58.446034+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:55.192996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:45:55.192996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.835989+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":851,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.618548+0000","last_change":"2024-03-12T03:50:59.586938+0000","last_active":"2024-03-12T03:55:02.618548+0000","last_peered":"2024-03-12T03:55:02.618548+0000","last_clean":"2024-03-12T03:55:02.618548+0000","last_became_active":"2024-03-12T03:50:58.840022+0000","last_became_peered":"2024-03-12T03:50:58.840022+0000","last_unstale":"2024-03-12T03:55:02.618548+0000","last_undegraded":"2024-03-12T03:55:02.618548+0000","last_fullsized":"2024-03-12T03:55:02.618548+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:59.586800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:59.586800+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"480'67","reported_seq":992,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.232468+0000","last_change":"2024-03-12T03:54:29.386308+0000","last_active":"2024-03-12T03:55:03.232468+0000","last_peered":"2024-03-12T03:55:03.232468+0000","last_clean":"2024-03-12T03:55:03.232468+0000","last_became_active":"2024-03-12T03:45:18.106503+0000","last_became_peered":"2024-03-12T03:45:18.106503+0000","last_unstale":"2024-03-12T03:55:03.232468+0000","last_undegraded":"2024-03-12T03:55:03.232468+0000","last_fullsized":"2024-03-12T03:55:03.232468+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'67","last_scrub_stamp":"2024-03-12T03:54:29.386139+0000","last_deep_scrub":"195'35","last_deep_scrub_stamp":"2024-03-12T03:39:26.925703+0000","last_clean_scrub_stamp":"2024-03-12T03:54:29.386139+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:57.167327+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":76,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":981,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899898+0000","last_change":"2024-03-12T03:54:47.095002+0000","last_active":"2024-03-12T03:54:58.899898+0000","last_peered":"2024-03-12T03:54:58.899898+0000","last_clean":"2024-03-12T03:54:58.899898+0000","last_became_active":"2024-03-12T03:54:47.094476+0000","last_became_peered":"2024-03-12T03:54:47.094476+0000","last_unstale":"2024-03-12T03:54:58.899898+0000","last_undegraded":"2024-03-12T03:54:58.899898+0000","last_fullsized":"2024-03-12T03:54:58.899898+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:34.290763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:51:34.290763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836162+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":686,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.817481+0000","last_change":"2024-03-12T03:54:48.109253+0000","last_active":"2024-03-12T03:54:58.817481+0000","last_peered":"2024-03-12T03:54:58.817481+0000","last_clean":"2024-03-12T03:54:58.817481+0000","last_became_active":"2024-03-12T03:54:48.108852+0000","last_became_peered":"2024-03-12T03:54:48.108852+0000","last_unstale":"2024-03-12T03:54:58.817481+0000","last_undegraded":"2024-03-12T03:54:58.817481+0000","last_fullsized":"2024-03-12T03:54:58.817481+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.954731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:11.497802+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.954731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.715028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7,5],"avail_no_missing":["2","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":993,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.898538+0000","last_change":"2024-03-12T03:54:33.626671+0000","last_active":"2024-03-12T03:54:58.898538+0000","last_peered":"2024-03-12T03:54:58.898538+0000","last_clean":"2024-03-12T03:54:58.898538+0000","last_became_active":"2024-03-12T03:46:25.292122+0000","last_became_peered":"2024-03-12T03:46:25.292122+0000","last_unstale":"2024-03-12T03:54:58.898538+0000","last_undegraded":"2024-03-12T03:54:58.898538+0000","last_fullsized":"2024-03-12T03:54:58.898538+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:33.626553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:54:33.626553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:46.561238+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":936,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.618324+0000","last_change":"2024-03-12T03:54:49.575171+0000","last_active":"2024-03-12T03:55:02.618324+0000","last_peered":"2024-03-12T03:55:02.618324+0000","last_clean":"2024-03-12T03:55:02.618324+0000","last_became_active":"2024-03-12T03:54:49.571379+0000","last_became_peered":"2024-03-12T03:54:49.571379+0000","last_unstale":"2024-03-12T03:55:02.618324+0000","last_undegraded":"2024-03-12T03:55:02.618324+0000","last_fullsized":"2024-03-12T03:55:02.618324+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:51:27.598607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:51:27.598607+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":698,"reported_epoch":513,"state":"active+undersized","last_fresh":"2024-03-12T03:54:58.817371+0000","last_change":"2024-03-12T03:48:58.461069+0000","last_active":"2024-03-12T03:54:58.817371+0000","last_peered":"2024-03-12T03:54:58.817371+0000","last_clean":"2024-03-12T03:48:53.445898+0000","last_became_active":"2024-03-12T03:48:58.461069+0000","last_became_peered":"2024-03-12T03:48:58.461069+0000","last_unstale":"2024-03-12T03:54:58.817371+0000","last_undegraded":"2024-03-12T03:54:58.817371+0000","last_fullsized":"2024-03-12T03:48:58.444677+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:15.981014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:15.981014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"480'36","reported_seq":690,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.231094+0000","last_change":"2024-03-12T03:54:13.482268+0000","last_active":"2024-03-12T03:55:03.231094+0000","last_peered":"2024-03-12T03:55:03.231094+0000","last_clean":"2024-03-12T03:55:03.231094+0000","last_became_active":"2024-03-12T03:51:42.836618+0000","last_became_peered":"2024-03-12T03:51:42.836618+0000","last_unstale":"2024-03-12T03:55:03.231094+0000","last_undegraded":"2024-03-12T03:55:03.231094+0000","last_fullsized":"2024-03-12T03:55:03.231094+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'36","last_scrub_stamp":"2024-03-12T03:54:13.482117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:54:13.482117+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:29.796933+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":983,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899997+0000","last_change":"2024-03-12T03:54:23.597096+0000","last_active":"2024-03-12T03:54:58.899997+0000","last_peered":"2024-03-12T03:54:58.899997+0000","last_clean":"2024-03-12T03:54:58.899997+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:54:58.899997+0000","last_undegraded":"2024-03-12T03:54:58.899997+0000","last_fullsized":"2024-03-12T03:54:58.899997+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:23.596882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:17.051487+0000","last_clean_scrub_stamp":"2024-03-12T03:54:23.596882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:40.475624+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":702,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.622183+0000","last_change":"2024-03-12T03:51:41.674084+0000","last_active":"2024-03-12T03:55:02.622183+0000","last_peered":"2024-03-12T03:55:02.622183+0000","last_clean":"2024-03-12T03:55:02.622183+0000","last_became_active":"2024-03-12T03:46:25.280784+0000","last_became_peered":"2024-03-12T03:46:25.280784+0000","last_unstale":"2024-03-12T03:55:02.622183+0000","last_undegraded":"2024-03-12T03:55:02.622183+0000","last_fullsized":"2024-03-12T03:55:02.622183+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:41.673933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:41.673933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":862,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.925756+0000","last_change":"2024-03-12T03:54:20.464111+0000","last_active":"2024-03-12T03:54:58.925756+0000","last_peered":"2024-03-12T03:54:58.925756+0000","last_clean":"2024-03-12T03:54:58.925756+0000","last_became_active":"2024-03-12T03:50:58.850740+0000","last_became_peered":"2024-03-12T03:50:58.850740+0000","last_unstale":"2024-03-12T03:54:58.925756+0000","last_undegraded":"2024-03-12T03:54:58.925756+0000","last_fullsized":"2024-03-12T03:54:58.925756+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:54:20.463880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:54:20.463880+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:48.581453+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":1236,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899154+0000","last_change":"2024-03-12T03:54:17.703498+0000","last_active":"2024-03-12T03:54:58.899154+0000","last_peered":"2024-03-12T03:54:58.899154+0000","last_clean":"2024-03-12T03:54:58.899154+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:54:58.899154+0000","last_undegraded":"2024-03-12T03:54:58.899154+0000","last_fullsized":"2024-03-12T03:54:58.899154+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:54:17.703373+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:54:17.703373+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:26.141018+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"480'60","reported_seq":1043,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.898911+0000","last_change":"2024-03-12T03:54:47.109626+0000","last_active":"2024-03-12T03:54:58.898911+0000","last_peered":"2024-03-12T03:54:58.898911+0000","last_clean":"2024-03-12T03:54:58.898911+0000","last_became_active":"2024-03-12T03:54:47.086821+0000","last_became_peered":"2024-03-12T03:54:47.086821+0000","last_unstale":"2024-03-12T03:54:58.898911+0000","last_undegraded":"2024-03-12T03:54:58.898911+0000","last_fullsized":"2024-03-12T03:54:58.898911+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'48","last_scrub_stamp":"2024-03-12T03:52:18.945763+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:52:18.945763+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":966,"reported_epoch":513,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:58.900135+0000","last_change":"2024-03-12T03:48:58.456357+0000","last_active":"2024-03-12T03:54:58.900135+0000","last_peered":"2024-03-12T03:54:58.900135+0000","last_clean":"2024-03-12T03:48:53.450296+0000","last_became_active":"2024-03-12T03:48:58.456357+0000","last_became_peered":"2024-03-12T03:48:58.456357+0000","last_unstale":"2024-03-12T03:54:58.900135+0000","last_undegraded":"2024-03-12T03:48:58.445880+0000","last_fullsized":"2024-03-12T03:48:58.445769+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:47:05.588582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:05.588582+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836048+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":26,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.817162+0000","last_change":"2024-03-12T03:54:48.116577+0000","last_active":"2024-03-12T03:54:58.817162+0000","last_peered":"2024-03-12T03:54:58.817162+0000","last_clean":"2024-03-12T03:54:58.817162+0000","last_became_active":"2024-03-12T03:54:48.115458+0000","last_became_peered":"2024-03-12T03:54:48.115458+0000","last_unstale":"2024-03-12T03:54:58.817162+0000","last_undegraded":"2024-03-12T03:54:58.817162+0000","last_fullsized":"2024-03-12T03:54:58.817162+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:59.567890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:45:59.567890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.714989+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6,5],"avail_no_missing":["2","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":521,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.898394+0000","last_change":"2024-03-12T03:54:46.728573+0000","last_active":"2024-03-12T03:54:58.898394+0000","last_peered":"2024-03-12T03:54:58.898394+0000","last_clean":"2024-03-12T03:54:58.898394+0000","last_became_active":"2024-03-12T03:50:58.836425+0000","last_became_peered":"2024-03-12T03:50:58.836425+0000","last_unstale":"2024-03-12T03:54:58.898394+0000","last_undegraded":"2024-03-12T03:54:58.898394+0000","last_fullsized":"2024-03-12T03:54:58.898394+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:54:46.728437+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:50:06.384808+0000","last_clean_scrub_stamp":"2024-03-12T03:54:46.728437+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:56:11.133167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,1],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"480'24","reported_seq":782,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.618854+0000","last_change":"2024-03-12T03:54:45.002893+0000","last_active":"2024-03-12T03:55:02.618854+0000","last_peered":"2024-03-12T03:55:02.618854+0000","last_clean":"2024-03-12T03:55:02.618854+0000","last_became_active":"2024-03-12T03:46:25.288573+0000","last_became_peered":"2024-03-12T03:46:25.288573+0000","last_unstale":"2024-03-12T03:55:02.618854+0000","last_undegraded":"2024-03-12T03:55:02.618854+0000","last_fullsized":"2024-03-12T03:55:02.618854+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'24","last_scrub_stamp":"2024-03-12T03:54:45.002753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:54:45.002753+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:45.885486+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":28,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:02.617580+0000","last_change":"2024-03-12T03:54:48.136604+0000","last_active":"2024-03-12T03:55:02.617580+0000","last_peered":"2024-03-12T03:55:02.617580+0000","last_clean":"2024-03-12T03:55:02.617580+0000","last_became_active":"2024-03-12T03:54:48.133060+0000","last_became_peered":"2024-03-12T03:54:48.133060+0000","last_unstale":"2024-03-12T03:55:02.617580+0000","last_undegraded":"2024-03-12T03:55:02.617580+0000","last_fullsized":"2024-03-12T03:55:02.617580+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:10.400135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:10.400135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":597,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619046+0000","last_change":"2024-03-12T03:54:47.090638+0000","last_active":"2024-03-12T03:55:02.619046+0000","last_peered":"2024-03-12T03:55:02.619046+0000","last_clean":"2024-03-12T03:55:02.619046+0000","last_became_active":"2024-03-12T03:54:47.090270+0000","last_became_peered":"2024-03-12T03:54:47.090270+0000","last_unstale":"2024-03-12T03:55:02.619046+0000","last_undegraded":"2024-03-12T03:55:02.619046+0000","last_fullsized":"2024-03-12T03:55:02.619046+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:27.297101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:54:27.297101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:56.684761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":962,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899435+0000","last_change":"2024-03-12T03:51:38.361542+0000","last_active":"2024-03-12T03:54:58.899435+0000","last_peered":"2024-03-12T03:54:58.899435+0000","last_clean":"2024-03-12T03:54:58.899435+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:54:58.899435+0000","last_undegraded":"2024-03-12T03:54:58.899435+0000","last_fullsized":"2024-03-12T03:54:58.899435+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:38.361430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:38.361430+0000","last_clean_scrub_stamp":"2024-03-12T03:51:38.361430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836177+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"480'33","reported_seq":900,"reported_epoch":514,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:55:02.619359+0000","last_change":"2024-03-12T03:48:58.462615+0000","last_active":"2024-03-12T03:55:02.619359+0000","last_peered":"2024-03-12T03:55:02.619359+0000","last_clean":"2024-03-12T03:43:13.333279+0000","last_became_active":"2024-03-12T03:48:58.462615+0000","last_became_peered":"2024-03-12T03:48:58.462615+0000","last_unstale":"2024-03-12T03:55:02.619359+0000","last_undegraded":"2024-03-12T03:48:58.443892+0000","last_fullsized":"2024-03-12T03:48:58.443761+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:48:06.104297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:48:06.104297+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":975,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899034+0000","last_change":"2024-03-12T03:54:21.630125+0000","last_active":"2024-03-12T03:54:58.899034+0000","last_peered":"2024-03-12T03:54:58.899034+0000","last_clean":"2024-03-12T03:54:58.899034+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:54:58.899034+0000","last_undegraded":"2024-03-12T03:54:58.899034+0000","last_fullsized":"2024-03-12T03:54:58.899034+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:21.630014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:54:21.630014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:42.845784+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":736,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619128+0000","last_change":"2024-03-12T03:52:19.693182+0000","last_active":"2024-03-12T03:55:02.619128+0000","last_peered":"2024-03-12T03:55:02.619128+0000","last_clean":"2024-03-12T03:55:02.619128+0000","last_became_active":"2024-03-12T03:46:25.286866+0000","last_became_peered":"2024-03-12T03:46:25.286866+0000","last_unstale":"2024-03-12T03:55:02.619128+0000","last_undegraded":"2024-03-12T03:55:02.619128+0000","last_fullsized":"2024-03-12T03:55:02.619128+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:52:19.693090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:52:19.693090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":702,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.818093+0000","last_change":"2024-03-12T03:54:19.463884+0000","last_active":"2024-03-12T03:54:58.818093+0000","last_peered":"2024-03-12T03:54:58.818093+0000","last_clean":"2024-03-12T03:54:58.818093+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:54:58.818093+0000","last_undegraded":"2024-03-12T03:54:58.818093+0000","last_fullsized":"2024-03-12T03:54:58.818093+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:19.463666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:54:19.463666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:21.867332+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":983,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.898521+0000","last_change":"2024-03-12T03:50:57.227680+0000","last_active":"2024-03-12T03:54:58.898521+0000","last_peered":"2024-03-12T03:54:58.898521+0000","last_clean":"2024-03-12T03:54:58.898521+0000","last_became_active":"2024-03-12T03:50:56.823948+0000","last_became_peered":"2024-03-12T03:50:56.823948+0000","last_unstale":"2024-03-12T03:54:58.898521+0000","last_undegraded":"2024-03-12T03:54:58.898521+0000","last_fullsized":"2024-03-12T03:54:58.898521+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:50:57.227429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:50:57.227429+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"480'27","reported_seq":1005,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899978+0000","last_change":"2024-03-12T03:54:47.105813+0000","last_active":"2024-03-12T03:54:58.899978+0000","last_peered":"2024-03-12T03:54:58.899978+0000","last_clean":"2024-03-12T03:54:58.899978+0000","last_became_active":"2024-03-12T03:54:47.093863+0000","last_became_peered":"2024-03-12T03:54:47.093863+0000","last_unstale":"2024-03-12T03:54:58.899978+0000","last_undegraded":"2024-03-12T03:54:58.899978+0000","last_fullsized":"2024-03-12T03:54:58.899978+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'27","last_scrub_stamp":"2024-03-12T03:54:29.259256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:54:29.259256+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:53.218223+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":843,"reported_epoch":513,"state":"active+undersized","last_fresh":"2024-03-12T03:54:58.898808+0000","last_change":"2024-03-12T03:48:58.462149+0000","last_active":"2024-03-12T03:54:58.898808+0000","last_peered":"2024-03-12T03:54:58.898808+0000","last_clean":"2024-03-12T03:43:13.331412+0000","last_became_active":"2024-03-12T03:48:58.462149+0000","last_became_peered":"2024-03-12T03:48:58.462149+0000","last_unstale":"2024-03-12T03:54:58.898808+0000","last_undegraded":"2024-03-12T03:54:58.898808+0000","last_fullsized":"2024-03-12T03:48:58.443909+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:16.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.753153+0000","last_clean_scrub_stamp":"2024-03-12T03:47:16.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836063+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"332'14","reported_seq":1019,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619259+0000","last_change":"2024-03-12T03:50:53.568427+0000","last_active":"2024-03-12T03:55:02.619259+0000","last_peered":"2024-03-12T03:55:02.619259+0000","last_clean":"2024-03-12T03:55:02.619259+0000","last_became_active":"2024-03-12T03:46:25.300463+0000","last_became_peered":"2024-03-12T03:46:25.300463+0000","last_unstale":"2024-03-12T03:55:02.619259+0000","last_undegraded":"2024-03-12T03:55:02.619259+0000","last_fullsized":"2024-03-12T03:55:02.619259+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'14","last_scrub_stamp":"2024-03-12T03:50:53.568307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.568307+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":460,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.817981+0000","last_change":"2024-03-12T03:54:48.121825+0000","last_active":"2024-03-12T03:54:58.817981+0000","last_peered":"2024-03-12T03:54:58.817981+0000","last_clean":"2024-03-12T03:54:58.817981+0000","last_became_active":"2024-03-12T03:54:48.117212+0000","last_became_peered":"2024-03-12T03:54:48.117212+0000","last_unstale":"2024-03-12T03:54:58.817981+0000","last_undegraded":"2024-03-12T03:54:58.817981+0000","last_fullsized":"2024-03-12T03:54:58.817981+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:15.793337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:54:15.793337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:43.903760+0000","scrub_duration":280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4,5],"avail_no_missing":["2","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":727,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.817323+0000","last_change":"2024-03-12T03:54:49.572770+0000","last_active":"2024-03-12T03:54:58.817323+0000","last_peered":"2024-03-12T03:54:58.817323+0000","last_clean":"2024-03-12T03:54:58.817323+0000","last_became_active":"2024-03-12T03:54:49.571966+0000","last_became_peered":"2024-03-12T03:54:49.571966+0000","last_unstale":"2024-03-12T03:54:58.817323+0000","last_undegraded":"2024-03-12T03:54:58.817323+0000","last_fullsized":"2024-03-12T03:54:58.817323+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:54:42.109513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:54:42.109513+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:54:59.710957+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4,5],"avail_no_missing":["2","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"480'49","reported_seq":850,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.618920+0000","last_change":"2024-03-12T03:51:40.548114+0000","last_active":"2024-03-12T03:55:02.618920+0000","last_peered":"2024-03-12T03:55:02.618920+0000","last_clean":"2024-03-12T03:55:02.618920+0000","last_became_active":"2024-03-12T03:46:25.299798+0000","last_became_peered":"2024-03-12T03:46:25.299798+0000","last_unstale":"2024-03-12T03:55:02.618920+0000","last_undegraded":"2024-03-12T03:55:02.618920+0000","last_fullsized":"2024-03-12T03:55:02.618920+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'39","last_scrub_stamp":"2024-03-12T03:51:40.547967+0000","last_deep_scrub":"403'39","last_deep_scrub_stamp":"2024-03-12T03:50:06.535772+0000","last_clean_scrub_stamp":"2024-03-12T03:51:40.547967+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":1002,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899047+0000","last_change":"2024-03-12T03:51:32.285686+0000","last_active":"2024-03-12T03:54:58.899047+0000","last_peered":"2024-03-12T03:54:58.899047+0000","last_clean":"2024-03-12T03:54:58.899047+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:54:58.899047+0000","last_undegraded":"2024-03-12T03:54:58.899047+0000","last_fullsized":"2024-03-12T03:54:58.899047+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:51:32.285604+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:51:32.285604+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"506'11","reported_seq":63,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:03.232611+0000","last_change":"2024-03-12T03:54:48.134382+0000","last_active":"2024-03-12T03:55:03.232611+0000","last_peered":"2024-03-12T03:55:03.232611+0000","last_clean":"2024-03-12T03:55:03.232611+0000","last_became_active":"2024-03-12T03:54:48.128641+0000","last_became_peered":"2024-03-12T03:54:48.128641+0000","last_unstale":"2024-03-12T03:55:03.232611+0000","last_undegraded":"2024-03-12T03:55:03.232611+0000","last_fullsized":"2024-03-12T03:55:03.232611+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-12T03:45:53.458170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.458170+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,5],"avail_no_missing":["1","3","5"],"object_location_counts":[{"shards":"1,3,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":831,"reported_epoch":513,"state":"active+undersized","last_fresh":"2024-03-12T03:54:58.899583+0000","last_change":"2024-03-12T03:48:58.463189+0000","last_active":"2024-03-12T03:54:58.899583+0000","last_peered":"2024-03-12T03:54:58.899583+0000","last_clean":"2024-03-12T03:43:13.332853+0000","last_became_active":"2024-03-12T03:48:58.463189+0000","last_became_peered":"2024-03-12T03:48:58.463189+0000","last_unstale":"2024-03-12T03:54:58.899583+0000","last_undegraded":"2024-03-12T03:54:58.899583+0000","last_fullsized":"2024-03-12T03:48:58.445060+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.079569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.079569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836071+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"480'27","reported_seq":1022,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.926620+0000","last_change":"2024-03-12T03:54:21.494844+0000","last_active":"2024-03-12T03:54:58.926620+0000","last_peered":"2024-03-12T03:54:58.926620+0000","last_clean":"2024-03-12T03:54:58.926620+0000","last_became_active":"2024-03-12T03:33:23.164172+0000","last_became_peered":"2024-03-12T03:33:23.164172+0000","last_unstale":"2024-03-12T03:54:58.926620+0000","last_undegraded":"2024-03-12T03:54:58.926620+0000","last_fullsized":"2024-03-12T03:54:58.926620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'27","last_scrub_stamp":"2024-03-12T03:54:21.494701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:54:21.494701+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:47.710993+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":39,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.618225+0000","last_change":"2024-03-12T03:54:29.584355+0000","last_active":"2024-03-12T03:55:02.618225+0000","last_peered":"2024-03-12T03:55:02.618225+0000","last_clean":"2024-03-12T03:55:02.618225+0000","last_became_active":"2024-03-12T03:54:29.584018+0000","last_became_peered":"2024-03-12T03:54:29.584018+0000","last_unstale":"2024-03-12T03:55:02.618225+0000","last_undegraded":"2024-03-12T03:55:02.618225+0000","last_fullsized":"2024-03-12T03:55:02.618225+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:46.344443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_clean_scrub_stamp":"2024-03-12T03:51:46.344443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":984,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.231100+0000","last_change":"2024-03-12T03:54:47.094138+0000","last_active":"2024-03-12T03:55:03.231100+0000","last_peered":"2024-03-12T03:55:03.231100+0000","last_clean":"2024-03-12T03:55:03.231100+0000","last_became_active":"2024-03-12T03:54:47.093214+0000","last_became_peered":"2024-03-12T03:54:47.093214+0000","last_unstale":"2024-03-12T03:55:03.231100+0000","last_undegraded":"2024-03-12T03:55:03.231100+0000","last_fullsized":"2024-03-12T03:55:03.231100+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:48.723073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:50:48.723073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261752+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":744,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:02.621863+0000","last_change":"2024-03-12T03:54:48.137897+0000","last_active":"2024-03-12T03:55:02.621863+0000","last_peered":"2024-03-12T03:55:02.621863+0000","last_clean":"2024-03-12T03:55:02.621863+0000","last_became_active":"2024-03-12T03:54:48.137063+0000","last_became_peered":"2024-03-12T03:54:48.137063+0000","last_unstale":"2024-03-12T03:55:02.621863+0000","last_undegraded":"2024-03-12T03:55:02.621863+0000","last_fullsized":"2024-03-12T03:55:02.621863+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:54.433184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:47:54.433184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,5],"avail_no_missing":["3","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"480'21","reported_seq":39,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.231681+0000","last_change":"2024-03-12T03:54:47.108378+0000","last_active":"2024-03-12T03:55:03.231681+0000","last_peered":"2024-03-12T03:55:03.231681+0000","last_clean":"2024-03-12T03:55:03.231681+0000","last_became_active":"2024-03-12T03:54:47.094495+0000","last_became_peered":"2024-03-12T03:54:47.094495+0000","last_unstale":"2024-03-12T03:55:03.231681+0000","last_undegraded":"2024-03-12T03:55:03.231681+0000","last_fullsized":"2024-03-12T03:55:03.231681+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'15","last_scrub_stamp":"2024-03-12T03:52:20.985668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:52:20.985668+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.262135+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":992,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899441+0000","last_change":"2024-03-12T03:51:47.306023+0000","last_active":"2024-03-12T03:54:58.899441+0000","last_peered":"2024-03-12T03:54:58.899441+0000","last_clean":"2024-03-12T03:54:58.899441+0000","last_became_active":"2024-03-12T03:46:25.283602+0000","last_became_peered":"2024-03-12T03:46:25.283602+0000","last_unstale":"2024-03-12T03:54:58.899441+0000","last_undegraded":"2024-03-12T03:54:58.899441+0000","last_fullsized":"2024-03-12T03:54:58.899441+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:47.305919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:41:18.030146+0000","last_clean_scrub_stamp":"2024-03-12T03:51:47.305919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":751,"reported_epoch":514,"state":"active+undersized","last_fresh":"2024-03-12T03:55:02.620815+0000","last_change":"2024-03-12T03:48:58.460970+0000","last_active":"2024-03-12T03:55:02.620815+0000","last_peered":"2024-03-12T03:55:02.620815+0000","last_clean":"2024-03-12T03:48:53.448497+0000","last_became_active":"2024-03-12T03:48:58.460970+0000","last_became_peered":"2024-03-12T03:48:58.460970+0000","last_unstale":"2024-03-12T03:55:02.620815+0000","last_undegraded":"2024-03-12T03:55:02.620815+0000","last_fullsized":"2024-03-12T03:48:58.444135+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:48:10.625582+0000","last_clean_scrub_stamp":"2024-03-12T03:48:10.625582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":31,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:03.231496+0000","last_change":"2024-03-12T03:54:48.115787+0000","last_active":"2024-03-12T03:55:03.231496+0000","last_peered":"2024-03-12T03:55:03.231496+0000","last_clean":"2024-03-12T03:55:03.231496+0000","last_became_active":"2024-03-12T03:54:48.115373+0000","last_became_peered":"2024-03-12T03:54:48.115373+0000","last_unstale":"2024-03-12T03:55:03.231496+0000","last_undegraded":"2024-03-12T03:55:03.231496+0000","last_fullsized":"2024-03-12T03:55:03.231496+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:13.228117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:48:13.228117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6,5],"avail_no_missing":["1","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"480'34","reported_seq":787,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.622218+0000","last_change":"2024-03-12T03:51:31.509254+0000","last_active":"2024-03-12T03:55:02.622218+0000","last_peered":"2024-03-12T03:55:02.622218+0000","last_clean":"2024-03-12T03:55:02.622218+0000","last_became_active":"2024-03-12T03:46:25.287095+0000","last_became_peered":"2024-03-12T03:46:25.287095+0000","last_unstale":"2024-03-12T03:55:02.622218+0000","last_undegraded":"2024-03-12T03:55:02.622218+0000","last_fullsized":"2024-03-12T03:55:02.622218+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'26","last_scrub_stamp":"2024-03-12T03:51:31.509156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:51:31.509156+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":28,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899996+0000","last_change":"2024-03-12T03:54:47.097340+0000","last_active":"2024-03-12T03:54:58.899996+0000","last_peered":"2024-03-12T03:54:58.899996+0000","last_clean":"2024-03-12T03:54:58.899996+0000","last_became_active":"2024-03-12T03:54:47.096994+0000","last_became_peered":"2024-03-12T03:54:47.096994+0000","last_unstale":"2024-03-12T03:54:58.899996+0000","last_undegraded":"2024-03-12T03:54:58.899996+0000","last_fullsized":"2024-03-12T03:54:58.899996+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:42.096972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:33.687198+0000","last_clean_scrub_stamp":"2024-03-12T03:51:42.096972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":967,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899116+0000","last_change":"2024-03-12T03:51:39.276331+0000","last_active":"2024-03-12T03:54:58.899116+0000","last_peered":"2024-03-12T03:54:58.899116+0000","last_clean":"2024-03-12T03:54:58.899116+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:54:58.899116+0000","last_undegraded":"2024-03-12T03:54:58.899116+0000","last_fullsized":"2024-03-12T03:54:58.899116+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:39.276250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:39.276250+0000","last_clean_scrub_stamp":"2024-03-12T03:51:39.276250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":766,"reported_epoch":514,"state":"active+undersized","last_fresh":"2024-03-12T03:55:02.622298+0000","last_change":"2024-03-12T03:48:58.449798+0000","last_active":"2024-03-12T03:55:02.622298+0000","last_peered":"2024-03-12T03:55:02.622298+0000","last_clean":"2024-03-12T03:48:53.448767+0000","last_became_active":"2024-03-12T03:48:58.449798+0000","last_became_peered":"2024-03-12T03:48:58.449798+0000","last_unstale":"2024-03-12T03:55:02.622298+0000","last_undegraded":"2024-03-12T03:55:02.622298+0000","last_fullsized":"2024-03-12T03:48:58.443857+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:39:38.034285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:39:38.034285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"480'14","reported_seq":760,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619791+0000","last_change":"2024-03-12T03:54:39.950753+0000","last_active":"2024-03-12T03:55:02.619791+0000","last_peered":"2024-03-12T03:55:02.619791+0000","last_clean":"2024-03-12T03:55:02.619791+0000","last_became_active":"2024-03-12T03:46:25.286315+0000","last_became_peered":"2024-03-12T03:46:25.286315+0000","last_unstale":"2024-03-12T03:55:02.619791+0000","last_undegraded":"2024-03-12T03:55:02.619791+0000","last_fullsized":"2024-03-12T03:55:02.619791+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'14","last_scrub_stamp":"2024-03-12T03:54:39.950625+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:54:39.950625+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:56:07.907032+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":607,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:02.618947+0000","last_change":"2024-03-12T03:54:48.130991+0000","last_active":"2024-03-12T03:55:02.618947+0000","last_peered":"2024-03-12T03:55:02.618947+0000","last_clean":"2024-03-12T03:55:02.618947+0000","last_became_active":"2024-03-12T03:54:48.130356+0000","last_became_peered":"2024-03-12T03:54:48.130356+0000","last_unstale":"2024-03-12T03:55:02.618947+0000","last_undegraded":"2024-03-12T03:55:02.618947+0000","last_fullsized":"2024-03-12T03:55:02.618947+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:47:11.851271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:47:11.851271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":438,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.925901+0000","last_change":"2024-03-12T03:54:39.521656+0000","last_active":"2024-03-12T03:54:58.925901+0000","last_peered":"2024-03-12T03:54:58.925901+0000","last_clean":"2024-03-12T03:54:58.925901+0000","last_became_active":"2024-03-12T03:45:18.109983+0000","last_became_peered":"2024-03-12T03:45:18.109983+0000","last_unstale":"2024-03-12T03:54:58.925901+0000","last_undegraded":"2024-03-12T03:54:58.925901+0000","last_fullsized":"2024-03-12T03:54:58.925901+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:39.521553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:54:39.521553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:56:07.797951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":975,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899137+0000","last_change":"2024-03-12T03:52:17.430546+0000","last_active":"2024-03-12T03:54:58.899137+0000","last_peered":"2024-03-12T03:54:58.899137+0000","last_clean":"2024-03-12T03:54:58.899137+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:54:58.899137+0000","last_undegraded":"2024-03-12T03:54:58.899137+0000","last_fullsized":"2024-03-12T03:54:58.899137+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:52:17.430447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:52:17.430447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836268+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"480'39","reported_seq":910,"reported_epoch":514,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:55:03.234807+0000","last_change":"2024-03-12T03:54:48.116035+0000","last_active":"2024-03-12T03:55:03.234807+0000","last_peered":"2024-03-12T03:55:03.234807+0000","last_clean":"2024-03-12T03:55:03.234807+0000","last_became_active":"2024-03-12T03:54:48.115256+0000","last_became_peered":"2024-03-12T03:54:48.115256+0000","last_unstale":"2024-03-12T03:55:03.234807+0000","last_undegraded":"2024-03-12T03:55:03.234807+0000","last_fullsized":"2024-03-12T03:55:03.234807+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'31","last_scrub_stamp":"2024-03-12T03:51:47.045769+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:51:47.045769+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.262003+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6,5],"avail_no_missing":["1","5","6"],"object_location_counts":[{"shards":"1,5,6","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":767,"reported_epoch":514,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:55:02.620639+0000","last_change":"2024-03-12T03:48:58.454467+0000","last_active":"2024-03-12T03:55:02.620639+0000","last_peered":"2024-03-12T03:55:02.620639+0000","last_clean":"2024-03-12T03:48:53.477110+0000","last_became_active":"2024-03-12T03:48:58.454467+0000","last_became_peered":"2024-03-12T03:48:58.454467+0000","last_unstale":"2024-03-12T03:55:02.620639+0000","last_undegraded":"2024-03-12T03:48:58.439427+0000","last_fullsized":"2024-03-12T03:48:58.439320+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:48:00.458460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:48:00.458460+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":555,"reported_epoch":513,"state":"active+undersized","last_fresh":"2024-03-12T03:54:58.900184+0000","last_change":"2024-03-12T03:48:58.461994+0000","last_active":"2024-03-12T03:54:58.900184+0000","last_peered":"2024-03-12T03:54:58.900184+0000","last_clean":"2024-03-12T03:39:31.213817+0000","last_became_active":"2024-03-12T03:48:58.461994+0000","last_became_peered":"2024-03-12T03:48:58.461994+0000","last_unstale":"2024-03-12T03:54:58.900184+0000","last_undegraded":"2024-03-12T03:54:58.900184+0000","last_fullsized":"2024-03-12T03:48:58.446456+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:09.160277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:48:09.160277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:05.915150+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":969,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.233032+0000","last_change":"2024-03-12T03:54:47.098785+0000","last_active":"2024-03-12T03:55:03.233032+0000","last_peered":"2024-03-12T03:55:03.233032+0000","last_clean":"2024-03-12T03:55:03.233032+0000","last_became_active":"2024-03-12T03:54:47.098444+0000","last_became_peered":"2024-03-12T03:54:47.098444+0000","last_unstale":"2024-03-12T03:55:03.233032+0000","last_undegraded":"2024-03-12T03:55:03.233032+0000","last_fullsized":"2024-03-12T03:55:03.233032+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:50:53.919661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:50:53.919661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261781+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"480'37","reported_seq":776,"reported_epoch":513,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:54:58.900282+0000","last_change":"2024-03-12T03:48:58.462533+0000","last_active":"2024-03-12T03:54:58.900282+0000","last_peered":"2024-03-12T03:54:58.900282+0000","last_clean":"2024-03-12T03:43:13.333148+0000","last_became_active":"2024-03-12T03:48:58.462533+0000","last_became_peered":"2024-03-12T03:48:58.462533+0000","last_unstale":"2024-03-12T03:54:58.900282+0000","last_undegraded":"2024-03-12T03:48:58.446989+0000","last_fullsized":"2024-03-12T03:48:58.446871+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-03-12T03:47:15.795515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:47:15.795515+0000","objects_scrubbed":6,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:05.915107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"351'2","reported_seq":987,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.231476+0000","last_change":"2024-03-12T03:54:23.368510+0000","last_active":"2024-03-12T03:55:03.231476+0000","last_peered":"2024-03-12T03:55:03.231476+0000","last_clean":"2024-03-12T03:55:03.231476+0000","last_became_active":"2024-03-12T03:46:25.283344+0000","last_became_peered":"2024-03-12T03:46:25.283344+0000","last_unstale":"2024-03-12T03:55:03.231476+0000","last_undegraded":"2024-03-12T03:55:03.231476+0000","last_fullsized":"2024-03-12T03:55:03.231476+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'2","last_scrub_stamp":"2024-03-12T03:54:23.368282+0000","last_deep_scrub":"351'2","last_deep_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_clean_scrub_stamp":"2024-03-12T03:54:23.368282+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:41.669033+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"441'10","reported_seq":233,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.900186+0000","last_change":"2024-03-12T03:54:19.622194+0000","last_active":"2024-03-12T03:54:58.900186+0000","last_peered":"2024-03-12T03:54:58.900186+0000","last_clean":"2024-03-12T03:54:58.900186+0000","last_became_active":"2024-03-12T03:51:42.833920+0000","last_became_peered":"2024-03-12T03:51:42.833920+0000","last_unstale":"2024-03-12T03:54:58.900186+0000","last_undegraded":"2024-03-12T03:54:58.900186+0000","last_fullsized":"2024-03-12T03:54:58.900186+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'10","last_scrub_stamp":"2024-03-12T03:54:19.622074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:54:19.622074+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:31.141348+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":993,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899316+0000","last_change":"2024-03-12T03:54:47.102409+0000","last_active":"2024-03-12T03:54:58.899316+0000","last_peered":"2024-03-12T03:54:58.899316+0000","last_clean":"2024-03-12T03:54:58.899316+0000","last_became_active":"2024-03-12T03:54:47.089002+0000","last_became_peered":"2024-03-12T03:54:47.089002+0000","last_unstale":"2024-03-12T03:54:58.899316+0000","last_undegraded":"2024-03-12T03:54:58.899316+0000","last_fullsized":"2024-03-12T03:54:58.899316+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:51:28.318860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:28.318860+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"480'30","reported_seq":821,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.618669+0000","last_change":"2024-03-12T03:54:31.125539+0000","last_active":"2024-03-12T03:55:02.618669+0000","last_peered":"2024-03-12T03:55:02.618669+0000","last_clean":"2024-03-12T03:55:02.618669+0000","last_became_active":"2024-03-12T03:46:25.283193+0000","last_became_peered":"2024-03-12T03:46:25.283193+0000","last_unstale":"2024-03-12T03:55:02.618669+0000","last_undegraded":"2024-03-12T03:55:02.618669+0000","last_fullsized":"2024-03-12T03:55:02.618669+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'30","last_scrub_stamp":"2024-03-12T03:54:31.125400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:54:31.125400+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:44.800840+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":994,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.926296+0000","last_change":"2024-03-12T03:54:25.533286+0000","last_active":"2024-03-12T03:54:58.926296+0000","last_peered":"2024-03-12T03:54:58.926296+0000","last_clean":"2024-03-12T03:54:58.926296+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:54:58.926296+0000","last_undegraded":"2024-03-12T03:54:58.926296+0000","last_fullsized":"2024-03-12T03:54:58.926296+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:25.533181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:54:25.533181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:37.275360+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":747,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619142+0000","last_change":"2024-03-12T03:51:42.841557+0000","last_active":"2024-03-12T03:55:02.619142+0000","last_peered":"2024-03-12T03:55:02.619142+0000","last_clean":"2024-03-12T03:55:02.619142+0000","last_became_active":"2024-03-12T03:51:42.841245+0000","last_became_peered":"2024-03-12T03:51:42.841245+0000","last_unstale":"2024-03-12T03:55:02.619142+0000","last_undegraded":"2024-03-12T03:55:02.619142+0000","last_fullsized":"2024-03-12T03:55:02.619142+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:34.600111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:51:34.600111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":788,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619605+0000","last_change":"2024-03-12T03:54:47.090557+0000","last_active":"2024-03-12T03:55:02.619605+0000","last_peered":"2024-03-12T03:55:02.619605+0000","last_clean":"2024-03-12T03:55:02.619605+0000","last_became_active":"2024-03-12T03:54:47.090156+0000","last_became_peered":"2024-03-12T03:54:47.090156+0000","last_unstale":"2024-03-12T03:55:02.619605+0000","last_undegraded":"2024-03-12T03:55:02.619605+0000","last_fullsized":"2024-03-12T03:55:02.619605+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:28.687482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:28.687482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"480'39","reported_seq":779,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.817991+0000","last_change":"2024-03-12T03:54:31.501041+0000","last_active":"2024-03-12T03:54:58.817991+0000","last_peered":"2024-03-12T03:54:58.817991+0000","last_clean":"2024-03-12T03:54:58.817991+0000","last_became_active":"2024-03-12T03:46:25.301700+0000","last_became_peered":"2024-03-12T03:46:25.301700+0000","last_unstale":"2024-03-12T03:54:58.817991+0000","last_undegraded":"2024-03-12T03:54:58.817991+0000","last_fullsized":"2024-03-12T03:54:58.817991+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:54:31.500902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:54:31.500902+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:51.612375+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":156,"reported_epoch":513,"state":"active+undersized","last_fresh":"2024-03-12T03:54:58.899218+0000","last_change":"2024-03-12T03:51:42.833539+0000","last_active":"2024-03-12T03:54:58.899218+0000","last_peered":"2024-03-12T03:54:58.899218+0000","last_clean":"2024-03-12T03:33:20.084273+0000","last_became_active":"2024-03-12T03:51:42.833539+0000","last_became_peered":"2024-03-12T03:51:42.833539+0000","last_unstale":"2024-03-12T03:54:58.899218+0000","last_undegraded":"2024-03-12T03:54:58.899218+0000","last_fullsized":"2024-03-12T03:51:42.829920+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:03.797296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:45:03.797296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.835977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":953,"reported_epoch":514,"state":"active+undersized","last_fresh":"2024-03-12T03:55:03.232743+0000","last_change":"2024-03-12T03:48:58.458438+0000","last_active":"2024-03-12T03:55:03.232743+0000","last_peered":"2024-03-12T03:55:03.232743+0000","last_clean":"2024-03-12T03:48:53.448127+0000","last_became_active":"2024-03-12T03:48:58.458438+0000","last_became_peered":"2024-03-12T03:48:58.458438+0000","last_unstale":"2024-03-12T03:55:03.232743+0000","last_undegraded":"2024-03-12T03:55:03.232743+0000","last_fullsized":"2024-03-12T03:48:58.440067+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:53.016030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:45:53.016030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261588+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":970,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.926597+0000","last_change":"2024-03-12T03:54:47.085880+0000","last_active":"2024-03-12T03:54:58.926597+0000","last_peered":"2024-03-12T03:54:58.926597+0000","last_clean":"2024-03-12T03:54:58.926597+0000","last_became_active":"2024-03-12T03:54:47.085558+0000","last_became_peered":"2024-03-12T03:54:47.085558+0000","last_unstale":"2024-03-12T03:54:58.926597+0000","last_undegraded":"2024-03-12T03:54:58.926597+0000","last_fullsized":"2024-03-12T03:54:58.926597+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:34.537196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:51.952189+0000","last_clean_scrub_stamp":"2024-03-12T03:54:34.537196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:56:02.876782+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"480'18","reported_seq":815,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.620714+0000","last_change":"2024-03-12T03:54:24.217033+0000","last_active":"2024-03-12T03:55:02.620714+0000","last_peered":"2024-03-12T03:55:02.620714+0000","last_clean":"2024-03-12T03:55:02.620714+0000","last_became_active":"2024-03-12T03:51:42.838579+0000","last_became_peered":"2024-03-12T03:51:42.838579+0000","last_unstale":"2024-03-12T03:55:02.620714+0000","last_undegraded":"2024-03-12T03:55:02.620714+0000","last_fullsized":"2024-03-12T03:55:02.620714+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'18","last_scrub_stamp":"2024-03-12T03:54:24.216682+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:54:24.216682+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:43.542006+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":28,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.926407+0000","last_change":"2024-03-12T03:54:47.091788+0000","last_active":"2024-03-12T03:54:58.926407+0000","last_peered":"2024-03-12T03:54:58.926407+0000","last_clean":"2024-03-12T03:54:58.926407+0000","last_became_active":"2024-03-12T03:54:47.091315+0000","last_became_peered":"2024-03-12T03:54:47.091315+0000","last_unstale":"2024-03-12T03:54:58.926407+0000","last_undegraded":"2024-03-12T03:54:58.926407+0000","last_fullsized":"2024-03-12T03:54:58.926407+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:12.075089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:54:12.075089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:20.769262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":966,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.899041+0000","last_change":"2024-03-12T03:54:28.615793+0000","last_active":"2024-03-12T03:54:58.899041+0000","last_peered":"2024-03-12T03:54:58.899041+0000","last_clean":"2024-03-12T03:54:58.899041+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:54:58.899041+0000","last_undegraded":"2024-03-12T03:54:58.899041+0000","last_fullsized":"2024-03-12T03:54:58.899041+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:28.615688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:54:28.615688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:45.737454+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":593,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.231087+0000","last_change":"2024-03-12T03:54:47.098875+0000","last_active":"2024-03-12T03:55:03.231087+0000","last_peered":"2024-03-12T03:55:03.231087+0000","last_clean":"2024-03-12T03:55:03.231087+0000","last_became_active":"2024-03-12T03:54:47.098498+0000","last_became_peered":"2024-03-12T03:54:47.098498+0000","last_unstale":"2024-03-12T03:55:03.231087+0000","last_undegraded":"2024-03-12T03:55:03.231087+0000","last_fullsized":"2024-03-12T03:55:03.231087+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:46.135156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:51:46.135156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.262032+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"480'39","reported_seq":843,"reported_epoch":514,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:55:02.620942+0000","last_change":"2024-03-12T03:51:42.838076+0000","last_active":"2024-03-12T03:55:02.620942+0000","last_peered":"2024-03-12T03:55:02.620942+0000","last_clean":"2024-03-12T03:47:02.086867+0000","last_became_active":"2024-03-12T03:51:42.838076+0000","last_became_peered":"2024-03-12T03:51:42.838076+0000","last_unstale":"2024-03-12T03:55:02.620942+0000","last_undegraded":"2024-03-12T03:51:42.833048+0000","last_fullsized":"2024-03-12T03:51:42.832909+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'33","last_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_deep_scrub":"345'33","last_deep_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_clean_scrub_stamp":"2024-03-12T03:47:02.110517+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.232058+0000","last_change":"2024-03-12T03:54:47.095300+0000","last_active":"2024-03-12T03:55:03.232058+0000","last_peered":"2024-03-12T03:55:03.232058+0000","last_clean":"2024-03-12T03:55:03.232058+0000","last_became_active":"2024-03-12T03:54:47.094791+0000","last_became_peered":"2024-03-12T03:54:47.094791+0000","last_unstale":"2024-03-12T03:55:03.232058+0000","last_undegraded":"2024-03-12T03:55:03.232058+0000","last_fullsized":"2024-03-12T03:55:03.232058+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:13.049449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:54:13.049449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:31.458634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":706,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.899571+0000","last_change":"2024-03-12T03:54:48.093378+0000","last_active":"2024-03-12T03:54:58.899571+0000","last_peered":"2024-03-12T03:54:58.899571+0000","last_clean":"2024-03-12T03:54:58.899571+0000","last_became_active":"2024-03-12T03:54:48.091690+0000","last_became_peered":"2024-03-12T03:54:48.091690+0000","last_unstale":"2024-03-12T03:54:58.899571+0000","last_undegraded":"2024-03-12T03:54:58.899571+0000","last_fullsized":"2024-03-12T03:54:58.899571+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:46:02.618599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:16.541173+0000","last_clean_scrub_stamp":"2024-03-12T03:46:02.618599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.836031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,5],"avail_no_missing":["7","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":269,"reported_epoch":514,"state":"active+undersized","last_fresh":"2024-03-12T03:55:03.231359+0000","last_change":"2024-03-12T03:48:58.463158+0000","last_active":"2024-03-12T03:55:03.231359+0000","last_peered":"2024-03-12T03:55:03.231359+0000","last_clean":"2024-03-12T03:33:26.164901+0000","last_became_active":"2024-03-12T03:48:58.463158+0000","last_became_peered":"2024-03-12T03:48:58.463158+0000","last_unstale":"2024-03-12T03:55:03.231359+0000","last_undegraded":"2024-03-12T03:55:03.231359+0000","last_fullsized":"2024-03-12T03:48:58.439250+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_clean_scrub_stamp":"2024-03-12T03:45:58.095196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"480'74","reported_seq":936,"reported_epoch":514,"state":"active+undersized+degraded","last_fresh":"2024-03-12T03:55:02.620919+0000","last_change":"2024-03-12T03:48:58.457134+0000","last_active":"2024-03-12T03:55:02.620919+0000","last_peered":"2024-03-12T03:55:02.620919+0000","last_clean":"2024-03-12T03:48:53.477142+0000","last_became_active":"2024-03-12T03:48:58.457134+0000","last_became_peered":"2024-03-12T03:48:58.457134+0000","last_unstale":"2024-03-12T03:55:02.620919+0000","last_undegraded":"2024-03-12T03:48:58.441299+0000","last_fullsized":"2024-03-12T03:48:58.441188+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'50","last_scrub_stamp":"2024-03-12T03:43:00.508427+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:43:00.508427+0000","objects_scrubbed":11,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":97,"num_read_kb":84,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":807,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.619250+0000","last_change":"2024-03-12T03:54:38.939844+0000","last_active":"2024-03-12T03:55:02.619250+0000","last_peered":"2024-03-12T03:55:02.619250+0000","last_clean":"2024-03-12T03:55:02.619250+0000","last_became_active":"2024-03-12T03:46:25.279131+0000","last_became_peered":"2024-03-12T03:46:25.279131+0000","last_unstale":"2024-03-12T03:55:02.619250+0000","last_undegraded":"2024-03-12T03:55:02.619250+0000","last_fullsized":"2024-03-12T03:55:02.619250+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:38.939724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:54:38.939724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:56:05.771227+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":167,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.231405+0000","last_change":"2024-03-12T03:54:46.309169+0000","last_active":"2024-03-12T03:55:03.231405+0000","last_peered":"2024-03-12T03:55:03.231405+0000","last_clean":"2024-03-12T03:55:03.231405+0000","last_became_active":"2024-03-12T03:51:42.832427+0000","last_became_peered":"2024-03-12T03:51:42.832427+0000","last_unstale":"2024-03-12T03:55:03.231405+0000","last_undegraded":"2024-03-12T03:55:03.231405+0000","last_fullsized":"2024-03-12T03:55:03.231405+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:46.309061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:54:46.309061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:56:06.523278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":757,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:02.622075+0000","last_change":"2024-03-12T03:54:34.279695+0000","last_active":"2024-03-12T03:55:02.622075+0000","last_peered":"2024-03-12T03:55:02.622075+0000","last_clean":"2024-03-12T03:55:02.622075+0000","last_became_active":"2024-03-12T03:46:25.281081+0000","last_became_peered":"2024-03-12T03:46:25.281081+0000","last_unstale":"2024-03-12T03:55:02.622075+0000","last_undegraded":"2024-03-12T03:55:02.622075+0000","last_fullsized":"2024-03-12T03:55:02.622075+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:34.279352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:54:34.279352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:53.626280+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"480'32","reported_seq":1021,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-03-12T03:54:58.925917+0000","last_change":"2024-03-12T03:54:35.522486+0000","last_active":"2024-03-12T03:54:58.925917+0000","last_peered":"2024-03-12T03:54:58.925917+0000","last_clean":"2024-03-12T03:54:58.925917+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:54:58.925917+0000","last_undegraded":"2024-03-12T03:54:58.925917+0000","last_fullsized":"2024-03-12T03:54:58.925917+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'32","last_scrub_stamp":"2024-03-12T03:54:35.522335+0000","last_deep_scrub":"218'20","last_deep_scrub_stamp":"2024-03-12T03:40:29.325128+0000","last_clean_scrub_stamp":"2024-03-12T03:54:35.522335+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:56.191206+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":323,"reported_epoch":513,"state":"active+clean+remapped","last_fresh":"2024-03-12T03:54:58.817843+0000","last_change":"2024-03-12T03:54:48.116408+0000","last_active":"2024-03-12T03:54:58.817843+0000","last_peered":"2024-03-12T03:54:58.817843+0000","last_clean":"2024-03-12T03:54:58.817843+0000","last_became_active":"2024-03-12T03:54:48.115756+0000","last_became_peered":"2024-03-12T03:54:48.115756+0000","last_unstale":"2024-03-12T03:54:58.817843+0000","last_undegraded":"2024-03-12T03:54:58.817843+0000","last_fullsized":"2024-03-12T03:54:58.817843+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:48:12.186146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:48:12.186146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:00.715137+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6,5],"avail_no_missing":["2","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":981,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.231577+0000","last_change":"2024-03-12T03:54:47.098296+0000","last_active":"2024-03-12T03:55:03.231577+0000","last_peered":"2024-03-12T03:55:03.231577+0000","last_clean":"2024-03-12T03:55:03.231577+0000","last_became_active":"2024-03-12T03:54:47.097792+0000","last_became_peered":"2024-03-12T03:54:47.097792+0000","last_unstale":"2024-03-12T03:55:03.231577+0000","last_undegraded":"2024-03-12T03:55:03.231577+0000","last_fullsized":"2024-03-12T03:55:03.231577+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:54:33.458460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_clean_scrub_stamp":"2024-03-12T03:54:33.458460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:35.621957+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":33,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-03-12T03:55:03.231870+0000","last_change":"2024-03-12T03:54:47.090704+0000","last_active":"2024-03-12T03:55:03.231870+0000","last_peered":"2024-03-12T03:55:03.231870+0000","last_clean":"2024-03-12T03:55:03.231870+0000","last_became_active":"2024-03-12T03:54:47.090372+0000","last_became_peered":"2024-03-12T03:54:47.090372+0000","last_unstale":"2024-03-12T03:55:03.231870+0000","last_undegraded":"2024-03-12T03:55:03.231870+0000","last_fullsized":"2024-03-12T03:55:03.231870+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:51:32.662924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_clean_scrub_stamp":"2024-03-12T03:51:32.662924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:55:07.261894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1291,"num_read_kb":1286,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":6,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":19,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1523928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2177,"ondisk_log_size":2177,"up":49,"acting":54,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":83,"acting":87,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":35,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1816,"num_read_kb":1541,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":3632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":11766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},"log_size":1251,"ondisk_log_size":1251,"up":84,"acting":89,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":81,"acting":90,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":86,"acting":91,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070729,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1002416,"kb_used_data":2212,"kb_used_omap":1,"kb_used_meta":1000190,"kb_avail":92742736,"statfs":{"total":95995035648,"available":94968561664,"internally_reserved":0,"allocated":2265088,"data_stored":1402717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1024195018},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332366,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":926296,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92818856,"statfs":{"total":95995035648,"available":95046508544,"internally_reserved":0,"allocated":2498560,"data_stored":1663222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":946010566},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575143,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1061852,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":1059902,"kb_avail":92683300,"statfs":{"total":95995035648,"available":94907699200,"internally_reserved":0,"allocated":1978368,"data_stored":1075282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1085340101},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738595,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3725208,"kb_used_data":3102608,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":90019944,"statfs":{"total":95995035648,"available":92180422656,"internally_reserved":0,"allocated":3177070592,"data_stored":3175997927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411617,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958796,"kb_used_data":3008,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92786356,"statfs":{"total":95995035648,"available":95013228544,"internally_reserved":0,"allocated":3080192,"data_stored":2253949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":330,"seq":1417339207785,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560012,"kb_used_data":28416,"kb_used_omap":13,"kb_used_meta":531570,"kb_avail":93185140,"statfs":{"total":95995035648,"available":95421583360,"internally_reserved":0,"allocated":29097984,"data_stored":28214886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13485,"internal_metadata":544328531},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953753,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":821616,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":819710,"kb_avail":92923536,"statfs":{"total":95995035648,"available":95153700864,"internally_reserved":0,"allocated":1933312,"data_stored":1074357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":839383493},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":176093659411,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:55:05.926 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:05 smithi173 ceph-mon[33544]: pgmap v1067: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:05.927 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:05 smithi173 ceph-mon[33544]: from='client.15945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:55:05.927 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:05 smithi173 ceph-mon[33544]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-12T03:55:06.501 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-12T03:55:06.528 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-12T03:55:07.847 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:55:07.847 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:55:07.848 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:55:07.848 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:55:07.848 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:55:07.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:07 smithi169 ceph-mon[29344]: pgmap v1068: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 528 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:07 smithi169 ceph-mon[39949]: pgmap v1068: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 528 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:07 smithi173 ceph-mon[33544]: pgmap v1068: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 528 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:08.635 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:55:08.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:08 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1154871716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:55:08.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:08 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 55/690 objects degraded (7.971%), 11 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-03-12T03:55:08.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-12T03:55:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:08 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1154871716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:55:08.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:08 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 55/690 objects degraded (7.971%), 11 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-03-12T03:55:08.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:08 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1154871716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:55:08.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:08 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 55/690 objects degraded (7.971%), 11 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-03-12T03:55:09.158 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:55:09.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:09 smithi169 ceph-mon[29344]: pgmap v1069: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:09.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:09 smithi169 ceph-mon[29344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:55:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:09 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1154871716' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:55:09.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:09 smithi169 ceph-mon[29344]: osdmap e516: 8 total, 7 up, 7 in 2024-03-12T03:55:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:09 smithi169 ceph-mon[39949]: pgmap v1069: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:09 smithi169 ceph-mon[39949]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:55:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:09 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1154871716' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:55:09.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:09 smithi169 ceph-mon[39949]: osdmap e516: 8 total, 7 up, 7 in 2024-03-12T03:55:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:09 smithi173 ceph-mon[33544]: pgmap v1069: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:09 smithi173 ceph-mon[33544]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-12T03:55:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:09 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1154871716' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:55:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:09 smithi173 ceph-mon[33544]: osdmap e516: 8 total, 7 up, 7 in 2024-03-12T03:55:10.870 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-12T03:55:10.871 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-03-12T03:55:10.871 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-03-12T03:55:10.871 DEBUG:teuthology.orchestra.run.smithi169:osd.0> sudo journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.0.service 2024-03-12T03:55:10.874 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl start ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.0 2024-03-12T03:55:10.901 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:10 smithi173 ceph-mon[33544]: 1.11 scrub starts 2024-03-12T03:55:10.902 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:10 smithi173 ceph-mon[33544]: 1.11 scrub ok 2024-03-12T03:55:10.902 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:10 smithi173 ceph-mon[33544]: 3.10 scrub starts 2024-03-12T03:55:10.902 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:10 smithi173 ceph-mon[33544]: 3.10 scrub ok 2024-03-12T03:55:10.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:10 smithi169 ceph-mon[39949]: 1.11 scrub starts 2024-03-12T03:55:10.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:10 smithi169 ceph-mon[39949]: 1.11 scrub ok 2024-03-12T03:55:10.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:10 smithi169 ceph-mon[39949]: 3.10 scrub starts 2024-03-12T03:55:10.903 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:10 smithi169 ceph-mon[39949]: 3.10 scrub ok 2024-03-12T03:55:10.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:10 smithi169 ceph-mon[29344]: 1.11 scrub starts 2024-03-12T03:55:10.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:10 smithi169 ceph-mon[29344]: 1.11 scrub ok 2024-03-12T03:55:10.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:10 smithi169 ceph-mon[29344]: 3.10 scrub starts 2024-03-12T03:55:10.903 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:10 smithi169 ceph-mon[29344]: 3.10 scrub ok 2024-03-12T03:55:11.171 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:79299 10.a 10:59849233:::benchmark_data_smithi169_162694_object79298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.568308+0000", 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 373.59896660800001, 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90207626699999999, 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:55:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 79299 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.568308+0000", 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.568308+0000", 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.568310+0000", 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7999999999999999e-06 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.568357+0000", 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6239999999999998e-05 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.568367+0000", 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0558e-05 2024-03-12T03:55:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.568377+0000", 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4949999999999997e-06 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.573277+0000", 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049008699999999999 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.573286+0000", 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7930000000000003e-06 2024-03-12T03:55:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.463254+0000", 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88996800499999995 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.465778+0000", 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025236220000000001 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6", 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.465958+0000", 2024-03-12T03:55:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018031899999999999 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.469732+0000", 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037739200000000001 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.470222+0000", 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00049023199999999995 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.470284+0000", 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1290000000000004e-05 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.470385+0000", 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101123 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:79304 10.6 10:608aa570:::benchmark_data_smithi169_162694_object79303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:55:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.577428+0000", 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 373.58984740599999, 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90310600699999999, 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 79304 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.577428+0000", 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.577428+0000", 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.577429+0000", 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5400000000000001e-06 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.577472+0000", 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2611999999999998e-05 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.577486+0000", 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3920000000000001e-05 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.577495+0000", 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7739999999999993e-06 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.577545+0000", 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0485999999999997e-05 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.577551+0000", 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2589999999999997e-06 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.467569+0000", 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89001797100000002 2024-03-12T03:55:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.467882+0000", 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031232100000000001 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4", 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.468088+0000", 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000206257 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.475433+0000", 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0073451990000000002 2024-03-12T03:55:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.480327+0000", 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048936279999999997 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.480429+0000", 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010192900000000001 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.480534+0000", 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000105111 2024-03-12T03:55:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25463.0:79312 10.6 10:66200d32:::benchmark_data_smithi169_162694_object79311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.587955+0000", 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 373.57932044699999, 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89280468199999996, 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25463", 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1331914179", 2024-03-12T03:55:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 79312 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.587955+0000", 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.587955+0000", 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.587956+0000", 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.666e-06 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.587998+0000", 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1808999999999999e-05 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.588012+0000", 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3553999999999999e-05 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.588019+0000", 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0210000000000002e-06 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.588072+0000", 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3491e-05 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.588080+0000", 2024-03-12T03:55:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8220000000000007e-06 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.471517+0000", 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88343746199999995 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.471643+0000", 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000125318 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4", 2024-03-12T03:55:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.471756+0000", 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011332099999999999 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.477028+0000", 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0052715699999999997 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.480602+0000", 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035745009999999999 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.480660+0000", 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8097999999999998e-05 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.480759+0000", 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9049000000000007e-05 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24359.0:3629 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94067641426944 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-12T03:55:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:58.204160+0000", 2024-03-12T03:55:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 372.96311472299999, 2024-03-12T03:55:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25559225099999999, 2024-03-12T03:55:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:55:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24359", 2024-03-12T03:55:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:55:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 3629 2024-03-12T03:55:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204160+0000", 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204160+0000", 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204161+0000", 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4000000000000001e-07 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204164+0000", 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3840000000000001e-06 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204171+0000", 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7039999999999997e-06 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204177+0000", 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0920000000000003e-06 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204224+0000", 2024-03-12T03:55:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7243999999999997e-05 2024-03-12T03:55:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.204231+0000", 2024-03-12T03:55:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2899999999999999e-06 2024-03-12T03:55:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.449790+0000", 2024-03-12T03:55:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.245559208 2024-03-12T03:55:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.449991+0000", 2024-03-12T03:55:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000200795 2024-03-12T03:55:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:58.459753+0000", 2024-03-12T03:55:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0097618940000000001 2024-03-12T03:55:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24359.0:3967 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94067641428096 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:51:41.845082+0000", 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 209.322193328, 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99121555699999997, 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24359", 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 3967 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845082+0000", 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845082+0000", 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845082+0000", 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0800000000000004e-07 2024-03-12T03:55:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.203 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:10 smithi169 systemd[1]: Starting Ceph osd.0 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:55:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845086+0000", 2024-03-12T03:55:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3919999999999999e-06 2024-03-12T03:55:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845093+0000", 2024-03-12T03:55:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6090000000000003e-06 2024-03-12T03:55:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845098+0000", 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8659999999999997e-06 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845140+0000", 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2109e-05 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845150+0000", 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3819999999999993e-06 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:42.834074+0000", 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98892400000000003 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:42.834258+0000", 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000183928 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:42.836297+0000", 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020396630000000002 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:69052 11.15 11:a84f5948:::benchmark_data_smithi169_185434_object69051:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:52:29.543404+0000", 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 161.62387104199999, 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16067119299999999, 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 69052 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.543404+0000", 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.543404+0000", 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.543406+0000", 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.511e-06 2024-03-12T03:55:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.543500+0000", 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4820000000000004e-05 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.543516+0000", 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5778999999999999e-05 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.543525+0000", 2024-03-12T03:55:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7329999999999992e-06 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.543593+0000", 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7731999999999995e-05 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.543744+0000", 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015118400000000001 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2,4", 2024-03-12T03:55:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.543902+0000", 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000158634 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.545244+0000", 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013418620000000001 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.545994+0000", 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00074939400000000001 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.546396+0000", 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040260600000000001 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.702363+0000", 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15596668299999999 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.702412+0000", 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8612999999999997e-05 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:29.704075+0000", 2024-03-12T03:55:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016636419999999999 2024-03-12T03:55:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:81075 11.7 11:e527d2f0:::benchmark_data_smithi169_185434_object81074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:55:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:52:42.840821+0000", 2024-03-12T03:55:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 148.32645391, 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20309632799999999, 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81075 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:42.840821+0000", 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:42.840821+0000", 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:42.840823+0000", 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6619999999999999e-06 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:42.840855+0000", 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2301999999999997e-05 2024-03-12T03:55:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:42.840865+0000", 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6180000000000001e-06 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:42.840871+0000", 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.545e-06 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:42.840998+0000", 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012707699999999999 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:42.841157+0000", 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015867999999999999 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:42.841304+0000", 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000147365 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:42.844604+0000", 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032996969999999999 2024-03-12T03:55:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:42.844677+0000", 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3386999999999994e-05 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.043842+0000", 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19916463500000001 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.043882+0000", 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9685999999999999e-05 2024-03-12T03:55:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.043917+0000", 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5673999999999998e-05 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:81374 11.15 11:a8ceb172:::benchmark_data_smithi169_185434_object81373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:52:43.313198+0000", 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 147.854077332, 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.124708394, 2024-03-12T03:55:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81374 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.313198+0000", 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.313198+0000", 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.313199+0000", 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.734e-06 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.313260+0000", 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0146999999999999e-05 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.313275+0000", 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5849e-05 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.313285+0000", 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3700000000000001e-06 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.313350+0000", 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4697999999999996e-05 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.313519+0000", 2024-03-12T03:55:11.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016992900000000001 2024-03-12T03:55:11.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2,4", 2024-03-12T03:55:11.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.313670+0000", 2024-03-12T03:55:11.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000150307 2024-03-12T03:55:11.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.315012+0000", 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001342193 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.316405+0000", 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013930699999999999 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.429538+0000", 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.113132623 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.436123+0000", 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0065851379999999999 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.436226+0000", 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103124 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.437906+0000", 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016802119999999999 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.225 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:81549 11.15 11:aa4d9791:::benchmark_data_smithi169_185434_object81548:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:52:43.647771+0000", 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 147.51950396800001, 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12817711400000001, 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81549 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.226 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.647771+0000", 2024-03-12T03:55:11.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.647771+0000", 2024-03-12T03:55:11.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.647773+0000", 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6789999999999999e-06 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.647803+0000", 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0363999999999999e-05 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.647814+0000", 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1246999999999999e-05 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.647822+0000", 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7999999999999999e-06 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.647884+0000", 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2181000000000007e-05 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.648083+0000", 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000198517 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2,4", 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.648274+0000", 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000191215 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.650056+0000", 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001781464 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.652198+0000", 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021423900000000001 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.660006+0000", 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0078081089999999997 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.774436+0000", 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.114429804 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.774500+0000", 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4306000000000004e-05 2024-03-12T03:55:11.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.775948+0000", 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001448038 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:81555 11.7 11:e1f60130:::benchmark_data_smithi169_185434_object81554:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:52:43.652634+0000", 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 147.51464132300001, 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.126047354, 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.232 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81555 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.652634+0000", 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.652634+0000", 2024-03-12T03:55:11.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.652635+0000", 2024-03-12T03:55:11.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.587e-06 2024-03-12T03:55:11.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.652696+0000", 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0915999999999997e-05 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.652707+0000", 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1138000000000001e-05 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.652715+0000", 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.055e-06 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.652771+0000", 2024-03-12T03:55:11.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5874000000000001e-05 2024-03-12T03:55:11.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.652921+0000", 2024-03-12T03:55:11.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015013299999999999 2024-03-12T03:55:11.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.653062+0000", 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000140092 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.654637+0000", 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015755249999999999 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.655504+0000", 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086684100000000003 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.778594+0000", 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.123089807 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.778639+0000", 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5008e-05 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:43.778681+0000", 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2378e-05 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:82143 11.15 11:ae581267:::benchmark_data_smithi169_185434_object82142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:52:44.412241+0000", 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 146.75503431199999, 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14284587900000001, 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 82143 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.412241+0000", 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.412241+0000", 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.412243+0000", 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8059999999999999e-06 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.412297+0000", 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4259999999999999e-05 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.412311+0000", 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3828999999999999e-05 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.412320+0000", 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6609999999999995e-06 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.412388+0000", 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8062e-05 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.412564+0000", 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000175513 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2,4", 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.412729+0000", 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016479999999999999 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.414170+0000", 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014408170000000001 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.415514+0000", 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013444640000000001 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.415725+0000", 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021080599999999999 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.553457+0000", 2024-03-12T03:55:11.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13773176600000001 2024-03-12T03:55:11.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.553519+0000", 2024-03-12T03:55:11.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2528999999999997e-05 2024-03-12T03:55:11.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:44.555087+0000", 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015675660000000001 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:93573 11.7 11:e19cd0fc:::benchmark_data_smithi169_185434_object93572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e465)", 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:52:56.811944+0000", 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 134.355330791, 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.120791565, 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.245 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93573 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.811944+0000", 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.811944+0000", 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.811946+0000", 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.547e-06 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.811993+0000", 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7561000000000001e-05 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.812005+0000", 2024-03-12T03:55:11.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2013e-05 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.812012+0000", 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2190000000000001e-06 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.812068+0000", 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6706999999999998e-05 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.812220+0000", 2024-03-12T03:55:11.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000151278 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.812602+0000", 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038240000000000003 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.813892+0000", 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012903179999999999 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.814767+0000", 2024-03-12T03:55:11.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00087478000000000002 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.932644+0000", 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.117876552 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.932693+0000", 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9137000000000001e-05 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:52:56.932736+0000", 2024-03-12T03:55:11.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3053000000000003e-05 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:98414 11.7 11:e0a845fe:::benchmark_data_smithi169_185434_object98413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e465)", 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:53:02.203025+0000", 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 128.96425040299999, 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13438682599999999, 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.251 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98414 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.203025+0000", 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.203025+0000", 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.203026+0000", 2024-03-12T03:55:11.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.762e-06 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.203091+0000", 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4868000000000003e-05 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.203106+0000", 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4209999999999999e-05 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.203114+0000", 2024-03-12T03:55:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5040000000000002e-06 2024-03-12T03:55:11.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.203176+0000", 2024-03-12T03:55:11.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2483999999999993e-05 2024-03-12T03:55:11.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.203340+0000", 2024-03-12T03:55:11.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016378200000000001 2024-03-12T03:55:11.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.203708+0000", 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000367456 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.205093+0000", 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013848840000000001 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.206307+0000", 2024-03-12T03:55:11.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012141280000000001 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.337277+0000", 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13097066900000001 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.337348+0000", 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0168999999999999e-05 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.337411+0000", 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3910000000000003e-05 2024-03-12T03:55:11.256 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:98640 11.7 11:e52908a7:::benchmark_data_smithi169_185434_object98639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e465)", 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:53:02.522808+0000", 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 128.64446695500001, 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11983316099999999, 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98640 2024-03-12T03:55:11.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.522808+0000", 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.522808+0000", 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.522810+0000", 2024-03-12T03:55:11.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4440000000000001e-06 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.522853+0000", 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3489999999999999e-05 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.522870+0000", 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7088e-05 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.522880+0000", 2024-03-12T03:55:11.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5889999999999992e-06 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.522941+0000", 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1179999999999994e-05 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.523116+0000", 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000175057 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-03-12T03:55:11.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.523512+0000", 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039604799999999997 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.527313+0000", 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038009430000000002 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.529113+0000", 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017995960000000001 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.642526+0000", 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11341340499999999 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.642588+0000", 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2349999999999998e-05 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.642641+0000", 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2970999999999998e-05 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.262 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:98861 11.15 11:a8b48c00:::benchmark_data_smithi169_185434_object98860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e465)", 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:53:02.816865+0000", 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 128.35041013599999, 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15503995300000001, 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98861 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.816865+0000", 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.816865+0000", 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.816866+0000", 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.395e-06 2024-03-12T03:55:11.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.816911+0000", 2024-03-12T03:55:11.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4790999999999999e-05 2024-03-12T03:55:11.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.816927+0000", 2024-03-12T03:55:11.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5553000000000002e-05 2024-03-12T03:55:11.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.816944+0000", 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7305000000000001e-05 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.817031+0000", 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6742000000000004e-05 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.817197+0000", 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016660500000000001 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2,4", 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.817574+0000", 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00037647599999999999 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.819818+0000", 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002244069 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.822627+0000", 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028087469999999999 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.823656+0000", 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001029692 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.970496+0000", 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14684012699999999 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.970564+0000", 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7788999999999998e-05 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:02.971905+0000", 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013406620000000001 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:101180 11.15 11:aa18c6aa:::benchmark_data_smithi169_185434_object101179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e465)", 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:53:05.408978+0000", 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 125.75829727, 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.122037281, 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.269 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 101180 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.408978+0000", 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.408978+0000", 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.408980+0000", 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2419999999999999e-06 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.409029+0000", 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9212000000000002e-05 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.409044+0000", 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5167999999999999e-05 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.409053+0000", 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3720000000000005e-06 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.415385+0000", 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0063321289999999997 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.415585+0000", 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000199717 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2,4", 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.415992+0000", 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040702300000000001 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.417471+0000", 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001479068 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.419922+0000", 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024511480000000002 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.419985+0000", 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2844999999999999e-05 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.529208+0000", 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10922353899999999 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.529283+0000", 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5176e-05 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.531015+0000", 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001731642 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:101183 11.7 11:e4de2d4c:::benchmark_data_smithi169_185434_object101182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e465)", 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:53:05.413995+0000", 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 125.753280438, 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13249688400000001, 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.275 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 101183 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.413995+0000", 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.413995+0000", 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.413997+0000", 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.871e-06 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.414044+0000", 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7311999999999997e-05 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.414060+0000", 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6314999999999999e-05 2024-03-12T03:55:11.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.414070+0000", 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9089999999999995e-06 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.414134+0000", 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4302999999999997e-05 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.414260+0000", 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012552599999999999 2024-03-12T03:55:11.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.414582+0000", 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032167699999999998 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.416220+0000", 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016381709999999999 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.416855+0000", 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00063510699999999999 2024-03-12T03:55:11.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.546375+0000", 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.129519882 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.546441+0000", 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6791999999999993e-05 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.546492+0000", 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0018999999999997e-05 2024-03-12T03:55:11.280 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:101407 11.7 11:e2eb8523:::benchmark_data_smithi169_185434_object101406:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e465)", 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:53:05.720732+0000", 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 125.446543006, 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17405741399999999, 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 101407 2024-03-12T03:55:11.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.282 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.720732+0000", 2024-03-12T03:55:11.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.720732+0000", 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.720734+0000", 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.522e-06 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.720799+0000", 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5592999999999994e-05 2024-03-12T03:55:11.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.720815+0000", 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5937999999999999e-05 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.720824+0000", 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2760000000000006e-06 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.720879+0000", 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4870999999999997e-05 2024-03-12T03:55:11.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.721028+0000", 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014836499999999999 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.721410+0000", 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038207600000000002 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.726141+0000", 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.004731369 2024-03-12T03:55:11.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.726578+0000", 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00043694099999999999 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.894682+0000", 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16810370999999999 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.894745+0000", 2024-03-12T03:55:11.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3124000000000001e-05 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:05.894789+0000", 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4629000000000002e-05 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:102638 11.7 11:e45f5fda:::benchmark_data_smithi169_185434_object102637:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:53:07.845314+0000", 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 123.321960619, 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.142757192, 2024-03-12T03:55:11.287 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102638 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.845314+0000", 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.845314+0000", 2024-03-12T03:55:11.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.845316+0000", 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3e-06 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.845410+0000", 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4678000000000004e-05 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.845425+0000", 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4348000000000001e-05 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.845433+0000", 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8949999999999998e-06 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.845486+0000", 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3029000000000003e-05 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:11.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.845621+0000", 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013502300000000001 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.846096+0000", 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047548499999999999 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.848049+0000", 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019528289999999999 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.849566+0000", 2024-03-12T03:55:11.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001517298 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.987976+0000", 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13841013799999999 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.988027+0000", 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0648e-05 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:53:07.988072+0000", 2024-03-12T03:55:11.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4521000000000003e-05 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:219715 11.13 11.48f7fff3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:46.130263+0000", 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 25.037011765999999, 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94481432499999995, 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:11.293 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 219715 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.130263+0000", 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.130263+0000", 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:11.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.130265+0000", 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9530000000000002e-06 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.130273+0000", 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0469999999999994e-06 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.130293+0000", 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9411000000000001e-05 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.130303+0000", 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0550000000000001e-05 2024-03-12T03:55:11.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.130378+0000", 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4678000000000006e-05 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.130389+0000", 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1464999999999999e-05 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.075060+0000", 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94467060700000005 2024-03-12T03:55:11.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:11.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:11.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:11.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.075078+0000", 2024-03-12T03:55:11.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7614000000000001e-05 2024-03-12T03:55:11.297 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.297 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.297 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.297 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:11.297 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:11.297 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:55:11.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-12T03:55:11.508 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 podman[201028]: 2024-03-12 03:55:11.118477852 +0000 UTC m=+0.017984591 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:55:11.509 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 podman[201028]: 2024-03-12 03:55:11.220486612 +0000 UTC m=+0.119993352 container create 545d63d50b48d6fff20776d31bb19e7e56fe7e9753327796c035763f8dd760e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-12T03:55:11.509 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 podman[201028]: 2024-03-12 03:55:11.42326599 +0000 UTC m=+0.322772745 container init 545d63d50b48d6fff20776d31bb19e7e56fe7e9753327796c035763f8dd760e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:55:11.509 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 podman[201028]: 2024-03-12 03:55:11.427386257 +0000 UTC m=+0.326893004 container start 545d63d50b48d6fff20776d31bb19e7e56fe7e9753327796c035763f8dd760e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_CLEAN=True) 2024-03-12T03:55:11.509 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 podman[201028]: 2024-03-12 03:55:11.506700417 +0000 UTC m=+0.406207173 container attach 545d63d50b48d6fff20776d31bb19e7e56fe7e9753327796c035763f8dd760e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-12T03:55:11.648 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:11.774 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-mon[29344]: pgmap v1071: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 665 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:11.775 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-mon[29344]: 1.1 scrub starts 2024-03-12T03:55:11.775 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-mon[29344]: 1.1 scrub ok 2024-03-12T03:55:11.775 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-mon[39949]: pgmap v1071: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 665 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:11.775 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-mon[39949]: 1.1 scrub starts 2024-03-12T03:55:11.775 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-mon[39949]: 1.1 scrub ok 2024-03-12T03:55:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:11 smithi173 ceph-mon[33544]: pgmap v1071: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 665 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:11 smithi173 ceph-mon[33544]: 1.1 scrub starts 2024-03-12T03:55:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:11 smithi173 ceph-mon[33544]: 1.1 scrub ok 2024-03-12T03:55:12.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:55:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:55:12.174 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate[201042]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-12T03:55:12.174 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate[201042]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-12T03:55:12.174 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 bash[201028]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-12T03:55:12.175 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 bash[201028]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-12T03:55:12.175 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 bash[201028]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-12T03:55:12.175 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 bash[201028]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-12T03:55:12.175 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 bash[201028]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-12T03:55:12.175 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 bash[201028]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-12T03:55:12.175 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 bash[201028]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-12T03:55:12.175 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate[201042]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-12T03:55:12.175 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate[201042]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-12T03:55:12.175 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate[201042]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-12T03:55:12.175 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate[201042]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-12T03:55:12.175 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate[201042]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-12T03:55:12.175 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:12 smithi169 podman[201028]: 2024-03-12 03:55:12.007485012 +0000 UTC m=+0.906991758 container died 545d63d50b48d6fff20776d31bb19e7e56fe7e9753327796c035763f8dd760e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-2b0101e) 2024-03-12T03:55:12.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-12T03:55:12.625 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:55:13.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:55:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:55:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:13 smithi173 ceph-mon[33544]: pgmap v1072: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:13 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:55:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:13 smithi173 ceph-mon[33544]: 4.3 scrub starts 2024-03-12T03:55:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:13 smithi173 ceph-mon[33544]: 4.3 scrub ok 2024-03-12T03:55:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:13 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:55:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:13 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:55:13.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:13 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:13.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:13 smithi173 ceph-mon[33544]: osdmap e517: 8 total, 7 up, 7 in 2024-03-12T03:55:13.923 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:13 smithi169 podman[201237]: 2024-03-12 03:55:13.703879524 +0000 UTC m=+1.691376463 container remove 545d63d50b48d6fff20776d31bb19e7e56fe7e9753327796c035763f8dd760e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-activate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-03-12T03:55:13.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[39949]: pgmap v1072: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:55:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[39949]: 4.3 scrub starts 2024-03-12T03:55:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[39949]: 4.3 scrub ok 2024-03-12T03:55:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:55:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:55:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:13.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[39949]: osdmap e517: 8 total, 7 up, 7 in 2024-03-12T03:55:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[29344]: pgmap v1072: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:55:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[29344]: 4.3 scrub starts 2024-03-12T03:55:13.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[29344]: 4.3 scrub ok 2024-03-12T03:55:13.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:55:13.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:55:13.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:13.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:13 smithi169 ceph-mon[29344]: osdmap e517: 8 total, 7 up, 7 in 2024-03-12T03:55:14.554 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:55:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:55:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-03-12T03:55:14.554 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:55:14.588 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:14 smithi169 podman[201497]: 2024-03-12 03:55:14.23316384 +0000 UTC m=+0.017972307 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:55:14.588 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:14 smithi169 podman[201497]: 2024-03-12 03:55:14.346269761 +0000 UTC m=+0.131078215 container create 7ff85389739434cf5ab42c0ec739c1b88b3d6aed59698a42c8d4c43876b0ee52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:55:14.589 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:14 smithi169 podman[201497]: 2024-03-12 03:55:14.586422716 +0000 UTC m=+0.371231170 container init 7ff85389739434cf5ab42c0ec739c1b88b3d6aed59698a42c8d4c43876b0ee52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-12T03:55:14.650 INFO:teuthology.orchestra.run.smithi169.stderr:noscrub is unset 2024-03-12T03:55:14.703 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-12T03:55:14.857 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:14 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:14.857 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:14 smithi173 ceph-mon[33544]: 3.17 scrub starts 2024-03-12T03:55:14.857 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:14 smithi173 ceph-mon[33544]: 3.17 scrub ok 2024-03-12T03:55:14.857 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:14 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3398375058' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:55:14.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:14 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:14.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:14 smithi169 ceph-mon[39949]: 3.17 scrub starts 2024-03-12T03:55:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:14 smithi169 ceph-mon[39949]: 3.17 scrub ok 2024-03-12T03:55:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:14 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3398375058' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:55:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:14 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:14 smithi169 ceph-mon[29344]: 3.17 scrub starts 2024-03-12T03:55:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:14 smithi169 ceph-mon[29344]: 3.17 scrub ok 2024-03-12T03:55:14.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:14 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3398375058' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-12T03:55:14.924 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:14 smithi169 podman[201497]: 2024-03-12 03:55:14.589744791 +0000 UTC m=+0.374553256 container start 7ff85389739434cf5ab42c0ec739c1b88b3d6aed59698a42c8d4c43876b0ee52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-12T03:55:14.924 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:14 smithi169 ceph-osd[201517]: -- 172.21.15.169:0/3505037545 <== mon.0 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f40b0d3680 con 0x55f40b09cc00 2024-03-12T03:55:14.924 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:14 smithi169 bash[201497]: 7ff85389739434cf5ab42c0ec739c1b88b3d6aed59698a42c8d4c43876b0ee52 2024-03-12T03:55:14.925 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:14 smithi169 systemd[1]: Started Ceph osd.0 for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:55:15.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-12T03:55:15.369 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-12T03:55:15.423 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0[201513]: 2024-03-12T03:55:15.161+0000 7fa78c2067c0 -1 Falling back to public interface 2024-03-12T03:55:15.471 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:55:15.483 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:15.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[39949]: pgmap v1074: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:15.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3398375058' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:55:15.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[39949]: osdmap e518: 8 total, 7 up, 7 in 2024-03-12T03:55:15.872 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[39949]: 6.a scrub starts 2024-03-12T03:55:15.872 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[39949]: 6.a scrub ok 2024-03-12T03:55:15.872 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[39949]: 6.11 scrub starts 2024-03-12T03:55:15.872 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[39949]: 6.11 scrub ok 2024-03-12T03:55:15.872 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[29344]: pgmap v1074: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:15.872 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3398375058' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:55:15.872 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[29344]: osdmap e518: 8 total, 7 up, 7 in 2024-03-12T03:55:15.872 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[29344]: 6.a scrub starts 2024-03-12T03:55:15.872 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[29344]: 6.a scrub ok 2024-03-12T03:55:15.872 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[29344]: 6.11 scrub starts 2024-03-12T03:55:15.872 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:15 smithi169 ceph-mon[29344]: 6.11 scrub ok 2024-03-12T03:55:15.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:15 smithi173 ceph-mon[33544]: pgmap v1074: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:15.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:15 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3398375058' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-12T03:55:15.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:15 smithi173 ceph-mon[33544]: osdmap e518: 8 total, 7 up, 7 in 2024-03-12T03:55:15.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:15 smithi173 ceph-mon[33544]: 6.a scrub starts 2024-03-12T03:55:15.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:15 smithi173 ceph-mon[33544]: 6.a scrub ok 2024-03-12T03:55:15.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:15 smithi173 ceph-mon[33544]: 6.11 scrub starts 2024-03-12T03:55:15.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:15 smithi173 ceph-mon[33544]: 6.11 scrub ok 2024-03-12T03:55:16.193 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:16.890 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:16 smithi173 ceph-mon[33544]: 5.15 scrub starts 2024-03-12T03:55:16.891 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:16 smithi173 ceph-mon[33544]: 5.15 scrub ok 2024-03-12T03:55:16.891 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:16 smithi173 ceph-mon[33544]: 5.b scrub starts 2024-03-12T03:55:16.891 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:16 smithi173 ceph-mon[33544]: 5.b scrub ok 2024-03-12T03:55:16.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:16 smithi169 ceph-mon[39949]: 5.15 scrub starts 2024-03-12T03:55:16.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:16 smithi169 ceph-mon[39949]: 5.15 scrub ok 2024-03-12T03:55:16.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:16 smithi169 ceph-mon[39949]: 5.b scrub starts 2024-03-12T03:55:16.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:16 smithi169 ceph-mon[39949]: 5.b scrub ok 2024-03-12T03:55:16.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:16 smithi169 ceph-mon[29344]: 5.15 scrub starts 2024-03-12T03:55:16.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:16 smithi169 ceph-mon[29344]: 5.15 scrub ok 2024-03-12T03:55:16.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:16 smithi169 ceph-mon[29344]: 5.b scrub starts 2024-03-12T03:55:16.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:16 smithi169 ceph-mon[29344]: 5.b scrub ok 2024-03-12T03:55:17.241 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:55:17.241 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:55:17.242 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:55:17.242 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:55:17.242 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:55:17.923 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:17 smithi173 ceph-mon[33544]: pgmap v1076: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 666 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:17.923 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:17 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:17.923 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:17 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:17.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:17 smithi169 ceph-mon[39949]: pgmap v1076: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 666 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:17.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:17 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:17.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:17 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:17.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:17 smithi169 ceph-mon[29344]: pgmap v1076: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 666 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:17.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:17 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:17.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:17 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:17.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-12T03:55:18.175 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:55:18.730 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[29344]: 5.7 scrub starts 2024-03-12T03:55:18.731 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[29344]: 5.7 scrub ok 2024-03-12T03:55:18.731 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[29344]: 5.2 scrub starts 2024-03-12T03:55:18.731 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[29344]: 5.2 scrub ok 2024-03-12T03:55:18.731 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[29344]: 5.3 scrub starts 2024-03-12T03:55:18.731 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[29344]: 5.3 scrub ok 2024-03-12T03:55:18.731 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[29344]: osdmap e519: 8 total, 7 up, 7 in 2024-03-12T03:55:18.731 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[39949]: 5.7 scrub starts 2024-03-12T03:55:18.731 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[39949]: 5.7 scrub ok 2024-03-12T03:55:18.731 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[39949]: 5.2 scrub starts 2024-03-12T03:55:18.731 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[39949]: 5.2 scrub ok 2024-03-12T03:55:18.731 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[39949]: 5.3 scrub starts 2024-03-12T03:55:18.731 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[39949]: 5.3 scrub ok 2024-03-12T03:55:18.732 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-mon[39949]: osdmap e519: 8 total, 7 up, 7 in 2024-03-12T03:55:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:18 smithi173 ceph-mon[33544]: 5.7 scrub starts 2024-03-12T03:55:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:18 smithi173 ceph-mon[33544]: 5.7 scrub ok 2024-03-12T03:55:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:18 smithi173 ceph-mon[33544]: 5.2 scrub starts 2024-03-12T03:55:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:18 smithi173 ceph-mon[33544]: 5.2 scrub ok 2024-03-12T03:55:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:18 smithi173 ceph-mon[33544]: 5.3 scrub starts 2024-03-12T03:55:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:18 smithi173 ceph-mon[33544]: 5.3 scrub ok 2024-03-12T03:55:18.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:18 smithi173 ceph-mon[33544]: osdmap e519: 8 total, 7 up, 7 in 2024-03-12T03:55:19.097 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:18 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0[201513]: 2024-03-12T03:55:18.875+0000 7fa78c2067c0 -1 osd.0 383 log_to_monitors true 2024-03-12T03:55:19.446 INFO:teuthology.orchestra.run.smithi169.stderr:no valid command found; 10 closest matches: 2024-03-12T03:55:19.447 INFO:teuthology.orchestra.run.smithi169.stderr:0 2024-03-12T03:55:19.447 INFO:teuthology.orchestra.run.smithi169.stderr:1 2024-03-12T03:55:19.447 INFO:teuthology.orchestra.run.smithi169.stderr:2 2024-03-12T03:55:19.447 INFO:teuthology.orchestra.run.smithi169.stderr:abort 2024-03-12T03:55:19.447 INFO:teuthology.orchestra.run.smithi169.stderr:assert 2024-03-12T03:55:19.447 INFO:teuthology.orchestra.run.smithi169.stderr:bluefs debug_inject_read_zeros 2024-03-12T03:55:19.447 INFO:teuthology.orchestra.run.smithi169.stderr:bluefs files list 2024-03-12T03:55:19.447 INFO:teuthology.orchestra.run.smithi169.stderr:bluefs stats 2024-03-12T03:55:19.447 INFO:teuthology.orchestra.run.smithi169.stderr:bluestore allocator dump block 2024-03-12T03:55:19.447 INFO:teuthology.orchestra.run.smithi169.stderr:bluestore allocator fragmentation block 2024-03-12T03:55:19.447 INFO:teuthology.orchestra.run.smithi169.stderr:admin_socket: invalid command 2024-03-12T03:55:19.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:19 smithi169 ceph-mon[29344]: pgmap v1077: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:19.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:19 smithi169 ceph-mon[29344]: 5.e scrub starts 2024-03-12T03:55:19.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:19 smithi169 ceph-mon[29344]: 5.e scrub ok 2024-03-12T03:55:19.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:19 smithi169 ceph-mon[29344]: from='osd.0 v2:172.21.15.169:6801/2113559059' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:55:19.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:19 smithi169 ceph-mon[29344]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:55:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:19 smithi169 ceph-mon[39949]: pgmap v1077: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:19 smithi169 ceph-mon[39949]: 5.e scrub starts 2024-03-12T03:55:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:19 smithi169 ceph-mon[39949]: 5.e scrub ok 2024-03-12T03:55:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:19 smithi169 ceph-mon[39949]: from='osd.0 v2:172.21.15.169:6801/2113559059' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:55:19.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:19 smithi169 ceph-mon[39949]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:55:19.942 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:19 smithi173 ceph-mon[33544]: pgmap v1077: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:19.942 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:19 smithi173 ceph-mon[33544]: 5.e scrub starts 2024-03-12T03:55:19.942 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:19 smithi173 ceph-mon[33544]: 5.e scrub ok 2024-03-12T03:55:19.942 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:19 smithi173 ceph-mon[33544]: from='osd.0 v2:172.21.15.169:6801/2113559059' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:55:19.942 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:19 smithi173 ceph-mon[33544]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:55:19.993 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:55:19.993 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:55:19.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:55:19.993 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15279.0:258442 9.1 9:8a150de2:::benchmark_data_smithi169_135229_object114227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:47:00.807043+0000", 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 499.18268000299997, 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.030208796, 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15279", 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 258442 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807043+0000", 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807043+0000", 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807045+0000", 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8300000000000001e-06 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807051+0000", 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.728e-06 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807062+0000", 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1297e-05 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807078+0000", 2024-03-12T03:55:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5784999999999999e-05 2024-03-12T03:55:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.807943+0000", 2024-03-12T03:55:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086520399999999995 2024-03-12T03:55:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.808076+0000", 2024-03-12T03:55:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000132724 2024-03-12T03:55:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-03-12T03:55:19.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.808206+0000", 2024-03-12T03:55:19.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013005100000000001 2024-03-12T03:55:19.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.810938+0000", 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027315830000000001 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.813484+0000", 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025461030000000001 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.836790+0000", 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.023306473000000001 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.836901+0000", 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011044699999999999 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:47:00.837252+0000", 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035157100000000001 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78615 10.e e383/366 10:7dfe4819:::benchmark_data_smithi169_162694_object78614:head v 383'4854, mlcod=383'4852)", 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.988349+0000", 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 383.00137439899999, 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.042519055, 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988349+0000", 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988349+0000", 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988350+0000", 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4499999999999995e-07 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988374+0000", 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4346999999999999e-05 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988385+0000", 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0669000000000001e-05 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988393+0000", 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1259999999999998e-06 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988481+0000", 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7696999999999997e-05 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988506+0000", 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4791999999999999e-05 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.990980+0000", 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024742309999999999 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030868+0000", 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039888248000000001 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78612 10.d e383/366 10:b3a4751d:::benchmark_data_smithi169_162694_object78611:head v 383'4904, mlcod=383'4902)", 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.988423+0000", 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 383.00130060800001, 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.042485090000000003, 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988423+0000", 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988423+0000", 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988423+0000", 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2500000000000001e-07 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988443+0000", 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9653e-05 2024-03-12T03:55:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988449+0000", 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5939999999999999e-06 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988455+0000", 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2120000000000002e-06 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988543+0000", 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8616999999999995e-05 2024-03-12T03:55:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988562+0000", 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8893999999999999e-05 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991063+0000", 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025009260000000001 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030908+0000", 2024-03-12T03:55:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039844769000000002 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78613 10.a e383/366 10:5ef8fbae:::benchmark_data_smithi169_162694_object78612:head v 383'4848, mlcod=383'4846)", 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.988636+0000", 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 383.00108734399998, 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.042224314999999998, 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988636+0000", 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988636+0000", 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988637+0000", 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4600000000000005e-07 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988733+0000", 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6521000000000006e-05 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988742+0000", 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1600000000000004e-06 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988749+0000", 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6409999999999996e-06 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988805+0000", 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6424000000000001e-05 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.988823+0000", 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7884e-05 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030828+0000", 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.042004527999999999 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030860+0000", 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2611000000000003e-05 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78614 10.f e383/366 10:fb4a5bd3:::benchmark_data_smithi169_162694_object78613:head v 383'4934, mlcod=383'4932)", 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.989287+0000", 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 383.00043600100003, 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041609650999999998, 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989287+0000", 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989287+0000", 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989288+0000", 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7199999999999998e-07 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989336+0000", 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7648999999999997e-05 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989345+0000", 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3889999999999994e-06 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989352+0000", 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7410000000000003e-06 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989426+0000", 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3819999999999995e-05 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.989444+0000", 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8717000000000001e-05 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030864+0000", 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041419639000000001 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030897+0000", 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3024000000000001e-05 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78618 10.7 e383/366 10:e87822aa:::benchmark_data_smithi169_162694_object78617:head v 383'4971, mlcod=383'4969)", 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:56.991518+0000", 2024-03-12T03:55:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 382.99820490299999, 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.051062718999999999, 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991518+0000", 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991518+0000", 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991519+0000", 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1699999999999997e-07 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:56.991552+0000", 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2230999999999997e-05 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030814+0000", 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039262496000000001 2024-03-12T03:55:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030828+0000", 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4209000000000001e-05 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030876+0000", 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8000000000000001e-05 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030898+0000", 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2101000000000001e-05 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.042488+0000", 2024-03-12T03:55:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011589868 2024-03-12T03:55:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.042581+0000", 2024-03-12T03:55:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2997000000000004e-05 2024-03-12T03:55:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:78634 10.a e383/366 10:57b48c2c:::benchmark_data_smithi169_162694_object78633:head v 383'4849, mlcod=383'4846)", 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.005400+0000", 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 382.98432336399998, 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.029465791000000002, 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.005400+0000", 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.005400+0000", 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.005401+0000", 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1039999999999999e-06 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.005443+0000", 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2264000000000001e-05 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030824+0000", 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025381048 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030835+0000", 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0369e-05 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030890+0000", 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4744999999999998e-05 2024-03-12T03:55:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.030908+0000", 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8261e-05 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.034823+0000", 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039151730000000001 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.034866+0000", 2024-03-12T03:55:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2827000000000002e-05 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:79202 10.b e383/366 10:dbb2f93d:::benchmark_data_smithi169_162694_object79201:head v 383'4898, mlcod=383'4896)", 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:48:57.472218+0000", 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 382.51750525300002, 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026937955999999999, 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.472218+0000", 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.472218+0000", 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.472219+0000", 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.54e-07 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491702+0000", 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.019482816999999999 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491709+0000", 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7909999999999999e-06 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491716+0000", 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1609999999999997e-06 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491764+0000", 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8424000000000003e-05 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.491783+0000", 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8453000000000001e-05 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.499104+0000", 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0073216410000000003 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:48:57.499156+0000", 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1915e-05 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:110617 10.a e389/384 10:5aa4436c:::benchmark_data_smithi169_162694_object110616:head v 389'6831, mlcod=389'6828)", 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:20.439153+0000", 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 359.55057059299997, 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027448368000000001, 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439153+0000", 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439153+0000", 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439154+0000", 2024-03-12T03:55:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1799999999999999e-06 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439180+0000", 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6109999999999999e-05 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439191+0000", 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1099e-05 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.439203+0000", 2024-03-12T03:55:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2242000000000001e-05 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.440972+0000", 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001768146 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.440997+0000", 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4956999999999998e-05 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.466546+0000", 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025549471000000001 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:20.466601+0000", 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5163e-05 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:114672 10.f e391/366 10:fe0e19ae:::benchmark_data_smithi169_162694_object114671:head v 391'7222, mlcod=391'7219)", 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:23.949154+0000", 2024-03-12T03:55:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 356.04056959899998, 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026819050000000001, 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949154+0000", 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949154+0000", 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949155+0000", 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.540000000000001e-07 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949190+0000", 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5561000000000001e-05 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949201+0000", 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0206e-05 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.949208+0000", 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8769999999999993e-06 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.950125+0000", 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091698799999999998 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.950147+0000", 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1537000000000001e-05 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.975879+0000", 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025731878 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:23.975973+0000", 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4048999999999993e-05 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:137648 10.f e396/366 10:f39db279:::benchmark_data_smithi169_162694_object137647:head v 396'8687, mlcod=396'8684)", 2024-03-12T03:55:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:42.087175+0000", 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 337.902548072, 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.030562865000000002, 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087175+0000", 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087175+0000", 2024-03-12T03:55:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087176+0000", 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7300000000000005e-07 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087211+0000", 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5135000000000002e-05 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087228+0000", 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6532999999999999e-05 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087236+0000", 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3359999999999995e-06 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088257+0000", 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001021271 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088285+0000", 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7358000000000001e-05 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.117619+0000", 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.029334494999999999 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.117738+0000", 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011886400000000001 2024-03-12T03:55:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:137650 10.7 e396/366 10:ea5697a5:::benchmark_data_smithi169_162694_object137649:head v 396'8710, mlcod=396'8708)", 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:42.087996+0000", 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 337.90172710799999, 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.030052637, 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087996+0000", 2024-03-12T03:55:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087996+0000", 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.087997+0000", 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3300000000000001e-07 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088032+0000", 2024-03-12T03:55:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4813e-05 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088042+0000", 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0505e-05 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088050+0000", 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.464e-06 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088331+0000", 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028071999999999998 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.088438+0000", 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107873 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.117985+0000", 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.029546161000000001 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.118049+0000", 2024-03-12T03:55:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4368000000000004e-05 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:137652 10.f e396/366 10:f0a37749:::benchmark_data_smithi169_162694_object137651:head v 396'8688, mlcod=396'8684)", 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:42.091280+0000", 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 337.89844353699999, 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026800033000000001, 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.091280+0000", 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.091280+0000", 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.091280+0000", 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.01e-07 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.091316+0000", 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5309999999999999e-05 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.091325+0000", 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1749999999999994e-06 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.091330+0000", 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4759999999999997e-06 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.097939+0000", 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0066088589999999999 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.097958+0000", 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9100999999999999e-05 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.118053+0000", 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020094615999999999 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:42.118080+0000", 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7095000000000001e-05 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:140402 10.b e397/384 10:dedafb12:::benchmark_data_smithi169_162694_object1407:head v 397'8640, mlcod=397'8638)", 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:43.998036+0000", 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 335.99168754800002, 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.034026572999999997, 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:43.998036+0000", 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:43.998036+0000", 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:43.998036+0000", 2024-03-12T03:55:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.65e-07 2024-03-12T03:55:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:43.998040+0000", 2024-03-12T03:55:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7740000000000002e-06 2024-03-12T03:55:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:43.998045+0000", 2024-03-12T03:55:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0370000000000002e-06 2024-03-12T03:55:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:43.998049+0000", 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1579999999999998e-06 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.000488+0000", 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024390480000000001 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.000540+0000", 2024-03-12T03:55:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1650000000000002e-05 2024-03-12T03:55:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.031911+0000", 2024-03-12T03:55:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.031371688000000002 2024-03-12T03:55:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.032062+0000", 2024-03-12T03:55:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000150953 2024-03-12T03:55:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:140408 10.b e397/384 10:deabafc9:::benchmark_data_smithi169_162694_object1413:head v 397'8641, mlcod=397'8638)", 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:44.003987+0000", 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 335.98573645499999, 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.028198746, 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.003987+0000", 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.003987+0000", 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.003988+0000", 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8599999999999997e-07 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.003997+0000", 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3719999999999994e-06 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.004012+0000", 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5160999999999999e-05 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.004024+0000", 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1907e-05 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.017637+0000", 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013612602999999999 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.017663+0000", 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6676e-05 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.032066+0000", 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014403009 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:44.032186+0000", 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119232 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:142066 10.0 e398/366 10:00965d4a:::benchmark_data_smithi169_162694_object3071:head v 398'8902, mlcod=398'8900)", 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:49:45.125283+0000", 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 334.864440124, 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.028954408000000001, 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.125283+0000", 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.125283+0000", 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.125284+0000", 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2500000000000005e-07 2024-03-12T03:55:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.125289+0000", 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4260000000000002e-06 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.125297+0000", 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5089999999999996e-06 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.125303+0000", 2024-03-12T03:55:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9499999999999998e-06 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.127506+0000", 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022029829999999999 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.127527+0000", 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0910000000000001e-05 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.154151+0000", 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026624655000000001 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:49:45.154238+0000", 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6349999999999998e-05 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25463.0:260216 10.7 e422/366 10:eb55b62d:::benchmark_data_smithi169_162694_object121221:head v 422'16421, mlcod=422'16414)", 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:51:04.946729+0000", 2024-03-12T03:55:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 255.04299477399999, 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.028606256, 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.946729+0000", 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.946729+0000", 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.946729+0000", 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8299999999999997e-07 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.946735+0000", 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3859999999999998e-06 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.946747+0000", 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2150000000000001e-05 2024-03-12T03:55:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.946753+0000", 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.438e-06 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.957653+0000", 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010899880000000001 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.957858+0000", 2024-03-12T03:55:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020488400000000001 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.975286+0000", 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.017428294 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:04.975335+0000", 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8640999999999997e-05 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15816.0:179199 11.d e496/438 11:b990c6e2:::benchmark_data_smithi169_185434_object67252:head v 496'11211, mlcod=496'11205)", 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:17.097280+0000", 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 62.892443647, 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.022040567, 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:17.097280+0000", 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:17.097280+0000", 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:17.097281+0000", 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7300000000000005e-07 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:17.097287+0000", 2024-03-12T03:55:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5429999999999999e-06 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:17.097297+0000", 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6050000000000004e-06 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:17.097304+0000", 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8719999999999996e-06 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:17.097876+0000", 2024-03-12T03:55:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057234500000000004 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:17.097896+0000", 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9721e-05 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:17.119246+0000", 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.021350422000000001 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:17.119320+0000", 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4185999999999997e-05 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15816.0:211046 11.d e501/438 11:b1d88e03:::benchmark_data_smithi169_185434_object99099:head v 501'13271, mlcod=501'13267)", 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:39.451238+0000", 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 40.538485479999999, 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.022003319, 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.451238+0000", 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.451238+0000", 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.451239+0000", 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.92e-07 2024-03-12T03:55:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.451257+0000", 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8396000000000001e-05 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.451267+0000", 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0229e-05 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.451274+0000", 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9330000000000004e-06 2024-03-12T03:55:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.455386+0000", 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041115960000000003 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.455407+0000", 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0927000000000001e-05 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.473212+0000", 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.017805629999999999 2024-03-12T03:55:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.473241+0000", 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8915999999999999e-05 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15816.0:211627 11.18 e501/488 11:1dcc6803:::benchmark_data_smithi169_185434_object99680:head v 501'8424, mlcod=501'8422)", 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:39.833993+0000", 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 40.155730235999997, 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020695855999999999, 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.833993+0000", 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.833993+0000", 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.833994+0000", 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3900000000000004e-07 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.834003+0000", 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4690000000000003e-06 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.834014+0000", 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0547000000000001e-05 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.834021+0000", 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8050000000000001e-06 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.836073+0000", 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020526659999999999 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.836093+0000", 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9352999999999999e-05 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.854614+0000", 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.018521525 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:39.854689+0000", 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4752000000000005e-05 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:20.082 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:55:20.423 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:55:19 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0[201513]: 2024-03-12T03:55:19.925+0000 7fa77d65e700 -1 osd.0 520 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:55:20.686 INFO:teuthology.orchestra.run.smithi169.stderr:nodeep-scrub is unset 2024-03-12T03:55:20.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-12T03:55:20.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[39949]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:55:20.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[39949]: osdmap e520: 8 total, 7 up, 7 in 2024-03-12T03:55:20.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[39949]: from='osd.0 v2:172.21.15.169:6801/2113559059' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:55:20.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[39949]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[39949]: 5.1b scrub starts 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[39949]: 5.1b scrub ok 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[39949]: 3.1f scrub starts 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[39949]: 3.1f scrub ok 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2843920039' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[39949]: 3.8 scrub starts 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[39949]: 3.8 scrub ok 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[29344]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[29344]: osdmap e520: 8 total, 7 up, 7 in 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[29344]: from='osd.0 v2:172.21.15.169:6801/2113559059' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[29344]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[29344]: 5.1b scrub starts 2024-03-12T03:55:20.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[29344]: 5.1b scrub ok 2024-03-12T03:55:20.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[29344]: 3.1f scrub starts 2024-03-12T03:55:20.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[29344]: 3.1f scrub ok 2024-03-12T03:55:20.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2843920039' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:55:20.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[29344]: 3.8 scrub starts 2024-03-12T03:55:20.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:20 smithi169 ceph-mon[29344]: 3.8 scrub ok 2024-03-12T03:55:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:20 smithi173 ceph-mon[33544]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:55:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:20 smithi173 ceph-mon[33544]: osdmap e520: 8 total, 7 up, 7 in 2024-03-12T03:55:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:20 smithi173 ceph-mon[33544]: from='osd.0 v2:172.21.15.169:6801/2113559059' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:55:20.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:20 smithi173 ceph-mon[33544]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-12T03:55:20.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:20 smithi173 ceph-mon[33544]: 5.1b scrub starts 2024-03-12T03:55:20.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:20 smithi173 ceph-mon[33544]: 5.1b scrub ok 2024-03-12T03:55:20.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:20 smithi173 ceph-mon[33544]: 3.1f scrub starts 2024-03-12T03:55:20.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:20 smithi173 ceph-mon[33544]: 3.1f scrub ok 2024-03-12T03:55:20.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:20 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2843920039' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-12T03:55:20.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:20 smithi173 ceph-mon[33544]: 3.8 scrub starts 2024-03-12T03:55:20.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:20 smithi173 ceph-mon[33544]: 3.8 scrub ok 2024-03-12T03:55:21.022 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-12T03:55:21.023 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-03-12T03:55:21.138 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:55:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:21 smithi173 ceph-mon[33544]: pgmap v1080: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:21 smithi173 ceph-mon[33544]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:55:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:21 smithi173 ceph-mon[33544]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T03:55:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:21 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2843920039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:55:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:21 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:55:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:21 smithi173 ceph-mon[33544]: osd.0 v2:172.21.15.169:6801/2113559059 boot 2024-03-12T03:55:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:21 smithi173 ceph-mon[33544]: osdmap e521: 8 total, 8 up, 7 in 2024-03-12T03:55:21.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:21 smithi173 ceph-mon[33544]: osdmap e522: 8 total, 8 up, 7 in 2024-03-12T03:55:22.013 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:55:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:55:22.014 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[29344]: pgmap v1080: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:22.014 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[29344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:55:22.014 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[29344]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T03:55:22.014 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2843920039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:55:22.014 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:55:22.014 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[29344]: osd.0 v2:172.21.15.169:6801/2113559059 boot 2024-03-12T03:55:22.014 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[29344]: osdmap e521: 8 total, 8 up, 7 in 2024-03-12T03:55:22.014 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[29344]: osdmap e522: 8 total, 8 up, 7 in 2024-03-12T03:55:22.014 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[39949]: pgmap v1080: 150 pgs: 29 active+clean+remapped, 11 active+undersized+degraded, 21 active+undersized, 89 active+clean; 609 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 55/690 objects degraded (7.971%); 40/690 objects misplaced (5.797%) 2024-03-12T03:55:22.015 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[39949]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-12T03:55:22.015 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[39949]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T03:55:22.015 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2843920039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-12T03:55:22.015 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:55:22.015 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[39949]: osd.0 v2:172.21.15.169:6801/2113559059 boot 2024-03-12T03:55:22.015 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[39949]: osdmap e521: 8 total, 8 up, 7 in 2024-03-12T03:55:22.015 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:21 smithi169 ceph-mon[39949]: osdmap e522: 8 total, 8 up, 7 in 2024-03-12T03:55:22.848 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:55:22.848 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:55:22.848 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-03-12T03:55:22.848 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:55:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:22 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:55:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:22 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:55:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:22 smithi173 ceph-mon[33544]: 3.1c scrub starts 2024-03-12T03:55:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:22 smithi173 ceph-mon[33544]: 3.1c scrub ok 2024-03-12T03:55:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:22 smithi173 ceph-mon[33544]: 3.f scrub starts 2024-03-12T03:55:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:22 smithi173 ceph-mon[33544]: 3.f scrub ok 2024-03-12T03:55:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:22 smithi173 ceph-mon[33544]: 2.1 scrub starts 2024-03-12T03:55:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:22 smithi173 ceph-mon[33544]: 2.1 scrub ok 2024-03-12T03:55:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:22 smithi173 ceph-mon[33544]: osdmap e523: 8 total, 8 up, 7 in 2024-03-12T03:55:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:55:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:55:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[29344]: 3.1c scrub starts 2024-03-12T03:55:23.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[29344]: 3.1c scrub ok 2024-03-12T03:55:23.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[29344]: 3.f scrub starts 2024-03-12T03:55:23.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[29344]: 3.f scrub ok 2024-03-12T03:55:23.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[29344]: 2.1 scrub starts 2024-03-12T03:55:23.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[29344]: 2.1 scrub ok 2024-03-12T03:55:23.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[29344]: osdmap e523: 8 total, 8 up, 7 in 2024-03-12T03:55:23.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:55:23.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:55:23.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[39949]: 3.1c scrub starts 2024-03-12T03:55:23.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[39949]: 3.1c scrub ok 2024-03-12T03:55:23.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[39949]: 3.f scrub starts 2024-03-12T03:55:23.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[39949]: 3.f scrub ok 2024-03-12T03:55:23.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[39949]: 2.1 scrub starts 2024-03-12T03:55:23.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[39949]: 2.1 scrub ok 2024-03-12T03:55:23.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:22 smithi169 ceph-mon[39949]: osdmap e523: 8 total, 8 up, 7 in 2024-03-12T03:55:23.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-12T03:55:23.751 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:55:23.878 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:55:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:55:23.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:23 smithi173 ceph-mon[33544]: 1.6 deep-scrub starts 2024-03-12T03:55:23.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:23 smithi173 ceph-mon[33544]: 1.6 deep-scrub ok 2024-03-12T03:55:23.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:23 smithi173 ceph-mon[33544]: pgmap v1083: 150 pgs: 6 remapped+peering, 4 peering, 23 active+clean+remapped, 9 active+undersized+degraded, 19 active+undersized, 89 active+clean; 609 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 767 B/s rd, 0 op/s; 52/690 objects degraded (7.536%); 23/690 objects misplaced (3.333%) 2024-03-12T03:55:23.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:23 smithi173 ceph-mon[33544]: 5.c deep-scrub starts 2024-03-12T03:55:23.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:23 smithi173 ceph-mon[33544]: 5.c deep-scrub ok 2024-03-12T03:55:23.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:23 smithi173 ceph-mon[33544]: Health check update: Degraded data redundancy: 52/690 objects degraded (7.536%), 9 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-12T03:55:23.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:23 smithi173 ceph-mon[33544]: 5.8 scrub starts 2024-03-12T03:55:23.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:23 smithi173 ceph-mon[33544]: 5.8 scrub ok 2024-03-12T03:55:23.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:23 smithi173 ceph-mon[33544]: 6.13 scrub starts 2024-03-12T03:55:23.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:23 smithi173 ceph-mon[33544]: 6.13 scrub ok 2024-03-12T03:55:23.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:23 smithi173 ceph-mon[33544]: osdmap e524: 8 total, 8 up, 7 in 2024-03-12T03:55:24.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[29344]: 1.6 deep-scrub starts 2024-03-12T03:55:24.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[29344]: 1.6 deep-scrub ok 2024-03-12T03:55:24.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[29344]: pgmap v1083: 150 pgs: 6 remapped+peering, 4 peering, 23 active+clean+remapped, 9 active+undersized+degraded, 19 active+undersized, 89 active+clean; 609 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 767 B/s rd, 0 op/s; 52/690 objects degraded (7.536%); 23/690 objects misplaced (3.333%) 2024-03-12T03:55:24.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[29344]: 5.c deep-scrub starts 2024-03-12T03:55:24.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[29344]: 5.c deep-scrub ok 2024-03-12T03:55:24.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[29344]: Health check update: Degraded data redundancy: 52/690 objects degraded (7.536%), 9 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-12T03:55:24.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[29344]: 5.8 scrub starts 2024-03-12T03:55:24.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[29344]: 5.8 scrub ok 2024-03-12T03:55:24.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[29344]: 6.13 scrub starts 2024-03-12T03:55:24.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[29344]: 6.13 scrub ok 2024-03-12T03:55:24.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[29344]: osdmap e524: 8 total, 8 up, 7 in 2024-03-12T03:55:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[39949]: 1.6 deep-scrub starts 2024-03-12T03:55:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[39949]: 1.6 deep-scrub ok 2024-03-12T03:55:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[39949]: pgmap v1083: 150 pgs: 6 remapped+peering, 4 peering, 23 active+clean+remapped, 9 active+undersized+degraded, 19 active+undersized, 89 active+clean; 609 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 767 B/s rd, 0 op/s; 52/690 objects degraded (7.536%); 23/690 objects misplaced (3.333%) 2024-03-12T03:55:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[39949]: 5.c deep-scrub starts 2024-03-12T03:55:24.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[39949]: 5.c deep-scrub ok 2024-03-12T03:55:24.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[39949]: Health check update: Degraded data redundancy: 52/690 objects degraded (7.536%), 9 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2024-03-12T03:55:24.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[39949]: 5.8 scrub starts 2024-03-12T03:55:24.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[39949]: 5.8 scrub ok 2024-03-12T03:55:24.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[39949]: 6.13 scrub starts 2024-03-12T03:55:24.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[39949]: 6.13 scrub ok 2024-03-12T03:55:24.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:23 smithi169 ceph-mon[39949]: osdmap e524: 8 total, 8 up, 7 in 2024-03-12T03:55:24.902 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:24.902 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:24 smithi173 ceph-mon[33544]: 5.1d scrub starts 2024-03-12T03:55:24.902 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:24 smithi173 ceph-mon[33544]: 5.1d scrub ok 2024-03-12T03:55:24.902 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:24 smithi173 ceph-mon[33544]: 2.0 scrub starts 2024-03-12T03:55:24.902 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:24 smithi173 ceph-mon[33544]: 2.0 scrub ok 2024-03-12T03:55:24.902 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:24 smithi173 ceph-mon[33544]: 4.19 scrub starts 2024-03-12T03:55:24.902 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:24 smithi173 ceph-mon[33544]: 4.19 scrub ok 2024-03-12T03:55:24.902 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:24 smithi173 ceph-mon[33544]: osdmap e525: 8 total, 8 up, 7 in 2024-03-12T03:55:25.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:25.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[39949]: 5.1d scrub starts 2024-03-12T03:55:25.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[39949]: 5.1d scrub ok 2024-03-12T03:55:25.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[39949]: 2.0 scrub starts 2024-03-12T03:55:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[39949]: 2.0 scrub ok 2024-03-12T03:55:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[39949]: 4.19 scrub starts 2024-03-12T03:55:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[39949]: 4.19 scrub ok 2024-03-12T03:55:25.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[39949]: osdmap e525: 8 total, 8 up, 7 in 2024-03-12T03:55:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[29344]: 5.1d scrub starts 2024-03-12T03:55:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[29344]: 5.1d scrub ok 2024-03-12T03:55:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[29344]: 2.0 scrub starts 2024-03-12T03:55:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[29344]: 2.0 scrub ok 2024-03-12T03:55:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[29344]: 4.19 scrub starts 2024-03-12T03:55:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[29344]: 4.19 scrub ok 2024-03-12T03:55:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:24 smithi169 ceph-mon[29344]: osdmap e525: 8 total, 8 up, 7 in 2024-03-12T03:55:25.438 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:55:25.438 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-03-12T03:55:25.438 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-03-12T03:55:25.438 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-03-12T03:55:25.438 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:55:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:25 smithi173 ceph-mon[33544]: pgmap v1086: 150 pgs: 6 remapped+peering, 7 peering, 11 active+clean+remapped, 5 active+undersized+degraded, 11 active+undersized, 110 active+clean; 609 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 767 B/s rd, 0 op/s; 24/690 objects degraded (3.478%); 2/690 objects misplaced (0.290%); 32 B/s, 7 objects/s recovering 2024-03-12T03:55:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:25 smithi173 ceph-mon[33544]: 6.14 scrub starts 2024-03-12T03:55:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:25 smithi173 ceph-mon[33544]: 6.14 scrub ok 2024-03-12T03:55:26.024 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-12T03:55:26.055 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:25 smithi169 ceph-mon[29344]: pgmap v1086: 150 pgs: 6 remapped+peering, 7 peering, 11 active+clean+remapped, 5 active+undersized+degraded, 11 active+undersized, 110 active+clean; 609 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 767 B/s rd, 0 op/s; 24/690 objects degraded (3.478%); 2/690 objects misplaced (0.290%); 32 B/s, 7 objects/s recovering 2024-03-12T03:55:26.056 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:25 smithi169 ceph-mon[29344]: 6.14 scrub starts 2024-03-12T03:55:26.056 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:25 smithi169 ceph-mon[29344]: 6.14 scrub ok 2024-03-12T03:55:26.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:25 smithi169 ceph-mon[39949]: pgmap v1086: 150 pgs: 6 remapped+peering, 7 peering, 11 active+clean+remapped, 5 active+undersized+degraded, 11 active+undersized, 110 active+clean; 609 KiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 767 B/s rd, 0 op/s; 24/690 objects degraded (3.478%); 2/690 objects misplaced (0.290%); 32 B/s, 7 objects/s recovering 2024-03-12T03:55:26.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:25 smithi169 ceph-mon[39949]: 6.14 scrub starts 2024-03-12T03:55:26.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:25 smithi169 ceph-mon[39949]: 6.14 scrub ok 2024-03-12T03:55:26.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-12T03:55:26.285 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:55:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:26 smithi173 ceph-mon[33544]: 4.b scrub starts 2024-03-12T03:55:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:26 smithi173 ceph-mon[33544]: 4.b scrub ok 2024-03-12T03:55:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:26 smithi173 ceph-mon[33544]: 1.5 scrub starts 2024-03-12T03:55:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:26 smithi173 ceph-mon[33544]: 1.5 scrub ok 2024-03-12T03:55:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:26 smithi173 ceph-mon[33544]: 4.7 scrub starts 2024-03-12T03:55:26.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:26 smithi173 ceph-mon[33544]: 4.7 scrub ok 2024-03-12T03:55:27.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:26 smithi169 ceph-mon[29344]: 4.b scrub starts 2024-03-12T03:55:27.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:26 smithi169 ceph-mon[29344]: 4.b scrub ok 2024-03-12T03:55:27.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:26 smithi169 ceph-mon[29344]: 1.5 scrub starts 2024-03-12T03:55:27.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:26 smithi169 ceph-mon[29344]: 1.5 scrub ok 2024-03-12T03:55:27.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:26 smithi169 ceph-mon[29344]: 4.7 scrub starts 2024-03-12T03:55:27.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:26 smithi169 ceph-mon[29344]: 4.7 scrub ok 2024-03-12T03:55:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:26 smithi169 ceph-mon[39949]: 4.b scrub starts 2024-03-12T03:55:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:26 smithi169 ceph-mon[39949]: 4.b scrub ok 2024-03-12T03:55:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:26 smithi169 ceph-mon[39949]: 1.5 scrub starts 2024-03-12T03:55:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:26 smithi169 ceph-mon[39949]: 1.5 scrub ok 2024-03-12T03:55:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:26 smithi169 ceph-mon[39949]: 4.7 scrub starts 2024-03-12T03:55:27.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:26 smithi169 ceph-mon[39949]: 4.7 scrub ok 2024-03-12T03:55:27.938 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:27 smithi173 ceph-mon[33544]: pgmap v1088: 150 pgs: 3 remapped+peering, 7 peering, 140 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s, 10 objects/s recovering 2024-03-12T03:55:27.938 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:27 smithi173 ceph-mon[33544]: 6.d scrub starts 2024-03-12T03:55:27.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:27 smithi173 ceph-mon[33544]: 6.d scrub ok 2024-03-12T03:55:27.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:27 smithi173 ceph-mon[33544]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/690 objects degraded (3.478%), 5 pgs degraded, 16 pgs undersized) 2024-03-12T03:55:27.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:27 smithi173 ceph-mon[33544]: Cluster is now healthy 2024-03-12T03:55:27.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:27 smithi173 ceph-mon[33544]: 1.2 scrub starts 2024-03-12T03:55:27.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:27 smithi173 ceph-mon[33544]: 1.2 scrub ok 2024-03-12T03:55:27.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:27 smithi173 ceph-mon[33544]: 4.1a scrub starts 2024-03-12T03:55:27.939 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:27 smithi173 ceph-mon[33544]: 4.1a scrub ok 2024-03-12T03:55:27.981 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-12T03:55:27.981 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-03-12T03:55:27.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24359.0:3966 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94067641424640 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:51:41.845078+0000", 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 226.132249522, 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99126123399999999, 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24359", 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 3966 2024-03-12T03:55:27.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845078+0000", 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845078+0000", 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:27.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845079+0000", 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.44e-07 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845083+0000", 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5899999999999999e-06 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845092+0000", 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3619999999999995e-06 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:27.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845099+0000", 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1199999999999996e-06 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845144+0000", 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5303999999999999e-05 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:41.845151+0000", 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7129999999999999e-06 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:42.834504+0000", 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98935289400000004 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:42.834717+0000", 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000212845 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:51:42.836340+0000", 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001622762 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:27.986 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:219693 11.17 11:ed0be8e2:::benchmark_data_smithi169_185434_object107746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:46.097706+0000", 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 41.879622298999998, 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2192092429999999, 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 219693 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.987 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.097706+0000", 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.097706+0000", 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.097707+0000", 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.933e-06 2024-03-12T03:55:27.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.097713+0000", 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1710000000000001e-06 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.097729+0000", 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6473e-05 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.097736+0000", 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.409e-06 2024-03-12T03:55:27.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.097775+0000", 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9941000000000003e-05 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.097783+0000", 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7249999999999997e-06 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.085034+0000", 2024-03-12T03:55:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98725115200000002 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.085049+0000", 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4516e-05 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.302135+0000", 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.217086501 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.302245+0000", 2024-03-12T03:55:27.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010989000000000001 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,5,6", 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.302621+0000", 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00037552499999999998 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.308952+0000", 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006330975 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.309659+0000", 2024-03-12T03:55:27.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00070733600000000003 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.309702+0000", 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.278e-05 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.316722+0000", 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.007019997 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.316786+0000", 2024-03-12T03:55:27.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4228999999999998e-05 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.316915+0000", 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012868999999999999 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:219694 11.17 11:e8d93dae:::benchmark_data_smithi169_185434_object107747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:46.105177+0000", 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 41.872150601000001, 2024-03-12T03:55:27.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2176624459999998, 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 219694 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.105177+0000", 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.105177+0000", 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.105178+0000", 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0999999999999997e-07 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.105182+0000", 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2590000000000002e-06 2024-03-12T03:55:27.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.105190+0000", 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8350000000000004e-06 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.105196+0000", 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4480000000000002e-06 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.105250+0000", 2024-03-12T03:55:27.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3996999999999998e-05 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.105262+0000", 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2312e-05 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.085904+0000", 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98064154999999997 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:27.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.085914+0000", 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0268000000000001e-05 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.310301+0000", 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2243871749999999 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.311053+0000", 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000752478 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,5,6", 2024-03-12T03:55:27.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.311193+0000", 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013957800000000001 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.316163+0000", 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049703799999999999 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.316955+0000", 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00079184100000000005 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.317006+0000", 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.096e-05 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.322668+0000", 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0056621850000000001 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.322727+0000", 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8091000000000003e-05 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.322840+0000", 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011317899999999999 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:219697 11.5 11:a6c348d1:::benchmark_data_smithi169_185434_object107750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:46.108008+0000", 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 41.869319705999999, 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.163305501, 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:28.002 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 219697 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.108008+0000", 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.108008+0000", 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.108010+0000", 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4270000000000001e-06 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.108015+0000", 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4249999999999997e-06 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.108030+0000", 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5328999999999998e-05 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.108039+0000", 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6959999999999994e-06 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.108110+0000", 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0692999999999996e-05 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.108122+0000", 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1926000000000001e-05 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.095096+0000", 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98697409300000005 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.095133+0000", 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6922000000000003e-05 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.245512+0000", 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1503793659999999 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.245635+0000", 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012337900000000001 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,5,6", 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.245984+0000", 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000348513 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.252787+0000", 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006803084 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.008 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[29344]: pgmap v1088: 150 pgs: 3 remapped+peering, 7 peering, 140 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s, 10 objects/s recovering 2024-03-12T03:55:28.008 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[29344]: 6.d scrub starts 2024-03-12T03:55:28.008 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[29344]: 6.d scrub ok 2024-03-12T03:55:28.008 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[29344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/690 objects degraded (3.478%), 5 pgs degraded, 16 pgs undersized) 2024-03-12T03:55:28.009 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[29344]: Cluster is now healthy 2024-03-12T03:55:28.009 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[29344]: 1.2 scrub starts 2024-03-12T03:55:28.009 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[29344]: 1.2 scrub ok 2024-03-12T03:55:28.009 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[29344]: 4.1a scrub starts 2024-03-12T03:55:28.009 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[29344]: 4.1a scrub ok 2024-03-12T03:55:28.009 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[39949]: pgmap v1088: 150 pgs: 3 remapped+peering, 7 peering, 140 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s, 10 objects/s recovering 2024-03-12T03:55:28.009 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[39949]: 6.d scrub starts 2024-03-12T03:55:28.009 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[39949]: 6.d scrub ok 2024-03-12T03:55:28.009 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[39949]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/690 objects degraded (3.478%), 5 pgs degraded, 16 pgs undersized) 2024-03-12T03:55:28.009 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[39949]: Cluster is now healthy 2024-03-12T03:55:28.010 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[39949]: 1.2 scrub starts 2024-03-12T03:55:28.010 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[39949]: 1.2 scrub ok 2024-03-12T03:55:28.010 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[39949]: 4.1a scrub starts 2024-03-12T03:55:28.010 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:27 smithi169 ceph-mon[39949]: 4.1a scrub ok 2024-03-12T03:55:28.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.252798+0000", 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1337e-05 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.252854+0000", 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5458000000000003e-05 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.271173+0000", 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.018319068000000001 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.271234+0000", 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0940999999999998e-05 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.271314+0000", 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9844000000000003e-05 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:219701 11.16 11:690daff6:::benchmark_data_smithi169_185434_object107754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:28.012 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:46.115129+0000", 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 41.862198991, 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1653066889999999, 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 219701 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.115129+0000", 2024-03-12T03:55:28.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.115129+0000", 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.115129+0000", 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7300000000000001e-07 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.115132+0000", 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5100000000000001e-06 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.115137+0000", 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6330000000000001e-06 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.115142+0000", 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5530000000000003e-06 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.115182+0000", 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0382000000000001e-05 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.115189+0000", 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4339999999999997e-06 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.085209+0000", 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97002036999999997 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.085220+0000", 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0543999999999999e-05 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.232132+0000", 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1469121019999999 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.232277+0000", 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014501499999999999 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3,5", 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.232672+0000", 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039498000000000002 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.241200+0000", 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0085280080000000001 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.245412+0000", 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042121820000000001 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.275748+0000", 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.030336067000000001 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.280241+0000", 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044926879999999999 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.280307+0000", 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6527999999999997e-05 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.280436+0000", 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012821999999999999 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:219707 11.16 11:6b79449b:::benchmark_data_smithi169_185434_object107760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:46.123769+0000", 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 41.853559169, 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.156977973, 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 219707 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.123769+0000", 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.123769+0000", 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.123769+0000", 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9100000000000003e-07 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.123773+0000", 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5870000000000001e-06 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.123781+0000", 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7570000000000004e-06 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.123787+0000", 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5570000000000001e-06 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.123836+0000", 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8523999999999998e-05 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.123846+0000", 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0292000000000001e-05 2024-03-12T03:55:28.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.086625+0000", 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96277897400000001 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.086634+0000", 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0110000000000006e-06 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.241237+0000", 2024-03-12T03:55:28.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.154602954 2024-03-12T03:55:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.241384+0000", 2024-03-12T03:55:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000146682 2024-03-12T03:55:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3,5", 2024-03-12T03:55:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.241542+0000", 2024-03-12T03:55:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015798 2024-03-12T03:55:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.245418+0000", 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038762950000000001 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.245627+0000", 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020932200000000001 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.275797+0000", 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.030169544 2024-03-12T03:55:28.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.280520+0000", 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.004722782 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.280600+0000", 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9887000000000003e-05 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.280747+0000", 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014703399999999999 2024-03-12T03:55:28.027 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:219718 11.16 11:6d1c885c:::benchmark_data_smithi169_185434_object107771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:46.135513+0000", 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 41.841815166000004, 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.145509009, 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:28.028 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 219718 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.135513+0000", 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.135513+0000", 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.135514+0000", 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4099999999999997e-07 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.135518+0000", 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0729999999999998e-06 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.135527+0000", 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5570000000000001e-06 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.135535+0000", 2024-03-12T03:55:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4889999999999998e-06 2024-03-12T03:55:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.135590+0000", 2024-03-12T03:55:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4982000000000003e-05 2024-03-12T03:55:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.135598+0000", 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0390000000000005e-06 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.086662+0000", 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95106420999999997 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.086672+0000", 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9690000000000006e-06 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.243372+0000", 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1567003790000001 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.243483+0000", 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011039000000000001 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3,5", 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.243573+0000", 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0285999999999996e-05 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.245424+0000", 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018512419999999999 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.245757+0000", 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00033315800000000001 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.275904+0000", 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.030146599999999999 2024-03-12T03:55:28.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.280798+0000", 2024-03-12T03:55:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048937470000000004 2024-03-12T03:55:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.280889+0000", 2024-03-12T03:55:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1514999999999994e-05 2024-03-12T03:55:28.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.281022+0000", 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013253200000000001 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:219719 11.5 11:a384e87e:::benchmark_data_smithi169_185434_object107772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:46.137902+0000", 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 41.839426281000001, 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1335791030000002, 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.036 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 219719 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.137902+0000", 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.137902+0000", 2024-03-12T03:55:28.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.137904+0000", 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0169999999999999e-06 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.137916+0000", 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2516e-05 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.137944+0000", 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.764e-05 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.137990+0000", 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6133e-05 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.138112+0000", 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121929 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:46.138119+0000", 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.61e-06 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.095167+0000", 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95704743299999995 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:47.095178+0000", 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1078e-05 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.251007+0000", 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1558288729999999 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.251123+0000", 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000116215 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,5,6", 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.251253+0000", 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012977600000000001 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.252886+0000", 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001632966 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.253426+0000", 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054026499999999995 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.253616+0000", 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019045299999999999 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.271359+0000", 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.017742806999999999 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.271397+0000", 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7793999999999998e-05 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:48.271481+0000", 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3597999999999995e-05 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.043 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:221607 11.16 11:6bad770b:::benchmark_data_smithi169_185434_object109660:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:49.563779+0000", 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.413549328999999, 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0443277470000001, 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 221607 2024-03-12T03:55:28.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.563779+0000", 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.563779+0000", 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.563780+0000", 2024-03-12T03:55:28.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3009999999999999e-06 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.563786+0000", 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3389999999999998e-06 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.563797+0000", 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0584999999999999e-05 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.563805+0000", 2024-03-12T03:55:28.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0630000000000006e-06 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.570765+0000", 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0069603089999999996 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.570776+0000", 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0661e-05 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.596485+0000", 2024-03-12T03:55:28.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.025709183 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.596594+0000", 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108878 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.596727+0000", 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013305000000000001 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.599438+0000", 2024-03-12T03:55:28.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027106840000000001 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.599481+0000", 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3220000000000003e-05 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.608026+0000", 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0085449129999999995 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.608068+0000", 2024-03-12T03:55:28.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2128000000000001e-05 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.608106+0000", 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8433000000000003e-05 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:221640 11.16 11:68070d05:::benchmark_data_smithi169_185434_object109693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:49.594890+0000", 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.382437611, 2024-03-12T03:55:28.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0144616930000001, 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 221640 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.594890+0000", 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.594890+0000", 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.594891+0000", 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4099999999999997e-07 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.594896+0000", 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0320000000000003e-06 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.594911+0000", 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4408e-05 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.594918+0000", 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6029999999999999e-06 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.594983+0000", 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4974000000000005e-05 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.594992+0000", 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3319999999999992e-06 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.599515+0000", 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0045236150000001 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.599616+0000", 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010045 2024-03-12T03:55:28.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.599728+0000", 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112386 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.608117+0000", 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0083887270000000003 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.608166+0000", 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9777000000000001e-05 2024-03-12T03:55:28.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.608205+0000", 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8241000000000002e-05 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.609301+0000", 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010964220000000001 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.609352+0000", 2024-03-12T03:55:28.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0785000000000003e-05 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:221653 11.5 11:a0c42d35:::benchmark_data_smithi169_185434_object109706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:49.602495+0000", 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.374832376999997, 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99496498099999997, 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:28.057 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 221653 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.602495+0000", 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.602495+0000", 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.602496+0000", 2024-03-12T03:55:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5499999999999997e-07 2024-03-12T03:55:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.602501+0000", 2024-03-12T03:55:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5839999999999996e-06 2024-03-12T03:55:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.602513+0000", 2024-03-12T03:55:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1885000000000001e-05 2024-03-12T03:55:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.602519+0000", 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3350000000000003e-06 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.602580+0000", 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1173000000000005e-05 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.602591+0000", 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1075e-05 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.589344+0000", 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98675288699999997 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.589453+0000", 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108829 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.589591+0000", 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000137496 2024-03-12T03:55:28.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.592395+0000", 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028044749999999998 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.592481+0000", 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5750999999999999e-05 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.597369+0000", 2024-03-12T03:55:28.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048879509999999998 2024-03-12T03:55:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.597414+0000", 2024-03-12T03:55:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5463999999999997e-05 2024-03-12T03:55:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.597460+0000", 2024-03-12T03:55:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6220999999999997e-05 2024-03-12T03:55:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:221701 11.5 11:a2102e0a:::benchmark_data_smithi169_185434_object109754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:49.633084+0000", 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.344244097000001, 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96759610100000004, 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 221701 2024-03-12T03:55:28.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.633084+0000", 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.633084+0000", 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.633084+0000", 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.68e-07 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.633089+0000", 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3640000000000001e-06 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.633098+0000", 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8689999999999993e-06 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.633103+0000", 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6010000000000002e-06 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.633150+0000", 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6764000000000002e-05 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.633159+0000", 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3139999999999993e-06 2024-03-12T03:55:28.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.592527+0000", 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95936794400000003 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.592649+0000", 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012159699999999999 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.592765+0000", 2024-03-12T03:55:28.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011630500000000001 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.600393+0000", 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0076273280000000001 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.600458+0000", 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5450999999999994e-05 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.600582+0000", 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012451500000000001 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.600636+0000", 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3999000000000003e-05 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.600680+0000", 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3381999999999999e-05 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.071 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:221718 11.16 11:6e7c1c4a:::benchmark_data_smithi169_185434_object109771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:49.641358+0000", 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.335969359000003, 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97265409599999997, 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 221718 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.072 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.641358+0000", 2024-03-12T03:55:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.641358+0000", 2024-03-12T03:55:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.641359+0000", 2024-03-12T03:55:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3399999999999998e-07 2024-03-12T03:55:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.641363+0000", 2024-03-12T03:55:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8489999999999999e-06 2024-03-12T03:55:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.641375+0000", 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.238e-05 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.641383+0000", 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5739999999999998e-06 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.641443+0000", 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0445000000000001e-05 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.641454+0000", 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0540000000000001e-05 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.601754+0000", 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96030041600000005 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.601866+0000", 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111291 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.601993+0000", 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000127279 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.608123+0000", 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0061297360000000002 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.609420+0000", 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012969800000000001 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.613926+0000", 2024-03-12T03:55:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045066969999999996 2024-03-12T03:55:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.613985+0000", 2024-03-12T03:55:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8845000000000003e-05 2024-03-12T03:55:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.614013+0000", 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7330000000000001e-05 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:221755 11.5 11:a372bb0e:::benchmark_data_smithi169_185434_object109808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:49.666643+0000", 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.310684905999999, 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93416744699999998, 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:28.079 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 221755 2024-03-12T03:55:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.666643+0000", 2024-03-12T03:55:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.666643+0000", 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.666644+0000", 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9800000000000002e-07 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.666649+0000", 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0810000000000001e-06 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.666661+0000", 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2334000000000001e-05 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.666669+0000", 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.345e-06 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.666710+0000", 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1773000000000001e-05 2024-03-12T03:55:28.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.666717+0000", 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9260000000000003e-06 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.594792+0000", 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.92807473799999995 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.594909+0000", 2024-03-12T03:55:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000116897 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.595034+0000", 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000124931 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.600400+0000", 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0053657419999999997 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.600539+0000", 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000139006 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.600727+0000", 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018797700000000001 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.600772+0000", 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5856999999999998e-05 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.600810+0000", 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7941999999999997e-05 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15816.0:221787 11.5 11:a3795f20:::benchmark_data_smithi169_185434_object109840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:54:49.690637+0000", 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.286690749000002, 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91731159500000004, 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.086 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15816", 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 221787 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.690637+0000", 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.690637+0000", 2024-03-12T03:55:28.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.690638+0000", 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3489999999999999e-06 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.690644+0000", 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6180000000000003e-06 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.690657+0000", 2024-03-12T03:55:28.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3141e-05 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.690665+0000", 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6490000000000008e-06 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.690720+0000", 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4700999999999997e-05 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:49.690729+0000", 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8449999999999998e-06 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.597576+0000", 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90684675100000001 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.597673+0000", 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6577999999999996e-05 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,6", 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.597773+0000", 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010024800000000001 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.600690+0000", 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029170490000000001 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.600850+0000", 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000160295 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.607803+0000", 2024-03-12T03:55:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0069529819999999999 2024-03-12T03:55:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-03-12T03:55:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.607900+0000", 2024-03-12T03:55:28.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7060999999999998e-05 2024-03-12T03:55:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:54:50.607949+0000", 2024-03-12T03:55:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8328000000000002e-05 2024-03-12T03:55:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14529.0:1302 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e518)", 2024-03-12T03:55:28.093 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:55:17.209240+0000", 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.768088196000001, 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000422075, 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14529", 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/951871534", 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 1302 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:17.209240+0000", 2024-03-12T03:55:28.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:17.209240+0000", 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:17.209242+0000", 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9530000000000002e-06 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:17.209253+0000", 2024-03-12T03:55:28.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1430999999999999e-05 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:17.209291+0000", 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7639999999999999e-05 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:17.209318+0000", 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7678000000000002e-05 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:17.209394+0000", 2024-03-12T03:55:28.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5420000000000006e-05 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:17.209501+0000", 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107182 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:17.209662+0000", 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016077100000000001 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.14529.0:1306 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e520)", 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:55:21.213933+0000", 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.7633944240000003, 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48315719099999999, 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.14529", 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/951871534", 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 1306 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:21.213933+0000", 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:21.213933+0000", 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:21.213936+0000", 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4739999999999999e-06 2024-03-12T03:55:28.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:21.213948+0000", 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2003e-05 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:21.213996+0000", 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7716000000000002e-05 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:21.214013+0000", 2024-03-12T03:55:28.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7088999999999999e-05 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:21.214104+0000", 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1181000000000007e-05 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:21.214115+0000", 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0723000000000001e-05 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:21.696830+0000", 2024-03-12T03:55:28.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48271566999999999 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:21.696957+0000", 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012625700000000001 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:21.697091+0000", 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013407800000000001 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "pg_update_log_missing(5.d epoch 523/522 rep_tid 132147 entries 523'13 (0'0) error 5:b4812045:::notify.4:head by client.24359.0:4854 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:55:22.702425+0000", 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.274902891, 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020687520000000001, 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.702425+0000", 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.702425+0000", 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.702426+0000", 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1900000000000001e-07 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.702450+0000", 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4627999999999999e-05 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.702512+0000", 2024-03-12T03:55:28.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1607999999999997e-05 2024-03-12T03:55:28.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.702539+0000", 2024-03-12T03:55:28.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6891000000000001e-05 2024-03-12T03:55:28.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.702688+0000", 2024-03-12T03:55:28.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014868100000000001 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.704494+0000", 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001806125 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "pg_update_log_missing(5.d epoch 523/522 rep_tid 132148 entries 523'14 (0'0) error 5:b4812045:::notify.4:head by client.24359.0:4858 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:55:22.704958+0000", 2024-03-12T03:55:28.106 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.2723701329999999, 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088091099999999996, 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.704958+0000", 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.704958+0000", 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.704958+0000", 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8400000000000005e-07 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.704973+0000", 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5294e-05 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.705351+0000", 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00037766399999999999 2024-03-12T03:55:28.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.705363+0000", 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.205e-05 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.705539+0000", 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017539000000000001 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:22.705839+0000", 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000300029 2024-03-12T03:55:28.109 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24359.0:4868 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94067641424640 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e523)", 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-03-12T03:55:26.887263+0000", 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.090065098, 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038994300000000002, 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24359", 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 4868 2024-03-12T03:55:28.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:26.887263+0000", 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:26.887263+0000", 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:26.887264+0000", 2024-03-12T03:55:28.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5030000000000001e-06 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:26.887273+0000", 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7469999999999994e-06 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:26.887293+0000", 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0112e-05 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:26.887318+0000", 2024-03-12T03:55:28.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.533e-05 2024-03-12T03:55:28.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-03-12T03:55:28.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:26.887491+0000", 2024-03-12T03:55:28.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017289000000000001 2024-03-12T03:55:28.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:26.887583+0000", 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1324999999999997e-05 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-03-12T03:55:26.887653+0000", 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0036000000000005e-05 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-12T03:55:28.114 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-12T03:55:28.158 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:55:28.158 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:55:28.158 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-03-12T03:55:28.158 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:55:28.782 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-12T03:55:28.783 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-12T03:55:28.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd in 5 2024-03-12T03:55:28.907 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:28 smithi173 ceph-mon[33544]: 5.12 scrub starts 2024-03-12T03:55:28.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:28 smithi173 ceph-mon[33544]: 5.12 scrub ok 2024-03-12T03:55:28.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:28 smithi173 ceph-mon[33544]: 1.0 deep-scrub starts 2024-03-12T03:55:28.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:28 smithi173 ceph-mon[33544]: 1.0 deep-scrub ok 2024-03-12T03:55:28.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:28 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-12T03:55:28.908 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:28 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-12T03:55:28.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-12T03:55:29.104 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:29.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:28 smithi169 ceph-mon[39949]: 5.12 scrub starts 2024-03-12T03:55:29.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:28 smithi169 ceph-mon[39949]: 5.12 scrub ok 2024-03-12T03:55:29.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:28 smithi169 ceph-mon[39949]: 1.0 deep-scrub starts 2024-03-12T03:55:29.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:28 smithi169 ceph-mon[39949]: 1.0 deep-scrub ok 2024-03-12T03:55:29.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:28 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-12T03:55:29.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:28 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-12T03:55:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:28 smithi169 ceph-mon[29344]: 5.12 scrub starts 2024-03-12T03:55:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:28 smithi169 ceph-mon[29344]: 5.12 scrub ok 2024-03-12T03:55:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:28 smithi169 ceph-mon[29344]: 1.0 deep-scrub starts 2024-03-12T03:55:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:28 smithi169 ceph-mon[29344]: 1.0 deep-scrub ok 2024-03-12T03:55:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:28 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-12T03:55:29.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:28 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-12T03:55:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:29 smithi169 ceph-mon[29344]: pgmap v1089: 150 pgs: 150 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s, 8 objects/s recovering 2024-03-12T03:55:29.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:29 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-12T03:55:29.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:29 smithi169 ceph-mon[29344]: osdmap e526: 8 total, 8 up, 7 in 2024-03-12T03:55:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:29 smithi169 ceph-mon[39949]: pgmap v1089: 150 pgs: 150 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s, 8 objects/s recovering 2024-03-12T03:55:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:29 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-12T03:55:29.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:29 smithi169 ceph-mon[39949]: osdmap e526: 8 total, 8 up, 7 in 2024-03-12T03:55:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:29 smithi173 ceph-mon[33544]: pgmap v1089: 150 pgs: 150 active+clean; 582 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s, 8 objects/s recovering 2024-03-12T03:55:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:29 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-12T03:55:29.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:29 smithi173 ceph-mon[33544]: osdmap e526: 8 total, 8 up, 7 in 2024-03-12T03:55:30.437 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:31.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[29344]: 3.1b scrub starts 2024-03-12T03:55:31.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[29344]: 3.1b scrub ok 2024-03-12T03:55:31.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[29344]: osdmap e527: 8 total, 8 up, 7 in 2024-03-12T03:55:31.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[29344]: 1.10 deep-scrub starts 2024-03-12T03:55:31.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[29344]: 1.10 deep-scrub ok 2024-03-12T03:55:31.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[29344]: 4.0 scrub starts 2024-03-12T03:55:31.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[29344]: 4.0 scrub ok 2024-03-12T03:55:31.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[39949]: 3.1b scrub starts 2024-03-12T03:55:31.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[39949]: 3.1b scrub ok 2024-03-12T03:55:31.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[39949]: osdmap e527: 8 total, 8 up, 7 in 2024-03-12T03:55:31.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[39949]: 1.10 deep-scrub starts 2024-03-12T03:55:31.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[39949]: 1.10 deep-scrub ok 2024-03-12T03:55:31.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[39949]: 4.0 scrub starts 2024-03-12T03:55:31.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:30 smithi169 ceph-mon[39949]: 4.0 scrub ok 2024-03-12T03:55:31.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:30 smithi173 ceph-mon[33544]: 3.1b scrub starts 2024-03-12T03:55:31.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:30 smithi173 ceph-mon[33544]: 3.1b scrub ok 2024-03-12T03:55:31.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:30 smithi173 ceph-mon[33544]: osdmap e527: 8 total, 8 up, 7 in 2024-03-12T03:55:31.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:30 smithi173 ceph-mon[33544]: 1.10 deep-scrub starts 2024-03-12T03:55:31.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:30 smithi173 ceph-mon[33544]: 1.10 deep-scrub ok 2024-03-12T03:55:31.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:30 smithi173 ceph-mon[33544]: 4.0 scrub starts 2024-03-12T03:55:31.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:30 smithi173 ceph-mon[33544]: 4.0 scrub ok 2024-03-12T03:55:32.151 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:31 smithi169 ceph-mon[39949]: pgmap v1092: 150 pgs: 150 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s, 3 objects/s recovering 2024-03-12T03:55:32.151 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:31 smithi169 ceph-mon[39949]: osdmap e528: 8 total, 8 up, 7 in 2024-03-12T03:55:32.151 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:31 smithi169 ceph-mon[29344]: pgmap v1092: 150 pgs: 150 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s, 3 objects/s recovering 2024-03-12T03:55:32.151 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:31 smithi169 ceph-mon[29344]: osdmap e528: 8 total, 8 up, 7 in 2024-03-12T03:55:32.151 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a[29340]: 2024-03-12T03:55:31.804+0000 7f7050ca9700 -1 mon.a@0(leader).osd e529 definitely_dead 0 2024-03-12T03:55:32.152 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:55:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:55:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:55:32.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:31 smithi173 ceph-mon[33544]: pgmap v1092: 150 pgs: 150 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s, 3 objects/s recovering 2024-03-12T03:55:32.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:31 smithi173 ceph-mon[33544]: osdmap e528: 8 total, 8 up, 7 in 2024-03-12T03:55:32.720 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:55:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:55:32.720 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-03-12T03:55:32.720 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:55:32.747 INFO:teuthology.orchestra.run.smithi169.stderr:marked in osd.5. 2024-03-12T03:55:33.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[39949]: 6.1d scrub starts 2024-03-12T03:55:33.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[39949]: osdmap e529: 8 total, 8 up, 7 in 2024-03-12T03:55:33.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[39949]: 6.1d scrub ok 2024-03-12T03:55:33.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2216319740' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-12T03:55:33.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:33.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:33.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-12T03:55:33.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-12T03:55:33.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[29344]: 6.1d scrub starts 2024-03-12T03:55:33.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[29344]: osdmap e529: 8 total, 8 up, 7 in 2024-03-12T03:55:33.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[29344]: 6.1d scrub ok 2024-03-12T03:55:33.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2216319740' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-12T03:55:33.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:33.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:33.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-12T03:55:33.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:32 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-12T03:55:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:32 smithi173 ceph-mon[33544]: 6.1d scrub starts 2024-03-12T03:55:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:32 smithi173 ceph-mon[33544]: osdmap e529: 8 total, 8 up, 7 in 2024-03-12T03:55:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:32 smithi173 ceph-mon[33544]: 6.1d scrub ok 2024-03-12T03:55:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:32 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2216319740' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-12T03:55:33.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:32 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:33.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:32 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:33.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:32 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-12T03:55:33.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:32 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-12T03:55:33.710 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-12T03:55:33.710 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-12T03:55:33.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-03-12T03:55:33.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[29344]: pgmap v1095: 149 pgs: 1 peering, 148 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:33.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2216319740' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-12T03:55:33.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-12T03:55:33.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[29344]: osdmap e530: 8 total, 8 up, 8 in 2024-03-12T03:55:33.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[29344]: 3.13 scrub starts 2024-03-12T03:55:33.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[29344]: 3.13 scrub ok 2024-03-12T03:55:33.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[29344]: osdmap e531: 8 total, 8 up, 8 in 2024-03-12T03:55:33.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[39949]: pgmap v1095: 149 pgs: 1 peering, 148 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:33.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2216319740' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-12T03:55:33.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-12T03:55:33.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[39949]: osdmap e530: 8 total, 8 up, 8 in 2024-03-12T03:55:33.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[39949]: 3.13 scrub starts 2024-03-12T03:55:33.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[39949]: 3.13 scrub ok 2024-03-12T03:55:33.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:33 smithi169 ceph-mon[39949]: osdmap e531: 8 total, 8 up, 8 in 2024-03-12T03:55:33.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:55:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:55:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:33 smithi173 ceph-mon[33544]: pgmap v1095: 149 pgs: 1 peering, 148 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:33 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2216319740' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-12T03:55:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:33 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-12T03:55:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:33 smithi173 ceph-mon[33544]: osdmap e530: 8 total, 8 up, 8 in 2024-03-12T03:55:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:33 smithi173 ceph-mon[33544]: 3.13 scrub starts 2024-03-12T03:55:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:33 smithi173 ceph-mon[33544]: 3.13 scrub ok 2024-03-12T03:55:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:33 smithi173 ceph-mon[33544]: osdmap e531: 8 total, 8 up, 8 in 2024-03-12T03:55:34.131 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-12T03:55:34.200 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:35.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:35.138 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:34 smithi169 ceph-mon[29344]: 6.1c scrub starts 2024-03-12T03:55:35.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:34 smithi169 ceph-mon[29344]: 6.1c scrub ok 2024-03-12T03:55:35.139 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:34 smithi169 ceph-mon[29344]: osdmap e532: 8 total, 8 up, 8 in 2024-03-12T03:55:35.139 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:35.139 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:34 smithi169 ceph-mon[39949]: 6.1c scrub starts 2024-03-12T03:55:35.139 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:34 smithi169 ceph-mon[39949]: 6.1c scrub ok 2024-03-12T03:55:35.139 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:34 smithi169 ceph-mon[39949]: osdmap e532: 8 total, 8 up, 8 in 2024-03-12T03:55:35.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:35.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:34 smithi173 ceph-mon[33544]: 6.1c scrub starts 2024-03-12T03:55:35.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:34 smithi173 ceph-mon[33544]: 6.1c scrub ok 2024-03-12T03:55:35.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:34 smithi173 ceph-mon[33544]: osdmap e532: 8 total, 8 up, 8 in 2024-03-12T03:55:35.592 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:35.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[29344]: 5.5 scrub starts 2024-03-12T03:55:35.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[29344]: 5.5 scrub ok 2024-03-12T03:55:35.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[29344]: pgmap v1098: 149 pgs: 2 remapped+peering, 8 peering, 139 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:35.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[29344]: 1.4 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-12T03:55:35.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:55:35.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:55:35.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[29344]: 6.15 scrub starts 2024-03-12T03:55:35.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[29344]: 6.15 scrub ok 2024-03-12T03:55:35.865 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[29344]: osdmap e533: 8 total, 8 up, 8 in 2024-03-12T03:55:35.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[39949]: 5.5 scrub starts 2024-03-12T03:55:35.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[39949]: 5.5 scrub ok 2024-03-12T03:55:35.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[39949]: pgmap v1098: 149 pgs: 2 remapped+peering, 8 peering, 139 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:35.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[39949]: 1.4 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-12T03:55:35.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:55:35.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:55:35.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[39949]: 6.15 scrub starts 2024-03-12T03:55:35.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[39949]: 6.15 scrub ok 2024-03-12T03:55:35.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:35 smithi169 ceph-mon[39949]: osdmap e533: 8 total, 8 up, 8 in 2024-03-12T03:55:36.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:35 smithi173 ceph-mon[33544]: 5.5 scrub starts 2024-03-12T03:55:36.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:35 smithi173 ceph-mon[33544]: 5.5 scrub ok 2024-03-12T03:55:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:35 smithi173 ceph-mon[33544]: pgmap v1098: 149 pgs: 2 remapped+peering, 8 peering, 139 active+clean; 582 KiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:35 smithi173 ceph-mon[33544]: 1.4 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-12T03:55:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:35 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:55:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:35 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:55:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:35 smithi173 ceph-mon[33544]: 6.15 scrub starts 2024-03-12T03:55:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:35 smithi173 ceph-mon[33544]: 6.15 scrub ok 2024-03-12T03:55:36.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:35 smithi173 ceph-mon[33544]: osdmap e533: 8 total, 8 up, 8 in 2024-03-12T03:55:36.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[29344]: 6.1 deep-scrub starts 2024-03-12T03:55:36.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[29344]: 6.1 deep-scrub ok 2024-03-12T03:55:36.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[29344]: 5.10 deep-scrub starts 2024-03-12T03:55:36.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[29344]: 5.10 deep-scrub ok 2024-03-12T03:55:36.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[29344]: 3.b scrub starts 2024-03-12T03:55:36.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[29344]: 3.b scrub ok 2024-03-12T03:55:36.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[29344]: osdmap e534: 8 total, 8 up, 8 in 2024-03-12T03:55:36.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[39949]: 6.1 deep-scrub starts 2024-03-12T03:55:36.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[39949]: 6.1 deep-scrub ok 2024-03-12T03:55:36.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[39949]: 5.10 deep-scrub starts 2024-03-12T03:55:36.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[39949]: 5.10 deep-scrub ok 2024-03-12T03:55:36.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[39949]: 3.b scrub starts 2024-03-12T03:55:36.895 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[39949]: 3.b scrub ok 2024-03-12T03:55:36.895 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:36 smithi169 ceph-mon[39949]: osdmap e534: 8 total, 8 up, 8 in 2024-03-12T03:55:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:36 smithi173 ceph-mon[33544]: 6.1 deep-scrub starts 2024-03-12T03:55:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:36 smithi173 ceph-mon[33544]: 6.1 deep-scrub ok 2024-03-12T03:55:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:36 smithi173 ceph-mon[33544]: 5.10 deep-scrub starts 2024-03-12T03:55:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:36 smithi173 ceph-mon[33544]: 5.10 deep-scrub ok 2024-03-12T03:55:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:36 smithi173 ceph-mon[33544]: 3.b scrub starts 2024-03-12T03:55:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:36 smithi173 ceph-mon[33544]: 3.b scrub ok 2024-03-12T03:55:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:36 smithi173 ceph-mon[33544]: osdmap e534: 8 total, 8 up, 8 in 2024-03-12T03:55:37.741 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:55:37.741 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:55:37.741 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-03-12T03:55:37.741 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-03-12T03:55:37.741 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:55:37.790 INFO:teuthology.orchestra.run.smithi169.stderr:set pool 1 pgp_num to 11 2024-03-12T03:55:38.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:37 smithi169 ceph-mon[39949]: pgmap v1101: 149 pgs: 2 undersized+remapped+premerge+peered, 2 remapped+peering, 8 peering, 137 active+clean; 582 KiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 82 B/s, 2 objects/s recovering 2024-03-12T03:55:38.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:37 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:55:38.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:37 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:55:38.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:37 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1613086978' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-12T03:55:38.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:37 smithi169 ceph-mon[39949]: 3.18 scrub starts 2024-03-12T03:55:38.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:37 smithi169 ceph-mon[39949]: 3.18 scrub ok 2024-03-12T03:55:38.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:37 smithi169 ceph-mon[29344]: pgmap v1101: 149 pgs: 2 undersized+remapped+premerge+peered, 2 remapped+peering, 8 peering, 137 active+clean; 582 KiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 82 B/s, 2 objects/s recovering 2024-03-12T03:55:38.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:37 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:55:38.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:37 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:55:38.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:37 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1613086978' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-12T03:55:38.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:37 smithi169 ceph-mon[29344]: 3.18 scrub starts 2024-03-12T03:55:38.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:37 smithi169 ceph-mon[29344]: 3.18 scrub ok 2024-03-12T03:55:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:37 smithi173 ceph-mon[33544]: pgmap v1101: 149 pgs: 2 undersized+remapped+premerge+peered, 2 remapped+peering, 8 peering, 137 active+clean; 582 KiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 82 B/s, 2 objects/s recovering 2024-03-12T03:55:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:37 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:55:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:37 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:55:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:37 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1613086978' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-12T03:55:38.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:37 smithi173 ceph-mon[33544]: 3.18 scrub starts 2024-03-12T03:55:38.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:37 smithi173 ceph-mon[33544]: 3.18 scrub ok 2024-03-12T03:55:38.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-12T03:55:39.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[29344]: 3.1a scrub starts 2024-03-12T03:55:39.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[29344]: 3.1a scrub ok 2024-03-12T03:55:39.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[29344]: 1.b scrub starts 2024-03-12T03:55:39.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[29344]: 1.b scrub ok 2024-03-12T03:55:39.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1613086978' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-12T03:55:39.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[29344]: osdmap e535: 8 total, 8 up, 8 in 2024-03-12T03:55:39.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-12T03:55:39.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-12T03:55:39.166 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[39949]: 3.1a scrub starts 2024-03-12T03:55:39.166 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[39949]: 3.1a scrub ok 2024-03-12T03:55:39.166 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[39949]: 1.b scrub starts 2024-03-12T03:55:39.166 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[39949]: 1.b scrub ok 2024-03-12T03:55:39.166 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1613086978' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-12T03:55:39.166 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[39949]: osdmap e535: 8 total, 8 up, 8 in 2024-03-12T03:55:39.166 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-12T03:55:39.166 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:38 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-12T03:55:39.194 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-12T03:55:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:38 smithi173 ceph-mon[33544]: 3.1a scrub starts 2024-03-12T03:55:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:38 smithi173 ceph-mon[33544]: 3.1a scrub ok 2024-03-12T03:55:39.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:38 smithi173 ceph-mon[33544]: 1.b scrub starts 2024-03-12T03:55:39.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:38 smithi173 ceph-mon[33544]: 1.b scrub ok 2024-03-12T03:55:39.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:38 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1613086978' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-12T03:55:39.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:38 smithi173 ceph-mon[33544]: osdmap e535: 8 total, 8 up, 8 in 2024-03-12T03:55:39.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:38 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-12T03:55:39.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:38 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-12T03:55:39.252 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:39.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:39 smithi173 ceph-mon[33544]: pgmap v1104: 149 pgs: 2 remapped+peering, 7 peering, 140 active+clean; 582 KiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 64 KiB/s, 5 objects/s recovering 2024-03-12T03:55:40.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:39 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-12T03:55:40.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:39 smithi173 ceph-mon[33544]: osdmap e536: 8 total, 8 up, 8 in 2024-03-12T03:55:40.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:39 smithi173 ceph-mon[33544]: 5.13 deep-scrub starts 2024-03-12T03:55:40.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:39 smithi173 ceph-mon[33544]: 5.13 deep-scrub ok 2024-03-12T03:55:40.006 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:39 smithi173 ceph-mon[33544]: osdmap e537: 8 total, 8 up, 8 in 2024-03-12T03:55:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:40.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:39 smithi169 ceph-mon[39949]: pgmap v1104: 149 pgs: 2 remapped+peering, 7 peering, 140 active+clean; 582 KiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 64 KiB/s, 5 objects/s recovering 2024-03-12T03:55:40.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:39 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-12T03:55:40.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:39 smithi169 ceph-mon[39949]: osdmap e536: 8 total, 8 up, 8 in 2024-03-12T03:55:40.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:39 smithi169 ceph-mon[39949]: 5.13 deep-scrub starts 2024-03-12T03:55:40.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:39 smithi169 ceph-mon[39949]: 5.13 deep-scrub ok 2024-03-12T03:55:40.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:39 smithi169 ceph-mon[39949]: osdmap e537: 8 total, 8 up, 8 in 2024-03-12T03:55:40.114 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:39 smithi169 ceph-mon[29344]: pgmap v1104: 149 pgs: 2 remapped+peering, 7 peering, 140 active+clean; 582 KiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 64 KiB/s, 5 objects/s recovering 2024-03-12T03:55:40.114 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:39 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-12T03:55:40.114 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:39 smithi169 ceph-mon[29344]: osdmap e536: 8 total, 8 up, 8 in 2024-03-12T03:55:40.114 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:39 smithi169 ceph-mon[29344]: 5.13 deep-scrub starts 2024-03-12T03:55:40.114 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:39 smithi169 ceph-mon[29344]: 5.13 deep-scrub ok 2024-03-12T03:55:40.114 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:39 smithi169 ceph-mon[29344]: osdmap e537: 8 total, 8 up, 8 in 2024-03-12T03:55:41.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[39949]: 3.3 scrub starts 2024-03-12T03:55:41.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[39949]: 3.3 scrub ok 2024-03-12T03:55:41.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[39949]: 5.6 scrub starts 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[39949]: 5.6 scrub ok 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[39949]: 4.4 scrub starts 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[39949]: 4.4 scrub ok 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[39949]: 3.d scrub starts 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[39949]: 3.d scrub ok 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[39949]: osdmap e538: 8 total, 8 up, 8 in 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[29344]: 3.3 scrub starts 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[29344]: 3.3 scrub ok 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[29344]: 5.6 scrub starts 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[29344]: 5.6 scrub ok 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[29344]: 4.4 scrub starts 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[29344]: 4.4 scrub ok 2024-03-12T03:55:41.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[29344]: 3.d scrub starts 2024-03-12T03:55:41.036 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[29344]: 3.d scrub ok 2024-03-12T03:55:41.036 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:40 smithi169 ceph-mon[29344]: osdmap e538: 8 total, 8 up, 8 in 2024-03-12T03:55:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:40 smithi173 ceph-mon[33544]: 3.3 scrub starts 2024-03-12T03:55:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:40 smithi173 ceph-mon[33544]: 3.3 scrub ok 2024-03-12T03:55:41.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:40 smithi173 ceph-mon[33544]: 5.6 scrub starts 2024-03-12T03:55:41.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:40 smithi173 ceph-mon[33544]: 5.6 scrub ok 2024-03-12T03:55:41.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:40 smithi173 ceph-mon[33544]: 4.4 scrub starts 2024-03-12T03:55:41.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:40 smithi173 ceph-mon[33544]: 4.4 scrub ok 2024-03-12T03:55:41.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:40 smithi173 ceph-mon[33544]: 3.d scrub starts 2024-03-12T03:55:41.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:40 smithi173 ceph-mon[33544]: 3.d scrub ok 2024-03-12T03:55:41.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:40 smithi173 ceph-mon[33544]: osdmap e538: 8 total, 8 up, 8 in 2024-03-12T03:55:42.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[39949]: pgmap v1107: 149 pgs: 1 active+remapped+backfill_toofull, 148 active+clean; 582 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 144 KiB/s, 6 objects/s recovering 2024-03-12T03:55:42.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[39949]: 5.a deep-scrub starts 2024-03-12T03:55:42.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[39949]: 5.a deep-scrub ok 2024-03-12T03:55:42.121 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[39949]: 4.12 scrub starts 2024-03-12T03:55:42.121 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[39949]: 4.12 scrub ok 2024-03-12T03:55:42.121 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[39949]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:55:42.121 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[39949]: osdmap e539: 8 total, 8 up, 8 in 2024-03-12T03:55:42.121 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[29344]: pgmap v1107: 149 pgs: 1 active+remapped+backfill_toofull, 148 active+clean; 582 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 144 KiB/s, 6 objects/s recovering 2024-03-12T03:55:42.121 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[29344]: 5.a deep-scrub starts 2024-03-12T03:55:42.121 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[29344]: 5.a deep-scrub ok 2024-03-12T03:55:42.121 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[29344]: 4.12 scrub starts 2024-03-12T03:55:42.121 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[29344]: 4.12 scrub ok 2024-03-12T03:55:42.121 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[29344]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:55:42.121 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-mon[29344]: osdmap e539: 8 total, 8 up, 8 in 2024-03-12T03:55:42.122 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:55:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:55:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:55:42.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:41 smithi173 ceph-mon[33544]: pgmap v1107: 149 pgs: 1 active+remapped+backfill_toofull, 148 active+clean; 582 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 144 KiB/s, 6 objects/s recovering 2024-03-12T03:55:42.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:41 smithi173 ceph-mon[33544]: 5.a deep-scrub starts 2024-03-12T03:55:42.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:41 smithi173 ceph-mon[33544]: 5.a deep-scrub ok 2024-03-12T03:55:42.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:41 smithi173 ceph-mon[33544]: 4.12 scrub starts 2024-03-12T03:55:42.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:41 smithi173 ceph-mon[33544]: 4.12 scrub ok 2024-03-12T03:55:42.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:41 smithi173 ceph-mon[33544]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-12T03:55:42.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:41 smithi173 ceph-mon[33544]: osdmap e539: 8 total, 8 up, 8 in 2024-03-12T03:55:42.359 INFO:teuthology.orchestra.run.smithi169.stdout:mon.a: {} 2024-03-12T03:55:42.369 INFO:teuthology.orchestra.run.smithi169.stdout:mon.b: {} 2024-03-12T03:55:42.379 INFO:teuthology.orchestra.run.smithi169.stdout:mon.c: {} 2024-03-12T03:55:42.380 INFO:teuthology.orchestra.run.smithi169.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-12T03:55:42.380 INFO:teuthology.orchestra.run.smithi169.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-12T03:55:42.380 INFO:teuthology.orchestra.run.smithi169.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-12T03:55:42.614 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:55:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:55:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:55:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:55:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261758 9.1 9:85260111:::benchmark_data_smithi169_135229_object117543:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:55:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.410205+0000", 2024-03-12T03:55:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 519.19990050700005, 2024-03-12T03:55:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0706916660000001, 2024-03-12T03:55:42.614 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:55:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:55:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261758 2024-03-12T03:55:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410205+0000", 2024-03-12T03:55:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410205+0000", 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410206+0000", 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5799999999999998e-07 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410211+0000", 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7169999999999999e-06 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410220+0000", 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2930000000000006e-06 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410228+0000", 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6899999999999992e-06 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410291+0000", 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3311000000000004e-05 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.410297+0000", 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6290000000000004e-06 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415951+0000", 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056530599999999997 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415960+0000", 2024-03-12T03:55:42.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3470000000000005e-06 2024-03-12T03:55:42.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.480813+0000", 2024-03-12T03:55:42.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0648534999999999 2024-03-12T03:55:42.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.480897+0000", 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3160999999999999e-05 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261761 9.1 9:8da2e467:::benchmark_data_smithi169_135229_object117546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.415027+0000", 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 519.19507842300004, 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.084977831, 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.620 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261761 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415027+0000", 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415027+0000", 2024-03-12T03:55:42.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415028+0000", 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.424e-06 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415034+0000", 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6450000000000002e-06 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415043+0000", 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3030000000000005e-06 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.415051+0000", 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2400000000000001e-06 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.416244+0000", 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001193345 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.416253+0000", 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7790000000000001e-06 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.481315+0000", 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0650625730000001 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.481432+0000", 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011627 2024-03-12T03:55:42.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.481880+0000", 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00044876900000000001 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.493626+0000", 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011745267 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.496512+0000", 2024-03-12T03:55:42.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028865090000000002 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.499861+0000", 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033486789999999998 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.499933+0000", 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2706000000000005e-05 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.500005+0000", 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1322000000000006e-05 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261765 9.1 9:8ad8db76:::benchmark_data_smithi169_135229_object117550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.417087+0000", 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 519.19301837499995, 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0890983059999999, 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:55:42.627 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:42.628 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261765 2024-03-12T03:55:42.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.628 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417087+0000", 2024-03-12T03:55:42.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417087+0000", 2024-03-12T03:55:42.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417088+0000", 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7199999999999998e-07 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417092+0000", 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9570000000000004e-06 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417102+0000", 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0461e-05 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417110+0000", 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8129999999999996e-06 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417160+0000", 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9874999999999999e-05 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.417168+0000", 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0830000000000004e-06 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.491925+0000", 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0747573530000001 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.492007+0000", 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1908000000000003e-05 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.492086+0000", 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8860000000000001e-05 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.496568+0000", 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004481652 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.498024+0000", 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014564669999999999 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506074+0000", 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080501300000000008 2024-03-12T03:55:42.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506130+0000", 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5436999999999998e-05 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506185+0000", 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5538000000000003e-05 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261783 9.1 9:898fe7fd:::benchmark_data_smithi169_135229_object117568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.437041+0000", 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 519.17306483200002, 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0694922330000001, 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261783 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437041+0000", 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437041+0000", 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437041+0000", 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0999999999999999e-07 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437044+0000", 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9220000000000001e-06 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437050+0000", 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4289999999999998e-06 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437056+0000", 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7620000000000001e-06 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437144+0000", 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8048999999999997e-05 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437153+0000", 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8300000000000002e-06 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.493683+0000", 2024-03-12T03:55:42.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0565300179999999 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.493780+0000", 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7338999999999998e-05 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.493902+0000", 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121987 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.498030+0000", 2024-03-12T03:55:42.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004127716 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.500102+0000", 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0020721350000000001 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506322+0000", 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062200040000000003 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506462+0000", 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013965099999999999 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506533+0000", 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0881000000000001e-05 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261785 9.1 9:86b9b242:::benchmark_data_smithi169_135229_object117570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.437071+0000", 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 519.17303464999998, 2024-03-12T03:55:42.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.070147272, 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261785 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437071+0000", 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437071+0000", 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437071+0000", 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9500000000000003e-07 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437074+0000", 2024-03-12T03:55:42.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9799999999999998e-06 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437081+0000", 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3620000000000002e-06 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437087+0000", 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9649999999999996e-06 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437175+0000", 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8422000000000001e-05 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.437181+0000", 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9359999999999996e-06 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.495018+0000", 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0578373510000001 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.495118+0000", 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9735000000000006e-05 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.495234+0000", 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011618200000000001 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.500119+0000", 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048847059999999999 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506031+0000", 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0059123939999999996 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506677+0000", 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00064582699999999997 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506726+0000", 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9202000000000001e-05 2024-03-12T03:55:42.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507218+0000", 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00049171499999999999 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261796 9.1 9:8f8556f2:::benchmark_data_smithi169_135229_object117581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.456145+0000", 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 519.15396004599995, 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.05128215, 2024-03-12T03:55:42.648 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261796 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456145+0000", 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456145+0000", 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456147+0000", 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3939999999999999e-06 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456152+0000", 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.49e-06 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456167+0000", 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4620999999999999e-05 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456176+0000", 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8009999999999993e-06 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456235+0000", 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9247999999999999e-05 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.456243+0000", 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4880000000000007e-06 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.496616+0000", 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.040372536 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.496720+0000", 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000104215 2024-03-12T03:55:42.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.496797+0000", 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6518000000000002e-05 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.503848+0000", 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0070512819999999999 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506248+0000", 2024-03-12T03:55:42.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024002730000000001 2024-03-12T03:55:42.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507334+0000", 2024-03-12T03:55:42.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0010858440000000001 2024-03-12T03:55:42.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507384+0000", 2024-03-12T03:55:42.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9749000000000001e-05 2024-03-12T03:55:42.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507427+0000", 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3690999999999999e-05 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261804 9.1 9:88b23f87:::benchmark_data_smithi169_135229_object117589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.460244+0000", 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 519.14986141899999, 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0476733279999999, 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.655 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261804 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460244+0000", 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460244+0000", 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460245+0000", 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5340000000000001e-06 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460251+0000", 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.169e-06 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460261+0000", 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0654e-05 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460270+0000", 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2060000000000007e-06 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460372+0000", 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000102212 2024-03-12T03:55:42.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460381+0000", 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5759999999999995e-06 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.498055+0000", 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0376738430000001 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.498145+0000", 2024-03-12T03:55:42.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9747999999999995e-05 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.498237+0000", 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2366000000000003e-05 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506628+0000", 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083906250000000005 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.506642+0000", 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4623999999999999e-05 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507621+0000", 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00097875799999999993 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507879+0000", 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00025774899999999999 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507917+0000", 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8263999999999998e-05 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261806 9.1 9:8d72a8bd:::benchmark_data_smithi169_135229_object117591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.460561+0000", 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 519.149543978, 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.047474284, 2024-03-12T03:55:42.662 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261806 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460561+0000", 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460561+0000", 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460562+0000", 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0999999999999999e-07 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460565+0000", 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0790000000000001e-06 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460571+0000", 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2519999999999996e-06 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460577+0000", 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6370000000000004e-06 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460623+0000", 2024-03-12T03:55:42.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6378999999999998e-05 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.460631+0000", 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9619999999999993e-06 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.500157+0000", 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0395256470000001 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.500271+0000", 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000113964 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.500388+0000", 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116964 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507233+0000", 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068449100000000001 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507274+0000", 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1075000000000002e-05 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507962+0000", 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00068805199999999998 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508004+0000", 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1888999999999998e-05 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508036+0000", 2024-03-12T03:55:42.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1964000000000001e-05 2024-03-12T03:55:42.669 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.669 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.669 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.669 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261826 9.1 9:8dda3b7f:::benchmark_data_smithi169_135229_object117611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:55:42.669 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.482519+0000", 2024-03-12T03:55:42.669 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 519.12758685200004, 2024-03-12T03:55:42.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0256452170000001, 2024-03-12T03:55:42.671 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.671 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.671 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.671 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:55:42.671 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:42.671 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261826 2024-03-12T03:55:42.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.671 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.671 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482519+0000", 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482519+0000", 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482572+0000", 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3956000000000003e-05 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482579+0000", 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7290000000000003e-06 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482591+0000", 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1640999999999999e-05 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482599+0000", 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8099999999999998e-06 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482685+0000", 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6614999999999997e-05 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.482693+0000", 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2060000000000007e-06 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.503955+0000", 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.021261559 2024-03-12T03:55:42.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.504048+0000", 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2949999999999996e-05 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.504132+0000", 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3923000000000003e-05 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507238+0000", 2024-03-12T03:55:42.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0031059769999999998 2024-03-12T03:55:42.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.507568+0000", 2024-03-12T03:55:42.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00032969500000000002 2024-03-12T03:55:42.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508093+0000", 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052585900000000005 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508131+0000", 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7798e-05 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508164+0000", 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2499000000000002e-05 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.678 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261854 9.1 9:827d7e25:::benchmark_data_smithi169_135229_object117639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.504936+0000", 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 519.10516894199998, 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0172641549999999, 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261854 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.504936+0000", 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.504936+0000", 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.504938+0000", 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.161e-06 2024-03-12T03:55:42.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.504941+0000", 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8589999999999998e-06 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.504950+0000", 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1929999999999993e-06 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.504957+0000", 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2640000000000002e-06 2024-03-12T03:55:42.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.505095+0000", 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013761399999999999 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.505105+0000", 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0474e-05 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508192+0000", 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0030875100000001 2024-03-12T03:55:42.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508397+0000", 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000204976 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.508499+0000", 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101908 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.516871+0000", 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083716069999999997 2024-03-12T03:55:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.516943+0000", 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2513000000000002e-05 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522079+0000", 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051355070000000001 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522141+0000", 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1593000000000005e-05 2024-03-12T03:55:42.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522201+0000", 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9976000000000003e-05 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261863 9.1 9:8e1fde2b:::benchmark_data_smithi169_135229_object117648:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.512523+0000", 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 519.09758245900002, 2024-03-12T03:55:42.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0097642929999999, 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261863 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512523+0000", 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512523+0000", 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512524+0000", 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.362e-06 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512529+0000", 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1379999999999998e-06 2024-03-12T03:55:42.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512542+0000", 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3050999999999999e-05 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512550+0000", 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4560000000000003e-06 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512612+0000", 2024-03-12T03:55:42.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2089999999999997e-05 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.512621+0000", 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5019999999999992e-06 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.510476+0000", 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.99785585899999996 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.510571+0000", 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4543000000000006e-05 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.510687+0000", 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115807 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.516909+0000", 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062220299999999999 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.521991+0000", 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050824800000000003 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522208+0000", 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00021681000000000001 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522249+0000", 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1390999999999997e-05 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522287+0000", 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7774000000000001e-05 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15279.0:261868 9.1 9:87e1d5c9:::benchmark_data_smithi169_135229_object117653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:47:03.517760+0000", 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 519.09234580099996, 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0123262909999999, 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.692 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15279", 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/564200296", 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 261868 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517760+0000", 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517760+0000", 2024-03-12T03:55:42.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517760+0000", 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4799999999999997e-07 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517764+0000", 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9430000000000002e-06 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517774+0000", 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2769999999999994e-06 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517779+0000", 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6350000000000003e-06 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517818+0000", 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8349999999999997e-05 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.695 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:03.517823+0000", 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5260000000000001e-06 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.517013+0000", 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.99919013499999998 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.517253+0000", 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00023998300000000001 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 6,7", 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.517365+0000", 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112104 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522065+0000", 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046995939999999996 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.522298+0000", 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00023326400000000001 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.529995+0000", 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076970169999999996 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.530044+0000", 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8473000000000002e-05 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:47:04.530086+0000", 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2141999999999997e-05 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.698 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:219680 11.8 11:107fae17:::benchmark_data_smithi169_185434_object107733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:46.060872+0000", 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 56.549233043999998, 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.100256006, 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 219680 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.699 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.060872+0000", 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.060872+0000", 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.060873+0000", 2024-03-12T03:55:42.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5700000000000002e-07 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.060878+0000", 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7729999999999999e-06 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.060892+0000", 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4235e-05 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.060900+0000", 2024-03-12T03:55:42.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0020000000000006e-06 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.060953+0000", 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3208000000000003e-05 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.061065+0000", 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011215899999999999 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.061168+0000", 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010237499999999999 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.064527+0000", 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033588590000000001 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.064567+0000", 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0716000000000001e-05 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.703 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.102222+0000", 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.037654947000000001 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.102234+0000", 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1751e-05 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.086502+0000", 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.98426835800000001 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.086557+0000", 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4765000000000002e-05 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.161034+0000", 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0744767909999999 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.161128+0000", 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4309999999999997e-05 2024-03-12T03:55:42.705 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:219681 11.a 11:56521322:::benchmark_data_smithi169_185434_object107734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:46.062637+0000", 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 56.547468141000003, 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2245621170000001, 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:42.706 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 219681 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.062637+0000", 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.062637+0000", 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.062639+0000", 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.424e-06 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.062644+0000", 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4979999999999997e-06 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.062655+0000", 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1319e-05 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.062664+0000", 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2060000000000007e-06 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.068889+0000", 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062250630000000003 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.069043+0000", 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000154007 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5,6", 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.069159+0000", 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116227 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.074391+0000", 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052317249999999996 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.095400+0000", 2024-03-12T03:55:42.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021009280000000002 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.095415+0000", 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4664000000000001e-05 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.076633+0000", 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.98121864999999997 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.076649+0000", 2024-03-12T03:55:42.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5342e-05 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.287124+0000", 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.210475583 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.287199+0000", 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5129000000000005e-05 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:219700 11.a 11:5c6bd02b:::benchmark_data_smithi169_185434_object107753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:46.115872+0000", 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 56.494233399999999, 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.196147598, 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 219700 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.115872+0000", 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.115872+0000", 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.115873+0000", 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.491e-06 2024-03-12T03:55:42.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.115880+0000", 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6800000000000004e-06 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.115890+0000", 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9189999999999994e-06 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.115897+0000", 2024-03-12T03:55:42.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2660000000000004e-06 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.120397+0000", 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044999439999999996 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.120407+0000", 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4879999999999996e-06 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.076712+0000", 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.95630565899999997 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.076724+0000", 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1905999999999999e-05 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.295207+0000", 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2184826179999999 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.717 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.295348+0000", 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014087799999999999 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,5,6", 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.295687+0000", 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00033941199999999998 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.308684+0000", 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012996387999999999 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.308770+0000", 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6300999999999999e-05 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.311155+0000", 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002384686 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.311767+0000", 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00061191100000000003 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.311872+0000", 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000105262 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.312020+0000", 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000147789 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.720 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:219704 11.a 11:5d67601e:::benchmark_data_smithi169_185434_object107757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:46.119503+0000", 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 56.490602135000003, 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.193686885, 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 219704 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.721 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.119503+0000", 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.119503+0000", 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.119505+0000", 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9010000000000001e-06 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.119513+0000", 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.559e-06 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.119530+0000", 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6872999999999999e-05 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.119542+0000", 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2271e-05 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.120429+0000", 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00088689200000000002 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.120435+0000", 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.776e-06 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.076793+0000", 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.95635745999999999 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.724 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.076803+0000", 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5689999999999994e-06 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.300759+0000", 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.223956241 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.300918+0000", 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015897099999999999 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,5,6", 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.301078+0000", 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000160746 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.308727+0000", 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076488140000000003 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.311198+0000", 2024-03-12T03:55:42.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002470786 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.312028+0000", 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00083001500000000003 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.312904+0000", 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00087621400000000005 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.313073+0000", 2024-03-12T03:55:42.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016906000000000001 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.313190+0000", 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116737 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:219736 11.a 11:5c314505:::benchmark_data_smithi169_185434_object107789:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:46.172851+0000", 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 56.437254527999997, 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1406221560000001, 2024-03-12T03:55:42.728 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 219736 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172851+0000", 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.729 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172851+0000", 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172852+0000", 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.423e-06 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172859+0000", 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7059999999999998e-06 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172871+0000", 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1931e-05 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172879+0000", 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2430000000000005e-06 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172931+0000", 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1913999999999998e-05 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172938+0000", 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4290000000000004e-06 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.076896+0000", 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.90395758599999998 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.076905+0000", 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6740000000000003e-06 2024-03-12T03:55:42.732 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.303186+0000", 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.226280912 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.303323+0000", 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013731 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,5,6", 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.303454+0000", 2024-03-12T03:55:42.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013134799999999999 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.309386+0000", 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0059319439999999998 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.311237+0000", 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018506639999999999 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.312037+0000", 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00079996699999999995 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.313268+0000", 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001231123 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.313353+0000", 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5050999999999996e-05 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.313473+0000", 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000119931 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15816.0:219737 11.a 11:57a664c8:::benchmark_data_smithi169_185434_object107790:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:46.172896+0000", 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 56.437209482, 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1442211539999998, 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15816", 2024-03-12T03:55:42.736 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1885020140", 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 219737 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172896+0000", 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172896+0000", 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172896+0000", 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3500000000000002e-07 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172898+0000", 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.187e-06 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172903+0000", 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4780000000000002e-06 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.172907+0000", 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0659999999999997e-06 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.173808+0000", 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00090081299999999998 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.173819+0000", 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0844e-05 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.076930+0000", 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.90311109700000003 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.076950+0000", 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0565999999999999e-05 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.305736+0000", 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.228786208 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.305945+0000", 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000208044 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1,5,6", 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.306124+0000", 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017978800000000001 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.311657+0000", 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055330099999999997 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.312042+0000", 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000384626 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.312386+0000", 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00034370599999999999 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.316683+0000", 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042972039999999998 2024-03-12T03:55:42.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.316864+0000", 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000181373 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.317117+0000", 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000252809 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24359.0:4776 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94067641420032 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:46.880773+0000", 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 55.729332276000001, 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.259092305, 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24359", 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4776 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.744 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880773+0000", 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880773+0000", 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880774+0000", 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8999999999999995e-07 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880778+0000", 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1479999999999999e-06 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880790+0000", 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1541e-05 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880798+0000", 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0790000000000001e-06 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880890+0000", 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2459000000000004e-05 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:46.880898+0000", 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4189999999999996e-06 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.080301+0000", 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.199403203 2024-03-12T03:55:42.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.080311+0000", 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0346999999999999e-05 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.132534+0000", 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0522231 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for missing object", 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.132678+0000", 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014366199999999999 2024-03-12T03:55:42.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.133643+0000", 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00096476699999999995 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.133731+0000", 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8335e-05 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.136344+0000", 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002613523 2024-03-12T03:55:42.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.136434+0000", 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9060000000000005e-05 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.139865+0000", 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034318719999999999 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24359.0:4780 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94067641420032 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e505)", 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:54:47.081374+0000", 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 55.528731155000003, 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0589057180000001, 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24359", 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4780 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:42.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.081374+0000", 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.081374+0000", 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.081376+0000", 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7749999999999999e-06 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:42.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.081381+0000", 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4280000000000004e-06 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.081392+0000", 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0277e-05 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.081403+0000", 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0976e-05 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.081460+0000", 2024-03-12T03:55:42.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7723000000000003e-05 2024-03-12T03:55:42.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:42.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:47.081468+0000", 2024-03-12T03:55:42.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8550000000000002e-06 2024-03-12T03:55:42.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.132711+0000", 2024-03-12T03:55:42.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.051243111 2024-03-12T03:55:42.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for missing object", 2024-03-12T03:55:42.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.132862+0000", 2024-03-12T03:55:42.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015103 2024-03-12T03:55:42.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.133763+0000", 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00090100599999999999 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.133850+0000", 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6361999999999997e-05 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.138229+0000", 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043791450000000001 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.138321+0000", 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2447000000000004e-05 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:54:48.140280+0000", 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0019585829999999999 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:42.757 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:55:42.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[29344]: 6.1a scrub starts 2024-03-12T03:55:42.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[29344]: 6.1a scrub ok 2024-03-12T03:55:42.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[29344]: 3.7 deep-scrub starts 2024-03-12T03:55:42.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[29344]: 3.7 deep-scrub ok 2024-03-12T03:55:42.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-12T03:55:42.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-12T03:55:42.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4198721376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:55:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[39949]: 6.1a scrub starts 2024-03-12T03:55:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[39949]: 6.1a scrub ok 2024-03-12T03:55:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[39949]: 3.7 deep-scrub starts 2024-03-12T03:55:42.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[39949]: 3.7 deep-scrub ok 2024-03-12T03:55:42.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-12T03:55:42.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-12T03:55:42.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:42 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4198721376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:55:43.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:42 smithi173 ceph-mon[33544]: 6.1a scrub starts 2024-03-12T03:55:43.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:42 smithi173 ceph-mon[33544]: 6.1a scrub ok 2024-03-12T03:55:43.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:42 smithi173 ceph-mon[33544]: 3.7 deep-scrub starts 2024-03-12T03:55:43.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:42 smithi173 ceph-mon[33544]: 3.7 deep-scrub ok 2024-03-12T03:55:43.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:42 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-12T03:55:43.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:42 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-12T03:55:43.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:42 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4198721376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:55:43.595 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-12T03:55:43.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[29344]: pgmap v1110: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 3 objects/s recovering 2024-03-12T03:55:43.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[29344]: 1.d scrub starts 2024-03-12T03:55:43.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[29344]: 1.d scrub ok 2024-03-12T03:55:43.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[29344]: 3.16 scrub starts 2024-03-12T03:55:43.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[29344]: 3.16 scrub ok 2024-03-12T03:55:43.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-12T03:55:43.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[29344]: osdmap e540: 8 total, 8 up, 8 in 2024-03-12T03:55:43.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[29344]: osdmap e541: 8 total, 8 up, 8 in 2024-03-12T03:55:43.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[39949]: pgmap v1110: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 3 objects/s recovering 2024-03-12T03:55:43.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[39949]: 1.d scrub starts 2024-03-12T03:55:43.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[39949]: 1.d scrub ok 2024-03-12T03:55:43.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[39949]: 3.16 scrub starts 2024-03-12T03:55:43.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[39949]: 3.16 scrub ok 2024-03-12T03:55:43.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-12T03:55:43.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[39949]: osdmap e540: 8 total, 8 up, 8 in 2024-03-12T03:55:43.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:43 smithi169 ceph-mon[39949]: osdmap e541: 8 total, 8 up, 8 in 2024-03-12T03:55:43.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:55:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:55:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:43 smithi173 ceph-mon[33544]: pgmap v1110: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 3 objects/s recovering 2024-03-12T03:55:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:43 smithi173 ceph-mon[33544]: 1.d scrub starts 2024-03-12T03:55:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:43 smithi173 ceph-mon[33544]: 1.d scrub ok 2024-03-12T03:55:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:43 smithi173 ceph-mon[33544]: 3.16 scrub starts 2024-03-12T03:55:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:43 smithi173 ceph-mon[33544]: 3.16 scrub ok 2024-03-12T03:55:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:43 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-12T03:55:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:43 smithi173 ceph-mon[33544]: osdmap e540: 8 total, 8 up, 8 in 2024-03-12T03:55:43.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:43 smithi173 ceph-mon[33544]: osdmap e541: 8 total, 8 up, 8 in 2024-03-12T03:55:44.022 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:44.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-12T03:55:45.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:45.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:44 smithi169 ceph-mon[29344]: 1.7 scrub starts 2024-03-12T03:55:45.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:44 smithi169 ceph-mon[29344]: 1.7 scrub ok 2024-03-12T03:55:45.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:44 smithi169 ceph-mon[29344]: osdmap e542: 8 total, 8 up, 8 in 2024-03-12T03:55:45.170 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:45.170 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:44 smithi169 ceph-mon[39949]: 1.7 scrub starts 2024-03-12T03:55:45.170 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:44 smithi169 ceph-mon[39949]: 1.7 scrub ok 2024-03-12T03:55:45.170 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:44 smithi169 ceph-mon[39949]: osdmap e542: 8 total, 8 up, 8 in 2024-03-12T03:55:45.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:45.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:44 smithi173 ceph-mon[33544]: 1.7 scrub starts 2024-03-12T03:55:45.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:44 smithi173 ceph-mon[33544]: 1.7 scrub ok 2024-03-12T03:55:45.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:44 smithi173 ceph-mon[33544]: osdmap e542: 8 total, 8 up, 8 in 2024-03-12T03:55:45.383 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:46.124 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:45 smithi169 ceph-mon[29344]: pgmap v1113: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:46.124 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:45 smithi169 ceph-mon[29344]: 6.17 scrub starts 2024-03-12T03:55:46.125 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:45 smithi169 ceph-mon[29344]: 6.17 scrub ok 2024-03-12T03:55:46.125 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:45 smithi169 ceph-mon[29344]: 1.f scrub starts 2024-03-12T03:55:46.125 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:45 smithi169 ceph-mon[29344]: 1.f scrub ok 2024-03-12T03:55:46.125 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:45 smithi169 ceph-mon[29344]: osdmap e543: 8 total, 8 up, 8 in 2024-03-12T03:55:46.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:45 smithi169 ceph-mon[39949]: pgmap v1113: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:46.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:45 smithi169 ceph-mon[39949]: 6.17 scrub starts 2024-03-12T03:55:46.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:45 smithi169 ceph-mon[39949]: 6.17 scrub ok 2024-03-12T03:55:46.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:45 smithi169 ceph-mon[39949]: 1.f scrub starts 2024-03-12T03:55:46.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:45 smithi169 ceph-mon[39949]: 1.f scrub ok 2024-03-12T03:55:46.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:45 smithi169 ceph-mon[39949]: osdmap e543: 8 total, 8 up, 8 in 2024-03-12T03:55:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:45 smithi173 ceph-mon[33544]: pgmap v1113: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:45 smithi173 ceph-mon[33544]: 6.17 scrub starts 2024-03-12T03:55:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:45 smithi173 ceph-mon[33544]: 6.17 scrub ok 2024-03-12T03:55:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:45 smithi173 ceph-mon[33544]: 1.f scrub starts 2024-03-12T03:55:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:45 smithi173 ceph-mon[33544]: 1.f scrub ok 2024-03-12T03:55:46.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:45 smithi173 ceph-mon[33544]: osdmap e543: 8 total, 8 up, 8 in 2024-03-12T03:55:47.104 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[39949]: 6.9 scrub starts 2024-03-12T03:55:47.104 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[39949]: 6.9 scrub ok 2024-03-12T03:55:47.104 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:55:47.104 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:55:47.104 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[39949]: 6.1e scrub starts 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[39949]: 6.1e scrub ok 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[39949]: 4.1f scrub starts 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[39949]: 4.1f scrub ok 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[39949]: 6.18 scrub starts 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[39949]: 6.18 scrub ok 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[29344]: 6.9 scrub starts 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[29344]: 6.9 scrub ok 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[29344]: 6.1e scrub starts 2024-03-12T03:55:47.105 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[29344]: 6.1e scrub ok 2024-03-12T03:55:47.106 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[29344]: 4.1f scrub starts 2024-03-12T03:55:47.106 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[29344]: 4.1f scrub ok 2024-03-12T03:55:47.106 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[29344]: 6.18 scrub starts 2024-03-12T03:55:47.106 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[29344]: 6.18 scrub ok 2024-03-12T03:55:47.106 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-12T03:55:47.106 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:46 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-12T03:55:47.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:46 smithi173 ceph-mon[33544]: 6.9 scrub starts 2024-03-12T03:55:47.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:46 smithi173 ceph-mon[33544]: 6.9 scrub ok 2024-03-12T03:55:47.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:46 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:55:47.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:46 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:55:47.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:46 smithi173 ceph-mon[33544]: 6.1e scrub starts 2024-03-12T03:55:47.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:46 smithi173 ceph-mon[33544]: 6.1e scrub ok 2024-03-12T03:55:47.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:46 smithi173 ceph-mon[33544]: 4.1f scrub starts 2024-03-12T03:55:47.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:46 smithi173 ceph-mon[33544]: 4.1f scrub ok 2024-03-12T03:55:47.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:46 smithi173 ceph-mon[33544]: 6.18 scrub starts 2024-03-12T03:55:47.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:46 smithi173 ceph-mon[33544]: 6.18 scrub ok 2024-03-12T03:55:47.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:46 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-12T03:55:47.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:46 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-12T03:55:47.829 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.1 to 1 (10000) 2024-03-12T03:55:48.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[29344]: pgmap v1116: 147 pgs: 1 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 8.7 GiB used, 707 GiB / 715 GiB avail 2024-03-12T03:55:48.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[29344]: 6.7 scrub starts 2024-03-12T03:55:48.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[29344]: 4.e scrub starts 2024-03-12T03:55:48.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[29344]: 4.e scrub ok 2024-03-12T03:55:48.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[29344]: 6.7 scrub ok 2024-03-12T03:55:48.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-12T03:55:48.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[29344]: osdmap e544: 8 total, 8 up, 8 in 2024-03-12T03:55:48.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[29344]: 3.6 scrub starts 2024-03-12T03:55:48.136 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[29344]: 3.6 scrub ok 2024-03-12T03:55:48.136 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:48.136 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:48.136 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2590467194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:55:48.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[39949]: pgmap v1116: 147 pgs: 1 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 8.7 GiB used, 707 GiB / 715 GiB avail 2024-03-12T03:55:48.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[39949]: 6.7 scrub starts 2024-03-12T03:55:48.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[39949]: 4.e scrub starts 2024-03-12T03:55:48.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[39949]: 4.e scrub ok 2024-03-12T03:55:48.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[39949]: 6.7 scrub ok 2024-03-12T03:55:48.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-12T03:55:48.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[39949]: osdmap e544: 8 total, 8 up, 8 in 2024-03-12T03:55:48.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[39949]: 3.6 scrub starts 2024-03-12T03:55:48.137 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[39949]: 3.6 scrub ok 2024-03-12T03:55:48.137 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:48.137 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:48.137 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:47 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2590467194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:55:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:47 smithi173 ceph-mon[33544]: pgmap v1116: 147 pgs: 1 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 8.7 GiB used, 707 GiB / 715 GiB avail 2024-03-12T03:55:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:47 smithi173 ceph-mon[33544]: 6.7 scrub starts 2024-03-12T03:55:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:47 smithi173 ceph-mon[33544]: 4.e scrub starts 2024-03-12T03:55:48.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:47 smithi173 ceph-mon[33544]: 4.e scrub ok 2024-03-12T03:55:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:47 smithi173 ceph-mon[33544]: 6.7 scrub ok 2024-03-12T03:55:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:47 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-12T03:55:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:47 smithi173 ceph-mon[33544]: osdmap e544: 8 total, 8 up, 8 in 2024-03-12T03:55:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:47 smithi173 ceph-mon[33544]: 3.6 scrub starts 2024-03-12T03:55:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:47 smithi173 ceph-mon[33544]: 3.6 scrub ok 2024-03-12T03:55:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:47 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:55:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:47 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:48.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:47 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2590467194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:55:48.551 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-12T03:55:49.066 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:49.067 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:55:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:55:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-03-12T03:55:49.067 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:55:49.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:48 smithi169 ceph-mon[29344]: 3.9 scrub starts 2024-03-12T03:55:49.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:48 smithi169 ceph-mon[29344]: 3.9 scrub ok 2024-03-12T03:55:49.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:48 smithi169 ceph-mon[29344]: 6.19 scrub starts 2024-03-12T03:55:49.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:48 smithi169 ceph-mon[29344]: 6.19 scrub ok 2024-03-12T03:55:49.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:48 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2590467194' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-12T03:55:49.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:48 smithi169 ceph-mon[29344]: osdmap e545: 8 total, 8 up, 8 in 2024-03-12T03:55:49.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:48 smithi169 ceph-mon[39949]: 3.9 scrub starts 2024-03-12T03:55:49.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:48 smithi169 ceph-mon[39949]: 3.9 scrub ok 2024-03-12T03:55:49.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:48 smithi169 ceph-mon[39949]: 6.19 scrub starts 2024-03-12T03:55:49.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:48 smithi169 ceph-mon[39949]: 6.19 scrub ok 2024-03-12T03:55:49.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:48 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2590467194' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-12T03:55:49.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:48 smithi169 ceph-mon[39949]: osdmap e545: 8 total, 8 up, 8 in 2024-03-12T03:55:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:48 smithi173 ceph-mon[33544]: 3.9 scrub starts 2024-03-12T03:55:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:48 smithi173 ceph-mon[33544]: 3.9 scrub ok 2024-03-12T03:55:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:48 smithi173 ceph-mon[33544]: 6.19 scrub starts 2024-03-12T03:55:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:48 smithi173 ceph-mon[33544]: 6.19 scrub ok 2024-03-12T03:55:49.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:48 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2590467194' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-12T03:55:49.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:48 smithi173 ceph-mon[33544]: osdmap e545: 8 total, 8 up, 8 in 2024-03-12T03:55:50.077 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-12T03:55:50.104 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:49 smithi169 ceph-mon[29344]: pgmap v1119: 147 pgs: 1 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:50.104 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:49 smithi169 ceph-mon[29344]: 4.9 scrub starts 2024-03-12T03:55:50.104 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:49 smithi169 ceph-mon[29344]: 4.9 scrub ok 2024-03-12T03:55:50.104 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:49 smithi169 ceph-mon[29344]: osdmap e546: 8 total, 8 up, 8 in 2024-03-12T03:55:50.104 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:49 smithi169 ceph-mon[29344]: 5.14 scrub starts 2024-03-12T03:55:50.104 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:49 smithi169 ceph-mon[29344]: 5.14 scrub ok 2024-03-12T03:55:50.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:49 smithi169 ceph-mon[39949]: pgmap v1119: 147 pgs: 1 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:50.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:49 smithi169 ceph-mon[39949]: 4.9 scrub starts 2024-03-12T03:55:50.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:49 smithi169 ceph-mon[39949]: 4.9 scrub ok 2024-03-12T03:55:50.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:49 smithi169 ceph-mon[39949]: osdmap e546: 8 total, 8 up, 8 in 2024-03-12T03:55:50.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:49 smithi169 ceph-mon[39949]: 5.14 scrub starts 2024-03-12T03:55:50.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:49 smithi169 ceph-mon[39949]: 5.14 scrub ok 2024-03-12T03:55:50.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:49 smithi173 ceph-mon[33544]: pgmap v1119: 147 pgs: 1 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:50.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:49 smithi173 ceph-mon[33544]: 4.9 scrub starts 2024-03-12T03:55:50.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:49 smithi173 ceph-mon[33544]: 4.9 scrub ok 2024-03-12T03:55:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:49 smithi173 ceph-mon[33544]: osdmap e546: 8 total, 8 up, 8 in 2024-03-12T03:55:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:49 smithi173 ceph-mon[33544]: 5.14 scrub starts 2024-03-12T03:55:50.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:49 smithi173 ceph-mon[33544]: 5.14 scrub ok 2024-03-12T03:55:51.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:50 smithi169 ceph-mon[29344]: 6.f deep-scrub starts 2024-03-12T03:55:51.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:50 smithi169 ceph-mon[29344]: osdmap e547: 8 total, 8 up, 8 in 2024-03-12T03:55:51.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:50 smithi169 ceph-mon[29344]: 6.f deep-scrub ok 2024-03-12T03:55:51.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:50 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-12T03:55:51.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:50 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-12T03:55:51.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:50 smithi169 ceph-mon[39949]: 6.f deep-scrub starts 2024-03-12T03:55:51.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:50 smithi169 ceph-mon[39949]: osdmap e547: 8 total, 8 up, 8 in 2024-03-12T03:55:51.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:50 smithi169 ceph-mon[39949]: 6.f deep-scrub ok 2024-03-12T03:55:51.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:50 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-12T03:55:51.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:50 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-12T03:55:51.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:50 smithi173 ceph-mon[33544]: 6.f deep-scrub starts 2024-03-12T03:55:51.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:50 smithi173 ceph-mon[33544]: osdmap e547: 8 total, 8 up, 8 in 2024-03-12T03:55:51.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:50 smithi173 ceph-mon[33544]: 6.f deep-scrub ok 2024-03-12T03:55:51.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:50 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-12T03:55:51.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:50 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-12T03:55:51.858 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:52.112 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:55:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:55:52.113 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[29344]: pgmap v1122: 146 pgs: 1 peering, 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:52.113 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[29344]: 4.8 scrub starts 2024-03-12T03:55:52.113 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[29344]: 4.8 scrub ok 2024-03-12T03:55:52.113 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[29344]: 4.d scrub starts 2024-03-12T03:55:52.113 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[29344]: 4.d scrub ok 2024-03-12T03:55:52.113 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-12T03:55:52.113 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[29344]: osdmap e548: 8 total, 8 up, 8 in 2024-03-12T03:55:52.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[39949]: pgmap v1122: 146 pgs: 1 peering, 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:52.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[39949]: 4.8 scrub starts 2024-03-12T03:55:52.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[39949]: 4.8 scrub ok 2024-03-12T03:55:52.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[39949]: 4.d scrub starts 2024-03-12T03:55:52.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[39949]: 4.d scrub ok 2024-03-12T03:55:52.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-12T03:55:52.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:51 smithi169 ceph-mon[39949]: osdmap e548: 8 total, 8 up, 8 in 2024-03-12T03:55:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:51 smithi173 ceph-mon[33544]: pgmap v1122: 146 pgs: 1 peering, 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:51 smithi173 ceph-mon[33544]: 4.8 scrub starts 2024-03-12T03:55:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:51 smithi173 ceph-mon[33544]: 4.8 scrub ok 2024-03-12T03:55:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:51 smithi173 ceph-mon[33544]: 4.d scrub starts 2024-03-12T03:55:52.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:51 smithi173 ceph-mon[33544]: 4.d scrub ok 2024-03-12T03:55:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:51 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-12T03:55:52.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:51 smithi173 ceph-mon[33544]: osdmap e548: 8 total, 8 up, 8 in 2024-03-12T03:55:52.885 INFO:teuthology.orchestra.run.smithi169.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-12T03:55:53.060 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[29344]: 4.1c scrub starts 2024-03-12T03:55:53.061 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[29344]: 4.1c scrub ok 2024-03-12T03:55:53.061 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[29344]: 6.0 scrub starts 2024-03-12T03:55:53.061 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[29344]: 6.0 scrub ok 2024-03-12T03:55:53.061 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[29344]: osdmap e549: 8 total, 8 up, 8 in 2024-03-12T03:55:53.061 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2401101338' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:55:53.061 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[29344]: 1.c scrub starts 2024-03-12T03:55:53.061 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[29344]: 1.c scrub ok 2024-03-12T03:55:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[39949]: 4.1c scrub starts 2024-03-12T03:55:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[39949]: 4.1c scrub ok 2024-03-12T03:55:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[39949]: 6.0 scrub starts 2024-03-12T03:55:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[39949]: 6.0 scrub ok 2024-03-12T03:55:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[39949]: osdmap e549: 8 total, 8 up, 8 in 2024-03-12T03:55:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2401101338' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:55:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[39949]: 1.c scrub starts 2024-03-12T03:55:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:52 smithi169 ceph-mon[39949]: 1.c scrub ok 2024-03-12T03:55:53.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:52 smithi173 ceph-mon[33544]: 4.1c scrub starts 2024-03-12T03:55:53.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:52 smithi173 ceph-mon[33544]: 4.1c scrub ok 2024-03-12T03:55:53.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:52 smithi173 ceph-mon[33544]: 6.0 scrub starts 2024-03-12T03:55:53.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:52 smithi173 ceph-mon[33544]: 6.0 scrub ok 2024-03-12T03:55:53.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:52 smithi173 ceph-mon[33544]: osdmap e549: 8 total, 8 up, 8 in 2024-03-12T03:55:53.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:52 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2401101338' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-12T03:55:53.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:52 smithi173 ceph-mon[33544]: 1.c scrub starts 2024-03-12T03:55:53.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:52 smithi173 ceph-mon[33544]: 1.c scrub ok 2024-03-12T03:55:53.803 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:55:53.803 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-03-12T03:55:53.803 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-03-12T03:55:53.803 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-03-12T03:55:53.803 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:55:53.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:55:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:55:54.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[39949]: pgmap v1125: 146 pgs: 1 peering, 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:54.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[39949]: 3.4 scrub starts 2024-03-12T03:55:54.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[39949]: 3.4 scrub ok 2024-03-12T03:55:54.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[39949]: 4.1e scrub starts 2024-03-12T03:55:54.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[39949]: 4.1e scrub ok 2024-03-12T03:55:54.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2401101338' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-12T03:55:54.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[39949]: osdmap e550: 8 total, 8 up, 8 in 2024-03-12T03:55:54.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[39949]: 4.11 scrub starts 2024-03-12T03:55:54.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[39949]: 4.11 scrub ok 2024-03-12T03:55:54.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[39949]: osdmap e551: 8 total, 8 up, 8 in 2024-03-12T03:55:54.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[29344]: pgmap v1125: 146 pgs: 1 peering, 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:54.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[29344]: 3.4 scrub starts 2024-03-12T03:55:54.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[29344]: 3.4 scrub ok 2024-03-12T03:55:54.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[29344]: 4.1e scrub starts 2024-03-12T03:55:54.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[29344]: 4.1e scrub ok 2024-03-12T03:55:54.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2401101338' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-12T03:55:54.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[29344]: osdmap e550: 8 total, 8 up, 8 in 2024-03-12T03:55:54.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[29344]: 4.11 scrub starts 2024-03-12T03:55:54.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[29344]: 4.11 scrub ok 2024-03-12T03:55:54.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:53 smithi169 ceph-mon[29344]: osdmap e551: 8 total, 8 up, 8 in 2024-03-12T03:55:54.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:53 smithi173 ceph-mon[33544]: pgmap v1125: 146 pgs: 1 peering, 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:54.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:53 smithi173 ceph-mon[33544]: 3.4 scrub starts 2024-03-12T03:55:54.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:53 smithi173 ceph-mon[33544]: 3.4 scrub ok 2024-03-12T03:55:54.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:53 smithi173 ceph-mon[33544]: 4.1e scrub starts 2024-03-12T03:55:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:53 smithi173 ceph-mon[33544]: 4.1e scrub ok 2024-03-12T03:55:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:53 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2401101338' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-12T03:55:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:53 smithi173 ceph-mon[33544]: osdmap e550: 8 total, 8 up, 8 in 2024-03-12T03:55:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:53 smithi173 ceph-mon[33544]: 4.11 scrub starts 2024-03-12T03:55:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:53 smithi173 ceph-mon[33544]: 4.11 scrub ok 2024-03-12T03:55:54.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:53 smithi173 ceph-mon[33544]: osdmap e551: 8 total, 8 up, 8 in 2024-03-12T03:55:54.280 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-12T03:55:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[29344]: 4.13 scrub starts 2024-03-12T03:55:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[29344]: 4.13 scrub ok 2024-03-12T03:55:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[29344]: 4.1 scrub starts 2024-03-12T03:55:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[29344]: 4.1 scrub ok 2024-03-12T03:55:54.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:54.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-12T03:55:54.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-12T03:55:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[39949]: 4.13 scrub starts 2024-03-12T03:55:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[39949]: 4.13 scrub ok 2024-03-12T03:55:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[39949]: 4.1 scrub starts 2024-03-12T03:55:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[39949]: 4.1 scrub ok 2024-03-12T03:55:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-12T03:55:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:54 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-12T03:55:55.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-12T03:55:55.095 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:54 smithi173 ceph-mon[33544]: 4.13 scrub starts 2024-03-12T03:55:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:54 smithi173 ceph-mon[33544]: 4.13 scrub ok 2024-03-12T03:55:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:54 smithi173 ceph-mon[33544]: 4.1 scrub starts 2024-03-12T03:55:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:54 smithi173 ceph-mon[33544]: 4.1 scrub ok 2024-03-12T03:55:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:54 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:54 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-12T03:55:55.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:54 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-12T03:55:56.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:55 smithi169 ceph-mon[39949]: pgmap v1128: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:56.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:55 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-12T03:55:56.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:55 smithi169 ceph-mon[39949]: osdmap e552: 8 total, 8 up, 8 in 2024-03-12T03:55:56.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:55 smithi169 ceph-mon[39949]: 3.15 scrub starts 2024-03-12T03:55:56.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:55 smithi169 ceph-mon[39949]: 3.15 scrub ok 2024-03-12T03:55:56.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:55 smithi169 ceph-mon[29344]: pgmap v1128: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:56.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:55 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-12T03:55:56.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:55 smithi169 ceph-mon[29344]: osdmap e552: 8 total, 8 up, 8 in 2024-03-12T03:55:56.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:55 smithi169 ceph-mon[29344]: 3.15 scrub starts 2024-03-12T03:55:56.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:55 smithi169 ceph-mon[29344]: 3.15 scrub ok 2024-03-12T03:55:56.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:55 smithi173 ceph-mon[33544]: pgmap v1128: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:56.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:55 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-12T03:55:56.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:55 smithi173 ceph-mon[33544]: osdmap e552: 8 total, 8 up, 8 in 2024-03-12T03:55:56.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:55 smithi173 ceph-mon[33544]: 3.15 scrub starts 2024-03-12T03:55:56.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:55 smithi173 ceph-mon[33544]: 3.15 scrub ok 2024-03-12T03:55:56.812 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:55:56.912 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:56 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:55:56.912 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:56 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:55:56.912 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:56 smithi169 ceph-mon[29344]: 5.19 scrub starts 2024-03-12T03:55:56.912 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:56 smithi169 ceph-mon[29344]: 5.19 scrub ok 2024-03-12T03:55:56.912 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:56 smithi169 ceph-mon[29344]: osdmap e553: 8 total, 8 up, 8 in 2024-03-12T03:55:57.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:56 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:55:57.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:56 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:55:57.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:56 smithi169 ceph-mon[39949]: 5.19 scrub starts 2024-03-12T03:55:57.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:56 smithi169 ceph-mon[39949]: 5.19 scrub ok 2024-03-12T03:55:57.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:56 smithi169 ceph-mon[39949]: osdmap e553: 8 total, 8 up, 8 in 2024-03-12T03:55:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:56 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:55:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:56 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:55:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:56 smithi173 ceph-mon[33544]: 5.19 scrub starts 2024-03-12T03:55:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:56 smithi173 ceph-mon[33544]: 5.19 scrub ok 2024-03-12T03:55:57.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:56 smithi173 ceph-mon[33544]: osdmap e553: 8 total, 8 up, 8 in 2024-03-12T03:55:57.915 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.2 to 1 (10000) 2024-03-12T03:55:58.077 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[29344]: pgmap v1131: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-03-12T03:55:58.077 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[29344]: 5.16 scrub starts 2024-03-12T03:55:58.077 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[29344]: 5.16 scrub ok 2024-03-12T03:55:58.077 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[29344]: 3.0 scrub starts 2024-03-12T03:55:58.077 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[29344]: 3.0 scrub ok 2024-03-12T03:55:58.077 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[29344]: osdmap e554: 8 total, 8 up, 8 in 2024-03-12T03:55:58.077 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[29344]: 6.3 scrub starts 2024-03-12T03:55:58.078 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2028749235' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:55:58.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[39949]: pgmap v1131: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-03-12T03:55:58.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[39949]: 5.16 scrub starts 2024-03-12T03:55:58.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[39949]: 5.16 scrub ok 2024-03-12T03:55:58.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[39949]: 3.0 scrub starts 2024-03-12T03:55:58.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[39949]: 3.0 scrub ok 2024-03-12T03:55:58.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[39949]: osdmap e554: 8 total, 8 up, 8 in 2024-03-12T03:55:58.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[39949]: 6.3 scrub starts 2024-03-12T03:55:58.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:57 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2028749235' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:55:58.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:57 smithi173 ceph-mon[33544]: pgmap v1131: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-03-12T03:55:58.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:57 smithi173 ceph-mon[33544]: 5.16 scrub starts 2024-03-12T03:55:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:57 smithi173 ceph-mon[33544]: 5.16 scrub ok 2024-03-12T03:55:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:57 smithi173 ceph-mon[33544]: 3.0 scrub starts 2024-03-12T03:55:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:57 smithi173 ceph-mon[33544]: 3.0 scrub ok 2024-03-12T03:55:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:57 smithi173 ceph-mon[33544]: osdmap e554: 8 total, 8 up, 8 in 2024-03-12T03:55:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:57 smithi173 ceph-mon[33544]: 6.3 scrub starts 2024-03-12T03:55:58.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:57 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2028749235' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:55:58.725 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-12T03:55:59.036 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: 6.3 scrub ok 2024-03-12T03:55:59.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: 3.a scrub starts 2024-03-12T03:55:59.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: 3.a scrub ok 2024-03-12T03:55:59.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: 1.a scrub starts 2024-03-12T03:55:59.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: 1.a scrub ok 2024-03-12T03:55:59.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:55:59.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:55:59.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2028749235' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-12T03:55:59.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: osdmap e555: 8 total, 8 up, 8 in 2024-03-12T03:55:59.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: 6.12 scrub starts 2024-03-12T03:55:59.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: 6.12 scrub ok 2024-03-12T03:55:59.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: 3.12 scrub starts 2024-03-12T03:55:59.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[29344]: 3.12 scrub ok 2024-03-12T03:55:59.038 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: 6.3 scrub ok 2024-03-12T03:55:59.038 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: 3.a scrub starts 2024-03-12T03:55:59.038 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: 3.a scrub ok 2024-03-12T03:55:59.038 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: 1.a scrub starts 2024-03-12T03:55:59.039 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: 1.a scrub ok 2024-03-12T03:55:59.039 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:55:59.039 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:55:59.039 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2028749235' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-12T03:55:59.039 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: osdmap e555: 8 total, 8 up, 8 in 2024-03-12T03:55:59.039 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: 6.12 scrub starts 2024-03-12T03:55:59.039 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: 6.12 scrub ok 2024-03-12T03:55:59.039 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: 3.12 scrub starts 2024-03-12T03:55:59.039 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:58 smithi169 ceph-mon[39949]: 3.12 scrub ok 2024-03-12T03:55:59.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: 6.3 scrub ok 2024-03-12T03:55:59.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: 3.a scrub starts 2024-03-12T03:55:59.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: 3.a scrub ok 2024-03-12T03:55:59.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: 1.a scrub starts 2024-03-12T03:55:59.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: 1.a scrub ok 2024-03-12T03:55:59.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:55:59.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:55:59.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2028749235' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-12T03:55:59.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: osdmap e555: 8 total, 8 up, 8 in 2024-03-12T03:55:59.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: 6.12 scrub starts 2024-03-12T03:55:59.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: 6.12 scrub ok 2024-03-12T03:55:59.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: 3.12 scrub starts 2024-03-12T03:55:59.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:58 smithi173 ceph-mon[33544]: 3.12 scrub ok 2024-03-12T03:55:59.286 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-12T03:55:59.287 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-03-12T03:55:59.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-03-12T03:55:59.287 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-03-12T03:55:59.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.287 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(4.6 522/521 [PushOp(4:66924383:reshard::reshard.0000000005:head, version: 479'26, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:66924383:reshard::reshard.0000000005:head@479'26, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-12T03:55:59.287 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:21.714404+0000", 2024-03-12T03:55:59.287 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.568868113000001, 2024-03-12T03:55:59.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0011463059999999999, 2024-03-12T03:55:59.287 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.287 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.287 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714404+0000", 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714404+0000", 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714405+0000", 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8800000000000002e-07 2024-03-12T03:55:59.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714421+0000", 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5671e-05 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714438+0000", 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.787e-05 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714455+0000", 2024-03-12T03:55:59.289 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6412000000000001e-05 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714551+0000", 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6013000000000004e-05 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714565+0000", 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4374e-05 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.715551+0000", 2024-03-12T03:55:59.290 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00098527800000000002 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(4.1a 522/521 [PushOp(4:5c7e8a6d:reshard::reshard.0000000015:head, version: 479'35, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:5c7e8a6d:reshard::reshard.0000000015:head@479'35, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:21.714745+0000", 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.568527119000002, 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00072578200000000001, 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.291 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714745+0000", 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714745+0000", 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714745+0000", 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0100000000000001e-07 2024-03-12T03:55:59.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714763+0000", 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7915e-05 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714776+0000", 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.225e-05 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714792+0000", 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6424000000000001e-05 2024-03-12T03:55:59.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714884+0000", 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2223000000000001e-05 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714897+0000", 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.235e-05 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.715471+0000", 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00057441899999999995 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(4.19 522/521 [PushOp(4:9ed7f5b7:::obj_delete_at_hint.0000000004:head, version: 479'64, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:9ed7f5b7:::obj_delete_at_hint.0000000004:head@479'64, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:21.714868+0000", 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.568404106000003, 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00072555000000000004, 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.296 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.296 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714868+0000", 2024-03-12T03:55:59.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.296 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.296 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714868+0000", 2024-03-12T03:55:59.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714869+0000", 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1600000000000002e-07 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714878+0000", 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4839999999999993e-06 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714888+0000", 2024-03-12T03:55:59.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0134e-05 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714902+0000", 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3895e-05 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714987+0000", 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4505000000000005e-05 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.715001+0000", 2024-03-12T03:55:59.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4143e-05 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.715594+0000", 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00059307299999999995 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(4.0 522/521 [PushOp(4:02683654:reshard::reshard.0000000010:head, version: 479'44, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:02683654:reshard::reshard.0000000010:head@479'44, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:21.714922+0000", 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.568350117000001, 2024-03-12T03:55:59.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00068967799999999999, 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714922+0000", 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714922+0000", 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714923+0000", 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.46e-07 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714936+0000", 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3543999999999999e-05 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714948+0000", 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1531e-05 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.714964+0000", 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6660999999999999e-05 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.715484+0000", 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052001 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.715501+0000", 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6487e-05 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.715612+0000", 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000111099 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(4.6 522/521 [PushOp(4:64735f45:::obj_delete_at_hint.0000000083:head, version: 480'34, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:64735f45:::obj_delete_at_hint.0000000083:head@480'34, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:21.721311+0000", 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.561960993, 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000957568, 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721311+0000", 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721311+0000", 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721312+0000", 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3200000000000005e-07 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721338+0000", 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6488000000000001e-05 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721367+0000", 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8809000000000001e-05 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721390+0000", 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.283e-05 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721628+0000", 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00023818900000000001 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721641+0000", 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3132999999999999e-05 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.722269+0000", 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00062748700000000003 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.306 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(4.1e 522/521 [PushOp(4:7a480f67:::obj_delete_at_hint.0000000079:head, version: 480'35, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:7a480f67:::obj_delete_at_hint.0000000079:head@480'35, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:21.721703+0000", 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.561568973999997, 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00066302400000000001, 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721703+0000", 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721703+0000", 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721704+0000", 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4000000000000001e-07 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721716+0000", 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1853e-05 2024-03-12T03:55:59.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721733+0000", 2024-03-12T03:55:59.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7387000000000001e-05 2024-03-12T03:55:59.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.721751+0000", 2024-03-12T03:55:59.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7365999999999999e-05 2024-03-12T03:55:59.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.722025+0000", 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00027487799999999999 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.722036+0000", 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0944000000000001e-05 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.722366+0000", 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00032995600000000001 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24359.0:4853 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94067641428096 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:21.887248+0000", 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.396024093999998, 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.82630078100000004, 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24359", 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4853 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887248+0000", 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887248+0000", 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887250+0000", 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.477e-06 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887254+0000", 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2130000000000002e-06 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887263+0000", 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.054e-06 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887270+0000", 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7089999999999996e-06 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887339+0000", 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9412999999999994e-05 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887353+0000", 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.36e-05 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.709734+0000", 2024-03-12T03:55:59.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.822380848 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.709890+0000", 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015626600000000001 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.713549+0000", 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036592009999999999 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24359.0:4855 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94067641420032 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:21.887284+0000", 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.395987929999997, 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.82221968999999995, 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24359", 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4855 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887284+0000", 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887284+0000", 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887285+0000", 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2199999999999999e-07 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.318 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887296+0000", 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1289e-05 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887301+0000", 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8890000000000001e-06 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887306+0000", 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5639999999999998e-06 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887366+0000", 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0164000000000002e-05 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887377+0000", 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1516e-05 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.701282+0000", 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.813904713 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.701875+0000", 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000593269 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.709504+0000", 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.007628864 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.321 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24359.0:4857 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94067641428096 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:21.887314+0000", 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.395958106999998, 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.82683443000000001, 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24359", 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4857 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887314+0000", 2024-03-12T03:55:59.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887314+0000", 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887315+0000", 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4200000000000002e-07 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887316+0000", 2024-03-12T03:55:59.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.517e-06 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887319+0000", 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1370000000000002e-06 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887322+0000", 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1190000000000001e-06 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887391+0000", 2024-03-12T03:55:59.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9119e-05 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887397+0000", 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.733e-06 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.712877+0000", 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.82547994499999999 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.712963+0000", 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5385999999999999e-05 2024-03-12T03:55:59.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.714149+0000", 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0011861319999999999 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24359.0:4859 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94067641420032 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:21.887329+0000", 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.395942945999998, 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.82228263199999996, 2024-03-12T03:55:59.326 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24359", 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4859 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887329+0000", 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887329+0000", 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887330+0000", 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5800000000000001e-07 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887331+0000", 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4109999999999999e-06 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887334+0000", 2024-03-12T03:55:59.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1099999999999999e-06 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887337+0000", 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0860000000000002e-06 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887405+0000", 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7611e-05 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:21.887414+0000", 2024-03-12T03:55:59.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5719999999999992e-06 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.702717+0000", 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81530251200000003 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703449+0000", 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000732543 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.709612+0000", 2024-03-12T03:55:59.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061625289999999999 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(2.2 523/522 [PushOp(2:5cea7035:::rbd_info:head, version: 70'2, data_included: [0~19], data_size: 19, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:5cea7035:::rbd_info:head@70'2, size: 19, copy_subset: [(0, 19)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 19, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:22.698038+0000", 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 36.585234186999998, 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0059241399999999996, 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698038+0000", 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698038+0000", 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698038+0000", 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8900000000000001e-07 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698055+0000", 2024-03-12T03:55:59.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.641e-05 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698068+0000", 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3536000000000001e-05 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698088+0000", 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9802000000000001e-05 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698316+0000", 2024-03-12T03:55:59.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000228316 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.699937+0000", 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0016201309999999999 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703962+0000", 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040256559999999999 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(5.d 523/522 [PushOp(5:b4812045:::notify.4:head, version: 74'8, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:b4812045:::notify.4:head@74'8, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:22.698143+0000", 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 36.585129651999999, 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00156207, 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698143+0000", 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698143+0000", 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698143+0000", 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7000000000000001e-07 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698155+0000", 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2038000000000001e-05 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698165+0000", 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0312999999999999e-05 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.698185+0000", 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9296999999999998e-05 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.699201+0000", 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001016884 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.699214+0000", 2024-03-12T03:55:59.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2680000000000001e-05 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.699705+0000", 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00049058800000000003 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(4.7 523/522 [PushOp(4:e1cdb60d:reshard::reshard.0000000008:head, version: 479'25, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:e1cdb60d:reshard::reshard.0000000008:head@479'25, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:22.699354+0000", 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 36.583918375000003, 2024-03-12T03:55:59.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067705409999999997, 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.699354+0000", 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.699354+0000", 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.699355+0000", 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.82e-07 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.699367+0000", 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.204e-05 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.699381+0000", 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3981e-05 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.699399+0000", 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8244999999999999e-05 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.704060+0000", 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046608580000000004 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.705987+0000", 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0019273280000000001 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.706124+0000", 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013750700000000001 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "pg_update_log_missing(5.15 epoch 523/522 rep_tid 237829 entries 523'12 (0'0) error 5:a93a5511:::notify.2:head by client.24359.0:4852 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:22.703441+0000", 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 36.579831374000001, 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002805185, 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:55:59.342 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703441+0000", 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703441+0000", 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703442+0000", 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6000000000000003e-07 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703450+0000", 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5730000000000008e-06 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703466+0000", 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5536000000000002e-05 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703479+0000", 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3509999999999999e-05 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.704003+0000", 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052404700000000003 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.706246+0000", 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022428589999999998 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "pg_update_log_missing(5.15 epoch 523/522 rep_tid 237830 entries 523'13 (0'0) error 5:a93a5511:::notify.2:head by client.24359.0:4856 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-12T03:55:59.345 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:22.703494+0000", 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 36.579778543000003, 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051617169999999997, 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703494+0000", 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703494+0000", 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703494+0000", 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8599999999999998e-07 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703500+0000", 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.913e-06 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703511+0000", 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.041e-05 2024-03-12T03:55:59.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.703521+0000", 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0637e-05 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.706034+0000", 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025129699999999998 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.708655+0000", 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026213009999999999 2024-03-12T03:55:59.348 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "pg_update_log_missing(5.d epoch 523/522 rep_tid 132147 entries 523'13 (0'0) error 5:b4812045:::notify.4:head by client.24359.0:4854 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:22.704068+0000", 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 36.579204294, 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004607496, 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.704068+0000", 2024-03-12T03:55:59.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.704068+0000", 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.704069+0000", 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7599999999999997e-07 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.704076+0000", 2024-03-12T03:55:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3980000000000002e-06 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.704091+0000", 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5102999999999999e-05 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.704104+0000", 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3069e-05 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.706160+0000", 2024-03-12T03:55:59.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0020554940000000002 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.708676+0000", 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025159560000000002 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "pg_update_log_missing(5.d epoch 523/522 rep_tid 132148 entries 523'14 (0'0) error 5:b4812045:::notify.4:head by client.24359.0:4858 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:22.705675+0000", 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 36.577597619000002, 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003012823, 2024-03-12T03:55:59.352 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.705675+0000", 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.705675+0000", 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.705675+0000", 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1799999999999995e-07 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.705683+0000", 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5059999999999998e-06 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.705703+0000", 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9794000000000001e-05 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.705718+0000", 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5026e-05 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.706283+0000", 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00056569300000000003 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.708688+0000", 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024042859999999998 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(4.7 523/522 [PushOp(4:e09767b9:::obj_delete_at_hint.0000000086:head, version: 480'27, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:e09767b9:::obj_delete_at_hint.0000000086:head@480'27, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:22.708848+0000", 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 36.574424528999998, 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000698644, 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.708848+0000", 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.708848+0000", 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.708848+0000", 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9100000000000004e-07 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.708859+0000", 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0572e-05 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.708872+0000", 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3189e-05 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.708886+0000", 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3995999999999999e-05 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.708952+0000", 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5747999999999995e-05 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.709434+0000", 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000481924 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.709546+0000", 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112724 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "pg_update_log_missing_reply(5.e epoch 523/522 rep_tid 2 lcod 523'13)", 2024-03-12T03:55:59.359 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:22.710787+0000", 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 36.57248577, 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002034335, 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.710787+0000", 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.710787+0000", 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.710787+0000", 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.61e-07 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.710791+0000", 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3739999999999998e-06 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.711504+0000", 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00071355400000000003 2024-03-12T03:55:59.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.711515+0000", 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0465999999999999e-05 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.712798+0000", 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001283855 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:22.712821+0000", 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2425e-05 2024-03-12T03:55:59.362 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.24359.0:4891 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94067641420032 gen 7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e534)", 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-03-12T03:55:36.889667+0000", 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 22.393605526999998, 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0016461169999999999, 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.24359", 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.169:0/1393676617", 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 4891 2024-03-12T03:55:59.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:36.889667+0000", 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:36.889667+0000", 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:36.889668+0000", 2024-03-12T03:55:59.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9999999999999995e-07 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:36.889674+0000", 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6490000000000002e-06 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:36.889688+0000", 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3226e-05 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:36.889697+0000", 2024-03-12T03:55:59.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3060000000000003e-06 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:36.889774+0000", 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7197e-05 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:36.889864+0000", 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9765999999999994e-05 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-03-12T03:55:36.891313+0000", 2024-03-12T03:55:59.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014489730000000001 2024-03-12T03:55:59.367 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.367 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.367 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.367 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-12T03:55:59.367 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-12T03:55:59.367 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-12T03:55:59.774 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:00.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:59 smithi169 ceph-mon[29344]: pgmap v1134: 144 pgs: 1 clean+premerge+peered, 1 active+remapped+backfill_toofull, 142 active+clean; 582 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:00.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:59 smithi169 ceph-mon[29344]: 3.19 scrub starts 2024-03-12T03:56:00.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:59 smithi169 ceph-mon[29344]: 3.19 scrub ok 2024-03-12T03:56:00.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:59 smithi169 ceph-mon[29344]: 6.c scrub starts 2024-03-12T03:56:00.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:55:59 smithi169 ceph-mon[29344]: 6.c scrub ok 2024-03-12T03:56:00.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:59 smithi169 ceph-mon[39949]: pgmap v1134: 144 pgs: 1 clean+premerge+peered, 1 active+remapped+backfill_toofull, 142 active+clean; 582 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:00.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:59 smithi169 ceph-mon[39949]: 3.19 scrub starts 2024-03-12T03:56:00.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:59 smithi169 ceph-mon[39949]: 3.19 scrub ok 2024-03-12T03:56:00.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:59 smithi169 ceph-mon[39949]: 6.c scrub starts 2024-03-12T03:56:00.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:55:59 smithi169 ceph-mon[39949]: 6.c scrub ok 2024-03-12T03:56:00.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:59 smithi173 ceph-mon[33544]: pgmap v1134: 144 pgs: 1 clean+premerge+peered, 1 active+remapped+backfill_toofull, 142 active+clean; 582 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:00.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:59 smithi173 ceph-mon[33544]: 3.19 scrub starts 2024-03-12T03:56:00.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:59 smithi173 ceph-mon[33544]: 3.19 scrub ok 2024-03-12T03:56:00.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:59 smithi173 ceph-mon[33544]: 6.c scrub starts 2024-03-12T03:56:00.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:55:59 smithi173 ceph-mon[33544]: 6.c scrub ok 2024-03-12T03:56:02.174 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:56:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:56:02.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[29344]: pgmap v1135: 144 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 142 active+clean; 582 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-12T03:56:02.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[29344]: 4.18 deep-scrub starts 2024-03-12T03:56:02.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[29344]: 4.18 deep-scrub ok 2024-03-12T03:56:02.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[29344]: 3.1d scrub starts 2024-03-12T03:56:02.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[29344]: 3.1d scrub ok 2024-03-12T03:56:02.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[29344]: 5.9 scrub starts 2024-03-12T03:56:02.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[29344]: 5.9 scrub ok 2024-03-12T03:56:02.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[29344]: 6.1f scrub starts 2024-03-12T03:56:02.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[29344]: 6.1f scrub ok 2024-03-12T03:56:02.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[29344]: 3.1 scrub starts 2024-03-12T03:56:02.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[29344]: 3.1 scrub ok 2024-03-12T03:56:02.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[39949]: pgmap v1135: 144 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 142 active+clean; 582 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-12T03:56:02.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[39949]: 4.18 deep-scrub starts 2024-03-12T03:56:02.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[39949]: 4.18 deep-scrub ok 2024-03-12T03:56:02.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[39949]: 3.1d scrub starts 2024-03-12T03:56:02.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[39949]: 3.1d scrub ok 2024-03-12T03:56:02.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[39949]: 5.9 scrub starts 2024-03-12T03:56:02.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[39949]: 5.9 scrub ok 2024-03-12T03:56:02.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[39949]: 6.1f scrub starts 2024-03-12T03:56:02.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[39949]: 6.1f scrub ok 2024-03-12T03:56:02.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[39949]: 3.1 scrub starts 2024-03-12T03:56:02.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:01 smithi169 ceph-mon[39949]: 3.1 scrub ok 2024-03-12T03:56:02.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:01 smithi173 ceph-mon[33544]: pgmap v1135: 144 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 142 active+clean; 582 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-12T03:56:02.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:01 smithi173 ceph-mon[33544]: 4.18 deep-scrub starts 2024-03-12T03:56:02.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:01 smithi173 ceph-mon[33544]: 4.18 deep-scrub ok 2024-03-12T03:56:02.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:01 smithi173 ceph-mon[33544]: 3.1d scrub starts 2024-03-12T03:56:02.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:01 smithi173 ceph-mon[33544]: 3.1d scrub ok 2024-03-12T03:56:02.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:01 smithi173 ceph-mon[33544]: 5.9 scrub starts 2024-03-12T03:56:02.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:01 smithi173 ceph-mon[33544]: 5.9 scrub ok 2024-03-12T03:56:02.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:01 smithi173 ceph-mon[33544]: 6.1f scrub starts 2024-03-12T03:56:02.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:01 smithi173 ceph-mon[33544]: 6.1f scrub ok 2024-03-12T03:56:02.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:01 smithi173 ceph-mon[33544]: 3.1 scrub starts 2024-03-12T03:56:02.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:01 smithi173 ceph-mon[33544]: 3.1 scrub ok 2024-03-12T03:56:02.943 INFO:teuthology.orchestra.run.smithi169.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-12T03:56:03.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:02 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:03.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:02 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1426284022' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:56:03.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:02 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:56:03.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:02 smithi173 ceph-mon[33544]: 6.16 scrub starts 2024-03-12T03:56:03.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:02 smithi173 ceph-mon[33544]: 6.16 scrub ok 2024-03-12T03:56:03.358 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:02 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:03.359 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:02 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1426284022' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:56:03.359 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:02 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:56:03.359 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:02 smithi169 ceph-mon[29344]: 6.16 scrub starts 2024-03-12T03:56:03.359 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:02 smithi169 ceph-mon[29344]: 6.16 scrub ok 2024-03-12T03:56:03.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:02 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:03.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:02 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1426284022' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:56:03.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:02 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-12T03:56:03.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:02 smithi169 ceph-mon[39949]: 6.16 scrub starts 2024-03-12T03:56:03.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:02 smithi169 ceph-mon[39949]: 6.16 scrub ok 2024-03-12T03:56:03.785 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-12T03:56:03.939 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:56:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:56:04.036 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:04.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:03 smithi169 ceph-mon[29344]: pgmap v1136: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-12T03:56:04.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:03 smithi169 ceph-mon[29344]: 5.f scrub starts 2024-03-12T03:56:04.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:03 smithi169 ceph-mon[29344]: 5.f scrub ok 2024-03-12T03:56:04.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:03 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-12T03:56:04.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:03 smithi169 ceph-mon[29344]: osdmap e556: 8 total, 8 up, 8 in 2024-03-12T03:56:04.158 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:03 smithi169 ceph-mon[29344]: osdmap e557: 8 total, 8 up, 8 in 2024-03-12T03:56:04.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:03 smithi169 ceph-mon[39949]: pgmap v1136: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-12T03:56:04.158 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:03 smithi169 ceph-mon[39949]: 5.f scrub starts 2024-03-12T03:56:04.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:03 smithi169 ceph-mon[39949]: 5.f scrub ok 2024-03-12T03:56:04.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:03 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-12T03:56:04.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:03 smithi169 ceph-mon[39949]: osdmap e556: 8 total, 8 up, 8 in 2024-03-12T03:56:04.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:03 smithi169 ceph-mon[39949]: osdmap e557: 8 total, 8 up, 8 in 2024-03-12T03:56:04.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:03 smithi173 ceph-mon[33544]: pgmap v1136: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-12T03:56:04.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:03 smithi173 ceph-mon[33544]: 5.f scrub starts 2024-03-12T03:56:04.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:03 smithi173 ceph-mon[33544]: 5.f scrub ok 2024-03-12T03:56:04.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:03 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-12T03:56:04.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:03 smithi173 ceph-mon[33544]: osdmap e556: 8 total, 8 up, 8 in 2024-03-12T03:56:04.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:03 smithi173 ceph-mon[33544]: osdmap e557: 8 total, 8 up, 8 in 2024-03-12T03:56:05.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:04 smithi169 ceph-mon[29344]: 6.4 scrub starts 2024-03-12T03:56:05.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:04 smithi169 ceph-mon[29344]: 6.4 scrub ok 2024-03-12T03:56:05.117 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:05.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:04 smithi169 ceph-mon[39949]: 6.4 scrub starts 2024-03-12T03:56:05.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:04 smithi169 ceph-mon[39949]: 6.4 scrub ok 2024-03-12T03:56:05.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:05.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:04 smithi173 ceph-mon[33544]: 6.4 scrub starts 2024-03-12T03:56:05.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:04 smithi173 ceph-mon[33544]: 6.4 scrub ok 2024-03-12T03:56:05.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:05.958 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.4 to 1 (10000) 2024-03-12T03:56:06.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:05 smithi173 ceph-mon[33544]: pgmap v1139: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-03-12T03:56:06.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:05 smithi173 ceph-mon[33544]: 5.1c scrub starts 2024-03-12T03:56:06.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:05 smithi173 ceph-mon[33544]: 5.1c scrub ok 2024-03-12T03:56:06.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:05 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1242917028' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:56:06.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:05 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:56:06.332 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:05 smithi169 ceph-mon[29344]: pgmap v1139: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-03-12T03:56:06.332 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:05 smithi169 ceph-mon[29344]: 5.1c scrub starts 2024-03-12T03:56:06.332 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:05 smithi169 ceph-mon[29344]: 5.1c scrub ok 2024-03-12T03:56:06.332 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:05 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1242917028' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:56:06.332 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:05 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:56:06.333 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:05 smithi169 ceph-mon[39949]: pgmap v1139: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-03-12T03:56:06.333 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:05 smithi169 ceph-mon[39949]: 5.1c scrub starts 2024-03-12T03:56:06.333 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:05 smithi169 ceph-mon[39949]: 5.1c scrub ok 2024-03-12T03:56:06.333 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:05 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1242917028' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:56:06.333 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:05 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:56:06.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-12T03:56:06.980 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:07.225 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:06 smithi169 ceph-mon[29344]: 4.f scrub starts 2024-03-12T03:56:07.225 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:06 smithi169 ceph-mon[29344]: 4.f scrub ok 2024-03-12T03:56:07.226 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:06 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-12T03:56:07.226 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:06 smithi169 ceph-mon[29344]: osdmap e558: 8 total, 8 up, 8 in 2024-03-12T03:56:07.226 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:06 smithi169 ceph-mon[29344]: 5.1f deep-scrub starts 2024-03-12T03:56:07.226 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:06 smithi169 ceph-mon[29344]: 5.1f deep-scrub ok 2024-03-12T03:56:07.226 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:06 smithi169 ceph-mon[39949]: 4.f scrub starts 2024-03-12T03:56:07.229 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:06 smithi169 ceph-mon[39949]: 4.f scrub ok 2024-03-12T03:56:07.229 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:06 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-12T03:56:07.230 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:06 smithi169 ceph-mon[39949]: osdmap e558: 8 total, 8 up, 8 in 2024-03-12T03:56:07.230 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:06 smithi169 ceph-mon[39949]: 5.1f deep-scrub starts 2024-03-12T03:56:07.230 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:06 smithi169 ceph-mon[39949]: 5.1f deep-scrub ok 2024-03-12T03:56:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:06 smithi173 ceph-mon[33544]: 4.f scrub starts 2024-03-12T03:56:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:06 smithi173 ceph-mon[33544]: 4.f scrub ok 2024-03-12T03:56:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:06 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-12T03:56:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:06 smithi173 ceph-mon[33544]: osdmap e558: 8 total, 8 up, 8 in 2024-03-12T03:56:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:06 smithi173 ceph-mon[33544]: 5.1f deep-scrub starts 2024-03-12T03:56:07.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:06 smithi173 ceph-mon[33544]: 5.1f deep-scrub ok 2024-03-12T03:56:08.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[29344]: pgmap v1141: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:08.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[29344]: 2.2 scrub starts 2024-03-12T03:56:08.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[29344]: 2.2 scrub ok 2024-03-12T03:56:08.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[29344]: 1.3 scrub starts 2024-03-12T03:56:08.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[29344]: 1.3 scrub ok 2024-03-12T03:56:08.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[29344]: 3.e deep-scrub starts 2024-03-12T03:56:08.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[29344]: 3.e deep-scrub ok 2024-03-12T03:56:08.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[29344]: 4.14 scrub starts 2024-03-12T03:56:08.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[29344]: 4.14 scrub ok 2024-03-12T03:56:08.100 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[29344]: 5.1e scrub starts 2024-03-12T03:56:08.101 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[29344]: 5.1e scrub ok 2024-03-12T03:56:08.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[39949]: pgmap v1141: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:08.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[39949]: 2.2 scrub starts 2024-03-12T03:56:08.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[39949]: 2.2 scrub ok 2024-03-12T03:56:08.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[39949]: 1.3 scrub starts 2024-03-12T03:56:08.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[39949]: 1.3 scrub ok 2024-03-12T03:56:08.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[39949]: 3.e deep-scrub starts 2024-03-12T03:56:08.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[39949]: 3.e deep-scrub ok 2024-03-12T03:56:08.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[39949]: 4.14 scrub starts 2024-03-12T03:56:08.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[39949]: 4.14 scrub ok 2024-03-12T03:56:08.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[39949]: 5.1e scrub starts 2024-03-12T03:56:08.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:07 smithi169 ceph-mon[39949]: 5.1e scrub ok 2024-03-12T03:56:08.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:07 smithi173 ceph-mon[33544]: pgmap v1141: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:08.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:07 smithi173 ceph-mon[33544]: 2.2 scrub starts 2024-03-12T03:56:08.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:07 smithi173 ceph-mon[33544]: 2.2 scrub ok 2024-03-12T03:56:08.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:07 smithi173 ceph-mon[33544]: 1.3 scrub starts 2024-03-12T03:56:08.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:07 smithi173 ceph-mon[33544]: 1.3 scrub ok 2024-03-12T03:56:08.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:07 smithi173 ceph-mon[33544]: 3.e deep-scrub starts 2024-03-12T03:56:08.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:07 smithi173 ceph-mon[33544]: 3.e deep-scrub ok 2024-03-12T03:56:08.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:07 smithi173 ceph-mon[33544]: 4.14 scrub starts 2024-03-12T03:56:08.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:07 smithi173 ceph-mon[33544]: 4.14 scrub ok 2024-03-12T03:56:08.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:07 smithi173 ceph-mon[33544]: 5.1e scrub starts 2024-03-12T03:56:08.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:07 smithi173 ceph-mon[33544]: 5.1e scrub ok 2024-03-12T03:56:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:08 smithi173 ceph-mon[33544]: 5.d scrub starts 2024-03-12T03:56:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:08 smithi173 ceph-mon[33544]: 5.d scrub ok 2024-03-12T03:56:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:08 smithi173 ceph-mon[33544]: osdmap e559: 8 total, 8 up, 8 in 2024-03-12T03:56:09.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:08 smithi173 ceph-mon[33544]: 4.16 scrub starts 2024-03-12T03:56:09.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:08 smithi173 ceph-mon[33544]: 4.16 scrub ok 2024-03-12T03:56:09.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:08 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2261634084' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:56:09.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:08 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:56:09.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[29344]: 5.d scrub starts 2024-03-12T03:56:09.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[29344]: 5.d scrub ok 2024-03-12T03:56:09.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[29344]: osdmap e559: 8 total, 8 up, 8 in 2024-03-12T03:56:09.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[29344]: 4.16 scrub starts 2024-03-12T03:56:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[29344]: 4.16 scrub ok 2024-03-12T03:56:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2261634084' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:56:09.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:56:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[39949]: 5.d scrub starts 2024-03-12T03:56:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[39949]: 5.d scrub ok 2024-03-12T03:56:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[39949]: osdmap e559: 8 total, 8 up, 8 in 2024-03-12T03:56:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[39949]: 4.16 scrub starts 2024-03-12T03:56:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[39949]: 4.16 scrub ok 2024-03-12T03:56:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2261634084' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:56:09.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:08 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-12T03:56:09.590 INFO:teuthology.orchestra.run.smithi169.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-12T03:56:10.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: pgmap v1142: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: 5.1 scrub starts 2024-03-12T03:56:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: 5.1 scrub ok 2024-03-12T03:56:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: 4.15 scrub starts 2024-03-12T03:56:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: 4.15 scrub ok 2024-03-12T03:56:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: 6.e scrub starts 2024-03-12T03:56:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: 6.e scrub ok 2024-03-12T03:56:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: 6.5 scrub starts 2024-03-12T03:56:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: 4.c scrub starts 2024-03-12T03:56:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: 4.c scrub ok 2024-03-12T03:56:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: 6.5 scrub ok 2024-03-12T03:56:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-12T03:56:10.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[29344]: osdmap e560: 8 total, 8 up, 8 in 2024-03-12T03:56:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: pgmap v1142: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: 5.1 scrub starts 2024-03-12T03:56:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: 5.1 scrub ok 2024-03-12T03:56:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: 4.15 scrub starts 2024-03-12T03:56:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: 4.15 scrub ok 2024-03-12T03:56:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: 6.e scrub starts 2024-03-12T03:56:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: 6.e scrub ok 2024-03-12T03:56:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: 6.5 scrub starts 2024-03-12T03:56:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: 4.c scrub starts 2024-03-12T03:56:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: 4.c scrub ok 2024-03-12T03:56:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: 6.5 scrub ok 2024-03-12T03:56:10.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-12T03:56:10.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:09 smithi169 ceph-mon[39949]: osdmap e560: 8 total, 8 up, 8 in 2024-03-12T03:56:10.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: pgmap v1142: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:10.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: 5.1 scrub starts 2024-03-12T03:56:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: 5.1 scrub ok 2024-03-12T03:56:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: 4.15 scrub starts 2024-03-12T03:56:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: 4.15 scrub ok 2024-03-12T03:56:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: 6.e scrub starts 2024-03-12T03:56:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: 6.e scrub ok 2024-03-12T03:56:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: 6.5 scrub starts 2024-03-12T03:56:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: 4.c scrub starts 2024-03-12T03:56:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: 4.c scrub ok 2024-03-12T03:56:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: 6.5 scrub ok 2024-03-12T03:56:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-12T03:56:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:09 smithi173 ceph-mon[33544]: osdmap e560: 8 total, 8 up, 8 in 2024-03-12T03:56:10.362 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-12T03:56:10.614 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:10.988 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:10 smithi169 ceph-mon[29344]: 4.2 scrub starts 2024-03-12T03:56:10.988 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:10 smithi169 ceph-mon[29344]: 4.2 scrub ok 2024-03-12T03:56:10.988 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:10 smithi169 ceph-mon[29344]: osdmap e561: 8 total, 8 up, 8 in 2024-03-12T03:56:10.988 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:10 smithi169 ceph-mon[29344]: 6.10 scrub starts 2024-03-12T03:56:10.988 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:10 smithi169 ceph-mon[29344]: 6.10 scrub ok 2024-03-12T03:56:10.989 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:10 smithi169 ceph-mon[39949]: 4.2 scrub starts 2024-03-12T03:56:10.989 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:10 smithi169 ceph-mon[39949]: 4.2 scrub ok 2024-03-12T03:56:10.989 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:10 smithi169 ceph-mon[39949]: osdmap e561: 8 total, 8 up, 8 in 2024-03-12T03:56:10.989 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:10 smithi169 ceph-mon[39949]: 6.10 scrub starts 2024-03-12T03:56:10.989 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:10 smithi169 ceph-mon[39949]: 6.10 scrub ok 2024-03-12T03:56:11.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:10 smithi173 ceph-mon[33544]: 4.2 scrub starts 2024-03-12T03:56:11.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:10 smithi173 ceph-mon[33544]: 4.2 scrub ok 2024-03-12T03:56:11.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:10 smithi173 ceph-mon[33544]: osdmap e561: 8 total, 8 up, 8 in 2024-03-12T03:56:11.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:10 smithi173 ceph-mon[33544]: 6.10 scrub starts 2024-03-12T03:56:11.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:10 smithi173 ceph-mon[33544]: 6.10 scrub ok 2024-03-12T03:56:12.135 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:11 smithi169 ceph-mon[29344]: pgmap v1145: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:12.136 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:11 smithi169 ceph-mon[29344]: 3.11 deep-scrub starts 2024-03-12T03:56:12.136 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:11 smithi169 ceph-mon[29344]: 3.11 deep-scrub ok 2024-03-12T03:56:12.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:11 smithi169 ceph-mon[39949]: pgmap v1145: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:12.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:11 smithi169 ceph-mon[39949]: 3.11 deep-scrub starts 2024-03-12T03:56:12.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:11 smithi169 ceph-mon[39949]: 3.11 deep-scrub ok 2024-03-12T03:56:12.136 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:56:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:56:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:56:12.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:11 smithi173 ceph-mon[33544]: pgmap v1145: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:12.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:11 smithi173 ceph-mon[33544]: 3.11 deep-scrub starts 2024-03-12T03:56:12.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:11 smithi173 ceph-mon[33544]: 3.11 deep-scrub ok 2024-03-12T03:56:12.995 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.5 to 1 (10000) 2024-03-12T03:56:13.382 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:12 smithi169 ceph-mon[29344]: 4.a deep-scrub starts 2024-03-12T03:56:13.382 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:12 smithi169 ceph-mon[29344]: 4.a deep-scrub ok 2024-03-12T03:56:13.383 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:12 smithi169 ceph-mon[29344]: 6.b scrub starts 2024-03-12T03:56:13.383 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:12 smithi169 ceph-mon[29344]: 6.b scrub ok 2024-03-12T03:56:13.383 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:12 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4116359973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:56:13.383 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:12 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:56:13.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:12 smithi169 ceph-mon[39949]: 4.a deep-scrub starts 2024-03-12T03:56:13.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:12 smithi169 ceph-mon[39949]: 4.a deep-scrub ok 2024-03-12T03:56:13.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:12 smithi169 ceph-mon[39949]: 6.b scrub starts 2024-03-12T03:56:13.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:12 smithi169 ceph-mon[39949]: 6.b scrub ok 2024-03-12T03:56:13.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:12 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4116359973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:56:13.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:12 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:56:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:12 smithi173 ceph-mon[33544]: 4.a deep-scrub starts 2024-03-12T03:56:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:12 smithi173 ceph-mon[33544]: 4.a deep-scrub ok 2024-03-12T03:56:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:12 smithi173 ceph-mon[33544]: 6.b scrub starts 2024-03-12T03:56:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:12 smithi173 ceph-mon[33544]: 6.b scrub ok 2024-03-12T03:56:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:12 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4116359973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:56:13.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:12 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:56:13.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-12T03:56:13.937 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:13.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:56:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:56:14.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[39949]: 6.2 scrub starts 2024-03-12T03:56:14.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[39949]: 6.2 scrub ok 2024-03-12T03:56:14.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[39949]: pgmap v1147: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:56:14.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-12T03:56:14.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[39949]: osdmap e562: 8 total, 8 up, 8 in 2024-03-12T03:56:14.280 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[39949]: osdmap e563: 8 total, 8 up, 8 in 2024-03-12T03:56:14.280 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:14.280 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:14.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[29344]: 6.2 scrub starts 2024-03-12T03:56:14.280 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[29344]: 6.2 scrub ok 2024-03-12T03:56:14.281 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[29344]: pgmap v1147: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:56:14.281 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-12T03:56:14.281 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[29344]: osdmap e562: 8 total, 8 up, 8 in 2024-03-12T03:56:14.281 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[29344]: osdmap e563: 8 total, 8 up, 8 in 2024-03-12T03:56:14.281 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:14.281 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:13 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:14.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:13 smithi173 ceph-mon[33544]: 6.2 scrub starts 2024-03-12T03:56:14.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:13 smithi173 ceph-mon[33544]: 6.2 scrub ok 2024-03-12T03:56:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:13 smithi173 ceph-mon[33544]: pgmap v1147: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:56:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:13 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-12T03:56:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:13 smithi173 ceph-mon[33544]: osdmap e562: 8 total, 8 up, 8 in 2024-03-12T03:56:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:13 smithi173 ceph-mon[33544]: osdmap e563: 8 total, 8 up, 8 in 2024-03-12T03:56:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:13 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:14.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:13 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:15.331 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[29344]: 4.1b scrub starts 2024-03-12T03:56:15.331 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[29344]: 4.1b scrub ok 2024-03-12T03:56:15.332 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[29344]: 3.14 scrub starts 2024-03-12T03:56:15.332 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[29344]: 3.14 scrub ok 2024-03-12T03:56:15.332 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[29344]: pgmap v1150: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail 2024-03-12T03:56:15.332 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:15.332 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:56:15.332 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:56:15.332 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[39949]: 4.1b scrub starts 2024-03-12T03:56:15.332 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[39949]: 4.1b scrub ok 2024-03-12T03:56:15.332 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[39949]: 3.14 scrub starts 2024-03-12T03:56:15.332 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[39949]: 3.14 scrub ok 2024-03-12T03:56:15.332 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[39949]: pgmap v1150: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail 2024-03-12T03:56:15.333 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:15.333 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:56:15.333 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:14 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:56:15.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:14 smithi173 ceph-mon[33544]: 4.1b scrub starts 2024-03-12T03:56:15.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:14 smithi173 ceph-mon[33544]: 4.1b scrub ok 2024-03-12T03:56:15.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:14 smithi173 ceph-mon[33544]: 3.14 scrub starts 2024-03-12T03:56:15.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:14 smithi173 ceph-mon[33544]: 3.14 scrub ok 2024-03-12T03:56:15.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:14 smithi173 ceph-mon[33544]: pgmap v1150: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail 2024-03-12T03:56:15.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:14 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:15.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:14 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:56:15.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:14 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:56:16.003 INFO:teuthology.orchestra.run.smithi169.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-12T03:56:16.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[39949]: 1.8 scrub starts 2024-03-12T03:56:16.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[39949]: 1.8 scrub ok 2024-03-12T03:56:16.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[39949]: 5.17 scrub starts 2024-03-12T03:56:16.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[39949]: 5.17 scrub ok 2024-03-12T03:56:16.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[39949]: 4.6 scrub starts 2024-03-12T03:56:16.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[39949]: 4.6 scrub ok 2024-03-12T03:56:16.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3742167800' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:56:16.299 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[29344]: 1.8 scrub starts 2024-03-12T03:56:16.299 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[29344]: 1.8 scrub ok 2024-03-12T03:56:16.299 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[29344]: 5.17 scrub starts 2024-03-12T03:56:16.299 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[29344]: 5.17 scrub ok 2024-03-12T03:56:16.299 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[29344]: 4.6 scrub starts 2024-03-12T03:56:16.299 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[29344]: 4.6 scrub ok 2024-03-12T03:56:16.299 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:16 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3742167800' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:56:16.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:16 smithi173 ceph-mon[33544]: 1.8 scrub starts 2024-03-12T03:56:16.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:16 smithi173 ceph-mon[33544]: 1.8 scrub ok 2024-03-12T03:56:16.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:16 smithi173 ceph-mon[33544]: 5.17 scrub starts 2024-03-12T03:56:16.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:16 smithi173 ceph-mon[33544]: 5.17 scrub ok 2024-03-12T03:56:16.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:16 smithi173 ceph-mon[33544]: 4.6 scrub starts 2024-03-12T03:56:16.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:16 smithi173 ceph-mon[33544]: 4.6 scrub ok 2024-03-12T03:56:16.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:16 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3742167800' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-12T03:56:16.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-12T03:56:16.882 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:17.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:17 smithi169 ceph-mon[29344]: 3.5 scrub starts 2024-03-12T03:56:17.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:17 smithi169 ceph-mon[29344]: 3.5 scrub ok 2024-03-12T03:56:17.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:17 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3742167800' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-12T03:56:17.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:17 smithi169 ceph-mon[29344]: osdmap e564: 8 total, 8 up, 8 in 2024-03-12T03:56:17.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:17 smithi169 ceph-mon[29344]: pgmap v1152: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-03-12T03:56:17.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:17 smithi169 ceph-mon[39949]: 3.5 scrub starts 2024-03-12T03:56:17.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:17 smithi169 ceph-mon[39949]: 3.5 scrub ok 2024-03-12T03:56:17.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:17 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3742167800' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-12T03:56:17.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:17 smithi169 ceph-mon[39949]: osdmap e564: 8 total, 8 up, 8 in 2024-03-12T03:56:17.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:17 smithi169 ceph-mon[39949]: pgmap v1152: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-03-12T03:56:17.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:17 smithi173 ceph-mon[33544]: 3.5 scrub starts 2024-03-12T03:56:17.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:17 smithi173 ceph-mon[33544]: 3.5 scrub ok 2024-03-12T03:56:17.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:17 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3742167800' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-12T03:56:17.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:17 smithi173 ceph-mon[33544]: osdmap e564: 8 total, 8 up, 8 in 2024-03-12T03:56:17.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:17 smithi173 ceph-mon[33544]: pgmap v1152: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-03-12T03:56:18.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[29344]: 3.c scrub starts 2024-03-12T03:56:18.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[29344]: 3.c scrub ok 2024-03-12T03:56:18.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:56:18.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:56:18.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[29344]: osdmap e565: 8 total, 8 up, 8 in 2024-03-12T03:56:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[29344]: 4.1d scrub starts 2024-03-12T03:56:18.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[29344]: 4.1d scrub ok 2024-03-12T03:56:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[39949]: 3.c scrub starts 2024-03-12T03:56:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[39949]: 3.c scrub ok 2024-03-12T03:56:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:56:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:56:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[39949]: osdmap e565: 8 total, 8 up, 8 in 2024-03-12T03:56:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[39949]: 4.1d scrub starts 2024-03-12T03:56:18.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:18 smithi169 ceph-mon[39949]: 4.1d scrub ok 2024-03-12T03:56:18.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:18 smithi173 ceph-mon[33544]: 3.c scrub starts 2024-03-12T03:56:18.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:18 smithi173 ceph-mon[33544]: 3.c scrub ok 2024-03-12T03:56:18.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:18 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:56:18.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:18 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:56:18.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:18 smithi173 ceph-mon[33544]: osdmap e565: 8 total, 8 up, 8 in 2024-03-12T03:56:18.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:18 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:18.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:18 smithi173 ceph-mon[33544]: 4.1d scrub starts 2024-03-12T03:56:18.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:18 smithi173 ceph-mon[33544]: 4.1d scrub ok 2024-03-12T03:56:19.026 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.6 to 1 (10000) 2024-03-12T03:56:19.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:19 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:56:19.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:19 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:56:19.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:19 smithi169 ceph-mon[39949]: pgmap v1154: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-12T03:56:19.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:19 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2946186242' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-12T03:56:19.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:19 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:56:19.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:19 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:56:19.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:19 smithi169 ceph-mon[29344]: pgmap v1154: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-12T03:56:19.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:19 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2946186242' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-12T03:56:19.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:19 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:56:19.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:19 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:56:19.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:19 smithi173 ceph-mon[33544]: pgmap v1154: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-12T03:56:19.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:19 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2946186242' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-12T03:56:19.635 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-12T03:56:19.883 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:20.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:20 smithi169 ceph-mon[39949]: 5.18 scrub starts 2024-03-12T03:56:20.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:20 smithi169 ceph-mon[39949]: 5.18 scrub ok 2024-03-12T03:56:20.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:20 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2946186242' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-12T03:56:20.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:20 smithi169 ceph-mon[39949]: osdmap e566: 8 total, 8 up, 8 in 2024-03-12T03:56:20.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:20 smithi169 ceph-mon[39949]: 4.17 scrub starts 2024-03-12T03:56:20.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:20 smithi169 ceph-mon[39949]: 4.17 scrub ok 2024-03-12T03:56:20.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:20 smithi169 ceph-mon[29344]: 5.18 scrub starts 2024-03-12T03:56:20.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:20 smithi169 ceph-mon[29344]: 5.18 scrub ok 2024-03-12T03:56:20.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:20 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2946186242' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-12T03:56:20.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:20 smithi169 ceph-mon[29344]: osdmap e566: 8 total, 8 up, 8 in 2024-03-12T03:56:20.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:20 smithi169 ceph-mon[29344]: 4.17 scrub starts 2024-03-12T03:56:20.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:20 smithi169 ceph-mon[29344]: 4.17 scrub ok 2024-03-12T03:56:20.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:20 smithi173 ceph-mon[33544]: 5.18 scrub starts 2024-03-12T03:56:20.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:20 smithi173 ceph-mon[33544]: 5.18 scrub ok 2024-03-12T03:56:20.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:20 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2946186242' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-12T03:56:20.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:20 smithi173 ceph-mon[33544]: osdmap e566: 8 total, 8 up, 8 in 2024-03-12T03:56:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:20 smithi173 ceph-mon[33544]: 4.17 scrub starts 2024-03-12T03:56:20.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:20 smithi173 ceph-mon[33544]: 4.17 scrub ok 2024-03-12T03:56:21.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:21 smithi169 ceph-mon[39949]: 6.1b deep-scrub starts 2024-03-12T03:56:21.390 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:21 smithi169 ceph-mon[39949]: 6.1b deep-scrub ok 2024-03-12T03:56:21.390 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:21 smithi169 ceph-mon[39949]: pgmap v1156: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:21.390 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:21 smithi169 ceph-mon[29344]: 6.1b deep-scrub starts 2024-03-12T03:56:21.390 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:21 smithi169 ceph-mon[29344]: 6.1b deep-scrub ok 2024-03-12T03:56:21.390 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:21 smithi169 ceph-mon[29344]: pgmap v1156: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:21.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:21 smithi173 ceph-mon[33544]: 6.1b deep-scrub starts 2024-03-12T03:56:21.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:21 smithi173 ceph-mon[33544]: 6.1b deep-scrub ok 2024-03-12T03:56:21.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:21 smithi173 ceph-mon[33544]: pgmap v1156: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:22.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:56:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:56:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:56:23.037 INFO:teuthology.orchestra.run.smithi169.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-12T03:56:23.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:23 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:56:23.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:23 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:56:23.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:23 smithi169 ceph-mon[29344]: osdmap e567: 8 total, 8 up, 8 in 2024-03-12T03:56:23.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:23 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/86745267' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-12T03:56:23.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:23 smithi169 ceph-mon[29344]: pgmap v1158: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:56:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:23 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:56:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:23 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:56:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:23 smithi169 ceph-mon[39949]: osdmap e567: 8 total, 8 up, 8 in 2024-03-12T03:56:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:23 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/86745267' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-12T03:56:23.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:23 smithi169 ceph-mon[39949]: pgmap v1158: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:56:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:23 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:56:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:23 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:56:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:23 smithi173 ceph-mon[33544]: osdmap e567: 8 total, 8 up, 8 in 2024-03-12T03:56:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:23 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/86745267' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-12T03:56:23.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:23 smithi173 ceph-mon[33544]: pgmap v1158: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:56:23.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-12T03:56:23.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:56:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:56:24.087 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:24.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:24 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/86745267' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-12T03:56:24.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:24 smithi169 ceph-mon[29344]: osdmap e568: 8 total, 8 up, 8 in 2024-03-12T03:56:24.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:24 smithi169 ceph-mon[29344]: osdmap e569: 8 total, 8 up, 8 in 2024-03-12T03:56:24.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:24 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/86745267' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-12T03:56:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:24 smithi169 ceph-mon[39949]: osdmap e568: 8 total, 8 up, 8 in 2024-03-12T03:56:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:24 smithi169 ceph-mon[39949]: osdmap e569: 8 total, 8 up, 8 in 2024-03-12T03:56:24.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:24.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:24 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/86745267' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-12T03:56:24.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:24 smithi173 ceph-mon[33544]: osdmap e568: 8 total, 8 up, 8 in 2024-03-12T03:56:24.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:24 smithi173 ceph-mon[33544]: osdmap e569: 8 total, 8 up, 8 in 2024-03-12T03:56:24.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:25.323 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:25 smithi169 ceph-mon[29344]: pgmap v1161: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-03-12T03:56:25.323 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:25 smithi169 ceph-mon[39949]: pgmap v1161: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-03-12T03:56:25.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:25 smithi173 ceph-mon[33544]: pgmap v1161: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-03-12T03:56:26.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:26 smithi169 ceph-mon[29344]: 5.1a scrub starts 2024-03-12T03:56:26.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:26 smithi169 ceph-mon[29344]: 5.1a scrub ok 2024-03-12T03:56:26.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:26 smithi169 ceph-mon[29344]: 1.6 scrub starts 2024-03-12T03:56:26.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:26 smithi169 ceph-mon[29344]: 1.6 scrub ok 2024-03-12T03:56:26.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:26 smithi169 ceph-mon[29344]: 5.0 deep-scrub starts 2024-03-12T03:56:26.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:26 smithi169 ceph-mon[29344]: 5.0 deep-scrub ok 2024-03-12T03:56:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:26 smithi169 ceph-mon[39949]: 5.1a scrub starts 2024-03-12T03:56:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:26 smithi169 ceph-mon[39949]: 5.1a scrub ok 2024-03-12T03:56:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:26 smithi169 ceph-mon[39949]: 1.6 scrub starts 2024-03-12T03:56:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:26 smithi169 ceph-mon[39949]: 1.6 scrub ok 2024-03-12T03:56:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:26 smithi169 ceph-mon[39949]: 5.0 deep-scrub starts 2024-03-12T03:56:26.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:26 smithi169 ceph-mon[39949]: 5.0 deep-scrub ok 2024-03-12T03:56:26.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:26 smithi173 ceph-mon[33544]: 5.1a scrub starts 2024-03-12T03:56:26.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:26 smithi173 ceph-mon[33544]: 5.1a scrub ok 2024-03-12T03:56:26.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:26 smithi173 ceph-mon[33544]: 1.6 scrub starts 2024-03-12T03:56:26.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:26 smithi173 ceph-mon[33544]: 1.6 scrub ok 2024-03-12T03:56:26.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:26 smithi173 ceph-mon[33544]: 5.0 deep-scrub starts 2024-03-12T03:56:26.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:26 smithi173 ceph-mon[33544]: 5.0 deep-scrub ok 2024-03-12T03:56:27.057 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.7 to 1 (10000) 2024-03-12T03:56:27.374 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:27 smithi169 ceph-mon[29344]: 1.7 deep-scrub starts 2024-03-12T03:56:27.374 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:27 smithi169 ceph-mon[29344]: 1.7 deep-scrub ok 2024-03-12T03:56:27.374 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:27 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/468469924' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:56:27.374 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:27 smithi169 ceph-mon[29344]: pgmap v1162: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:27.374 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:27 smithi169 ceph-mon[39949]: 1.7 deep-scrub starts 2024-03-12T03:56:27.374 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:27 smithi169 ceph-mon[39949]: 1.7 deep-scrub ok 2024-03-12T03:56:27.374 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:27 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/468469924' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:56:27.374 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:27 smithi169 ceph-mon[39949]: pgmap v1162: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:27.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:27 smithi173 ceph-mon[33544]: 1.7 deep-scrub starts 2024-03-12T03:56:27.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:27 smithi173 ceph-mon[33544]: 1.7 deep-scrub ok 2024-03-12T03:56:27.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:27 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/468469924' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:56:27.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:27 smithi173 ceph-mon[33544]: pgmap v1162: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:27.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-12T03:56:28.056 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:28.315 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:28 smithi169 ceph-mon[29344]: 5.11 scrub starts 2024-03-12T03:56:28.316 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:28 smithi169 ceph-mon[29344]: 5.11 scrub ok 2024-03-12T03:56:28.316 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:28 smithi169 ceph-mon[29344]: 3.1e scrub starts 2024-03-12T03:56:28.316 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:28 smithi169 ceph-mon[29344]: 3.1e scrub ok 2024-03-12T03:56:28.316 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:28 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/468469924' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-12T03:56:28.316 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:28 smithi169 ceph-mon[29344]: osdmap e570: 8 total, 8 up, 8 in 2024-03-12T03:56:28.316 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:28 smithi169 ceph-mon[39949]: 5.11 scrub starts 2024-03-12T03:56:28.316 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:28 smithi169 ceph-mon[39949]: 5.11 scrub ok 2024-03-12T03:56:28.316 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:28 smithi169 ceph-mon[39949]: 3.1e scrub starts 2024-03-12T03:56:28.316 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:28 smithi169 ceph-mon[39949]: 3.1e scrub ok 2024-03-12T03:56:28.316 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:28 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/468469924' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-12T03:56:28.316 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:28 smithi169 ceph-mon[39949]: osdmap e570: 8 total, 8 up, 8 in 2024-03-12T03:56:28.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:28 smithi173 ceph-mon[33544]: 5.11 scrub starts 2024-03-12T03:56:28.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:28 smithi173 ceph-mon[33544]: 5.11 scrub ok 2024-03-12T03:56:28.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:28 smithi173 ceph-mon[33544]: 3.1e scrub starts 2024-03-12T03:56:28.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:28 smithi173 ceph-mon[33544]: 3.1e scrub ok 2024-03-12T03:56:28.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:28 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/468469924' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-12T03:56:28.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:28 smithi173 ceph-mon[33544]: osdmap e570: 8 total, 8 up, 8 in 2024-03-12T03:56:29.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:29 smithi169 ceph-mon[39949]: 4.5 scrub starts 2024-03-12T03:56:29.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:29 smithi169 ceph-mon[39949]: 4.5 scrub ok 2024-03-12T03:56:29.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:29 smithi169 ceph-mon[39949]: 3.2 scrub starts 2024-03-12T03:56:29.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:29 smithi169 ceph-mon[39949]: 3.2 scrub ok 2024-03-12T03:56:29.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:29 smithi169 ceph-mon[39949]: pgmap v1164: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:56:29.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:29 smithi169 ceph-mon[39949]: osdmap e571: 8 total, 8 up, 8 in 2024-03-12T03:56:29.378 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:29 smithi169 ceph-mon[29344]: 4.5 scrub starts 2024-03-12T03:56:29.378 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:29 smithi169 ceph-mon[29344]: 4.5 scrub ok 2024-03-12T03:56:29.378 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:29 smithi169 ceph-mon[29344]: 3.2 scrub starts 2024-03-12T03:56:29.378 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:29 smithi169 ceph-mon[29344]: 3.2 scrub ok 2024-03-12T03:56:29.378 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:29 smithi169 ceph-mon[29344]: pgmap v1164: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:56:29.378 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:29 smithi169 ceph-mon[29344]: osdmap e571: 8 total, 8 up, 8 in 2024-03-12T03:56:29.387 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:29 smithi173 ceph-mon[33544]: 4.5 scrub starts 2024-03-12T03:56:29.387 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:29 smithi173 ceph-mon[33544]: 4.5 scrub ok 2024-03-12T03:56:29.388 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:29 smithi173 ceph-mon[33544]: 3.2 scrub starts 2024-03-12T03:56:29.388 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:29 smithi173 ceph-mon[33544]: 3.2 scrub ok 2024-03-12T03:56:29.388 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:29 smithi173 ceph-mon[33544]: pgmap v1164: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:56:29.388 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:29 smithi173 ceph-mon[33544]: osdmap e571: 8 total, 8 up, 8 in 2024-03-12T03:56:30.068 INFO:teuthology.orchestra.run.smithi169.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-12T03:56:30.357 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[39949]: 5.e scrub starts 2024-03-12T03:56:30.357 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[39949]: 5.e scrub ok 2024-03-12T03:56:30.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[39949]: 6.14 deep-scrub starts 2024-03-12T03:56:30.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[39949]: 6.14 deep-scrub ok 2024-03-12T03:56:30.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[39949]: 6.8 scrub starts 2024-03-12T03:56:30.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[39949]: 6.8 scrub ok 2024-03-12T03:56:30.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/888675728' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:56:30.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:56:30.358 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[29344]: 5.e scrub starts 2024-03-12T03:56:30.358 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[29344]: 5.e scrub ok 2024-03-12T03:56:30.358 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[29344]: 6.14 deep-scrub starts 2024-03-12T03:56:30.358 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[29344]: 6.14 deep-scrub ok 2024-03-12T03:56:30.358 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[29344]: 6.8 scrub starts 2024-03-12T03:56:30.358 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[29344]: 6.8 scrub ok 2024-03-12T03:56:30.359 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/888675728' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:56:30.359 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:30 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:56:30.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:30 smithi173 ceph-mon[33544]: 5.e scrub starts 2024-03-12T03:56:30.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:30 smithi173 ceph-mon[33544]: 5.e scrub ok 2024-03-12T03:56:30.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:30 smithi173 ceph-mon[33544]: 6.14 deep-scrub starts 2024-03-12T03:56:30.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:30 smithi173 ceph-mon[33544]: 6.14 deep-scrub ok 2024-03-12T03:56:30.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:30 smithi173 ceph-mon[33544]: 6.8 scrub starts 2024-03-12T03:56:30.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:30 smithi173 ceph-mon[33544]: 6.8 scrub ok 2024-03-12T03:56:30.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:30 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/888675728' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:56:30.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:30 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-12T03:56:30.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-12T03:56:31.099 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:31.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:31 smithi169 ceph-mon[39949]: 1.9 scrub starts 2024-03-12T03:56:31.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:31 smithi169 ceph-mon[39949]: 1.9 scrub ok 2024-03-12T03:56:31.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:31 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-12T03:56:31.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:31 smithi169 ceph-mon[39949]: osdmap e572: 8 total, 8 up, 8 in 2024-03-12T03:56:31.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:31 smithi169 ceph-mon[39949]: pgmap v1167: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:56:31.344 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:31 smithi169 ceph-mon[29344]: 1.9 scrub starts 2024-03-12T03:56:31.344 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:31 smithi169 ceph-mon[29344]: 1.9 scrub ok 2024-03-12T03:56:31.344 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:31 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-12T03:56:31.344 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:31 smithi169 ceph-mon[29344]: osdmap e572: 8 total, 8 up, 8 in 2024-03-12T03:56:31.344 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:31 smithi169 ceph-mon[29344]: pgmap v1167: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:56:31.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:31 smithi173 ceph-mon[33544]: 1.9 scrub starts 2024-03-12T03:56:31.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:31 smithi173 ceph-mon[33544]: 1.9 scrub ok 2024-03-12T03:56:31.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:31 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-12T03:56:31.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:31 smithi173 ceph-mon[33544]: osdmap e572: 8 total, 8 up, 8 in 2024-03-12T03:56:31.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:31 smithi173 ceph-mon[33544]: pgmap v1167: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:56:32.173 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:56:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:56:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:56:32.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[39949]: 4.1a scrub starts 2024-03-12T03:56:32.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[39949]: 4.1a scrub ok 2024-03-12T03:56:32.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[39949]: 3.10 scrub starts 2024-03-12T03:56:32.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[39949]: 3.10 scrub ok 2024-03-12T03:56:32.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[39949]: 5.7 scrub starts 2024-03-12T03:56:32.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[39949]: 5.7 scrub ok 2024-03-12T03:56:32.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[39949]: 5.b deep-scrub starts 2024-03-12T03:56:32.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[39949]: 5.b deep-scrub ok 2024-03-12T03:56:32.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[39949]: 6.6 scrub starts 2024-03-12T03:56:32.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[39949]: 6.6 scrub ok 2024-03-12T03:56:32.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[29344]: 4.1a scrub starts 2024-03-12T03:56:32.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[29344]: 4.1a scrub ok 2024-03-12T03:56:32.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[29344]: 3.10 scrub starts 2024-03-12T03:56:32.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[29344]: 3.10 scrub ok 2024-03-12T03:56:32.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[29344]: 5.7 scrub starts 2024-03-12T03:56:32.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[29344]: 5.7 scrub ok 2024-03-12T03:56:32.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[29344]: 5.b deep-scrub starts 2024-03-12T03:56:32.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[29344]: 5.b deep-scrub ok 2024-03-12T03:56:32.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[29344]: 6.6 scrub starts 2024-03-12T03:56:32.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:32 smithi169 ceph-mon[29344]: 6.6 scrub ok 2024-03-12T03:56:32.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:32 smithi173 ceph-mon[33544]: 4.1a scrub starts 2024-03-12T03:56:32.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:32 smithi173 ceph-mon[33544]: 4.1a scrub ok 2024-03-12T03:56:32.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:32 smithi173 ceph-mon[33544]: 3.10 scrub starts 2024-03-12T03:56:32.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:32 smithi173 ceph-mon[33544]: 3.10 scrub ok 2024-03-12T03:56:32.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:32 smithi173 ceph-mon[33544]: 5.7 scrub starts 2024-03-12T03:56:32.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:32 smithi173 ceph-mon[33544]: 5.7 scrub ok 2024-03-12T03:56:32.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:32 smithi173 ceph-mon[33544]: 5.b deep-scrub starts 2024-03-12T03:56:32.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:32 smithi173 ceph-mon[33544]: 5.b deep-scrub ok 2024-03-12T03:56:32.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:32 smithi173 ceph-mon[33544]: 6.6 scrub starts 2024-03-12T03:56:32.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:32 smithi173 ceph-mon[33544]: 6.6 scrub ok 2024-03-12T03:56:33.089 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.3 to 1 (10000) 2024-03-12T03:56:33.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:33 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:33.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:33 smithi169 ceph-mon[29344]: 4.10 scrub starts 2024-03-12T03:56:33.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:33 smithi169 ceph-mon[29344]: 4.10 scrub ok 2024-03-12T03:56:33.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:33 smithi169 ceph-mon[29344]: pgmap v1168: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T03:56:33.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:33 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1431426914' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:56:33.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:33 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:56:33.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:33 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:33.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:33 smithi169 ceph-mon[39949]: 4.10 scrub starts 2024-03-12T03:56:33.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:33 smithi169 ceph-mon[39949]: 4.10 scrub ok 2024-03-12T03:56:33.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:33 smithi169 ceph-mon[39949]: pgmap v1168: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T03:56:33.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:33 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1431426914' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:56:33.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:33 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:56:33.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:33 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:33.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:33 smithi173 ceph-mon[33544]: 4.10 scrub starts 2024-03-12T03:56:33.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:33 smithi173 ceph-mon[33544]: 4.10 scrub ok 2024-03-12T03:56:33.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:33 smithi173 ceph-mon[33544]: pgmap v1168: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T03:56:33.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:33 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1431426914' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:56:33.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:33 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:56:33.965 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-12T03:56:33.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:56:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:56:34.216 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:34.334 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:34 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-12T03:56:34.334 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:34 smithi169 ceph-mon[39949]: osdmap e573: 8 total, 8 up, 8 in 2024-03-12T03:56:34.334 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:34 smithi169 ceph-mon[39949]: 4.3 scrub starts 2024-03-12T03:56:34.334 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:34 smithi169 ceph-mon[39949]: 4.3 scrub ok 2024-03-12T03:56:34.334 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:34.335 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:34 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-12T03:56:34.335 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:34 smithi169 ceph-mon[29344]: osdmap e573: 8 total, 8 up, 8 in 2024-03-12T03:56:34.335 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:34 smithi169 ceph-mon[29344]: 4.3 scrub starts 2024-03-12T03:56:34.335 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:34 smithi169 ceph-mon[29344]: 4.3 scrub ok 2024-03-12T03:56:34.335 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:34.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:34 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-12T03:56:34.489 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:34 smithi173 ceph-mon[33544]: osdmap e573: 8 total, 8 up, 8 in 2024-03-12T03:56:34.489 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:34 smithi173 ceph-mon[33544]: 4.3 scrub starts 2024-03-12T03:56:34.489 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:34 smithi173 ceph-mon[33544]: 4.3 scrub ok 2024-03-12T03:56:34.490 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:35.349 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:35 smithi169 ceph-mon[29344]: 5.4 scrub starts 2024-03-12T03:56:35.350 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:35 smithi169 ceph-mon[29344]: 5.4 scrub ok 2024-03-12T03:56:35.350 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:35 smithi169 ceph-mon[29344]: pgmap v1170: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:35.350 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:35 smithi169 ceph-mon[29344]: 1.1 scrub starts 2024-03-12T03:56:35.350 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:35 smithi169 ceph-mon[39949]: 5.4 scrub starts 2024-03-12T03:56:35.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:35 smithi169 ceph-mon[39949]: 5.4 scrub ok 2024-03-12T03:56:35.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:35 smithi169 ceph-mon[39949]: pgmap v1170: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:35.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:35 smithi169 ceph-mon[39949]: 1.1 scrub starts 2024-03-12T03:56:35.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:35 smithi173 ceph-mon[33544]: 5.4 scrub starts 2024-03-12T03:56:35.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:35 smithi173 ceph-mon[33544]: 5.4 scrub ok 2024-03-12T03:56:35.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:35 smithi173 ceph-mon[33544]: pgmap v1170: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:35.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:35 smithi173 ceph-mon[33544]: 1.1 scrub starts 2024-03-12T03:56:36.101 INFO:teuthology.orchestra.run.smithi169.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-12T03:56:36.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:36 smithi169 ceph-mon[39949]: 1.1 scrub ok 2024-03-12T03:56:36.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:36 smithi169 ceph-mon[39949]: osdmap e574: 8 total, 8 up, 8 in 2024-03-12T03:56:36.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:36 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1171437051' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:56:36.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:36 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:56:36.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:36 smithi169 ceph-mon[29344]: 1.1 scrub ok 2024-03-12T03:56:36.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:36 smithi169 ceph-mon[29344]: osdmap e574: 8 total, 8 up, 8 in 2024-03-12T03:56:36.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:36 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1171437051' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:56:36.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:36 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:56:36.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:36 smithi173 ceph-mon[33544]: 1.1 scrub ok 2024-03-12T03:56:36.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:36 smithi173 ceph-mon[33544]: osdmap e574: 8 total, 8 up, 8 in 2024-03-12T03:56:36.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:36 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1171437051' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:56:36.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:36 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-12T03:56:36.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-12T03:56:37.038 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:37.351 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-12T03:56:37.351 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[29344]: osdmap e575: 8 total, 8 up, 8 in 2024-03-12T03:56:37.351 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[29344]: pgmap v1173: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:37.351 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[29344]: 6.a scrub starts 2024-03-12T03:56:37.351 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[29344]: 6.a scrub ok 2024-03-12T03:56:37.351 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[29344]: 3.17 scrub starts 2024-03-12T03:56:37.352 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[29344]: 3.17 scrub ok 2024-03-12T03:56:37.352 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-12T03:56:37.352 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[39949]: osdmap e575: 8 total, 8 up, 8 in 2024-03-12T03:56:37.352 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[39949]: pgmap v1173: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:37.352 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[39949]: 6.a scrub starts 2024-03-12T03:56:37.352 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[39949]: 6.a scrub ok 2024-03-12T03:56:37.352 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[39949]: 3.17 scrub starts 2024-03-12T03:56:37.352 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:37 smithi169 ceph-mon[39949]: 3.17 scrub ok 2024-03-12T03:56:37.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:37 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-12T03:56:37.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:37 smithi173 ceph-mon[33544]: osdmap e575: 8 total, 8 up, 8 in 2024-03-12T03:56:37.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:37 smithi173 ceph-mon[33544]: pgmap v1173: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:37.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:37 smithi173 ceph-mon[33544]: 6.a scrub starts 2024-03-12T03:56:37.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:37 smithi173 ceph-mon[33544]: 6.a scrub ok 2024-03-12T03:56:37.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:37 smithi173 ceph-mon[33544]: 3.17 scrub starts 2024-03-12T03:56:37.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:37 smithi173 ceph-mon[33544]: 3.17 scrub ok 2024-03-12T03:56:38.330 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:38 smithi169 ceph-mon[29344]: 1.4 scrub starts 2024-03-12T03:56:38.330 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:38 smithi169 ceph-mon[29344]: 1.4 scrub ok 2024-03-12T03:56:38.330 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:38 smithi169 ceph-mon[39949]: 1.4 scrub starts 2024-03-12T03:56:38.330 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:38 smithi169 ceph-mon[39949]: 1.4 scrub ok 2024-03-12T03:56:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:38 smithi173 ceph-mon[33544]: 1.4 scrub starts 2024-03-12T03:56:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:38 smithi173 ceph-mon[33544]: 1.4 scrub ok 2024-03-12T03:56:39.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[29344]: 3.1c scrub starts 2024-03-12T03:56:39.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[29344]: 3.1c scrub ok 2024-03-12T03:56:39.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[29344]: 6.11 scrub starts 2024-03-12T03:56:39.425 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[29344]: 6.11 scrub ok 2024-03-12T03:56:39.426 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[29344]: pgmap v1174: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:39.426 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[29344]: 3.1f scrub starts 2024-03-12T03:56:39.426 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[29344]: 3.1f scrub ok 2024-03-12T03:56:39.426 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2757739592' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:56:39.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[39949]: 3.1c scrub starts 2024-03-12T03:56:39.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[39949]: 3.1c scrub ok 2024-03-12T03:56:39.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[39949]: 6.11 scrub starts 2024-03-12T03:56:39.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[39949]: 6.11 scrub ok 2024-03-12T03:56:39.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[39949]: pgmap v1174: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:39.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[39949]: 3.1f scrub starts 2024-03-12T03:56:39.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[39949]: 3.1f scrub ok 2024-03-12T03:56:39.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:39 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2757739592' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:56:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:39 smithi173 ceph-mon[33544]: 3.1c scrub starts 2024-03-12T03:56:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:39 smithi173 ceph-mon[33544]: 3.1c scrub ok 2024-03-12T03:56:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:39 smithi173 ceph-mon[33544]: 6.11 scrub starts 2024-03-12T03:56:39.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:39 smithi173 ceph-mon[33544]: 6.11 scrub ok 2024-03-12T03:56:39.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:39 smithi173 ceph-mon[33544]: pgmap v1174: 144 pgs: 1 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:39.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:39 smithi173 ceph-mon[33544]: 3.1f scrub starts 2024-03-12T03:56:39.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:39 smithi173 ceph-mon[33544]: 3.1f scrub ok 2024-03-12T03:56:39.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:39 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2757739592' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:56:39.599 INFO:teuthology.orchestra.run.smithi169.stderr:reweighted osd.0 to 1 (10000) 2024-03-12T03:56:40.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:40 smithi169 ceph-mon[29344]: 3.f scrub starts 2024-03-12T03:56:40.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:40 smithi169 ceph-mon[29344]: 3.f scrub ok 2024-03-12T03:56:40.173 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:40 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2757739592' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-12T03:56:40.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:40 smithi169 ceph-mon[29344]: osdmap e576: 8 total, 8 up, 8 in 2024-03-12T03:56:40.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:40 smithi169 ceph-mon[39949]: 3.f scrub starts 2024-03-12T03:56:40.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:40 smithi169 ceph-mon[39949]: 3.f scrub ok 2024-03-12T03:56:40.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:40 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2757739592' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-12T03:56:40.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:40 smithi169 ceph-mon[39949]: osdmap e576: 8 total, 8 up, 8 in 2024-03-12T03:56:40.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-12T03:56:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:40 smithi173 ceph-mon[33544]: 3.f scrub starts 2024-03-12T03:56:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:40 smithi173 ceph-mon[33544]: 3.f scrub ok 2024-03-12T03:56:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:40 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2757739592' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-12T03:56:40.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:40 smithi173 ceph-mon[33544]: osdmap e576: 8 total, 8 up, 8 in 2024-03-12T03:56:40.572 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:41.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: 5.2 scrub starts 2024-03-12T03:56:41.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: 5.2 scrub ok 2024-03-12T03:56:41.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: 5.15 deep-scrub starts 2024-03-12T03:56:41.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: 5.15 deep-scrub ok 2024-03-12T03:56:41.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: 3.8 scrub starts 2024-03-12T03:56:41.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: 3.8 scrub ok 2024-03-12T03:56:41.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: 1.2 scrub starts 2024-03-12T03:56:41.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: 1.2 scrub ok 2024-03-12T03:56:41.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: 3.1b deep-scrub starts 2024-03-12T03:56:41.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: 3.1b deep-scrub ok 2024-03-12T03:56:41.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: pgmap v1176: 144 pgs: 144 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:41.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-12T03:56:41.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-12T03:56:41.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: 5.2 scrub starts 2024-03-12T03:56:41.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: 5.2 scrub ok 2024-03-12T03:56:41.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: 5.15 deep-scrub starts 2024-03-12T03:56:41.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: 5.15 deep-scrub ok 2024-03-12T03:56:41.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: 3.8 scrub starts 2024-03-12T03:56:41.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: 3.8 scrub ok 2024-03-12T03:56:41.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: 1.2 scrub starts 2024-03-12T03:56:41.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: 1.2 scrub ok 2024-03-12T03:56:41.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: 3.1b deep-scrub starts 2024-03-12T03:56:41.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: 3.1b deep-scrub ok 2024-03-12T03:56:41.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: pgmap v1176: 144 pgs: 144 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:41.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-12T03:56:41.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-12T03:56:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: 5.2 scrub starts 2024-03-12T03:56:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: 5.2 scrub ok 2024-03-12T03:56:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: 5.15 deep-scrub starts 2024-03-12T03:56:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: 5.15 deep-scrub ok 2024-03-12T03:56:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: 3.8 scrub starts 2024-03-12T03:56:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: 3.8 scrub ok 2024-03-12T03:56:41.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: 1.2 scrub starts 2024-03-12T03:56:41.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: 1.2 scrub ok 2024-03-12T03:56:41.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: 3.1b deep-scrub starts 2024-03-12T03:56:41.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: 3.1b deep-scrub ok 2024-03-12T03:56:41.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: pgmap v1176: 144 pgs: 144 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:41.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-12T03:56:41.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:41 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-12T03:56:42.020 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:56:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:56:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:56:42.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[39949]: 5.13 scrub starts 2024-03-12T03:56:42.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[39949]: 5.13 scrub ok 2024-03-12T03:56:42.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[39949]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-12T03:56:42.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[39949]: Cluster is now healthy 2024-03-12T03:56:42.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-12T03:56:42.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[39949]: osdmap e577: 8 total, 8 up, 8 in 2024-03-12T03:56:42.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[39949]: 5.8 scrub starts 2024-03-12T03:56:42.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[39949]: 5.8 scrub ok 2024-03-12T03:56:42.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[29344]: 5.13 scrub starts 2024-03-12T03:56:42.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[29344]: 5.13 scrub ok 2024-03-12T03:56:42.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[29344]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-12T03:56:42.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[29344]: Cluster is now healthy 2024-03-12T03:56:42.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-12T03:56:42.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[29344]: osdmap e577: 8 total, 8 up, 8 in 2024-03-12T03:56:42.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[29344]: 5.8 scrub starts 2024-03-12T03:56:42.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:42 smithi169 ceph-mon[29344]: 5.8 scrub ok 2024-03-12T03:56:42.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:42 smithi173 ceph-mon[33544]: 5.13 scrub starts 2024-03-12T03:56:42.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:42 smithi173 ceph-mon[33544]: 5.13 scrub ok 2024-03-12T03:56:42.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:42 smithi173 ceph-mon[33544]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-12T03:56:42.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:42 smithi173 ceph-mon[33544]: Cluster is now healthy 2024-03-12T03:56:42.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:42 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-12T03:56:42.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:42 smithi173 ceph-mon[33544]: osdmap e577: 8 total, 8 up, 8 in 2024-03-12T03:56:42.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:42 smithi173 ceph-mon[33544]: 5.8 scrub starts 2024-03-12T03:56:42.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:42 smithi173 ceph-mon[33544]: 5.8 scrub ok 2024-03-12T03:56:43.136 INFO:teuthology.orchestra.run.smithi169.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-12T03:56:43.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[29344]: 1.5 scrub starts 2024-03-12T03:56:43.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[29344]: 1.5 scrub ok 2024-03-12T03:56:43.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[29344]: osdmap e578: 8 total, 8 up, 8 in 2024-03-12T03:56:43.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[29344]: pgmap v1179: 144 pgs: 144 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:56:43.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1752170272' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:56:43.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:56:43.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[29344]: 5.a scrub starts 2024-03-12T03:56:43.424 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[29344]: 5.a scrub ok 2024-03-12T03:56:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[39949]: 1.5 scrub starts 2024-03-12T03:56:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[39949]: 1.5 scrub ok 2024-03-12T03:56:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[39949]: osdmap e578: 8 total, 8 up, 8 in 2024-03-12T03:56:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[39949]: pgmap v1179: 144 pgs: 144 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:56:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1752170272' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:56:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:56:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[39949]: 5.a scrub starts 2024-03-12T03:56:43.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:43 smithi169 ceph-mon[39949]: 5.a scrub ok 2024-03-12T03:56:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:43 smithi173 ceph-mon[33544]: 1.5 scrub starts 2024-03-12T03:56:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:43 smithi173 ceph-mon[33544]: 1.5 scrub ok 2024-03-12T03:56:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:43 smithi173 ceph-mon[33544]: osdmap e578: 8 total, 8 up, 8 in 2024-03-12T03:56:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:43 smithi173 ceph-mon[33544]: pgmap v1179: 144 pgs: 144 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:56:43.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:43 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1752170272' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:56:43.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:43 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-12T03:56:43.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:43 smithi173 ceph-mon[33544]: 5.a scrub starts 2024-03-12T03:56:43.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:43 smithi173 ceph-mon[33544]: 5.a scrub ok 2024-03-12T03:56:43.825 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-12T03:56:43.825 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-12T03:56:43.825 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-12T03:56:43.826 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-12T03:56:43.826 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-12T03:56:43.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:56:43.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:56:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:56:44.072 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:44.413 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:44 smithi169 ceph-mon[29344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-12T03:56:44.414 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:44 smithi169 ceph-mon[29344]: osdmap e579: 8 total, 8 up, 8 in 2024-03-12T03:56:44.414 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:44 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:44.414 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:44 smithi169 ceph-mon[39949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-12T03:56:44.414 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:44 smithi169 ceph-mon[39949]: osdmap e579: 8 total, 8 up, 8 in 2024-03-12T03:56:44.414 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:44 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:44.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:44 smithi173 ceph-mon[33544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-12T03:56:44.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:44 smithi173 ceph-mon[33544]: osdmap e579: 8 total, 8 up, 8 in 2024-03-12T03:56:44.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:44 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:45.411 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:45 smithi169 ceph-mon[29344]: osdmap e580: 8 total, 8 up, 8 in 2024-03-12T03:56:45.411 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:45 smithi169 ceph-mon[29344]: 2.0 scrub starts 2024-03-12T03:56:45.412 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:45 smithi169 ceph-mon[29344]: 2.0 scrub ok 2024-03-12T03:56:45.412 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:45 smithi169 ceph-mon[29344]: pgmap v1182: 143 pgs: 1 peering, 142 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:45.412 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:45 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-12T03:56:45.412 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:45 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-12T03:56:45.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:45 smithi169 ceph-mon[39949]: osdmap e580: 8 total, 8 up, 8 in 2024-03-12T03:56:45.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:45 smithi169 ceph-mon[39949]: 2.0 scrub starts 2024-03-12T03:56:45.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:45 smithi169 ceph-mon[39949]: 2.0 scrub ok 2024-03-12T03:56:45.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:45 smithi169 ceph-mon[39949]: pgmap v1182: 143 pgs: 1 peering, 142 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:45.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:45 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-12T03:56:45.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:45 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-12T03:56:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:45 smithi173 ceph-mon[33544]: osdmap e580: 8 total, 8 up, 8 in 2024-03-12T03:56:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:45 smithi173 ceph-mon[33544]: 2.0 scrub starts 2024-03-12T03:56:45.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:45 smithi173 ceph-mon[33544]: 2.0 scrub ok 2024-03-12T03:56:45.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:45 smithi173 ceph-mon[33544]: pgmap v1182: 143 pgs: 1 peering, 142 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:45.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:45 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-12T03:56:45.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:45 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-12T03:56:45.995 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:56:45.995 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":581,"fsid":"677d624a-e020-11ee-95c8-87774f69a715","created":"2024-03-12T03:27:53.319986+0000","modified":"2024-03-12T03:56:45.133877+0000","last_up_change":"2024-03-12T03:55:20.666629+0000","last_in_change":"2024-03-12T03:55:32.740461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":21,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:31:04.863493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":12,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.c","ready_epoch":579,"last_epoch_started":578,"last_epoch_clean":578,"source_version":"83'128","target_version":"83'128"},"last_change":"581","last_force_op_resend":"0","last_force_op_resend_prenautilus":"581","last_force_op_resend_preluminous":"581","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-12T03:33:18.983144+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-12T03:33:19.343301+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-12T03:33:21.864621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-12T03:33:23.416265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-12T03:33:25.476511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c44466b0-3112-44dc-b5f3-d6494b958761","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":383,"up_from":521,"up_thru":550,"down_at":384,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6801","nonce":2113559059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":2113559059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":2113559059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6803","nonce":2113559059}]},"public_addr":"172.21.15.169:6801/2113559059","cluster_addr":"172.21.15.169:6802/2113559059","heartbeat_back_addr":"172.21.15.169:6804/2113559059","heartbeat_front_addr":"172.21.15.169:6803/2113559059","state":["exists","up"]},{"osd":1,"uuid":"184c5c97-a992-4be9-bcd3-c08cb58a0e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":530,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":130042595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":130042595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":130042595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6807","nonce":130042595}]},"public_addr":"172.21.15.169:6805/130042595","cluster_addr":"172.21.15.169:6806/130042595","heartbeat_back_addr":"172.21.15.169:6808/130042595","heartbeat_front_addr":"172.21.15.169:6807/130042595","state":["exists","up"]},{"osd":2,"uuid":"beb15e06-1ca4-46a7-b640-efd1991367e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":2751816702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":2751816702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":2751816702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6811","nonce":2751816702}]},"public_addr":"172.21.15.169:6809/2751816702","cluster_addr":"172.21.15.169:6810/2751816702","heartbeat_back_addr":"172.21.15.169:6812/2751816702","heartbeat_front_addr":"172.21.15.169:6811/2751816702","state":["exists","up"]},{"osd":3,"uuid":"a00b4c50-e064-4c9e-85dc-b0a75ea5bf37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":222,"last_clean_end":264,"up_from":330,"up_thru":579,"down_at":265,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6813","nonce":3056769071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3056769071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3056769071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6815","nonce":3056769071}]},"public_addr":"172.21.15.169:6813/3056769071","cluster_addr":"172.21.15.169:6814/3056769071","heartbeat_back_addr":"172.21.15.169:6816/3056769071","heartbeat_front_addr":"172.21.15.169:6815/3056769071","state":["exists","up"]},{"osd":4,"uuid":"3a6121ef-6309-41b7-a964-6772c010b3a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":560,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1397212305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":1397212305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":1397212305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1397212305}]},"public_addr":"172.21.15.173:6800/1397212305","cluster_addr":"172.21.15.173:6801/1397212305","heartbeat_back_addr":"172.21.15.173:6803/1397212305","heartbeat_front_addr":"172.21.15.173:6802/1397212305","state":["exists","up"]},{"osd":5,"uuid":"3c2f6dd9-9784-4a07-8710-55ac3de82964","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":538,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3525888417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":3525888417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":3525888417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3525888417}]},"public_addr":"172.21.15.173:6804/3525888417","cluster_addr":"172.21.15.173:6805/3525888417","heartbeat_back_addr":"172.21.15.173:6807/3525888417","heartbeat_front_addr":"172.21.15.173:6806/3525888417","state":["exists","up"]},{"osd":6,"uuid":"d7759df7-1978-4d70-94a2-721540804f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":568,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2147192629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":2147192629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":2147192629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2147192629}]},"public_addr":"172.21.15.173:6808/2147192629","cluster_addr":"172.21.15.173:6809/2147192629","heartbeat_back_addr":"172.21.15.173:6811/2147192629","heartbeat_front_addr":"172.21.15.173:6810/2147192629","state":["exists","up"]},{"osd":7,"uuid":"8843039e-2b17-4230-9cca-5788a2294d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":542,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":4014976876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":4014976876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6815","nonce":4014976876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":4014976876}]},"public_addr":"172.21.15.173:6812/4014976876","cluster_addr":"172.21.15.173:6813/4014976876","heartbeat_back_addr":"172.21.15.173:6815/4014976876","heartbeat_front_addr":"172.21.15.173:6814/4014976876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:56:21.890212+0000","dead_epoch":383},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:55:34.644873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:55:55.614142+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:55:57.498801+0000","dead_epoch":264},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:56:16.985498+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:56:18.300221+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:55:36.403557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:55:45.371244+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.8","osds":[4,5,0]},{"pgid":"3.14","osds":[3,6,1]}],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/4041747161":"2024-03-13T03:33:59.734991+0000","172.21.15.169:6800/2779395456":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/555866474":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/342292643":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/197210445":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/1430250501":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/3918553041":"2024-03-13T03:28:48.518732+0000","172.21.15.169:0/1020514981":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/284191239":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/1212459137":"2024-03-13T03:28:48.518732+0000","172.21.15.169:6800/204892396":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/2751007072":"2024-03-13T03:28:48.518732+0000","172.21.15.169:0/2349376510":"2024-03-13T03:28:21.868923+0000","172.21.15.169:6800/108891420":"2024-03-13T03:28:48.518732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:56:46.369 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-12T03:56:46.369 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[29344]: osdmap e581: 8 total, 8 up, 8 in 2024-03-12T03:56:46.369 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[29344]: 5.c deep-scrub starts 2024-03-12T03:56:46.369 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[29344]: 5.c deep-scrub ok 2024-03-12T03:56:46.369 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[29344]: 4.b scrub starts 2024-03-12T03:56:46.369 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[29344]: 4.b scrub ok 2024-03-12T03:56:46.369 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2248458077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:56:46.369 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-12T03:56:46.369 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[39949]: osdmap e581: 8 total, 8 up, 8 in 2024-03-12T03:56:46.370 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[39949]: 5.c deep-scrub starts 2024-03-12T03:56:46.370 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[39949]: 5.c deep-scrub ok 2024-03-12T03:56:46.370 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[39949]: 4.b scrub starts 2024-03-12T03:56:46.370 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[39949]: 4.b scrub ok 2024-03-12T03:56:46.370 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:46 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2248458077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:56:46.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:46 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-12T03:56:46.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:46 smithi173 ceph-mon[33544]: osdmap e581: 8 total, 8 up, 8 in 2024-03-12T03:56:46.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:46 smithi173 ceph-mon[33544]: 5.c deep-scrub starts 2024-03-12T03:56:46.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:46 smithi173 ceph-mon[33544]: 5.c deep-scrub ok 2024-03-12T03:56:46.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:46 smithi173 ceph-mon[33544]: 4.b scrub starts 2024-03-12T03:56:46.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:46 smithi173 ceph-mon[33544]: 4.b scrub ok 2024-03-12T03:56:46.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:46 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2248458077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:56:46.783 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-12T03:56:46.784 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:56:47.033 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:47.415 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[29344]: 4.19 scrub starts 2024-03-12T03:56:47.416 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[29344]: 4.19 scrub ok 2024-03-12T03:56:47.416 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[29344]: osdmap e582: 8 total, 8 up, 8 in 2024-03-12T03:56:47.416 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[29344]: 6.9 scrub starts 2024-03-12T03:56:47.416 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[29344]: pgmap v1185: 143 pgs: 1 peering, 142 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-03-12T03:56:47.416 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[29344]: 6.9 scrub ok 2024-03-12T03:56:47.416 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[29344]: 2.1 scrub starts 2024-03-12T03:56:47.416 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[29344]: 2.1 scrub ok 2024-03-12T03:56:47.416 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:56:47.416 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:47.416 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[29344]: osdmap e583: 8 total, 8 up, 8 in 2024-03-12T03:56:47.416 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[39949]: 4.19 scrub starts 2024-03-12T03:56:47.417 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[39949]: 4.19 scrub ok 2024-03-12T03:56:47.417 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[39949]: osdmap e582: 8 total, 8 up, 8 in 2024-03-12T03:56:47.417 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[39949]: 6.9 scrub starts 2024-03-12T03:56:47.417 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[39949]: pgmap v1185: 143 pgs: 1 peering, 142 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-03-12T03:56:47.417 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[39949]: 6.9 scrub ok 2024-03-12T03:56:47.417 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[39949]: 2.1 scrub starts 2024-03-12T03:56:47.417 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[39949]: 2.1 scrub ok 2024-03-12T03:56:47.417 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:56:47.417 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:47.417 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:47 smithi169 ceph-mon[39949]: osdmap e583: 8 total, 8 up, 8 in 2024-03-12T03:56:47.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:47 smithi173 ceph-mon[33544]: 4.19 scrub starts 2024-03-12T03:56:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:47 smithi173 ceph-mon[33544]: 4.19 scrub ok 2024-03-12T03:56:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:47 smithi173 ceph-mon[33544]: osdmap e582: 8 total, 8 up, 8 in 2024-03-12T03:56:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:47 smithi173 ceph-mon[33544]: 6.9 scrub starts 2024-03-12T03:56:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:47 smithi173 ceph-mon[33544]: pgmap v1185: 143 pgs: 1 peering, 142 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-03-12T03:56:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:47 smithi173 ceph-mon[33544]: 6.9 scrub ok 2024-03-12T03:56:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:47 smithi173 ceph-mon[33544]: 2.1 scrub starts 2024-03-12T03:56:47.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:47 smithi173 ceph-mon[33544]: 2.1 scrub ok 2024-03-12T03:56:47.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:47 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:56:47.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:47 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:47.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:47 smithi173 ceph-mon[33544]: osdmap e583: 8 total, 8 up, 8 in 2024-03-12T03:56:48.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:48 smithi169 ceph-mon[29344]: 6.13 scrub starts 2024-03-12T03:56:48.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:48 smithi169 ceph-mon[29344]: 6.13 scrub ok 2024-03-12T03:56:48.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:48 smithi169 ceph-mon[29344]: 5.1d scrub starts 2024-03-12T03:56:48.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:48 smithi169 ceph-mon[29344]: 5.1d scrub ok 2024-03-12T03:56:48.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:48 smithi169 ceph-mon[29344]: osdmap e584: 8 total, 8 up, 8 in 2024-03-12T03:56:48.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:48 smithi169 ceph-mon[39949]: 6.13 scrub starts 2024-03-12T03:56:48.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:48 smithi169 ceph-mon[39949]: 6.13 scrub ok 2024-03-12T03:56:48.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:48 smithi169 ceph-mon[39949]: 5.1d scrub starts 2024-03-12T03:56:48.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:48 smithi169 ceph-mon[39949]: 5.1d scrub ok 2024-03-12T03:56:48.424 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:48 smithi169 ceph-mon[39949]: osdmap e584: 8 total, 8 up, 8 in 2024-03-12T03:56:48.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:48 smithi173 ceph-mon[33544]: 6.13 scrub starts 2024-03-12T03:56:48.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:48 smithi173 ceph-mon[33544]: 6.13 scrub ok 2024-03-12T03:56:48.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:48 smithi173 ceph-mon[33544]: 5.1d scrub starts 2024-03-12T03:56:48.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:48 smithi173 ceph-mon[33544]: 5.1d scrub ok 2024-03-12T03:56:48.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:48 smithi173 ceph-mon[33544]: osdmap e584: 8 total, 8 up, 8 in 2024-03-12T03:56:49.059 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:56:49.059 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":585,"fsid":"677d624a-e020-11ee-95c8-87774f69a715","created":"2024-03-12T03:27:53.319986+0000","modified":"2024-03-12T03:56:48.586041+0000","last_up_change":"2024-03-12T03:55:20.666629+0000","last_in_change":"2024-03-12T03:55:32.740461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":21,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:31:04.863493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":583,"last_epoch_started":582,"last_epoch_clean":582,"source_version":"83'128","target_version":"83'128"},"last_change":"583","last_force_op_resend":"0","last_force_op_resend_prenautilus":"583","last_force_op_resend_preluminous":"581","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-12T03:33:18.983144+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-12T03:33:19.343301+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-12T03:33:21.864621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-12T03:33:23.416265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-12T03:33:25.476511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c44466b0-3112-44dc-b5f3-d6494b958761","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":383,"up_from":521,"up_thru":550,"down_at":384,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6801","nonce":2113559059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":2113559059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":2113559059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6803","nonce":2113559059}]},"public_addr":"172.21.15.169:6801/2113559059","cluster_addr":"172.21.15.169:6802/2113559059","heartbeat_back_addr":"172.21.15.169:6804/2113559059","heartbeat_front_addr":"172.21.15.169:6803/2113559059","state":["exists","up"]},{"osd":1,"uuid":"184c5c97-a992-4be9-bcd3-c08cb58a0e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":583,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":130042595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":130042595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":130042595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6807","nonce":130042595}]},"public_addr":"172.21.15.169:6805/130042595","cluster_addr":"172.21.15.169:6806/130042595","heartbeat_back_addr":"172.21.15.169:6808/130042595","heartbeat_front_addr":"172.21.15.169:6807/130042595","state":["exists","up"]},{"osd":2,"uuid":"beb15e06-1ca4-46a7-b640-efd1991367e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":2751816702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":2751816702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":2751816702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6811","nonce":2751816702}]},"public_addr":"172.21.15.169:6809/2751816702","cluster_addr":"172.21.15.169:6810/2751816702","heartbeat_back_addr":"172.21.15.169:6812/2751816702","heartbeat_front_addr":"172.21.15.169:6811/2751816702","state":["exists","up"]},{"osd":3,"uuid":"a00b4c50-e064-4c9e-85dc-b0a75ea5bf37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":222,"last_clean_end":264,"up_from":330,"up_thru":579,"down_at":265,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6813","nonce":3056769071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3056769071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3056769071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6815","nonce":3056769071}]},"public_addr":"172.21.15.169:6813/3056769071","cluster_addr":"172.21.15.169:6814/3056769071","heartbeat_back_addr":"172.21.15.169:6816/3056769071","heartbeat_front_addr":"172.21.15.169:6815/3056769071","state":["exists","up"]},{"osd":4,"uuid":"3a6121ef-6309-41b7-a964-6772c010b3a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":560,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1397212305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":1397212305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":1397212305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1397212305}]},"public_addr":"172.21.15.173:6800/1397212305","cluster_addr":"172.21.15.173:6801/1397212305","heartbeat_back_addr":"172.21.15.173:6803/1397212305","heartbeat_front_addr":"172.21.15.173:6802/1397212305","state":["exists","up"]},{"osd":5,"uuid":"3c2f6dd9-9784-4a07-8710-55ac3de82964","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":538,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3525888417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":3525888417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":3525888417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3525888417}]},"public_addr":"172.21.15.173:6804/3525888417","cluster_addr":"172.21.15.173:6805/3525888417","heartbeat_back_addr":"172.21.15.173:6807/3525888417","heartbeat_front_addr":"172.21.15.173:6806/3525888417","state":["exists","up"]},{"osd":6,"uuid":"d7759df7-1978-4d70-94a2-721540804f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":568,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2147192629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":2147192629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":2147192629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2147192629}]},"public_addr":"172.21.15.173:6808/2147192629","cluster_addr":"172.21.15.173:6809/2147192629","heartbeat_back_addr":"172.21.15.173:6811/2147192629","heartbeat_front_addr":"172.21.15.173:6810/2147192629","state":["exists","up"]},{"osd":7,"uuid":"8843039e-2b17-4230-9cca-5788a2294d8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":542,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":4014976876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":4014976876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6815","nonce":4014976876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":4014976876}]},"public_addr":"172.21.15.173:6812/4014976876","cluster_addr":"172.21.15.173:6813/4014976876","heartbeat_back_addr":"172.21.15.173:6815/4014976876","heartbeat_front_addr":"172.21.15.173:6814/4014976876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:56:21.890212+0000","dead_epoch":383},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:55:34.644873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:55:55.614142+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:55:57.498801+0000","dead_epoch":264},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:56:16.985498+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:56:18.300221+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:55:36.403557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:55:45.371244+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.8","osds":[4,5,0]},{"pgid":"3.14","osds":[3,6,1]}],"pg_upmap_items":[{"pgid":"4.14","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/4041747161":"2024-03-13T03:33:59.734991+0000","172.21.15.169:6800/2779395456":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/555866474":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/342292643":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/197210445":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/1430250501":"2024-03-13T03:33:59.734991+0000","172.21.15.169:0/3918553041":"2024-03-13T03:28:48.518732+0000","172.21.15.169:0/1020514981":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/284191239":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/1212459137":"2024-03-13T03:28:48.518732+0000","172.21.15.169:6800/204892396":"2024-03-13T03:28:21.868923+0000","172.21.15.169:0/2751007072":"2024-03-13T03:28:48.518732+0000","172.21.15.169:0/2349376510":"2024-03-13T03:28:21.868923+0000","172.21.15.169:6800/108891420":"2024-03-13T03:28:48.518732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:56:49.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:49 smithi169 ceph-mon[29344]: 3.13 scrub starts 2024-03-12T03:56:49.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:49 smithi169 ceph-mon[29344]: 3.13 scrub ok 2024-03-12T03:56:49.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:49 smithi169 ceph-mon[29344]: pgmap v1188: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:49.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:49 smithi169 ceph-mon[29344]: osdmap e585: 8 total, 8 up, 8 in 2024-03-12T03:56:49.863 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:49 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/3247066859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:56:49.863 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:49 smithi169 ceph-mon[39949]: 3.13 scrub starts 2024-03-12T03:56:49.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:49 smithi169 ceph-mon[39949]: 3.13 scrub ok 2024-03-12T03:56:49.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:49 smithi169 ceph-mon[39949]: pgmap v1188: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:49.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:49 smithi169 ceph-mon[39949]: osdmap e585: 8 total, 8 up, 8 in 2024-03-12T03:56:49.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:49 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/3247066859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:56:49.892 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-12T03:56:49.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-12T03:56:49.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-12T03:56:49.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-12T03:56:49.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-12T03:56:49.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-12T03:56:49.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-12T03:56:49.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-12T03:56:49.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:49 smithi173 ceph-mon[33544]: 3.13 scrub starts 2024-03-12T03:56:49.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:49 smithi173 ceph-mon[33544]: 3.13 scrub ok 2024-03-12T03:56:49.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:49 smithi173 ceph-mon[33544]: pgmap v1188: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:49.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:49 smithi173 ceph-mon[33544]: osdmap e585: 8 total, 8 up, 8 in 2024-03-12T03:56:49.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:49 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/3247066859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:56:50.416 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:50.417 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:50.417 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:50.418 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:50.419 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:50.420 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:50.428 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:50.461 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:56:50.852 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:50 smithi169 ceph-mon[39949]: 6.1d scrub starts 2024-03-12T03:56:50.852 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:50 smithi169 ceph-mon[39949]: 6.1d scrub ok 2024-03-12T03:56:50.852 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:50 smithi169 ceph-mon[29344]: 6.1d scrub starts 2024-03-12T03:56:50.852 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:50 smithi169 ceph-mon[29344]: 6.1d scrub ok 2024-03-12T03:56:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:50 smithi173 ceph-mon[33544]: 6.1d scrub starts 2024-03-12T03:56:50.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:50 smithi173 ceph-mon[33544]: 6.1d scrub ok 2024-03-12T03:56:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[39949]: 3.d scrub starts 2024-03-12T03:56:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[39949]: 3.d scrub ok 2024-03-12T03:56:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[39949]: pgmap v1190: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[39949]: 5.3 scrub starts 2024-03-12T03:56:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[39949]: 5.3 scrub ok 2024-03-12T03:56:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[39949]: 3.1a scrub starts 2024-03-12T03:56:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[39949]: 3.1a scrub ok 2024-03-12T03:56:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[39949]: 3.7 scrub starts 2024-03-12T03:56:51.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[39949]: 3.7 scrub ok 2024-03-12T03:56:51.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[29344]: 3.d scrub starts 2024-03-12T03:56:51.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[29344]: 3.d scrub ok 2024-03-12T03:56:51.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[29344]: pgmap v1190: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:51.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[29344]: 5.3 scrub starts 2024-03-12T03:56:51.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[29344]: 5.3 scrub ok 2024-03-12T03:56:51.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[29344]: 3.1a scrub starts 2024-03-12T03:56:51.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[29344]: 3.1a scrub ok 2024-03-12T03:56:51.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[29344]: 3.7 scrub starts 2024-03-12T03:56:51.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-mon[29344]: 3.7 scrub ok 2024-03-12T03:56:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:51 smithi173 ceph-mon[33544]: 3.d scrub starts 2024-03-12T03:56:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:51 smithi173 ceph-mon[33544]: 3.d scrub ok 2024-03-12T03:56:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:51 smithi173 ceph-mon[33544]: pgmap v1190: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:51 smithi173 ceph-mon[33544]: 5.3 scrub starts 2024-03-12T03:56:51.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:51 smithi173 ceph-mon[33544]: 5.3 scrub ok 2024-03-12T03:56:51.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:51 smithi173 ceph-mon[33544]: 3.1a scrub starts 2024-03-12T03:56:51.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:51 smithi173 ceph-mon[33544]: 3.1a scrub ok 2024-03-12T03:56:51.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:51 smithi173 ceph-mon[33544]: 3.7 scrub starts 2024-03-12T03:56:51.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:51 smithi173 ceph-mon[33544]: 3.7 scrub ok 2024-03-12T03:56:52.134 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:56:51 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:56:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:56:53.818 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[29344]: pgmap v1191: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:56:53.818 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[29344]: 4.12 scrub starts 2024-03-12T03:56:53.818 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[29344]: 4.12 scrub ok 2024-03-12T03:56:53.818 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[29344]: 3.16 deep-scrub starts 2024-03-12T03:56:53.818 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[29344]: 3.16 deep-scrub ok 2024-03-12T03:56:53.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:56:53.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:56:53.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[29344]: 4.0 scrub starts 2024-03-12T03:56:53.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[29344]: 4.0 scrub ok 2024-03-12T03:56:53.819 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[39949]: pgmap v1191: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:56:53.819 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[39949]: 4.12 scrub starts 2024-03-12T03:56:53.819 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[39949]: 4.12 scrub ok 2024-03-12T03:56:53.819 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[39949]: 3.16 deep-scrub starts 2024-03-12T03:56:53.819 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[39949]: 3.16 deep-scrub ok 2024-03-12T03:56:53.819 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:56:53.819 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:56:53.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[39949]: 4.0 scrub starts 2024-03-12T03:56:53.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:53 smithi169 ceph-mon[39949]: 4.0 scrub ok 2024-03-12T03:56:53.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:56:53 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:56:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:53 smithi173 ceph-mon[33544]: pgmap v1191: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-12T03:56:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:53 smithi173 ceph-mon[33544]: 4.12 scrub starts 2024-03-12T03:56:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:53 smithi173 ceph-mon[33544]: 4.12 scrub ok 2024-03-12T03:56:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:53 smithi173 ceph-mon[33544]: 3.16 deep-scrub starts 2024-03-12T03:56:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:53 smithi173 ceph-mon[33544]: 3.16 deep-scrub ok 2024-03-12T03:56:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:53 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:56:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:53 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:56:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:53 smithi173 ceph-mon[33544]: 4.0 scrub starts 2024-03-12T03:56:53.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:53 smithi173 ceph-mon[33544]: 4.0 scrub ok 2024-03-12T03:56:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:54 smithi169 ceph-mon[29344]: 5.5 scrub starts 2024-03-12T03:56:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:54 smithi169 ceph-mon[29344]: 5.5 scrub ok 2024-03-12T03:56:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:54 smithi169 ceph-mon[29344]: osdmap e586: 8 total, 8 up, 8 in 2024-03-12T03:56:54.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:54 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:54.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:54 smithi169 ceph-mon[39949]: 5.5 scrub starts 2024-03-12T03:56:54.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:54 smithi169 ceph-mon[39949]: 5.5 scrub ok 2024-03-12T03:56:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:54 smithi169 ceph-mon[39949]: osdmap e586: 8 total, 8 up, 8 in 2024-03-12T03:56:54.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:54 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:54 smithi173 ceph-mon[33544]: 5.5 scrub starts 2024-03-12T03:56:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:54 smithi173 ceph-mon[33544]: 5.5 scrub ok 2024-03-12T03:56:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:54 smithi173 ceph-mon[33544]: osdmap e586: 8 total, 8 up, 8 in 2024-03-12T03:56:54.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:54 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:55.880 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[29344]: 1.0 scrub starts 2024-03-12T03:56:55.880 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[29344]: 1.0 scrub ok 2024-03-12T03:56:55.880 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[29344]: 6.1 scrub starts 2024-03-12T03:56:55.881 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[29344]: pgmap v1193: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 826 B/s rd, 0 op/s 2024-03-12T03:56:55.881 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[29344]: 6.1 scrub ok 2024-03-12T03:56:55.881 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[29344]: 5.1b deep-scrub starts 2024-03-12T03:56:55.881 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[29344]: 5.1b deep-scrub ok 2024-03-12T03:56:55.881 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[29344]: 3.9 scrub starts 2024-03-12T03:56:55.881 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[29344]: 3.9 scrub ok 2024-03-12T03:56:55.881 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[29344]: 3.b scrub starts 2024-03-12T03:56:55.881 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[29344]: 3.b scrub ok 2024-03-12T03:56:55.881 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[39949]: 1.0 scrub starts 2024-03-12T03:56:55.881 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[39949]: 1.0 scrub ok 2024-03-12T03:56:55.881 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[39949]: 6.1 scrub starts 2024-03-12T03:56:55.882 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[39949]: pgmap v1193: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 826 B/s rd, 0 op/s 2024-03-12T03:56:55.882 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[39949]: 6.1 scrub ok 2024-03-12T03:56:55.882 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[39949]: 5.1b deep-scrub starts 2024-03-12T03:56:55.882 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[39949]: 5.1b deep-scrub ok 2024-03-12T03:56:55.882 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[39949]: 3.9 scrub starts 2024-03-12T03:56:55.882 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[39949]: 3.9 scrub ok 2024-03-12T03:56:55.882 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[39949]: 3.b scrub starts 2024-03-12T03:56:55.882 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:55 smithi169 ceph-mon[39949]: 3.b scrub ok 2024-03-12T03:56:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:55 smithi173 ceph-mon[33544]: 1.0 scrub starts 2024-03-12T03:56:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:55 smithi173 ceph-mon[33544]: 1.0 scrub ok 2024-03-12T03:56:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:55 smithi173 ceph-mon[33544]: 6.1 scrub starts 2024-03-12T03:56:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:55 smithi173 ceph-mon[33544]: pgmap v1193: 142 pgs: 142 active+clean; 582 KiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 826 B/s rd, 0 op/s 2024-03-12T03:56:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:55 smithi173 ceph-mon[33544]: 6.1 scrub ok 2024-03-12T03:56:55.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:55 smithi173 ceph-mon[33544]: 5.1b deep-scrub starts 2024-03-12T03:56:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:55 smithi173 ceph-mon[33544]: 5.1b deep-scrub ok 2024-03-12T03:56:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:55 smithi173 ceph-mon[33544]: 3.9 scrub starts 2024-03-12T03:56:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:55 smithi173 ceph-mon[33544]: 3.9 scrub ok 2024-03-12T03:56:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:55 smithi173 ceph-mon[33544]: 3.b scrub starts 2024-03-12T03:56:55.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:55 smithi173 ceph-mon[33544]: 3.b scrub ok 2024-03-12T03:56:56.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:56 smithi169 ceph-mon[39949]: 6.d scrub starts 2024-03-12T03:56:56.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:56 smithi169 ceph-mon[39949]: 6.d scrub ok 2024-03-12T03:56:56.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:56 smithi169 ceph-mon[39949]: 3.18 scrub starts 2024-03-12T03:56:56.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:56 smithi169 ceph-mon[39949]: 3.18 scrub ok 2024-03-12T03:56:56.884 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:56 smithi169 ceph-mon[29344]: 6.d scrub starts 2024-03-12T03:56:56.884 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:56 smithi169 ceph-mon[29344]: 6.d scrub ok 2024-03-12T03:56:56.884 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:56 smithi169 ceph-mon[29344]: 3.18 scrub starts 2024-03-12T03:56:56.884 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:56 smithi169 ceph-mon[29344]: 3.18 scrub ok 2024-03-12T03:56:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:56 smithi173 ceph-mon[33544]: 6.d scrub starts 2024-03-12T03:56:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:56 smithi173 ceph-mon[33544]: 6.d scrub ok 2024-03-12T03:56:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:56 smithi173 ceph-mon[33544]: 3.18 scrub starts 2024-03-12T03:56:56.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:56 smithi173 ceph-mon[33544]: 3.18 scrub ok 2024-03-12T03:56:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:57 smithi169 ceph-mon[29344]: 5.10 scrub starts 2024-03-12T03:56:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:57 smithi169 ceph-mon[29344]: 5.10 scrub ok 2024-03-12T03:56:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:57 smithi169 ceph-mon[29344]: pgmap v1194: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:56:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:57 smithi169 ceph-mon[29344]: 6.f scrub starts 2024-03-12T03:56:57.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:57 smithi169 ceph-mon[29344]: 6.f scrub ok 2024-03-12T03:56:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:57 smithi169 ceph-mon[39949]: 5.10 scrub starts 2024-03-12T03:56:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:57 smithi169 ceph-mon[39949]: 5.10 scrub ok 2024-03-12T03:56:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:57 smithi169 ceph-mon[39949]: pgmap v1194: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:56:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:57 smithi169 ceph-mon[39949]: 6.f scrub starts 2024-03-12T03:56:57.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:57 smithi169 ceph-mon[39949]: 6.f scrub ok 2024-03-12T03:56:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:57 smithi173 ceph-mon[33544]: 5.10 scrub starts 2024-03-12T03:56:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:57 smithi173 ceph-mon[33544]: 5.10 scrub ok 2024-03-12T03:56:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:57 smithi173 ceph-mon[33544]: pgmap v1194: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:56:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:57 smithi173 ceph-mon[33544]: 6.f scrub starts 2024-03-12T03:56:57.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:57 smithi173 ceph-mon[33544]: 6.f scrub ok 2024-03-12T03:56:59.874 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:59 smithi169 ceph-mon[29344]: 6.15 scrub starts 2024-03-12T03:56:59.874 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:59 smithi169 ceph-mon[29344]: pgmap v1195: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T03:56:59.874 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:59 smithi169 ceph-mon[29344]: 6.15 scrub ok 2024-03-12T03:56:59.874 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:56:59 smithi169 ceph-mon[29344]: osdmap e587: 8 total, 8 up, 8 in 2024-03-12T03:56:59.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:59 smithi169 ceph-mon[39949]: 6.15 scrub starts 2024-03-12T03:56:59.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:59 smithi169 ceph-mon[39949]: pgmap v1195: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T03:56:59.875 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:59 smithi169 ceph-mon[39949]: 6.15 scrub ok 2024-03-12T03:56:59.875 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:56:59 smithi169 ceph-mon[39949]: osdmap e587: 8 total, 8 up, 8 in 2024-03-12T03:56:59.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:59 smithi173 ceph-mon[33544]: 6.15 scrub starts 2024-03-12T03:56:59.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:59 smithi173 ceph-mon[33544]: pgmap v1195: 142 pgs: 142 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T03:56:59.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:59 smithi173 ceph-mon[33544]: 6.15 scrub ok 2024-03-12T03:56:59.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:56:59 smithi173 ceph-mon[33544]: osdmap e587: 8 total, 8 up, 8 in 2024-03-12T03:57:00.895 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:57:00.895 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:57:00.895 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[29344]: 4.13 deep-scrub starts 2024-03-12T03:57:00.896 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[29344]: 4.13 deep-scrub ok 2024-03-12T03:57:00.896 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[29344]: osdmap e588: 8 total, 8 up, 8 in 2024-03-12T03:57:00.896 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-12T03:57:00.896 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-03-12T03:57:00.896 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-12T03:57:00.896 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-03-12T03:57:00.896 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:57:00.896 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:57:00.896 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[39949]: 4.13 deep-scrub starts 2024-03-12T03:57:00.897 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[39949]: 4.13 deep-scrub ok 2024-03-12T03:57:00.897 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[39949]: osdmap e588: 8 total, 8 up, 8 in 2024-03-12T03:57:00.897 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-12T03:57:00.897 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-03-12T03:57:00.897 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-12T03:57:00.897 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:00 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-03-12T03:57:00.898 INFO:teuthology.orchestra.run.smithi169.stdout:176093659435 2024-03-12T03:57:00.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-12T03:57:00.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:00 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:57:00.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:00 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:57:00.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:00 smithi173 ceph-mon[33544]: 4.13 deep-scrub starts 2024-03-12T03:57:00.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:00 smithi173 ceph-mon[33544]: 4.13 deep-scrub ok 2024-03-12T03:57:00.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:00 smithi173 ceph-mon[33544]: osdmap e588: 8 total, 8 up, 8 in 2024-03-12T03:57:00.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-12T03:57:00.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:00 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-03-12T03:57:00.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:00 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-12T03:57:00.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:00 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-03-12T03:57:01.542 INFO:teuthology.orchestra.run.smithi169.stdout:2237677961237 2024-03-12T03:57:01.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-12T03:57:01.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-mon[39949]: pgmap v1198: 142 pgs: 142 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 761 B/s rd, 0 op/s 2024-03-12T03:57:01.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-mon[39949]: 3.a scrub starts 2024-03-12T03:57:01.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-mon[39949]: 3.a scrub ok 2024-03-12T03:57:01.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-03-12T03:57:01.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]': finished 2024-03-12T03:57:01.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-mon[39949]: osdmap e589: 8 total, 8 up, 8 in 2024-03-12T03:57:01.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-mon[29344]: pgmap v1198: 142 pgs: 142 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 761 B/s rd, 0 op/s 2024-03-12T03:57:01.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-mon[29344]: 3.a scrub starts 2024-03-12T03:57:01.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-mon[29344]: 3.a scrub ok 2024-03-12T03:57:01.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-03-12T03:57:01.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]': finished 2024-03-12T03:57:01.825 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-mon[29344]: osdmap e589: 8 total, 8 up, 8 in 2024-03-12T03:57:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:01 smithi173 ceph-mon[33544]: pgmap v1198: 142 pgs: 142 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 761 B/s rd, 0 op/s 2024-03-12T03:57:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:01 smithi173 ceph-mon[33544]: 3.a scrub starts 2024-03-12T03:57:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:01 smithi173 ceph-mon[33544]: 3.a scrub ok 2024-03-12T03:57:01.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:01 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-03-12T03:57:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:01 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]': finished 2024-03-12T03:57:01.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:01 smithi173 ceph-mon[33544]: osdmap e589: 8 total, 8 up, 8 in 2024-03-12T03:57:02.112 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:01 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:57:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:57:02.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[39949]: 4.7 scrub starts 2024-03-12T03:57:02.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[39949]: 4.7 scrub ok 2024-03-12T03:57:02.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[39949]: 6.1e scrub starts 2024-03-12T03:57:02.673 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[39949]: 6.1e scrub ok 2024-03-12T03:57:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[39949]: osdmap e590: 8 total, 8 up, 8 in 2024-03-12T03:57:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[39949]: 5.6 scrub starts 2024-03-12T03:57:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[39949]: 5.6 scrub ok 2024-03-12T03:57:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:57:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:57:02.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[29344]: 4.7 scrub starts 2024-03-12T03:57:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[29344]: 4.7 scrub ok 2024-03-12T03:57:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[29344]: 6.1e scrub starts 2024-03-12T03:57:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[29344]: 6.1e scrub ok 2024-03-12T03:57:02.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[29344]: osdmap e590: 8 total, 8 up, 8 in 2024-03-12T03:57:02.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[29344]: 5.6 scrub starts 2024-03-12T03:57:02.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[29344]: 5.6 scrub ok 2024-03-12T03:57:02.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:57:02.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:57:02.675 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:02 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:02 smithi173 ceph-mon[33544]: 4.7 scrub starts 2024-03-12T03:57:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:02 smithi173 ceph-mon[33544]: 4.7 scrub ok 2024-03-12T03:57:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:02 smithi173 ceph-mon[33544]: 6.1e scrub starts 2024-03-12T03:57:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:02 smithi173 ceph-mon[33544]: 6.1e scrub ok 2024-03-12T03:57:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:02 smithi173 ceph-mon[33544]: osdmap e590: 8 total, 8 up, 8 in 2024-03-12T03:57:02.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:02 smithi173 ceph-mon[33544]: 5.6 scrub starts 2024-03-12T03:57:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:02 smithi173 ceph-mon[33544]: 5.6 scrub ok 2024-03-12T03:57:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:02 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:57:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:02 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:57:02.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:02 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:03.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:03 smithi169 ceph-mon[29344]: pgmap v1201: 142 pgs: 2 peering, 140 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:03.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:03 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:57:03.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:03 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:57:03.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:03 smithi169 ceph-mon[29344]: osdmap e591: 8 total, 8 up, 8 in 2024-03-12T03:57:03.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:03 smithi169 ceph-mon[39949]: pgmap v1201: 142 pgs: 2 peering, 140 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:03.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:03 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:57:03.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:03 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:57:03.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:03 smithi169 ceph-mon[39949]: osdmap e591: 8 total, 8 up, 8 in 2024-03-12T03:57:03.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:57:03 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:57:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:03 smithi173 ceph-mon[33544]: pgmap v1201: 142 pgs: 2 peering, 140 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:03 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:57:03.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:03 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:57:03.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:03 smithi173 ceph-mon[33544]: osdmap e591: 8 total, 8 up, 8 in 2024-03-12T03:57:04.228 INFO:teuthology.orchestra.run.smithi169.stdout:219043332391 2024-03-12T03:57:04.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-12T03:57:04.830 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:04 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:04.830 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:04 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:04.859 INFO:teuthology.orchestra.run.smithi169.stdout:77309411642 2024-03-12T03:57:04.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-12T03:57:04.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:04 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:05.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:05 smithi169 ceph-mon[29344]: 3.4 scrub starts 2024-03-12T03:57:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:05 smithi169 ceph-mon[29344]: 3.4 scrub ok 2024-03-12T03:57:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:05 smithi169 ceph-mon[29344]: pgmap v1203: 142 pgs: 2 peering, 140 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:57:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:05 smithi169 ceph-mon[29344]: 4.e scrub starts 2024-03-12T03:57:05.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:05 smithi169 ceph-mon[29344]: 4.e scrub ok 2024-03-12T03:57:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:05 smithi169 ceph-mon[39949]: 3.4 scrub starts 2024-03-12T03:57:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:05 smithi169 ceph-mon[39949]: 3.4 scrub ok 2024-03-12T03:57:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:05 smithi169 ceph-mon[39949]: pgmap v1203: 142 pgs: 2 peering, 140 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:57:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:05 smithi169 ceph-mon[39949]: 4.e scrub starts 2024-03-12T03:57:05.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:05 smithi169 ceph-mon[39949]: 4.e scrub ok 2024-03-12T03:57:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:05 smithi173 ceph-mon[33544]: 3.4 scrub starts 2024-03-12T03:57:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:05 smithi173 ceph-mon[33544]: 3.4 scrub ok 2024-03-12T03:57:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:05 smithi173 ceph-mon[33544]: pgmap v1203: 142 pgs: 2 peering, 140 active+clean; 582 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:57:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:05 smithi173 ceph-mon[33544]: 4.e scrub starts 2024-03-12T03:57:05.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:05 smithi173 ceph-mon[33544]: 4.e scrub ok 2024-03-12T03:57:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:06 smithi169 ceph-mon[29344]: 6.18 scrub starts 2024-03-12T03:57:06.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:06 smithi169 ceph-mon[29344]: 6.18 scrub ok 2024-03-12T03:57:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:06 smithi169 ceph-mon[39949]: 6.18 scrub starts 2024-03-12T03:57:06.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:06 smithi169 ceph-mon[39949]: 6.18 scrub ok 2024-03-12T03:57:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:06 smithi173 ceph-mon[33544]: 6.18 scrub starts 2024-03-12T03:57:06.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:06 smithi173 ceph-mon[33544]: 6.18 scrub ok 2024-03-12T03:57:07.046 INFO:teuthology.orchestra.run.smithi169.stdout:137438953777 2024-03-12T03:57:07.046 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-12T03:57:07.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[39949]: pgmap v1204: 142 pgs: 2 peering, 140 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:57:07.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[39949]: 6.19 scrub starts 2024-03-12T03:57:07.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[39949]: 3.3 scrub starts 2024-03-12T03:57:07.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[39949]: 3.3 scrub ok 2024-03-12T03:57:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[39949]: 6.19 scrub ok 2024-03-12T03:57:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[39949]: 4.1c scrub starts 2024-03-12T03:57:07.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[39949]: 4.1c scrub ok 2024-03-12T03:57:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[29344]: pgmap v1204: 142 pgs: 2 peering, 140 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:57:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[29344]: 6.19 scrub starts 2024-03-12T03:57:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[29344]: 3.3 scrub starts 2024-03-12T03:57:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[29344]: 3.3 scrub ok 2024-03-12T03:57:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[29344]: 6.19 scrub ok 2024-03-12T03:57:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[29344]: 4.1c scrub starts 2024-03-12T03:57:07.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:07 smithi169 ceph-mon[29344]: 4.1c scrub ok 2024-03-12T03:57:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:07 smithi173 ceph-mon[33544]: pgmap v1204: 142 pgs: 2 peering, 140 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:57:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:07 smithi173 ceph-mon[33544]: 6.19 scrub starts 2024-03-12T03:57:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:07 smithi173 ceph-mon[33544]: 3.3 scrub starts 2024-03-12T03:57:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:07 smithi173 ceph-mon[33544]: 3.3 scrub ok 2024-03-12T03:57:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:07 smithi173 ceph-mon[33544]: 6.19 scrub ok 2024-03-12T03:57:07.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:07 smithi173 ceph-mon[33544]: 4.1c scrub starts 2024-03-12T03:57:07.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:07 smithi173 ceph-mon[33544]: 4.1c scrub ok 2024-03-12T03:57:07.987 INFO:teuthology.orchestra.run.smithi169.stdout:253403070754 2024-03-12T03:57:07.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-12T03:57:08.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:08 smithi169 ceph-mon[29344]: 6.7 scrub starts 2024-03-12T03:57:08.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:08 smithi169 ceph-mon[29344]: 6.7 scrub ok 2024-03-12T03:57:08.809 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:08 smithi169 ceph-mon[39949]: 6.7 scrub starts 2024-03-12T03:57:08.809 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:08 smithi169 ceph-mon[39949]: 6.7 scrub ok 2024-03-12T03:57:08.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:08 smithi173 ceph-mon[33544]: 6.7 scrub starts 2024-03-12T03:57:08.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:08 smithi173 ceph-mon[33544]: 6.7 scrub ok 2024-03-12T03:57:09.215 INFO:teuthology.orchestra.run.smithi169.stdout:55834575168 2024-03-12T03:57:09.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-12T03:57:09.283 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:57:09.285 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:57:09.286 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:57:09.736 INFO:teuthology.orchestra.run.smithi169.stdout:1417339207810 2024-03-12T03:57:09.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-12T03:57:09.769 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:57:09.777 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:57:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:09 smithi173 ceph-mon[33544]: pgmap v1205: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 28 B/s, 1 objects/s recovering 2024-03-12T03:57:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:09 smithi173 ceph-mon[33544]: 3.6 deep-scrub starts 2024-03-12T03:57:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:09 smithi173 ceph-mon[33544]: 3.6 deep-scrub ok 2024-03-12T03:57:09.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:09 smithi173 ceph-mon[33544]: 4.11 scrub starts 2024-03-12T03:57:09.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:09 smithi173 ceph-mon[33544]: 4.11 scrub ok 2024-03-12T03:57:10.016 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:09 smithi169 ceph-mon[29344]: pgmap v1205: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 28 B/s, 1 objects/s recovering 2024-03-12T03:57:10.016 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:09 smithi169 ceph-mon[29344]: 3.6 deep-scrub starts 2024-03-12T03:57:10.016 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:09 smithi169 ceph-mon[29344]: 3.6 deep-scrub ok 2024-03-12T03:57:10.017 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:09 smithi169 ceph-mon[29344]: 4.11 scrub starts 2024-03-12T03:57:10.017 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:09 smithi169 ceph-mon[29344]: 4.11 scrub ok 2024-03-12T03:57:10.017 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:09 smithi169 ceph-mon[39949]: pgmap v1205: 142 pgs: 142 active+clean; 582 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 28 B/s, 1 objects/s recovering 2024-03-12T03:57:10.017 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:09 smithi169 ceph-mon[39949]: 3.6 deep-scrub starts 2024-03-12T03:57:10.017 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:09 smithi169 ceph-mon[39949]: 3.6 deep-scrub ok 2024-03-12T03:57:10.017 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:09 smithi169 ceph-mon[39949]: 4.11 scrub starts 2024-03-12T03:57:10.017 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:09 smithi169 ceph-mon[39949]: 4.11 scrub ok 2024-03-12T03:57:11.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:57:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:57:11.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[39949]: 5.12 scrub starts 2024-03-12T03:57:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[39949]: 5.12 scrub ok 2024-03-12T03:57:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[39949]: 6.3 scrub starts 2024-03-12T03:57:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[39949]: 6.3 scrub ok 2024-03-12T03:57:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[39949]: pgmap v1206: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 25 B/s, 1 objects/s recovering 2024-03-12T03:57:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[39949]: 6.0 scrub starts 2024-03-12T03:57:11.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[39949]: 6.0 scrub ok 2024-03-12T03:57:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[29344]: 5.12 scrub starts 2024-03-12T03:57:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[29344]: 5.12 scrub ok 2024-03-12T03:57:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[29344]: 6.3 scrub starts 2024-03-12T03:57:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[29344]: 6.3 scrub ok 2024-03-12T03:57:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[29344]: pgmap v1206: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 25 B/s, 1 objects/s recovering 2024-03-12T03:57:11.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[29344]: 6.0 scrub starts 2024-03-12T03:57:11.925 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:11 smithi169 ceph-mon[29344]: 6.0 scrub ok 2024-03-12T03:57:11.953 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:57:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:11 smithi173 ceph-mon[33544]: 5.12 scrub starts 2024-03-12T03:57:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:11 smithi173 ceph-mon[33544]: 5.12 scrub ok 2024-03-12T03:57:11.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:11 smithi173 ceph-mon[33544]: 6.3 scrub starts 2024-03-12T03:57:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:11 smithi173 ceph-mon[33544]: 6.3 scrub ok 2024-03-12T03:57:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:11 smithi173 ceph-mon[33544]: pgmap v1206: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 25 B/s, 1 objects/s recovering 2024-03-12T03:57:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:11 smithi173 ceph-mon[33544]: 6.0 scrub starts 2024-03-12T03:57:11.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:11 smithi173 ceph-mon[33544]: 6.0 scrub ok 2024-03-12T03:57:12.761 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:12 smithi169 ceph-mon[29344]: 3.0 scrub starts 2024-03-12T03:57:12.761 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:12 smithi169 ceph-mon[29344]: 3.0 scrub ok 2024-03-12T03:57:12.761 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:12 smithi169 ceph-mon[29344]: 6.1c scrub starts 2024-03-12T03:57:12.761 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:12 smithi169 ceph-mon[29344]: 6.1c scrub ok 2024-03-12T03:57:12.761 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:12 smithi169 ceph-mon[39949]: 3.0 scrub starts 2024-03-12T03:57:12.761 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:12 smithi169 ceph-mon[39949]: 3.0 scrub ok 2024-03-12T03:57:12.761 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:12 smithi169 ceph-mon[39949]: 6.1c scrub starts 2024-03-12T03:57:12.761 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:12 smithi169 ceph-mon[39949]: 6.1c scrub ok 2024-03-12T03:57:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:12 smithi173 ceph-mon[33544]: 3.0 scrub starts 2024-03-12T03:57:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:12 smithi173 ceph-mon[33544]: 3.0 scrub ok 2024-03-12T03:57:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:12 smithi173 ceph-mon[33544]: 6.1c scrub starts 2024-03-12T03:57:12.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:12 smithi173 ceph-mon[33544]: 6.1c scrub ok 2024-03-12T03:57:13.959 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[29344]: 3.15 deep-scrub starts 2024-03-12T03:57:13.959 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[29344]: 3.15 deep-scrub ok 2024-03-12T03:57:13.960 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[29344]: pgmap v1207: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-12T03:57:13.960 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[29344]: 2.2 deep-scrub starts 2024-03-12T03:57:13.960 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[29344]: 2.2 deep-scrub ok 2024-03-12T03:57:13.960 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[29344]: 4.16 scrub starts 2024-03-12T03:57:13.960 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[29344]: 4.16 scrub ok 2024-03-12T03:57:13.960 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[29344]: 4.4 scrub starts 2024-03-12T03:57:13.960 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[29344]: 4.4 scrub ok 2024-03-12T03:57:13.960 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[39949]: 3.15 deep-scrub starts 2024-03-12T03:57:13.960 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[39949]: 3.15 deep-scrub ok 2024-03-12T03:57:13.960 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[39949]: pgmap v1207: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-12T03:57:13.960 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[39949]: 2.2 deep-scrub starts 2024-03-12T03:57:13.961 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[39949]: 2.2 deep-scrub ok 2024-03-12T03:57:13.961 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[39949]: 4.16 scrub starts 2024-03-12T03:57:13.961 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[39949]: 4.16 scrub ok 2024-03-12T03:57:13.961 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[39949]: 4.4 scrub starts 2024-03-12T03:57:13.961 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:13 smithi169 ceph-mon[39949]: 4.4 scrub ok 2024-03-12T03:57:13.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:57:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:57:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:13 smithi173 ceph-mon[33544]: 3.15 deep-scrub starts 2024-03-12T03:57:13.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:13 smithi173 ceph-mon[33544]: 3.15 deep-scrub ok 2024-03-12T03:57:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:13 smithi173 ceph-mon[33544]: pgmap v1207: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-12T03:57:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:13 smithi173 ceph-mon[33544]: 2.2 deep-scrub starts 2024-03-12T03:57:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:13 smithi173 ceph-mon[33544]: 2.2 deep-scrub ok 2024-03-12T03:57:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:13 smithi173 ceph-mon[33544]: 4.16 scrub starts 2024-03-12T03:57:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:13 smithi173 ceph-mon[33544]: 4.16 scrub ok 2024-03-12T03:57:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:13 smithi173 ceph-mon[33544]: 4.4 scrub starts 2024-03-12T03:57:13.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:13 smithi173 ceph-mon[33544]: 4.4 scrub ok 2024-03-12T03:57:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:14 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:14.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:14 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:14 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:14.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:14 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:14 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:14.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:14 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:15.654 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:57:15.655 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:57:15.908 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:15 smithi169 ceph-mon[29344]: pgmap v1208: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 875 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-12T03:57:15.908 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:15.908 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:15 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:15.909 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:15 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:57:15.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:15 smithi169 ceph-mon[39949]: pgmap v1208: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 875 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-12T03:57:15.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:15.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:15 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:15.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:15 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:57:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:15 smithi173 ceph-mon[33544]: pgmap v1208: 142 pgs: 142 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 875 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-12T03:57:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:15 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:15.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:15 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:57:16.394 INFO:teuthology.orchestra.run.smithi169.stdout:219043332393 2024-03-12T03:57:16.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:16 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:57:16.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:16 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:57:16.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:16 smithi169 ceph-mon[39949]: 3.1 scrub starts 2024-03-12T03:57:16.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:16 smithi169 ceph-mon[39949]: 3.1 scrub ok 2024-03-12T03:57:16.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:16 smithi169 ceph-mon[39949]: osdmap e592: 8 total, 8 up, 8 in 2024-03-12T03:57:16.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:16 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/4275877796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T03:57:16.795 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:16 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:57:16.795 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:16 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:57:16.795 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:16 smithi169 ceph-mon[29344]: 3.1 scrub starts 2024-03-12T03:57:16.795 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:16 smithi169 ceph-mon[29344]: 3.1 scrub ok 2024-03-12T03:57:16.795 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:16 smithi169 ceph-mon[29344]: osdmap e592: 8 total, 8 up, 8 in 2024-03-12T03:57:16.795 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:16 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/4275877796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T03:57:16.824 INFO:teuthology.orchestra.run.smithi169.stdout:176093659438 2024-03-12T03:57:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:16 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:57:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:16 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:57:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:16 smithi173 ceph-mon[33544]: 3.1 scrub starts 2024-03-12T03:57:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:16 smithi173 ceph-mon[33544]: 3.1 scrub ok 2024-03-12T03:57:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:16 smithi173 ceph-mon[33544]: osdmap e592: 8 total, 8 up, 8 in 2024-03-12T03:57:16.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:16 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/4275877796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T03:57:17.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[29344]: pgmap v1210: 142 pgs: 142 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 614 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-12T03:57:17.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[29344]: 6.17 scrub starts 2024-03-12T03:57:17.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[29344]: 4.8 scrub starts 2024-03-12T03:57:17.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[29344]: 4.8 scrub ok 2024-03-12T03:57:17.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[29344]: 5.d scrub starts 2024-03-12T03:57:17.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[29344]: 5.d scrub ok 2024-03-12T03:57:17.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[29344]: 6.16 scrub starts 2024-03-12T03:57:17.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[29344]: 6.16 scrub ok 2024-03-12T03:57:17.820 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[29344]: 6.17 scrub ok 2024-03-12T03:57:17.820 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2091822463' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:57:17.820 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:17.820 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2935523654' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:57:17.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[39949]: pgmap v1210: 142 pgs: 142 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 614 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-12T03:57:17.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[39949]: 6.17 scrub starts 2024-03-12T03:57:17.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[39949]: 4.8 scrub starts 2024-03-12T03:57:17.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[39949]: 4.8 scrub ok 2024-03-12T03:57:17.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[39949]: 5.d scrub starts 2024-03-12T03:57:17.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[39949]: 5.d scrub ok 2024-03-12T03:57:17.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[39949]: 6.16 scrub starts 2024-03-12T03:57:17.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[39949]: 6.16 scrub ok 2024-03-12T03:57:17.821 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[39949]: 6.17 scrub ok 2024-03-12T03:57:17.821 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2091822463' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:57:17.821 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:17.821 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:17 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2935523654' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:57:17.839 INFO:teuthology.orchestra.run.smithi169.stdout:2237677961241 2024-03-12T03:57:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:17 smithi173 ceph-mon[33544]: pgmap v1210: 142 pgs: 142 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 614 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-12T03:57:17.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:17 smithi173 ceph-mon[33544]: 6.17 scrub starts 2024-03-12T03:57:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:17 smithi173 ceph-mon[33544]: 4.8 scrub starts 2024-03-12T03:57:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:17 smithi173 ceph-mon[33544]: 4.8 scrub ok 2024-03-12T03:57:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:17 smithi173 ceph-mon[33544]: 5.d scrub starts 2024-03-12T03:57:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:17 smithi173 ceph-mon[33544]: 5.d scrub ok 2024-03-12T03:57:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:17 smithi173 ceph-mon[33544]: 6.16 scrub starts 2024-03-12T03:57:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:17 smithi173 ceph-mon[33544]: 6.16 scrub ok 2024-03-12T03:57:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:17 smithi173 ceph-mon[33544]: 6.17 scrub ok 2024-03-12T03:57:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:17 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2091822463' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:57:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:17 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:17.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:17 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2935523654' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:57:18.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:18 smithi169 ceph-mon[29344]: 4.1f scrub starts 2024-03-12T03:57:18.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:18 smithi169 ceph-mon[29344]: 4.1f scrub ok 2024-03-12T03:57:18.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:18 smithi169 ceph-mon[29344]: 4.d scrub starts 2024-03-12T03:57:18.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:18 smithi169 ceph-mon[29344]: 4.d scrub ok 2024-03-12T03:57:18.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:18 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/2483157951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:57:18.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:18 smithi169 ceph-mon[29344]: osdmap e593: 8 total, 8 up, 8 in 2024-03-12T03:57:18.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:18 smithi169 ceph-mon[39949]: 4.1f scrub starts 2024-03-12T03:57:18.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:18 smithi169 ceph-mon[39949]: 4.1f scrub ok 2024-03-12T03:57:18.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:18 smithi169 ceph-mon[39949]: 4.d scrub starts 2024-03-12T03:57:18.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:18 smithi169 ceph-mon[39949]: 4.d scrub ok 2024-03-12T03:57:18.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:18 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/2483157951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:57:18.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:18 smithi169 ceph-mon[39949]: osdmap e593: 8 total, 8 up, 8 in 2024-03-12T03:57:18.959 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332391 got 219043332393 for osd.6 2024-03-12T03:57:18.960 DEBUG:teuthology.parallel:result is None 2024-03-12T03:57:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:18 smithi173 ceph-mon[33544]: 4.1f scrub starts 2024-03-12T03:57:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:18 smithi173 ceph-mon[33544]: 4.1f scrub ok 2024-03-12T03:57:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:18 smithi173 ceph-mon[33544]: 4.d scrub starts 2024-03-12T03:57:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:18 smithi173 ceph-mon[33544]: 4.d scrub ok 2024-03-12T03:57:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:18 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/2483157951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:57:18.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:18 smithi173 ceph-mon[33544]: osdmap e593: 8 total, 8 up, 8 in 2024-03-12T03:57:19.891 INFO:teuthology.orchestra.run.smithi169.stdout:77309411644 2024-03-12T03:57:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:19 smithi173 ceph-mon[33544]: 5.9 scrub starts 2024-03-12T03:57:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:19 smithi173 ceph-mon[33544]: 5.9 scrub ok 2024-03-12T03:57:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:19 smithi173 ceph-mon[33544]: pgmap v1211: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:57:19.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:19 smithi173 ceph-mon[33544]: 5.1f scrub starts 2024-03-12T03:57:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:19 smithi173 ceph-mon[33544]: 5.1f scrub ok 2024-03-12T03:57:19.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:19 smithi173 ceph-mon[33544]: 6.1f scrub starts 2024-03-12T03:57:20.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:19 smithi169 ceph-mon[29344]: 5.9 scrub starts 2024-03-12T03:57:20.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:19 smithi169 ceph-mon[29344]: 5.9 scrub ok 2024-03-12T03:57:20.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:19 smithi169 ceph-mon[29344]: pgmap v1211: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:57:20.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:19 smithi169 ceph-mon[29344]: 5.1f scrub starts 2024-03-12T03:57:20.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:19 smithi169 ceph-mon[29344]: 5.1f scrub ok 2024-03-12T03:57:20.149 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:19 smithi169 ceph-mon[29344]: 6.1f scrub starts 2024-03-12T03:57:20.149 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:19 smithi169 ceph-mon[39949]: 5.9 scrub starts 2024-03-12T03:57:20.149 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:19 smithi169 ceph-mon[39949]: 5.9 scrub ok 2024-03-12T03:57:20.149 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:19 smithi169 ceph-mon[39949]: pgmap v1211: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:57:20.150 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:19 smithi169 ceph-mon[39949]: 5.1f scrub starts 2024-03-12T03:57:20.150 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:19 smithi169 ceph-mon[39949]: 5.1f scrub ok 2024-03-12T03:57:20.150 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:19 smithi169 ceph-mon[39949]: 6.1f scrub starts 2024-03-12T03:57:20.700 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411642 got 77309411644 for osd.2 2024-03-12T03:57:20.701 DEBUG:teuthology.parallel:result is None 2024-03-12T03:57:20.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:20 smithi169 ceph-mon[29344]: 6.1f scrub ok 2024-03-12T03:57:20.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:20 smithi169 ceph-mon[39949]: 6.1f scrub ok 2024-03-12T03:57:20.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:20 smithi173 ceph-mon[33544]: 6.1f scrub ok 2024-03-12T03:57:21.575 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659435 got 176093659438 for osd.5 2024-03-12T03:57:21.576 DEBUG:teuthology.parallel:result is None 2024-03-12T03:57:21.823 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:21 smithi169 ceph-mon[29344]: pgmap v1213: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:57:21.823 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:21 smithi169 ceph-mon[39949]: pgmap v1213: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:57:21.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:21 smithi173 ceph-mon[33544]: pgmap v1213: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:57:22.081 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:21 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:57:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:57:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:22 smithi173 ceph-mon[33544]: 6.12 scrub starts 2024-03-12T03:57:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:22 smithi173 ceph-mon[33544]: 6.12 scrub ok 2024-03-12T03:57:22.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:22 smithi173 ceph-mon[33544]: 4.a deep-scrub starts 2024-03-12T03:57:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:22 smithi173 ceph-mon[33544]: 4.a deep-scrub ok 2024-03-12T03:57:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:22 smithi173 ceph-mon[33544]: 4.9 scrub starts 2024-03-12T03:57:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:22 smithi173 ceph-mon[33544]: 4.9 scrub ok 2024-03-12T03:57:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:22 smithi173 ceph-mon[33544]: 5.f scrub starts 2024-03-12T03:57:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:22 smithi173 ceph-mon[33544]: 5.f scrub ok 2024-03-12T03:57:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:22 smithi173 ceph-mon[33544]: 5.14 scrub starts 2024-03-12T03:57:22.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:22 smithi173 ceph-mon[33544]: 5.14 scrub ok 2024-03-12T03:57:23.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[29344]: 6.12 scrub starts 2024-03-12T03:57:23.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[29344]: 6.12 scrub ok 2024-03-12T03:57:23.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[29344]: 4.a deep-scrub starts 2024-03-12T03:57:23.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[29344]: 4.a deep-scrub ok 2024-03-12T03:57:23.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[29344]: 4.9 scrub starts 2024-03-12T03:57:23.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[29344]: 4.9 scrub ok 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[29344]: 5.f scrub starts 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[29344]: 5.f scrub ok 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[29344]: 5.14 scrub starts 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[29344]: 5.14 scrub ok 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[39949]: 6.12 scrub starts 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[39949]: 6.12 scrub ok 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[39949]: 4.a deep-scrub starts 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[39949]: 4.a deep-scrub ok 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[39949]: 4.9 scrub starts 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[39949]: 4.9 scrub ok 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[39949]: 5.f scrub starts 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[39949]: 5.f scrub ok 2024-03-12T03:57:23.171 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[39949]: 5.14 scrub starts 2024-03-12T03:57:23.172 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:22 smithi169 ceph-mon[39949]: 5.14 scrub ok 2024-03-12T03:57:23.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:23 smithi169 ceph-mon[29344]: pgmap v1214: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:23.870 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:23 smithi169 ceph-mon[29344]: 6.1b scrub starts 2024-03-12T03:57:23.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:23 smithi169 ceph-mon[39949]: pgmap v1214: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:23.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:23 smithi169 ceph-mon[39949]: 6.1b scrub starts 2024-03-12T03:57:23.879 INFO:teuthology.orchestra.run.smithi169.stdout:137438953782 2024-03-12T03:57:23.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:23 smithi173 ceph-mon[33544]: pgmap v1214: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:23.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:23 smithi173 ceph-mon[33544]: 6.1b scrub starts 2024-03-12T03:57:23.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:57:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:57:24.427 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2237677961237 got 2237677961241 for osd.0 2024-03-12T03:57:24.427 DEBUG:teuthology.parallel:result is None 2024-03-12T03:57:24.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:24 smithi169 ceph-mon[29344]: 6.1b scrub ok 2024-03-12T03:57:24.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:24 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:24.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:24 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/158626188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:57:24.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:24 smithi169 ceph-mon[29344]: 6.4 deep-scrub starts 2024-03-12T03:57:24.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:24 smithi169 ceph-mon[29344]: 6.4 deep-scrub ok 2024-03-12T03:57:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:24 smithi169 ceph-mon[39949]: 6.1b scrub ok 2024-03-12T03:57:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:24 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:24 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/158626188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:57:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:24 smithi169 ceph-mon[39949]: 6.4 deep-scrub starts 2024-03-12T03:57:24.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:24 smithi169 ceph-mon[39949]: 6.4 deep-scrub ok 2024-03-12T03:57:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:24 smithi173 ceph-mon[33544]: 6.1b scrub ok 2024-03-12T03:57:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:24 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:24 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/158626188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:57:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:24 smithi173 ceph-mon[33544]: 6.4 deep-scrub starts 2024-03-12T03:57:24.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:24 smithi173 ceph-mon[33544]: 6.4 deep-scrub ok 2024-03-12T03:57:25.048 INFO:teuthology.orchestra.run.smithi169.stdout:55834575172 2024-03-12T03:57:25.693 INFO:teuthology.orchestra.run.smithi169.stdout:1417339207814 2024-03-12T03:57:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:25 smithi173 ceph-mon[33544]: pgmap v1215: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:25 smithi173 ceph-mon[33544]: 5.1e scrub starts 2024-03-12T03:57:25.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:25 smithi173 ceph-mon[33544]: 5.1e scrub ok 2024-03-12T03:57:25.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:25 smithi173 ceph-mon[33544]: 3.19 scrub starts 2024-03-12T03:57:25.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:25 smithi173 ceph-mon[33544]: 3.19 scrub ok 2024-03-12T03:57:25.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:25 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/140681934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:57:25.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:25 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1916844525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:57:25.995 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[29344]: pgmap v1215: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:25.995 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[29344]: 5.1e scrub starts 2024-03-12T03:57:25.995 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[29344]: 5.1e scrub ok 2024-03-12T03:57:25.995 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[29344]: 3.19 scrub starts 2024-03-12T03:57:25.995 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[29344]: 3.19 scrub ok 2024-03-12T03:57:25.995 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/140681934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:57:25.995 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1916844525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:57:25.996 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[39949]: pgmap v1215: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:25.996 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[39949]: 5.1e scrub starts 2024-03-12T03:57:25.996 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[39949]: 5.1e scrub ok 2024-03-12T03:57:25.996 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[39949]: 3.19 scrub starts 2024-03-12T03:57:25.996 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[39949]: 3.19 scrub ok 2024-03-12T03:57:25.996 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/140681934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:57:25.996 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:25 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1916844525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:57:26.338 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575168 got 55834575172 for osd.1 2024-03-12T03:57:26.338 DEBUG:teuthology.parallel:result is None 2024-03-12T03:57:26.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:26 smithi169 ceph-mon[29344]: 4.1e scrub starts 2024-03-12T03:57:26.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:26 smithi169 ceph-mon[29344]: 4.1e scrub ok 2024-03-12T03:57:26.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:26 smithi169 ceph-mon[29344]: 1.7 scrub starts 2024-03-12T03:57:26.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:26 smithi169 ceph-mon[29344]: 1.7 scrub ok 2024-03-12T03:57:26.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:26 smithi169 ceph-mon[39949]: 4.1e scrub starts 2024-03-12T03:57:26.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:26 smithi169 ceph-mon[39949]: 4.1e scrub ok 2024-03-12T03:57:26.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:26 smithi169 ceph-mon[39949]: 1.7 scrub starts 2024-03-12T03:57:26.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:26 smithi169 ceph-mon[39949]: 1.7 scrub ok 2024-03-12T03:57:26.978 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1417339207810 got 1417339207814 for osd.3 2024-03-12T03:57:26.978 DEBUG:teuthology.parallel:result is None 2024-03-12T03:57:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:26 smithi173 ceph-mon[33544]: 4.1e scrub starts 2024-03-12T03:57:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:26 smithi173 ceph-mon[33544]: 4.1e scrub ok 2024-03-12T03:57:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:26 smithi173 ceph-mon[33544]: 1.7 scrub starts 2024-03-12T03:57:26.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:26 smithi173 ceph-mon[33544]: 1.7 scrub ok 2024-03-12T03:57:27.353 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953777 got 137438953782 for osd.4 2024-03-12T03:57:27.353 DEBUG:teuthology.parallel:result is None 2024-03-12T03:57:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[29344]: pgmap v1216: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:57:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[29344]: purged_snaps scrub starts 2024-03-12T03:57:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[29344]: purged_snaps scrub ok 2024-03-12T03:57:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[29344]: 3.12 deep-scrub starts 2024-03-12T03:57:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[29344]: 3.12 deep-scrub ok 2024-03-12T03:57:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[29344]: 1.a scrub starts 2024-03-12T03:57:27.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[29344]: 1.a scrub ok 2024-03-12T03:57:27.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[29344]: osdmap e594: 8 total, 8 up, 8 in 2024-03-12T03:57:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[39949]: pgmap v1216: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:57:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[39949]: purged_snaps scrub starts 2024-03-12T03:57:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[39949]: purged_snaps scrub ok 2024-03-12T03:57:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[39949]: 3.12 deep-scrub starts 2024-03-12T03:57:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[39949]: 3.12 deep-scrub ok 2024-03-12T03:57:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[39949]: 1.a scrub starts 2024-03-12T03:57:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[39949]: 1.a scrub ok 2024-03-12T03:57:27.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:27 smithi169 ceph-mon[39949]: osdmap e594: 8 total, 8 up, 8 in 2024-03-12T03:57:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:27 smithi173 ceph-mon[33544]: pgmap v1216: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:57:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:27 smithi173 ceph-mon[33544]: purged_snaps scrub starts 2024-03-12T03:57:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:27 smithi173 ceph-mon[33544]: purged_snaps scrub ok 2024-03-12T03:57:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:27 smithi173 ceph-mon[33544]: 3.12 deep-scrub starts 2024-03-12T03:57:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:27 smithi173 ceph-mon[33544]: 3.12 deep-scrub ok 2024-03-12T03:57:27.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:27 smithi173 ceph-mon[33544]: 1.a scrub starts 2024-03-12T03:57:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:27 smithi173 ceph-mon[33544]: 1.a scrub ok 2024-03-12T03:57:27.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:27 smithi173 ceph-mon[33544]: osdmap e594: 8 total, 8 up, 8 in 2024-03-12T03:57:28.012 INFO:teuthology.orchestra.run.smithi169.stdout:253403070758 2024-03-12T03:57:28.672 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070754 got 253403070758 for osd.7 2024-03-12T03:57:28.672 DEBUG:teuthology.parallel:result is None 2024-03-12T03:57:28.672 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-12T03:57:28.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:57:28.922 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:57:28.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:28 smithi169 ceph-mon[39949]: 6.10 scrub starts 2024-03-12T03:57:28.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:28 smithi169 ceph-mon[39949]: 6.10 scrub ok 2024-03-12T03:57:28.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:28 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1276805901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T03:57:28.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:28 smithi169 ceph-mon[39949]: osdmap e595: 8 total, 8 up, 8 in 2024-03-12T03:57:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:28 smithi169 ceph-mon[29344]: 6.10 scrub starts 2024-03-12T03:57:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:28 smithi169 ceph-mon[29344]: 6.10 scrub ok 2024-03-12T03:57:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:28 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1276805901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T03:57:28.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:28 smithi169 ceph-mon[29344]: osdmap e595: 8 total, 8 up, 8 in 2024-03-12T03:57:29.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:28 smithi173 ceph-mon[33544]: 6.10 scrub starts 2024-03-12T03:57:29.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:28 smithi173 ceph-mon[33544]: 6.10 scrub ok 2024-03-12T03:57:29.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:28 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1276805901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T03:57:29.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:28 smithi173 ceph-mon[33544]: osdmap e595: 8 total, 8 up, 8 in 2024-03-12T03:57:30.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:29 smithi169 ceph-mon[39949]: pgmap v1218: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T03:57:30.035 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:29 smithi169 ceph-mon[29344]: pgmap v1218: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T03:57:30.137 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:29 smithi173 ceph-mon[33544]: pgmap v1218: 142 pgs: 142 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T03:57:30.775 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:57:30.778 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:57:31.406 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":1220,"stamp":"2024-03-12T03:57:30.348252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":3258,"num_read_kb":2978,"num_write":1200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":2041295,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1772,"ondisk_log_size":1772,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":436,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7090392,"kb_used_data":706444,"kb_used_omap":26,"kb_used_meta":6383845,"kb_avail":742870824,"statfs":{"total":767960285184,"available":760699723776,"internally_reserved":0,"allocated":723398656,"data_stored":713016866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27493,"internal_metadata":6537057435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002501"},"pg_stats":[{"pgid":"1.9","version":"83'128","reported_seq":1478,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696018+0000","last_change":"2024-03-12T03:56:30.032245+0000","last_active":"2024-03-12T03:57:18.696018+0000","last_peered":"2024-03-12T03:57:18.696018+0000","last_clean":"2024-03-12T03:57:18.696018+0000","last_became_active":"2024-03-12T03:46:25.289005+0000","last_became_peered":"2024-03-12T03:46:25.289005+0000","last_unstale":"2024-03-12T03:57:18.696018+0000","last_undegraded":"2024-03-12T03:57:18.696018+0000","last_fullsized":"2024-03-12T03:57:18.696018+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:30.032166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:56:30.032166+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:36.663339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"480'35","reported_seq":848,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.697744+0000","last_change":"2024-03-12T03:56:10.592384+0000","last_active":"2024-03-12T03:57:18.697744+0000","last_peered":"2024-03-12T03:57:18.697744+0000","last_clean":"2024-03-12T03:57:18.697744+0000","last_became_active":"2024-03-12T03:56:10.592008+0000","last_became_peered":"2024-03-12T03:56:10.592008+0000","last_unstale":"2024-03-12T03:57:18.697744+0000","last_undegraded":"2024-03-12T03:57:18.697744+0000","last_fullsized":"2024-03-12T03:57:18.697744+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'35","last_scrub_stamp":"2024-03-12T03:56:09.034513+0000","last_deep_scrub":"344'25","last_deep_scrub_stamp":"2024-03-12T03:51:46.582596+0000","last_clean_scrub_stamp":"2024-03-12T03:56:09.034513+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:21.657159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":916,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898987+0000","last_change":"2024-03-12T03:56:55.006481+0000","last_active":"2024-03-12T03:57:28.898987+0000","last_peered":"2024-03-12T03:57:28.898987+0000","last_clean":"2024-03-12T03:57:28.898987+0000","last_became_active":"2024-03-12T03:55:21.712481+0000","last_became_peered":"2024-03-12T03:55:21.712481+0000","last_unstale":"2024-03-12T03:57:28.898987+0000","last_undegraded":"2024-03-12T03:57:28.898987+0000","last_fullsized":"2024-03-12T03:57:28.898987+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:55.006399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:55.006399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:12.462881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"534'15","reported_seq":1129,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910153+0000","last_change":"2024-03-12T03:57:16.543011+0000","last_active":"2024-03-12T03:57:26.910153+0000","last_peered":"2024-03-12T03:57:26.910153+0000","last_clean":"2024-03-12T03:57:26.910153+0000","last_became_active":"2024-03-12T03:55:33.605103+0000","last_became_peered":"2024-03-12T03:55:33.605103+0000","last_unstale":"2024-03-12T03:57:26.910153+0000","last_undegraded":"2024-03-12T03:57:26.910153+0000","last_fullsized":"2024-03-12T03:57:26.910153+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'15","last_scrub_stamp":"2024-03-12T03:57:16.542923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:57:16.542923+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:34.874127+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":892,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698720+0000","last_change":"2024-03-12T03:56:10.598958+0000","last_active":"2024-03-12T03:57:18.698720+0000","last_peered":"2024-03-12T03:57:18.698720+0000","last_clean":"2024-03-12T03:57:18.698720+0000","last_became_active":"2024-03-12T03:56:10.598335+0000","last_became_peered":"2024-03-12T03:56:10.598335+0000","last_unstale":"2024-03-12T03:57:18.698720+0000","last_undegraded":"2024-03-12T03:57:18.698720+0000","last_fullsized":"2024-03-12T03:57:18.698720+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:08.628663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:08.628663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:24.722308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"480'45","reported_seq":734,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698547+0000","last_change":"2024-03-12T03:57:17.402882+0000","last_active":"2024-03-12T03:57:18.698547+0000","last_peered":"2024-03-12T03:57:18.698547+0000","last_clean":"2024-03-12T03:57:18.698547+0000","last_became_active":"2024-03-12T03:56:10.596339+0000","last_became_peered":"2024-03-12T03:56:10.596339+0000","last_unstale":"2024-03-12T03:57:18.698547+0000","last_undegraded":"2024-03-12T03:57:18.698547+0000","last_fullsized":"2024-03-12T03:57:18.698547+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'45","last_scrub_stamp":"2024-03-12T03:57:17.402760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:17.402760+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:44.777798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":617,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.697169+0000","last_change":"2024-03-12T03:57:00.461615+0000","last_active":"2024-03-12T03:57:18.697169+0000","last_peered":"2024-03-12T03:57:18.697169+0000","last_clean":"2024-03-12T03:57:18.697169+0000","last_became_active":"2024-03-12T03:56:23.593598+0000","last_became_peered":"2024-03-12T03:56:23.593598+0000","last_unstale":"2024-03-12T03:57:18.697169+0000","last_undegraded":"2024-03-12T03:57:18.697169+0000","last_fullsized":"2024-03-12T03:57:18.697169+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:00.461483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:57:00.461483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:19.367641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1136,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910328+0000","last_change":"2024-03-12T03:56:45.251733+0000","last_active":"2024-03-12T03:57:26.910328+0000","last_peered":"2024-03-12T03:57:26.910328+0000","last_clean":"2024-03-12T03:57:26.910328+0000","last_became_active":"2024-03-12T03:55:21.693557+0000","last_became_peered":"2024-03-12T03:55:21.693557+0000","last_unstale":"2024-03-12T03:57:26.910328+0000","last_undegraded":"2024-03-12T03:57:26.910328+0000","last_fullsized":"2024-03-12T03:57:26.910328+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:45.251645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:45.251645+0000","last_clean_scrub_stamp":"2024-03-12T03:56:45.251645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:11.281164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":894,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910529+0000","last_change":"2024-03-12T03:56:56.850437+0000","last_active":"2024-03-12T03:57:26.910529+0000","last_peered":"2024-03-12T03:57:26.910529+0000","last_clean":"2024-03-12T03:57:26.910529+0000","last_became_active":"2024-03-12T03:46:25.293293+0000","last_became_peered":"2024-03-12T03:46:25.293293+0000","last_unstale":"2024-03-12T03:57:26.910529+0000","last_undegraded":"2024-03-12T03:57:26.910529+0000","last_fullsized":"2024-03-12T03:57:26.910529+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:56.850341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:55:49.916580+0000","last_clean_scrub_stamp":"2024-03-12T03:56:56.850341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:58.133532+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"388'129","reported_seq":1076,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.732281+0000","last_change":"2024-03-12T03:56:14.471554+0000","last_active":"2024-03-12T03:57:26.732281+0000","last_peered":"2024-03-12T03:57:26.732281+0000","last_clean":"2024-03-12T03:57:26.732281+0000","last_became_active":"2024-03-12T03:55:36.583655+0000","last_became_peered":"2024-03-12T03:55:36.583655+0000","last_unstale":"2024-03-12T03:57:26.732281+0000","last_undegraded":"2024-03-12T03:57:26.732281+0000","last_fullsized":"2024-03-12T03:57:26.732281+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":4,"last_scrub":"388'129","last_scrub_stamp":"2024-03-12T03:56:14.471428+0000","last_deep_scrub":"388'129","last_deep_scrub_stamp":"2024-03-12T03:49:17.168603+0000","last_clean_scrub_stamp":"2024-03-12T03:56:14.471428+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1050752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"480'39","reported_seq":697,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698474+0000","last_change":"2024-03-12T03:57:04.552469+0000","last_active":"2024-03-12T03:57:18.698474+0000","last_peered":"2024-03-12T03:57:18.698474+0000","last_clean":"2024-03-12T03:57:18.698474+0000","last_became_active":"2024-03-12T03:57:01.702972+0000","last_became_peered":"2024-03-12T03:57:01.702972+0000","last_unstale":"2024-03-12T03:57:18.698474+0000","last_undegraded":"2024-03-12T03:57:18.698474+0000","last_fullsized":"2024-03-12T03:57:18.698474+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:57:04.552347+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:57:04.552347+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:24.574091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1164,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.697119+0000","last_change":"2024-03-12T03:56:54.585455+0000","last_active":"2024-03-12T03:57:18.697119+0000","last_peered":"2024-03-12T03:57:18.697119+0000","last_clean":"2024-03-12T03:57:18.697119+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:57:18.697119+0000","last_undegraded":"2024-03-12T03:57:18.697119+0000","last_fullsized":"2024-03-12T03:57:18.697119+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:54.585345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_clean_scrub_stamp":"2024-03-12T03:56:54.585345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:06.172179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":151,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.733166+0000","last_change":"2024-03-12T03:57:21.505521+0000","last_active":"2024-03-12T03:57:26.733166+0000","last_peered":"2024-03-12T03:57:26.733166+0000","last_clean":"2024-03-12T03:57:26.733166+0000","last_became_active":"2024-03-12T03:55:33.603297+0000","last_became_peered":"2024-03-12T03:55:33.603297+0000","last_unstale":"2024-03-12T03:57:26.733166+0000","last_undegraded":"2024-03-12T03:57:26.733166+0000","last_fullsized":"2024-03-12T03:57:26.733166+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:21.505391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:57:21.505391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:24.482071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":937,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899726+0000","last_change":"2024-03-12T03:55:59.072404+0000","last_active":"2024-03-12T03:57:28.899726+0000","last_peered":"2024-03-12T03:57:28.899726+0000","last_clean":"2024-03-12T03:57:28.899726+0000","last_became_active":"2024-03-12T03:55:33.592064+0000","last_became_peered":"2024-03-12T03:55:33.592064+0000","last_unstale":"2024-03-12T03:57:28.899726+0000","last_undegraded":"2024-03-12T03:57:28.899726+0000","last_fullsized":"2024-03-12T03:57:28.899726+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:55:59.072302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:55:59.072302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":639,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.732414+0000","last_change":"2024-03-12T03:57:26.629490+0000","last_active":"2024-03-12T03:57:26.732414+0000","last_peered":"2024-03-12T03:57:26.732414+0000","last_clean":"2024-03-12T03:57:26.732414+0000","last_became_active":"2024-03-12T03:55:36.584043+0000","last_became_peered":"2024-03-12T03:55:36.584043+0000","last_unstale":"2024-03-12T03:57:26.732414+0000","last_undegraded":"2024-03-12T03:57:26.732414+0000","last_fullsized":"2024-03-12T03:57:26.732414+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:57:26.629393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:57:26.629393+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:38.494636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"480'39","reported_seq":1018,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.911191+0000","last_change":"2024-03-12T03:56:05.475015+0000","last_active":"2024-03-12T03:57:26.911191+0000","last_peered":"2024-03-12T03:57:26.911191+0000","last_clean":"2024-03-12T03:57:26.911191+0000","last_became_active":"2024-03-12T03:46:25.287373+0000","last_became_peered":"2024-03-12T03:46:25.287373+0000","last_unstale":"2024-03-12T03:57:26.911191+0000","last_undegraded":"2024-03-12T03:57:26.911191+0000","last_fullsized":"2024-03-12T03:57:26.911191+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:56:05.474875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:05.474875+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:30.806764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":84,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696146+0000","last_change":"2024-03-12T03:56:39.478674+0000","last_active":"2024-03-12T03:57:18.696146+0000","last_peered":"2024-03-12T03:57:18.696146+0000","last_clean":"2024-03-12T03:57:18.696146+0000","last_became_active":"2024-03-12T03:56:10.594485+0000","last_became_peered":"2024-03-12T03:56:10.594485+0000","last_unstale":"2024-03-12T03:57:18.696146+0000","last_undegraded":"2024-03-12T03:57:18.696146+0000","last_fullsized":"2024-03-12T03:57:18.696146+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:39.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:39.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:54.448383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"561'16","reported_seq":1131,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698622+0000","last_change":"2024-03-12T03:56:28.303821+0000","last_active":"2024-03-12T03:57:18.698622+0000","last_peered":"2024-03-12T03:57:18.698622+0000","last_clean":"2024-03-12T03:57:18.698622+0000","last_became_active":"2024-03-12T03:56:10.598971+0000","last_became_peered":"2024-03-12T03:56:10.598971+0000","last_unstale":"2024-03-12T03:57:18.698622+0000","last_undegraded":"2024-03-12T03:57:18.698622+0000","last_fullsized":"2024-03-12T03:57:18.698622+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"561'16","last_scrub_stamp":"2024-03-12T03:56:28.303716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:28.303716+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:35.829965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":143,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.732860+0000","last_change":"2024-03-12T03:56:55.718648+0000","last_active":"2024-03-12T03:57:26.732860+0000","last_peered":"2024-03-12T03:57:26.732860+0000","last_clean":"2024-03-12T03:57:26.732860+0000","last_became_active":"2024-03-12T03:55:33.598673+0000","last_became_peered":"2024-03-12T03:55:33.598673+0000","last_unstale":"2024-03-12T03:57:26.732860+0000","last_undegraded":"2024-03-12T03:57:26.732860+0000","last_fullsized":"2024-03-12T03:57:26.732860+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:55.718508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:55.718508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:17.770072+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1146,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696994+0000","last_change":"2024-03-12T03:56:08.360269+0000","last_active":"2024-03-12T03:57:18.696994+0000","last_peered":"2024-03-12T03:57:18.696994+0000","last_clean":"2024-03-12T03:57:18.696994+0000","last_became_active":"2024-03-12T03:46:25.286985+0000","last_became_peered":"2024-03-12T03:46:25.286985+0000","last_unstale":"2024-03-12T03:57:18.696994+0000","last_undegraded":"2024-03-12T03:57:18.696994+0000","last_fullsized":"2024-03-12T03:57:18.696994+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:08.360125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:08.360125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"480'54","reported_seq":1038,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898629+0000","last_change":"2024-03-12T03:56:53.075930+0000","last_active":"2024-03-12T03:57:28.898629+0000","last_peered":"2024-03-12T03:57:28.898629+0000","last_clean":"2024-03-12T03:57:28.898629+0000","last_became_active":"2024-03-12T03:55:21.699982+0000","last_became_peered":"2024-03-12T03:55:21.699982+0000","last_unstale":"2024-03-12T03:57:28.898629+0000","last_undegraded":"2024-03-12T03:57:28.898629+0000","last_fullsized":"2024-03-12T03:57:28.898629+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'54","last_scrub_stamp":"2024-03-12T03:56:53.075818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:53.075818+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:03.040971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":86,"num_read_kb":72,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":921,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899480+0000","last_change":"2024-03-12T03:56:51.027613+0000","last_active":"2024-03-12T03:57:28.899480+0000","last_peered":"2024-03-12T03:57:28.899480+0000","last_clean":"2024-03-12T03:57:28.899480+0000","last_became_active":"2024-03-12T03:55:21.699521+0000","last_became_peered":"2024-03-12T03:55:21.699521+0000","last_unstale":"2024-03-12T03:57:28.899480+0000","last_undegraded":"2024-03-12T03:57:28.899480+0000","last_fullsized":"2024-03-12T03:57:28.899480+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:51.027531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:55:42.078673+0000","last_clean_scrub_stamp":"2024-03-12T03:56:51.027531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:58.955126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":910,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698341+0000","last_change":"2024-03-12T03:56:12.307612+0000","last_active":"2024-03-12T03:57:18.698341+0000","last_peered":"2024-03-12T03:57:18.698341+0000","last_clean":"2024-03-12T03:57:18.698341+0000","last_became_active":"2024-03-12T03:56:10.591396+0000","last_became_peered":"2024-03-12T03:56:10.591396+0000","last_unstale":"2024-03-12T03:57:18.698341+0000","last_undegraded":"2024-03-12T03:57:18.698341+0000","last_fullsized":"2024-03-12T03:57:18.698341+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:12.307536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_clean_scrub_stamp":"2024-03-12T03:56:12.307536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:21.925624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":1328,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698045+0000","last_change":"2024-03-12T03:56:41.481708+0000","last_active":"2024-03-12T03:57:18.698045+0000","last_peered":"2024-03-12T03:57:18.698045+0000","last_clean":"2024-03-12T03:57:18.698045+0000","last_became_active":"2024-03-12T03:55:57.919888+0000","last_became_peered":"2024-03-12T03:55:57.919888+0000","last_unstale":"2024-03-12T03:57:18.698045+0000","last_undegraded":"2024-03-12T03:57:18.698045+0000","last_fullsized":"2024-03-12T03:57:18.698045+0000","mapping_epoch":200,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":555,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:41.481584+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:15.895638+0000","last_clean_scrub_stamp":"2024-03-12T03:56:41.481584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:49.341253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"501'13","reported_seq":1211,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898578+0000","last_change":"2024-03-12T03:56:25.043446+0000","last_active":"2024-03-12T03:57:28.898578+0000","last_peered":"2024-03-12T03:57:28.898578+0000","last_clean":"2024-03-12T03:57:28.898578+0000","last_became_active":"2024-03-12T03:46:25.288674+0000","last_became_peered":"2024-03-12T03:46:25.288674+0000","last_unstale":"2024-03-12T03:57:28.898578+0000","last_undegraded":"2024-03-12T03:57:28.898578+0000","last_fullsized":"2024-03-12T03:57:28.898578+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-03-12T03:56:25.043305+0000","last_deep_scrub":"501'13","last_deep_scrub_stamp":"2024-03-12T03:56:25.043305+0000","last_clean_scrub_stamp":"2024-03-12T03:56:25.043305+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:42.133137+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"480'56","reported_seq":732,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.697354+0000","last_change":"2024-03-12T03:56:10.593554+0000","last_active":"2024-03-12T03:57:18.697354+0000","last_peered":"2024-03-12T03:57:18.697354+0000","last_clean":"2024-03-12T03:57:18.697354+0000","last_became_active":"2024-03-12T03:56:10.593076+0000","last_became_peered":"2024-03-12T03:56:10.593076+0000","last_unstale":"2024-03-12T03:57:18.697354+0000","last_undegraded":"2024-03-12T03:57:18.697354+0000","last_fullsized":"2024-03-12T03:57:18.697354+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'56","last_scrub_stamp":"2024-03-12T03:55:53.492164+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:55:53.492164+0000","objects_scrubbed":8,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:21.635208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":62,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"320'2","reported_seq":823,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600490+0000","last_change":"2024-03-12T03:57:08.510133+0000","last_active":"2024-03-12T03:57:28.600490+0000","last_peered":"2024-03-12T03:57:28.600490+0000","last_clean":"2024-03-12T03:57:28.600490+0000","last_became_active":"2024-03-12T03:55:21.706162+0000","last_became_peered":"2024-03-12T03:55:21.706162+0000","last_unstale":"2024-03-12T03:57:28.600490+0000","last_undegraded":"2024-03-12T03:57:28.600490+0000","last_fullsized":"2024-03-12T03:57:28.600490+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'2","last_scrub_stamp":"2024-03-12T03:57:08.510029+0000","last_deep_scrub":"320'2","last_deep_scrub_stamp":"2024-03-12T03:57:08.510029+0000","last_clean_scrub_stamp":"2024-03-12T03:57:08.510029+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:27.698894+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1150,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696277+0000","last_change":"2024-03-12T03:57:10.342740+0000","last_active":"2024-03-12T03:57:18.696277+0000","last_peered":"2024-03-12T03:57:18.696277+0000","last_clean":"2024-03-12T03:57:18.696277+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:57:18.696277+0000","last_undegraded":"2024-03-12T03:57:18.696277+0000","last_fullsized":"2024-03-12T03:57:18.696277+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:10.342625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:49:54.471078+0000","last_clean_scrub_stamp":"2024-03-12T03:57:10.342625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:34.149055+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":1174,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899331+0000","last_change":"2024-03-12T03:56:44.141366+0000","last_active":"2024-03-12T03:57:28.899331+0000","last_peered":"2024-03-12T03:57:28.899331+0000","last_clean":"2024-03-12T03:57:28.899331+0000","last_became_active":"2024-03-12T03:56:44.141085+0000","last_became_peered":"2024-03-12T03:56:44.141085+0000","last_unstale":"2024-03-12T03:57:28.899331+0000","last_undegraded":"2024-03-12T03:57:28.899331+0000","last_fullsized":"2024-03-12T03:57:28.899331+0000","mapping_epoch":574,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:37.088818+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_clean_scrub_stamp":"2024-03-12T03:56:37.088818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:53.781815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"534'14","reported_seq":1056,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600225+0000","last_change":"2024-03-12T03:56:50.346734+0000","last_active":"2024-03-12T03:57:28.600225+0000","last_peered":"2024-03-12T03:57:28.600225+0000","last_clean":"2024-03-12T03:57:28.600225+0000","last_became_active":"2024-03-12T03:55:33.590974+0000","last_became_peered":"2024-03-12T03:55:33.590974+0000","last_unstale":"2024-03-12T03:57:28.600225+0000","last_undegraded":"2024-03-12T03:57:28.600225+0000","last_fullsized":"2024-03-12T03:57:28.600225+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'14","last_scrub_stamp":"2024-03-12T03:56:50.346128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:50.346128+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:05.737400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"480'36","reported_seq":1249,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910789+0000","last_change":"2024-03-12T03:56:09.526675+0000","last_active":"2024-03-12T03:57:26.910789+0000","last_peered":"2024-03-12T03:57:26.910789+0000","last_clean":"2024-03-12T03:57:26.910789+0000","last_became_active":"2024-03-12T03:55:33.603004+0000","last_became_peered":"2024-03-12T03:55:33.603004+0000","last_unstale":"2024-03-12T03:57:26.910789+0000","last_undegraded":"2024-03-12T03:57:26.910789+0000","last_fullsized":"2024-03-12T03:57:26.910789+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'36","last_scrub_stamp":"2024-03-12T03:56:09.526562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:09.526562+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:32.351498+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":143,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.733118+0000","last_change":"2024-03-12T03:56:15.488701+0000","last_active":"2024-03-12T03:57:26.733118+0000","last_peered":"2024-03-12T03:57:26.733118+0000","last_clean":"2024-03-12T03:57:26.733118+0000","last_became_active":"2024-03-12T03:55:33.599234+0000","last_became_peered":"2024-03-12T03:55:33.599234+0000","last_unstale":"2024-03-12T03:57:26.733118+0000","last_undegraded":"2024-03-12T03:57:26.733118+0000","last_fullsized":"2024-03-12T03:57:26.733118+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:15.488604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:54:40.122207+0000","last_clean_scrub_stamp":"2024-03-12T03:56:15.488604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":809,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600001+0000","last_change":"2024-03-12T03:57:10.590959+0000","last_active":"2024-03-12T03:57:28.600001+0000","last_peered":"2024-03-12T03:57:28.600001+0000","last_clean":"2024-03-12T03:57:28.600001+0000","last_became_active":"2024-03-12T03:55:21.708570+0000","last_became_peered":"2024-03-12T03:55:21.708570+0000","last_unstale":"2024-03-12T03:57:28.600001+0000","last_undegraded":"2024-03-12T03:57:28.600001+0000","last_fullsized":"2024-03-12T03:57:28.600001+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:10.590876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:10.590876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:13.222979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":752,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.305906+0000","last_change":"2024-03-12T03:57:26.305933+0000","last_active":"2024-03-12T03:57:26.305906+0000","last_peered":"2024-03-12T03:57:26.305906+0000","last_clean":"2024-03-12T03:57:26.305906+0000","last_became_active":"2024-03-12T03:56:23.595099+0000","last_became_peered":"2024-03-12T03:56:23.595099+0000","last_unstale":"2024-03-12T03:57:26.305906+0000","last_undegraded":"2024-03-12T03:57:26.305906+0000","last_fullsized":"2024-03-12T03:57:26.305906+0000","mapping_epoch":568,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:57:26.305830+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:56:25.421410+0000","last_clean_scrub_stamp":"2024-03-12T03:57:26.305830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:28.545922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":617,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.697550+0000","last_change":"2024-03-12T03:56:39.402275+0000","last_active":"2024-03-12T03:57:18.697550+0000","last_peered":"2024-03-12T03:57:18.697550+0000","last_clean":"2024-03-12T03:57:18.697550+0000","last_became_active":"2024-03-12T03:56:23.588475+0000","last_became_peered":"2024-03-12T03:56:23.588475+0000","last_unstale":"2024-03-12T03:57:18.697550+0000","last_undegraded":"2024-03-12T03:57:18.697550+0000","last_fullsized":"2024-03-12T03:57:18.697550+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:39.402174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:56:39.402174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:50.792838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"480'45","reported_seq":909,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600380+0000","last_change":"2024-03-12T03:56:33.463204+0000","last_active":"2024-03-12T03:57:28.600380+0000","last_peered":"2024-03-12T03:57:28.600380+0000","last_clean":"2024-03-12T03:57:28.600380+0000","last_became_active":"2024-03-12T03:55:33.594248+0000","last_became_peered":"2024-03-12T03:55:33.594248+0000","last_unstale":"2024-03-12T03:57:28.600380+0000","last_undegraded":"2024-03-12T03:57:28.600380+0000","last_fullsized":"2024-03-12T03:57:28.600380+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'45","last_scrub_stamp":"2024-03-12T03:56:33.463059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:33.463059+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:50.125553+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":61,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1188,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910243+0000","last_change":"2024-03-12T03:57:04.227889+0000","last_active":"2024-03-12T03:57:26.910243+0000","last_peered":"2024-03-12T03:57:26.910243+0000","last_clean":"2024-03-12T03:57:26.910243+0000","last_became_active":"2024-03-12T03:55:33.599159+0000","last_became_peered":"2024-03-12T03:55:33.599159+0000","last_unstale":"2024-03-12T03:57:26.910243+0000","last_undegraded":"2024-03-12T03:57:26.910243+0000","last_fullsized":"2024-03-12T03:57:26.910243+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:04.227792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:57:04.227792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:20.574992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1153,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909819+0000","last_change":"2024-03-12T03:56:54.348787+0000","last_active":"2024-03-12T03:57:26.909819+0000","last_peered":"2024-03-12T03:57:26.909819+0000","last_clean":"2024-03-12T03:57:26.909819+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:57:26.909819+0000","last_undegraded":"2024-03-12T03:57:26.909819+0000","last_fullsized":"2024-03-12T03:57:26.909819+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:54.348690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:55:35.758421+0000","last_clean_scrub_stamp":"2024-03-12T03:56:54.348690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:08.111339+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":744,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.697357+0000","last_change":"2024-03-12T03:56:24.375447+0000","last_active":"2024-03-12T03:57:18.697357+0000","last_peered":"2024-03-12T03:57:18.697357+0000","last_clean":"2024-03-12T03:57:18.697357+0000","last_became_active":"2024-03-12T03:56:23.588993+0000","last_became_peered":"2024-03-12T03:56:23.588993+0000","last_unstale":"2024-03-12T03:57:18.697357+0000","last_undegraded":"2024-03-12T03:57:18.697357+0000","last_fullsized":"2024-03-12T03:57:18.697357+0000","mapping_epoch":568,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:24.375354+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:55:21.883409+0000","last_clean_scrub_stamp":"2024-03-12T03:56:24.375354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:28.403824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":143,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.733359+0000","last_change":"2024-03-12T03:56:30.427738+0000","last_active":"2024-03-12T03:57:26.733359+0000","last_peered":"2024-03-12T03:57:26.733359+0000","last_clean":"2024-03-12T03:57:26.733359+0000","last_became_active":"2024-03-12T03:55:33.594941+0000","last_became_peered":"2024-03-12T03:55:33.594941+0000","last_unstale":"2024-03-12T03:57:26.733359+0000","last_undegraded":"2024-03-12T03:57:26.733359+0000","last_fullsized":"2024-03-12T03:57:26.733359+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:30.427669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:56:30.427669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:38.116057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"480'36","reported_seq":928,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600636+0000","last_change":"2024-03-12T03:56:15.547771+0000","last_active":"2024-03-12T03:57:28.600636+0000","last_peered":"2024-03-12T03:57:28.600636+0000","last_clean":"2024-03-12T03:57:28.600636+0000","last_became_active":"2024-03-12T03:55:21.704595+0000","last_became_peered":"2024-03-12T03:55:21.704595+0000","last_unstale":"2024-03-12T03:57:28.600636+0000","last_undegraded":"2024-03-12T03:57:28.600636+0000","last_fullsized":"2024-03-12T03:57:28.600636+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'36","last_scrub_stamp":"2024-03-12T03:56:15.547648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:15.547648+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:41.592339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":2593,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:25.894580+0000","last_change":"2024-03-12T03:56:44.159681+0000","last_active":"2024-03-12T03:57:25.894580+0000","last_peered":"2024-03-12T03:57:25.894580+0000","last_clean":"2024-03-12T03:57:25.894580+0000","last_became_active":"2024-03-12T03:55:21.696299+0000","last_became_peered":"2024-03-12T03:55:21.696299+0000","last_unstale":"2024-03-12T03:57:25.894580+0000","last_undegraded":"2024-03-12T03:57:25.894580+0000","last_fullsized":"2024-03-12T03:57:25.894580+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:56:44.159579+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:56:44.159579+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:03.675553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1434,"num_read_kb":1429,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":842,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599619+0000","last_change":"2024-03-12T03:57:15.593193+0000","last_active":"2024-03-12T03:57:28.599619+0000","last_peered":"2024-03-12T03:57:28.599619+0000","last_clean":"2024-03-12T03:57:28.599619+0000","last_became_active":"2024-03-12T03:55:21.707906+0000","last_became_peered":"2024-03-12T03:55:21.707906+0000","last_unstale":"2024-03-12T03:57:28.599619+0000","last_undegraded":"2024-03-12T03:57:28.599619+0000","last_fullsized":"2024-03-12T03:57:28.599619+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:15.593111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:57:15.593111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:43.830117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1170,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910049+0000","last_change":"2024-03-12T03:57:23.999876+0000","last_active":"2024-03-12T03:57:26.910049+0000","last_peered":"2024-03-12T03:57:26.910049+0000","last_clean":"2024-03-12T03:57:26.910049+0000","last_became_active":"2024-03-12T03:55:33.597111+0000","last_became_peered":"2024-03-12T03:55:33.597111+0000","last_unstale":"2024-03-12T03:57:26.910049+0000","last_undegraded":"2024-03-12T03:57:26.910049+0000","last_fullsized":"2024-03-12T03:57:26.910049+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:23.999783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:57:23.999783+0000","last_clean_scrub_stamp":"2024-03-12T03:57:23.999783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:43.215614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1313,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909935+0000","last_change":"2024-03-12T03:56:48.152070+0000","last_active":"2024-03-12T03:57:26.909935+0000","last_peered":"2024-03-12T03:57:26.909935+0000","last_clean":"2024-03-12T03:57:26.909935+0000","last_became_active":"2024-03-12T03:56:48.151784+0000","last_became_peered":"2024-03-12T03:56:48.151784+0000","last_unstale":"2024-03-12T03:57:26.909935+0000","last_undegraded":"2024-03-12T03:57:26.909935+0000","last_fullsized":"2024-03-12T03:57:26.909935+0000","mapping_epoch":154,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:06.515586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:56:06.515586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1162,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.699065+0000","last_change":"2024-03-12T03:56:34.028265+0000","last_active":"2024-03-12T03:57:18.699065+0000","last_peered":"2024-03-12T03:57:18.699065+0000","last_clean":"2024-03-12T03:57:18.699065+0000","last_became_active":"2024-03-12T03:55:33.606416+0000","last_became_peered":"2024-03-12T03:55:33.606416+0000","last_unstale":"2024-03-12T03:57:18.699065+0000","last_undegraded":"2024-03-12T03:57:18.699065+0000","last_fullsized":"2024-03-12T03:57:18.699065+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:34.028165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:54:38.655865+0000","last_clean_scrub_stamp":"2024-03-12T03:56:34.028165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:36.072198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"569'42","reported_seq":1178,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.697696+0000","last_change":"2024-03-12T03:56:27.426330+0000","last_active":"2024-03-12T03:57:18.697696+0000","last_peered":"2024-03-12T03:57:18.697696+0000","last_clean":"2024-03-12T03:57:18.697696+0000","last_became_active":"2024-03-12T03:56:23.594618+0000","last_became_peered":"2024-03-12T03:56:23.594618+0000","last_unstale":"2024-03-12T03:57:18.697696+0000","last_undegraded":"2024-03-12T03:57:18.697696+0000","last_fullsized":"2024-03-12T03:57:18.697696+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'42","last_scrub_stamp":"2024-03-12T03:56:27.426212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:27.426212+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:27.659756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":931,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898722+0000","last_change":"2024-03-12T03:56:28.079262+0000","last_active":"2024-03-12T03:57:28.898722+0000","last_peered":"2024-03-12T03:57:28.898722+0000","last_clean":"2024-03-12T03:57:28.898722+0000","last_became_active":"2024-03-12T03:55:33.583727+0000","last_became_peered":"2024-03-12T03:55:33.583727+0000","last_unstale":"2024-03-12T03:57:28.898722+0000","last_undegraded":"2024-03-12T03:57:28.898722+0000","last_fullsized":"2024-03-12T03:57:28.898722+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:28.079184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:28.079184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:35.089911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1452,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696636+0000","last_change":"2024-03-12T03:56:54.242374+0000","last_active":"2024-03-12T03:57:18.696636+0000","last_peered":"2024-03-12T03:57:18.696636+0000","last_clean":"2024-03-12T03:57:18.696636+0000","last_became_active":"2024-03-12T03:55:45.582168+0000","last_became_peered":"2024-03-12T03:55:45.582168+0000","last_unstale":"2024-03-12T03:57:18.696636+0000","last_undegraded":"2024-03-12T03:57:18.696636+0000","last_fullsized":"2024-03-12T03:57:18.696636+0000","mapping_epoch":521,"log_start":"83'128","ondisk_log_start":"83'128","created":20,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:54.242270+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:55:27.740894+0000","last_clean_scrub_stamp":"2024-03-12T03:56:54.242270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:20.112391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":151,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.733298+0000","last_change":"2024-03-12T03:57:07.689399+0000","last_active":"2024-03-12T03:57:26.733298+0000","last_peered":"2024-03-12T03:57:26.733298+0000","last_clean":"2024-03-12T03:57:26.733298+0000","last_became_active":"2024-03-12T03:55:33.594123+0000","last_became_peered":"2024-03-12T03:55:33.594123+0000","last_unstale":"2024-03-12T03:57:26.733298+0000","last_undegraded":"2024-03-12T03:57:26.733298+0000","last_fullsized":"2024-03-12T03:57:26.733298+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:07.689305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:07.689305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:33.080629+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":897,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910170+0000","last_change":"2024-03-12T03:57:01.702738+0000","last_active":"2024-03-12T03:57:26.910170+0000","last_peered":"2024-03-12T03:57:26.910170+0000","last_clean":"2024-03-12T03:57:26.910170+0000","last_became_active":"2024-03-12T03:55:33.605033+0000","last_became_peered":"2024-03-12T03:55:33.605033+0000","last_unstale":"2024-03-12T03:57:26.910170+0000","last_undegraded":"2024-03-12T03:57:26.910170+0000","last_fullsized":"2024-03-12T03:57:26.910170+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:01.702666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:57:01.702666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:10.606822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"480'27","reported_seq":1237,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.911120+0000","last_change":"2024-03-12T03:57:01.294235+0000","last_active":"2024-03-12T03:57:26.911120+0000","last_peered":"2024-03-12T03:57:26.911120+0000","last_clean":"2024-03-12T03:57:26.911120+0000","last_became_active":"2024-03-12T03:55:33.597995+0000","last_became_peered":"2024-03-12T03:55:33.597995+0000","last_unstale":"2024-03-12T03:57:26.911120+0000","last_undegraded":"2024-03-12T03:57:26.911120+0000","last_fullsized":"2024-03-12T03:57:26.911120+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'27","last_scrub_stamp":"2024-03-12T03:57:01.294138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:01.294138+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:25.799015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":872,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910663+0000","last_change":"2024-03-12T03:56:46.662889+0000","last_active":"2024-03-12T03:57:26.910663+0000","last_peered":"2024-03-12T03:57:26.910663+0000","last_clean":"2024-03-12T03:57:26.910663+0000","last_became_active":"2024-03-12T03:55:21.695439+0000","last_became_peered":"2024-03-12T03:55:21.695439+0000","last_unstale":"2024-03-12T03:57:26.910663+0000","last_undegraded":"2024-03-12T03:57:26.910663+0000","last_fullsized":"2024-03-12T03:57:26.910663+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:46.662767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:56:46.662767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:57.506544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1173,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910064+0000","last_change":"2024-03-12T03:57:11.174578+0000","last_active":"2024-03-12T03:57:26.910064+0000","last_peered":"2024-03-12T03:57:26.910064+0000","last_clean":"2024-03-12T03:57:26.910064+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:57:26.910064+0000","last_undegraded":"2024-03-12T03:57:26.910064+0000","last_fullsized":"2024-03-12T03:57:26.910064+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:11.174451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:57:11.174451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:36.638785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":1150,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.699353+0000","last_change":"2024-03-12T03:56:09.083821+0000","last_active":"2024-03-12T03:57:18.699353+0000","last_peered":"2024-03-12T03:57:18.699353+0000","last_clean":"2024-03-12T03:57:18.699353+0000","last_became_active":"2024-03-12T03:46:25.284168+0000","last_became_peered":"2024-03-12T03:46:25.284168+0000","last_unstale":"2024-03-12T03:57:18.699353+0000","last_undegraded":"2024-03-12T03:57:18.699353+0000","last_fullsized":"2024-03-12T03:57:18.699353+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:56:09.083707+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:49:18.522103+0000","last_clean_scrub_stamp":"2024-03-12T03:56:09.083707+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:38.582457+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1467,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696524+0000","last_change":"2024-03-12T03:56:40.160673+0000","last_active":"2024-03-12T03:57:18.696524+0000","last_peered":"2024-03-12T03:57:18.696524+0000","last_clean":"2024-03-12T03:57:18.696524+0000","last_became_active":"2024-03-12T03:55:31.751073+0000","last_became_peered":"2024-03-12T03:55:31.751073+0000","last_unstale":"2024-03-12T03:57:18.696524+0000","last_undegraded":"2024-03-12T03:57:18.696524+0000","last_fullsized":"2024-03-12T03:57:18.696524+0000","mapping_epoch":521,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:40.160527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:56:40.160527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:56.297858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":789,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600432+0000","last_change":"2024-03-12T03:56:53.379669+0000","last_active":"2024-03-12T03:57:28.600432+0000","last_peered":"2024-03-12T03:57:28.600432+0000","last_clean":"2024-03-12T03:57:28.600432+0000","last_became_active":"2024-03-12T03:55:21.699805+0000","last_became_peered":"2024-03-12T03:55:21.699805+0000","last_unstale":"2024-03-12T03:57:28.600432+0000","last_undegraded":"2024-03-12T03:57:28.600432+0000","last_fullsized":"2024-03-12T03:57:28.600432+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:53.379580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:53.379580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:01.283519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"480'66","reported_seq":617,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909988+0000","last_change":"2024-03-12T03:57:13.124386+0000","last_active":"2024-03-12T03:57:26.909988+0000","last_peered":"2024-03-12T03:57:26.909988+0000","last_clean":"2024-03-12T03:57:26.909988+0000","last_became_active":"2024-03-12T03:48:48.462111+0000","last_became_peered":"2024-03-12T03:48:48.462111+0000","last_unstale":"2024-03-12T03:57:26.909988+0000","last_undegraded":"2024-03-12T03:57:26.909988+0000","last_fullsized":"2024-03-12T03:57:26.909988+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'66","last_scrub_stamp":"2024-03-12T03:57:13.124253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:13.124253+0000","objects_scrubbed":11,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:18.729923+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":76,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1059,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696412+0000","last_change":"2024-03-12T03:57:06.566628+0000","last_active":"2024-03-12T03:57:18.696412+0000","last_peered":"2024-03-12T03:57:18.696412+0000","last_clean":"2024-03-12T03:57:18.696412+0000","last_became_active":"2024-03-12T03:56:10.597221+0000","last_became_peered":"2024-03-12T03:56:10.597221+0000","last_unstale":"2024-03-12T03:57:18.696412+0000","last_undegraded":"2024-03-12T03:57:18.696412+0000","last_fullsized":"2024-03-12T03:57:18.696412+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:06.566531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:57:06.566531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:08.850445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"591'3","reported_seq":156,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.732271+0000","last_change":"2024-03-12T03:57:12.507635+0000","last_active":"2024-03-12T03:57:26.732271+0000","last_peered":"2024-03-12T03:57:26.732271+0000","last_clean":"2024-03-12T03:57:26.732271+0000","last_became_active":"2024-03-12T03:55:33.600104+0000","last_became_peered":"2024-03-12T03:55:33.600104+0000","last_unstale":"2024-03-12T03:57:26.732271+0000","last_undegraded":"2024-03-12T03:57:26.732271+0000","last_fullsized":"2024-03-12T03:57:26.732271+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"591'3","last_scrub_stamp":"2024-03-12T03:57:12.507496+0000","last_deep_scrub":"591'3","last_deep_scrub_stamp":"2024-03-12T03:57:12.507496+0000","last_clean_scrub_stamp":"2024-03-12T03:57:12.507496+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:24.442235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":959,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899448+0000","last_change":"2024-03-12T03:56:31.149845+0000","last_active":"2024-03-12T03:57:28.899448+0000","last_peered":"2024-03-12T03:57:28.899448+0000","last_clean":"2024-03-12T03:57:28.899448+0000","last_became_active":"2024-03-12T03:46:25.284416+0000","last_became_peered":"2024-03-12T03:46:25.284416+0000","last_unstale":"2024-03-12T03:57:28.899448+0000","last_undegraded":"2024-03-12T03:57:28.899448+0000","last_fullsized":"2024-03-12T03:57:28.899448+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:31.149736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:31.149736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:38.499883+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":496,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.732256+0000","last_change":"2024-03-12T03:56:34.407728+0000","last_active":"2024-03-12T03:57:26.732256+0000","last_peered":"2024-03-12T03:57:26.732256+0000","last_clean":"2024-03-12T03:57:26.732256+0000","last_became_active":"2024-03-12T03:55:41.798428+0000","last_became_peered":"2024-03-12T03:55:41.798428+0000","last_unstale":"2024-03-12T03:57:26.732256+0000","last_undegraded":"2024-03-12T03:57:26.732256+0000","last_fullsized":"2024-03-12T03:57:26.732256+0000","mapping_epoch":533,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:34.407625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:56:34.407625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:01.957883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"480'32","reported_seq":850,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600686+0000","last_change":"2024-03-12T03:56:45.357569+0000","last_active":"2024-03-12T03:57:28.600686+0000","last_peered":"2024-03-12T03:57:28.600686+0000","last_clean":"2024-03-12T03:57:28.600686+0000","last_became_active":"2024-03-12T03:55:21.711814+0000","last_became_peered":"2024-03-12T03:55:21.711814+0000","last_unstale":"2024-03-12T03:57:28.600686+0000","last_undegraded":"2024-03-12T03:57:28.600686+0000","last_fullsized":"2024-03-12T03:57:28.600686+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'32","last_scrub_stamp":"2024-03-12T03:56:45.357462+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:56:45.357462+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:06.317851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":143,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.732860+0000","last_change":"2024-03-12T03:56:16.473068+0000","last_active":"2024-03-12T03:57:26.732860+0000","last_peered":"2024-03-12T03:57:26.732860+0000","last_clean":"2024-03-12T03:57:26.732860+0000","last_became_active":"2024-03-12T03:55:33.600251+0000","last_became_peered":"2024-03-12T03:55:33.600251+0000","last_unstale":"2024-03-12T03:57:26.732860+0000","last_undegraded":"2024-03-12T03:57:26.732860+0000","last_fullsized":"2024-03-12T03:57:26.732860+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:16.472956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:56:16.472956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":887,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910773+0000","last_change":"2024-03-12T03:56:42.622502+0000","last_active":"2024-03-12T03:57:26.910773+0000","last_peered":"2024-03-12T03:57:26.910773+0000","last_clean":"2024-03-12T03:57:26.910773+0000","last_became_active":"2024-03-12T03:46:25.284556+0000","last_became_peered":"2024-03-12T03:46:25.284556+0000","last_unstale":"2024-03-12T03:57:26.910773+0000","last_undegraded":"2024-03-12T03:57:26.910773+0000","last_fullsized":"2024-03-12T03:57:26.910773+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:42.622408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:55:40.638645+0000","last_clean_scrub_stamp":"2024-03-12T03:56:42.622408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:52.326675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":819,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599960+0000","last_change":"2024-03-12T03:56:46.353344+0000","last_active":"2024-03-12T03:57:28.599960+0000","last_peered":"2024-03-12T03:57:28.599960+0000","last_clean":"2024-03-12T03:57:28.599960+0000","last_became_active":"2024-03-12T03:55:21.704614+0000","last_became_peered":"2024-03-12T03:55:21.704614+0000","last_unstale":"2024-03-12T03:57:28.599960+0000","last_undegraded":"2024-03-12T03:57:28.599960+0000","last_fullsized":"2024-03-12T03:57:28.599960+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:46.353210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:46.353210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:05.838936+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"480'67","reported_seq":980,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:21.328340+0000","last_change":"2024-03-12T03:57:21.328416+0000","last_active":"2024-03-12T03:57:21.328340+0000","last_peered":"2024-03-12T03:57:21.328340+0000","last_clean":"2024-03-12T03:57:21.328340+0000","last_became_active":"2024-03-12T03:56:23.593912+0000","last_became_peered":"2024-03-12T03:56:23.593912+0000","last_unstale":"2024-03-12T03:57:21.328340+0000","last_undegraded":"2024-03-12T03:57:21.328340+0000","last_fullsized":"2024-03-12T03:57:21.328340+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'67","last_scrub_stamp":"2024-03-12T03:57:21.328261+0000","last_deep_scrub":"480'67","last_deep_scrub_stamp":"2024-03-12T03:57:21.328261+0000","last_clean_scrub_stamp":"2024-03-12T03:57:21.328261+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:28.167134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":75,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1161,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.699280+0000","last_change":"2024-03-12T03:56:50.267570+0000","last_active":"2024-03-12T03:57:18.699280+0000","last_peered":"2024-03-12T03:57:18.699280+0000","last_clean":"2024-03-12T03:57:18.699280+0000","last_became_active":"2024-03-12T03:55:33.608753+0000","last_became_peered":"2024-03-12T03:55:33.608753+0000","last_unstale":"2024-03-12T03:57:18.699280+0000","last_undegraded":"2024-03-12T03:57:18.699280+0000","last_fullsized":"2024-03-12T03:57:18.699280+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:50.267429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:50.267429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:18.288518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":871,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909924+0000","last_change":"2024-03-12T03:56:30.638765+0000","last_active":"2024-03-12T03:57:26.909924+0000","last_peered":"2024-03-12T03:57:26.909924+0000","last_clean":"2024-03-12T03:57:26.909924+0000","last_became_active":"2024-03-12T03:55:33.605330+0000","last_became_peered":"2024-03-12T03:55:33.605330+0000","last_unstale":"2024-03-12T03:57:26.909924+0000","last_undegraded":"2024-03-12T03:57:26.909924+0000","last_fullsized":"2024-03-12T03:57:26.909924+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:30.638685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:30.638685+0000","last_clean_scrub_stamp":"2024-03-12T03:56:30.638685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:59.877289+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1160,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.695894+0000","last_change":"2024-03-12T03:56:29.070314+0000","last_active":"2024-03-12T03:57:18.695894+0000","last_peered":"2024-03-12T03:57:18.695894+0000","last_clean":"2024-03-12T03:57:18.695894+0000","last_became_active":"2024-03-12T03:46:25.292122+0000","last_became_peered":"2024-03-12T03:46:25.292122+0000","last_unstale":"2024-03-12T03:57:18.695894+0000","last_undegraded":"2024-03-12T03:57:18.695894+0000","last_fullsized":"2024-03-12T03:57:18.695894+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:29.070214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:29.070214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:52.987816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":878,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909795+0000","last_change":"2024-03-12T03:56:41.632325+0000","last_active":"2024-03-12T03:57:26.909795+0000","last_peered":"2024-03-12T03:57:26.909795+0000","last_clean":"2024-03-12T03:57:26.909795+0000","last_became_active":"2024-03-12T03:55:21.696267+0000","last_became_peered":"2024-03-12T03:55:21.696267+0000","last_unstale":"2024-03-12T03:57:26.909795+0000","last_undegraded":"2024-03-12T03:57:26.909795+0000","last_fullsized":"2024-03-12T03:57:26.909795+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:41.632248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:41.632248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:02.586420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"480'36","reported_seq":693,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:21.405690+0000","last_change":"2024-03-12T03:57:21.405755+0000","last_active":"2024-03-12T03:57:21.405690+0000","last_peered":"2024-03-12T03:57:21.405690+0000","last_clean":"2024-03-12T03:57:21.405690+0000","last_became_active":"2024-03-12T03:56:10.596698+0000","last_became_peered":"2024-03-12T03:56:10.596698+0000","last_unstale":"2024-03-12T03:57:21.405690+0000","last_undegraded":"2024-03-12T03:57:21.405690+0000","last_fullsized":"2024-03-12T03:57:21.405690+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'36","last_scrub_stamp":"2024-03-12T03:57:21.405624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:21.405624+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:23.762568+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":48,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1149,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698922+0000","last_change":"2024-03-12T03:56:07.020102+0000","last_active":"2024-03-12T03:57:18.698922+0000","last_peered":"2024-03-12T03:57:18.698922+0000","last_clean":"2024-03-12T03:57:18.698922+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:57:18.698922+0000","last_undegraded":"2024-03-12T03:57:18.698922+0000","last_fullsized":"2024-03-12T03:57:18.698922+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:07.020001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:07.020001+0000","last_clean_scrub_stamp":"2024-03-12T03:56:07.020001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:32.228800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":873,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899158+0000","last_change":"2024-03-12T03:56:12.268938+0000","last_active":"2024-03-12T03:57:28.899158+0000","last_peered":"2024-03-12T03:57:28.899158+0000","last_clean":"2024-03-12T03:57:28.899158+0000","last_became_active":"2024-03-12T03:46:25.280784+0000","last_became_peered":"2024-03-12T03:46:25.280784+0000","last_unstale":"2024-03-12T03:57:28.899158+0000","last_undegraded":"2024-03-12T03:57:28.899158+0000","last_fullsized":"2024-03-12T03:57:28.899158+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:12.268821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:12.268821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:30.062989+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":1440,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:21.667100+0000","last_change":"2024-03-12T03:57:18.230392+0000","last_active":"2024-03-12T03:57:21.667100+0000","last_peered":"2024-03-12T03:57:21.667100+0000","last_clean":"2024-03-12T03:57:21.667100+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:57:21.667100+0000","last_undegraded":"2024-03-12T03:57:21.667100+0000","last_fullsized":"2024-03-12T03:57:21.667100+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:57:18.230267+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:57:18.230267+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:42.715131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"480'60","reported_seq":171,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.732133+0000","last_change":"2024-03-12T03:57:16.473157+0000","last_active":"2024-03-12T03:57:26.732133+0000","last_peered":"2024-03-12T03:57:26.732133+0000","last_clean":"2024-03-12T03:57:26.732133+0000","last_became_active":"2024-03-12T03:55:33.602816+0000","last_became_peered":"2024-03-12T03:55:33.602816+0000","last_unstale":"2024-03-12T03:57:26.732133+0000","last_undegraded":"2024-03-12T03:57:26.732133+0000","last_fullsized":"2024-03-12T03:57:26.732133+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'60","last_scrub_stamp":"2024-03-12T03:57:16.473039+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:57:16.473039+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:21.899462+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1146,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698998+0000","last_change":"2024-03-12T03:56:38.190361+0000","last_active":"2024-03-12T03:57:18.698998+0000","last_peered":"2024-03-12T03:57:18.698998+0000","last_clean":"2024-03-12T03:57:18.698998+0000","last_became_active":"2024-03-12T03:55:21.696530+0000","last_became_peered":"2024-03-12T03:55:21.696530+0000","last_unstale":"2024-03-12T03:57:18.698998+0000","last_undegraded":"2024-03-12T03:57:18.698998+0000","last_fullsized":"2024-03-12T03:57:18.698998+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:56:38.190211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:38.190211+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:47.569585+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":143,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.732509+0000","last_change":"2024-03-12T03:56:36.388606+0000","last_active":"2024-03-12T03:57:26.732509+0000","last_peered":"2024-03-12T03:57:26.732509+0000","last_clean":"2024-03-12T03:57:26.732509+0000","last_became_active":"2024-03-12T03:55:33.603264+0000","last_became_peered":"2024-03-12T03:55:33.603264+0000","last_unstale":"2024-03-12T03:57:26.732509+0000","last_undegraded":"2024-03-12T03:57:26.732509+0000","last_fullsized":"2024-03-12T03:57:26.732509+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:36.388511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:56:36.388511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:47.725284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"480'24","reported_seq":953,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899564+0000","last_change":"2024-03-12T03:56:19.085452+0000","last_active":"2024-03-12T03:57:28.899564+0000","last_peered":"2024-03-12T03:57:28.899564+0000","last_clean":"2024-03-12T03:57:28.899564+0000","last_became_active":"2024-03-12T03:46:25.288573+0000","last_became_peered":"2024-03-12T03:46:25.288573+0000","last_unstale":"2024-03-12T03:57:28.899564+0000","last_undegraded":"2024-03-12T03:57:28.899564+0000","last_fullsized":"2024-03-12T03:57:28.899564+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'24","last_scrub_stamp":"2024-03-12T03:56:19.085344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:19.085344+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:32.626679+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":625,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.695898+0000","last_change":"2024-03-12T03:56:30.402057+0000","last_active":"2024-03-12T03:57:18.695898+0000","last_peered":"2024-03-12T03:57:18.695898+0000","last_clean":"2024-03-12T03:57:18.695898+0000","last_became_active":"2024-03-12T03:56:23.588721+0000","last_became_peered":"2024-03-12T03:56:23.588721+0000","last_unstale":"2024-03-12T03:57:18.695898+0000","last_undegraded":"2024-03-12T03:57:18.695898+0000","last_fullsized":"2024-03-12T03:57:18.695898+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:30.401941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:30.401941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:42.819705+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":143,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.733088+0000","last_change":"2024-03-12T03:55:56.532593+0000","last_active":"2024-03-12T03:57:26.733088+0000","last_peered":"2024-03-12T03:57:26.733088+0000","last_clean":"2024-03-12T03:57:26.733088+0000","last_became_active":"2024-03-12T03:55:33.592131+0000","last_became_peered":"2024-03-12T03:55:33.592131+0000","last_unstale":"2024-03-12T03:57:26.733088+0000","last_undegraded":"2024-03-12T03:57:26.733088+0000","last_fullsized":"2024-03-12T03:57:26.733088+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:55:56.532493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:55:56.532493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1136,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698714+0000","last_change":"2024-03-12T03:56:58.369068+0000","last_active":"2024-03-12T03:57:18.698714+0000","last_peered":"2024-03-12T03:57:18.698714+0000","last_clean":"2024-03-12T03:57:18.698714+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:57:18.698714+0000","last_undegraded":"2024-03-12T03:57:18.698714+0000","last_fullsized":"2024-03-12T03:57:18.698714+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:58.368911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:38.361430+0000","last_clean_scrub_stamp":"2024-03-12T03:56:58.368911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:01.957783+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"480'33","reported_seq":907,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600598+0000","last_change":"2024-03-12T03:57:12.618050+0000","last_active":"2024-03-12T03:57:28.600598+0000","last_peered":"2024-03-12T03:57:28.600598+0000","last_clean":"2024-03-12T03:57:28.600598+0000","last_became_active":"2024-03-12T03:55:21.711074+0000","last_became_peered":"2024-03-12T03:55:21.711074+0000","last_unstale":"2024-03-12T03:57:28.600598+0000","last_undegraded":"2024-03-12T03:57:28.600598+0000","last_fullsized":"2024-03-12T03:57:28.600598+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'33","last_scrub_stamp":"2024-03-12T03:57:12.617956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:12.617956+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:30.621978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1141,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696455+0000","last_change":"2024-03-12T03:56:11.088835+0000","last_active":"2024-03-12T03:57:18.696455+0000","last_peered":"2024-03-12T03:57:18.696455+0000","last_clean":"2024-03-12T03:57:18.696455+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:57:18.696455+0000","last_undegraded":"2024-03-12T03:57:18.696455+0000","last_fullsized":"2024-03-12T03:57:18.696455+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:11.088730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:11.088730+0000","last_clean_scrub_stamp":"2024-03-12T03:56:11.088730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:38.835221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":907,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899702+0000","last_change":"2024-03-12T03:56:15.154769+0000","last_active":"2024-03-12T03:57:28.899702+0000","last_peered":"2024-03-12T03:57:28.899702+0000","last_clean":"2024-03-12T03:57:28.899702+0000","last_became_active":"2024-03-12T03:46:25.286866+0000","last_became_peered":"2024-03-12T03:46:25.286866+0000","last_unstale":"2024-03-12T03:57:28.899702+0000","last_undegraded":"2024-03-12T03:57:28.899702+0000","last_fullsized":"2024-03-12T03:57:28.899702+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:15.154653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:56:15.154653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":876,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910682+0000","last_change":"2024-03-12T03:56:28.668411+0000","last_active":"2024-03-12T03:57:26.910682+0000","last_peered":"2024-03-12T03:57:26.910682+0000","last_clean":"2024-03-12T03:57:26.910682+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:57:26.910682+0000","last_undegraded":"2024-03-12T03:57:26.910682+0000","last_fullsized":"2024-03-12T03:57:26.910682+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:28.668329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:28.668329+0000","last_clean_scrub_stamp":"2024-03-12T03:56:28.668329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:35.848173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"480'27","reported_seq":155,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.733123+0000","last_change":"2024-03-12T03:56:08.461559+0000","last_active":"2024-03-12T03:57:26.733123+0000","last_peered":"2024-03-12T03:57:26.733123+0000","last_clean":"2024-03-12T03:57:26.733123+0000","last_became_active":"2024-03-12T03:55:33.604743+0000","last_became_peered":"2024-03-12T03:55:33.604743+0000","last_unstale":"2024-03-12T03:57:26.733123+0000","last_undegraded":"2024-03-12T03:57:26.733123+0000","last_fullsized":"2024-03-12T03:57:26.733123+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'27","last_scrub_stamp":"2024-03-12T03:56:08.461437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:08.461437+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":807,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600075+0000","last_change":"2024-03-12T03:57:26.545702+0000","last_active":"2024-03-12T03:57:28.600075+0000","last_peered":"2024-03-12T03:57:28.600075+0000","last_clean":"2024-03-12T03:57:28.600075+0000","last_became_active":"2024-03-12T03:55:21.705922+0000","last_became_peered":"2024-03-12T03:55:21.705922+0000","last_unstale":"2024-03-12T03:57:28.600075+0000","last_undegraded":"2024-03-12T03:57:28.600075+0000","last_fullsized":"2024-03-12T03:57:28.600075+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:26.545600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:57:26.545600+0000","last_clean_scrub_stamp":"2024-03-12T03:57:26.545600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:43.240467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"332'14","reported_seq":1227,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899942+0000","last_change":"2024-03-12T03:57:22.208353+0000","last_active":"2024-03-12T03:57:28.899942+0000","last_peered":"2024-03-12T03:57:28.899942+0000","last_clean":"2024-03-12T03:57:28.899942+0000","last_became_active":"2024-03-12T03:46:25.300463+0000","last_became_peered":"2024-03-12T03:46:25.300463+0000","last_unstale":"2024-03-12T03:57:28.899942+0000","last_undegraded":"2024-03-12T03:57:28.899942+0000","last_fullsized":"2024-03-12T03:57:28.899942+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'14","last_scrub_stamp":"2024-03-12T03:57:22.208259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:57:22.208259+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:49.861049+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":900,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698416+0000","last_change":"2024-03-12T03:57:16.632153+0000","last_active":"2024-03-12T03:57:18.698416+0000","last_peered":"2024-03-12T03:57:18.698416+0000","last_clean":"2024-03-12T03:57:18.698416+0000","last_became_active":"2024-03-12T03:56:10.594927+0000","last_became_peered":"2024-03-12T03:56:10.594927+0000","last_unstale":"2024-03-12T03:57:18.698416+0000","last_undegraded":"2024-03-12T03:57:18.698416+0000","last_fullsized":"2024-03-12T03:57:18.698416+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:16.632055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:16.632055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:35.118892+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"480'49","reported_seq":1021,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899629+0000","last_change":"2024-03-12T03:56:07.038711+0000","last_active":"2024-03-12T03:57:28.899629+0000","last_peered":"2024-03-12T03:57:28.899629+0000","last_clean":"2024-03-12T03:57:28.899629+0000","last_became_active":"2024-03-12T03:46:25.299798+0000","last_became_peered":"2024-03-12T03:46:25.299798+0000","last_unstale":"2024-03-12T03:57:28.899629+0000","last_undegraded":"2024-03-12T03:57:28.899629+0000","last_fullsized":"2024-03-12T03:57:28.899629+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'49","last_scrub_stamp":"2024-03-12T03:56:07.038607+0000","last_deep_scrub":"403'39","last_deep_scrub_stamp":"2024-03-12T03:50:06.535772+0000","last_clean_scrub_stamp":"2024-03-12T03:56:07.038607+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":1176,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696436+0000","last_change":"2024-03-12T03:56:48.221182+0000","last_active":"2024-03-12T03:57:18.696436+0000","last_peered":"2024-03-12T03:57:18.696436+0000","last_clean":"2024-03-12T03:57:18.696436+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:57:18.696436+0000","last_undegraded":"2024-03-12T03:57:18.696436+0000","last_fullsized":"2024-03-12T03:57:18.696436+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:56:48.221074+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:56:48.221074+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:52.324561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"575'15","reported_seq":170,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.733280+0000","last_change":"2024-03-12T03:56:39.471736+0000","last_active":"2024-03-12T03:57:26.733280+0000","last_peered":"2024-03-12T03:57:26.733280+0000","last_clean":"2024-03-12T03:57:26.733280+0000","last_became_active":"2024-03-12T03:55:33.595131+0000","last_became_peered":"2024-03-12T03:55:33.595131+0000","last_unstale":"2024-03-12T03:57:26.733280+0000","last_undegraded":"2024-03-12T03:57:26.733280+0000","last_fullsized":"2024-03-12T03:57:26.733280+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'15","last_scrub_stamp":"2024-03-12T03:56:39.471637+0000","last_deep_scrub":"575'15","last_deep_scrub_stamp":"2024-03-12T03:56:39.471637+0000","last_clean_scrub_stamp":"2024-03-12T03:56:39.471637+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:45.962312+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":823,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600779+0000","last_change":"2024-03-12T03:57:16.609050+0000","last_active":"2024-03-12T03:57:28.600779+0000","last_peered":"2024-03-12T03:57:28.600779+0000","last_clean":"2024-03-12T03:57:28.600779+0000","last_became_active":"2024-03-12T03:55:21.711184+0000","last_became_peered":"2024-03-12T03:55:21.711184+0000","last_unstale":"2024-03-12T03:57:28.600779+0000","last_undegraded":"2024-03-12T03:57:28.600779+0000","last_fullsized":"2024-03-12T03:57:28.600779+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:16.608920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:16.608920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:38.533448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"587'32","reported_seq":1221,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696725+0000","last_change":"2024-03-12T03:57:01.710768+0000","last_active":"2024-03-12T03:57:18.696725+0000","last_peered":"2024-03-12T03:57:18.696725+0000","last_clean":"2024-03-12T03:57:18.696725+0000","last_became_active":"2024-03-12T03:57:01.704851+0000","last_became_peered":"2024-03-12T03:57:01.704851+0000","last_unstale":"2024-03-12T03:57:18.696725+0000","last_undegraded":"2024-03-12T03:57:18.696725+0000","last_fullsized":"2024-03-12T03:57:18.696725+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"587'32","last_scrub_stamp":"2024-03-12T03:56:59.573734+0000","last_deep_scrub":"587'32","last_deep_scrub_stamp":"2024-03-12T03:56:59.573734+0000","last_clean_scrub_stamp":"2024-03-12T03:56:59.573734+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:16.440341+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":210,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899245+0000","last_change":"2024-03-12T03:56:14.120675+0000","last_active":"2024-03-12T03:57:28.899245+0000","last_peered":"2024-03-12T03:57:28.899245+0000","last_clean":"2024-03-12T03:57:28.899245+0000","last_became_active":"2024-03-12T03:54:29.584018+0000","last_became_peered":"2024-03-12T03:54:29.584018+0000","last_unstale":"2024-03-12T03:57:28.899245+0000","last_undegraded":"2024-03-12T03:57:28.899245+0000","last_fullsized":"2024-03-12T03:57:28.899245+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:14.120574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_clean_scrub_stamp":"2024-03-12T03:56:14.120574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:37.900989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1165,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910702+0000","last_change":"2024-03-12T03:57:10.202708+0000","last_active":"2024-03-12T03:57:26.910702+0000","last_peered":"2024-03-12T03:57:26.910702+0000","last_clean":"2024-03-12T03:57:26.910702+0000","last_became_active":"2024-03-12T03:55:33.599723+0000","last_became_peered":"2024-03-12T03:55:33.599723+0000","last_unstale":"2024-03-12T03:57:26.910702+0000","last_undegraded":"2024-03-12T03:57:26.910702+0000","last_fullsized":"2024-03-12T03:57:26.910702+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:10.202628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:57:10.202628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:29.203028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":934,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898930+0000","last_change":"2024-03-12T03:56:38.047368+0000","last_active":"2024-03-12T03:57:28.898930+0000","last_peered":"2024-03-12T03:57:28.898930+0000","last_clean":"2024-03-12T03:57:28.898930+0000","last_became_active":"2024-03-12T03:55:33.580129+0000","last_became_peered":"2024-03-12T03:55:33.580129+0000","last_unstale":"2024-03-12T03:57:28.898930+0000","last_undegraded":"2024-03-12T03:57:28.898930+0000","last_fullsized":"2024-03-12T03:57:28.898930+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:38.047286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:38.047286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:57.054659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"480'21","reported_seq":214,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910282+0000","last_change":"2024-03-12T03:56:52.356434+0000","last_active":"2024-03-12T03:57:26.910282+0000","last_peered":"2024-03-12T03:57:26.910282+0000","last_clean":"2024-03-12T03:57:26.910282+0000","last_became_active":"2024-03-12T03:54:47.094495+0000","last_became_peered":"2024-03-12T03:54:47.094495+0000","last_unstale":"2024-03-12T03:57:26.910282+0000","last_undegraded":"2024-03-12T03:57:26.910282+0000","last_fullsized":"2024-03-12T03:57:26.910282+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'21","last_scrub_stamp":"2024-03-12T03:56:52.356337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:52.356337+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:16.038565+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1166,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.699004+0000","last_change":"2024-03-12T03:57:12.257796+0000","last_active":"2024-03-12T03:57:18.699004+0000","last_peered":"2024-03-12T03:57:18.699004+0000","last_clean":"2024-03-12T03:57:18.699004+0000","last_became_active":"2024-03-12T03:46:25.283602+0000","last_became_peered":"2024-03-12T03:46:25.283602+0000","last_unstale":"2024-03-12T03:57:18.699004+0000","last_undegraded":"2024-03-12T03:57:18.699004+0000","last_fullsized":"2024-03-12T03:57:18.699004+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:12.257657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:57:12.257657+0000","last_clean_scrub_stamp":"2024-03-12T03:57:12.257657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:25.214738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":936,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898889+0000","last_change":"2024-03-12T03:56:41.034611+0000","last_active":"2024-03-12T03:57:28.898889+0000","last_peered":"2024-03-12T03:57:28.898889+0000","last_clean":"2024-03-12T03:57:28.898889+0000","last_became_active":"2024-03-12T03:55:21.712301+0000","last_became_peered":"2024-03-12T03:55:21.712301+0000","last_unstale":"2024-03-12T03:57:28.898889+0000","last_undegraded":"2024-03-12T03:57:28.898889+0000","last_fullsized":"2024-03-12T03:57:28.898889+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:41.034530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:55:38.964306+0000","last_clean_scrub_stamp":"2024-03-12T03:56:41.034530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:06.694594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":838,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599718+0000","last_change":"2024-03-12T03:57:27.505214+0000","last_active":"2024-03-12T03:57:28.599718+0000","last_peered":"2024-03-12T03:57:28.599718+0000","last_clean":"2024-03-12T03:57:28.599718+0000","last_became_active":"2024-03-12T03:55:33.590692+0000","last_became_peered":"2024-03-12T03:55:33.590692+0000","last_unstale":"2024-03-12T03:57:28.599718+0000","last_undegraded":"2024-03-12T03:57:28.599718+0000","last_fullsized":"2024-03-12T03:57:28.599718+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:27.505113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:57:27.505113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:43.444121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"480'34","reported_seq":966,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899150+0000","last_change":"2024-03-12T03:57:09.158380+0000","last_active":"2024-03-12T03:57:28.899150+0000","last_peered":"2024-03-12T03:57:28.899150+0000","last_clean":"2024-03-12T03:57:28.899150+0000","last_became_active":"2024-03-12T03:46:25.287095+0000","last_became_peered":"2024-03-12T03:46:25.287095+0000","last_unstale":"2024-03-12T03:57:28.899150+0000","last_undegraded":"2024-03-12T03:57:28.899150+0000","last_fullsized":"2024-03-12T03:57:28.899150+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'34","last_scrub_stamp":"2024-03-12T03:57:09.158292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:09.158292+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:24.056512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":151,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.733190+0000","last_change":"2024-03-12T03:56:52.549765+0000","last_active":"2024-03-12T03:57:26.733190+0000","last_peered":"2024-03-12T03:57:26.733190+0000","last_clean":"2024-03-12T03:57:26.733190+0000","last_became_active":"2024-03-12T03:55:33.588975+0000","last_became_peered":"2024-03-12T03:55:33.588975+0000","last_unstale":"2024-03-12T03:57:26.733190+0000","last_undegraded":"2024-03-12T03:57:26.733190+0000","last_fullsized":"2024-03-12T03:57:26.733190+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:52.549647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:52.549647+0000","last_clean_scrub_stamp":"2024-03-12T03:56:52.549647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:55.472330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1141,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696858+0000","last_change":"2024-03-12T03:56:56.241810+0000","last_active":"2024-03-12T03:57:18.696858+0000","last_peered":"2024-03-12T03:57:18.696858+0000","last_clean":"2024-03-12T03:57:18.696858+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:57:18.696858+0000","last_undegraded":"2024-03-12T03:57:18.696858+0000","last_fullsized":"2024-03-12T03:57:18.696858+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:56.241710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:55:35.857943+0000","last_clean_scrub_stamp":"2024-03-12T03:56:56.241710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:24.155864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":951,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899268+0000","last_change":"2024-03-12T03:56:47.140349+0000","last_active":"2024-03-12T03:57:28.899268+0000","last_peered":"2024-03-12T03:57:28.899268+0000","last_clean":"2024-03-12T03:57:28.899268+0000","last_became_active":"2024-03-12T03:55:21.711506+0000","last_became_peered":"2024-03-12T03:55:21.711506+0000","last_unstale":"2024-03-12T03:57:28.899268+0000","last_undegraded":"2024-03-12T03:57:28.899268+0000","last_fullsized":"2024-03-12T03:57:28.899268+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:47.140244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:56:47.140244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:04.904871+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"480'14","reported_seq":931,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898349+0000","last_change":"2024-03-12T03:56:32.140402+0000","last_active":"2024-03-12T03:57:28.898349+0000","last_peered":"2024-03-12T03:57:28.898349+0000","last_clean":"2024-03-12T03:57:28.898349+0000","last_became_active":"2024-03-12T03:46:25.286315+0000","last_became_peered":"2024-03-12T03:46:25.286315+0000","last_unstale":"2024-03-12T03:57:28.898349+0000","last_undegraded":"2024-03-12T03:57:28.898349+0000","last_fullsized":"2024-03-12T03:57:28.898349+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'14","last_scrub_stamp":"2024-03-12T03:56:32.140315+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:56:32.140315+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:43.658047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":802,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600957+0000","last_change":"2024-03-12T03:56:36.490463+0000","last_active":"2024-03-12T03:57:28.600957+0000","last_peered":"2024-03-12T03:57:28.600957+0000","last_clean":"2024-03-12T03:57:28.600957+0000","last_became_active":"2024-03-12T03:55:33.590905+0000","last_became_peered":"2024-03-12T03:55:33.590905+0000","last_unstale":"2024-03-12T03:57:28.600957+0000","last_undegraded":"2024-03-12T03:57:28.600957+0000","last_fullsized":"2024-03-12T03:57:28.600957+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:36.490334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:36.490334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:51.862253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":620,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.697145+0000","last_change":"2024-03-12T03:56:26.377525+0000","last_active":"2024-03-12T03:57:18.697145+0000","last_peered":"2024-03-12T03:57:18.697145+0000","last_clean":"2024-03-12T03:57:18.697145+0000","last_became_active":"2024-03-12T03:56:23.594174+0000","last_became_peered":"2024-03-12T03:56:23.594174+0000","last_unstale":"2024-03-12T03:57:18.697145+0000","last_undegraded":"2024-03-12T03:57:18.697145+0000","last_fullsized":"2024-03-12T03:57:18.697145+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:26.377419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:26.377419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:54.712974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1149,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:21.282678+0000","last_change":"2024-03-12T03:57:21.282705+0000","last_active":"2024-03-12T03:57:21.282678+0000","last_peered":"2024-03-12T03:57:21.282678+0000","last_clean":"2024-03-12T03:57:21.282678+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:57:21.282678+0000","last_undegraded":"2024-03-12T03:57:21.282678+0000","last_fullsized":"2024-03-12T03:57:21.282678+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:21.282582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:21.282582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:23.718535+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"480'39","reported_seq":948,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.697916+0000","last_change":"2024-03-12T03:57:17.350440+0000","last_active":"2024-03-12T03:57:18.697916+0000","last_peered":"2024-03-12T03:57:18.697916+0000","last_clean":"2024-03-12T03:57:18.697916+0000","last_became_active":"2024-03-12T03:56:23.595378+0000","last_became_peered":"2024-03-12T03:56:23.595378+0000","last_unstale":"2024-03-12T03:57:18.697916+0000","last_undegraded":"2024-03-12T03:57:18.697916+0000","last_fullsized":"2024-03-12T03:57:18.697916+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:57:17.350340+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:57:17.350340+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:17.597381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":952,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898520+0000","last_change":"2024-03-12T03:56:56.024267+0000","last_active":"2024-03-12T03:57:28.898520+0000","last_peered":"2024-03-12T03:57:28.898520+0000","last_clean":"2024-03-12T03:57:28.898520+0000","last_became_active":"2024-03-12T03:55:21.699340+0000","last_became_peered":"2024-03-12T03:55:21.699340+0000","last_unstale":"2024-03-12T03:57:28.898520+0000","last_undegraded":"2024-03-12T03:57:28.898520+0000","last_fullsized":"2024-03-12T03:57:28.898520+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:56:56.024179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:56.024179+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:12.112374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":843,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600698+0000","last_change":"2024-03-12T03:57:24.572147+0000","last_active":"2024-03-12T03:57:28.600698+0000","last_peered":"2024-03-12T03:57:28.600698+0000","last_clean":"2024-03-12T03:57:28.600698+0000","last_became_active":"2024-03-12T03:55:21.707893+0000","last_became_peered":"2024-03-12T03:55:21.707893+0000","last_unstale":"2024-03-12T03:57:28.600698+0000","last_undegraded":"2024-03-12T03:57:28.600698+0000","last_fullsized":"2024-03-12T03:57:28.600698+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:24.572062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:57:24.572062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:26.719140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1150,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.911006+0000","last_change":"2024-03-12T03:56:49.483218+0000","last_active":"2024-03-12T03:57:26.911006+0000","last_peered":"2024-03-12T03:57:26.911006+0000","last_clean":"2024-03-12T03:57:26.911006+0000","last_became_active":"2024-03-12T03:55:33.604326+0000","last_became_peered":"2024-03-12T03:55:33.604326+0000","last_unstale":"2024-03-12T03:57:26.911006+0000","last_undegraded":"2024-03-12T03:57:26.911006+0000","last_fullsized":"2024-03-12T03:57:26.911006+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:49.483108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:49.483108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:13.402853+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"480'37","reported_seq":873,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600910+0000","last_change":"2024-03-12T03:57:25.584319+0000","last_active":"2024-03-12T03:57:28.600910+0000","last_peered":"2024-03-12T03:57:28.600910+0000","last_clean":"2024-03-12T03:57:28.600910+0000","last_became_active":"2024-03-12T03:55:21.711857+0000","last_became_peered":"2024-03-12T03:55:21.711857+0000","last_unstale":"2024-03-12T03:57:28.600910+0000","last_undegraded":"2024-03-12T03:57:28.600910+0000","last_fullsized":"2024-03-12T03:57:28.600910+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'37","last_scrub_stamp":"2024-03-12T03:57:25.584207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:25.584207+0000","objects_scrubbed":6,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:29.836546+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"351'2","reported_seq":1162,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909678+0000","last_change":"2024-03-12T03:57:25.002147+0000","last_active":"2024-03-12T03:57:26.909678+0000","last_peered":"2024-03-12T03:57:26.909678+0000","last_clean":"2024-03-12T03:57:26.909678+0000","last_became_active":"2024-03-12T03:46:25.283344+0000","last_became_peered":"2024-03-12T03:46:25.283344+0000","last_unstale":"2024-03-12T03:57:26.909678+0000","last_undegraded":"2024-03-12T03:57:26.909678+0000","last_fullsized":"2024-03-12T03:57:26.909678+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'2","last_scrub_stamp":"2024-03-12T03:57:25.002070+0000","last_deep_scrub":"351'2","last_deep_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_clean_scrub_stamp":"2024-03-12T03:57:25.002070+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:40.109854+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"569'13","reported_seq":414,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.697872+0000","last_change":"2024-03-12T03:57:18.355377+0000","last_active":"2024-03-12T03:57:18.697872+0000","last_peered":"2024-03-12T03:57:18.697872+0000","last_clean":"2024-03-12T03:57:18.697872+0000","last_became_active":"2024-03-12T03:56:23.594849+0000","last_became_peered":"2024-03-12T03:56:23.594849+0000","last_unstale":"2024-03-12T03:57:18.697872+0000","last_undegraded":"2024-03-12T03:57:18.697872+0000","last_fullsized":"2024-03-12T03:57:18.697872+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'13","last_scrub_stamp":"2024-03-12T03:57:18.355246+0000","last_deep_scrub":"557'11","last_deep_scrub_stamp":"2024-03-12T03:56:06.046550+0000","last_clean_scrub_stamp":"2024-03-12T03:57:18.355246+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:31.020050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":1177,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698774+0000","last_change":"2024-03-12T03:57:11.234883+0000","last_active":"2024-03-12T03:57:18.698774+0000","last_peered":"2024-03-12T03:57:18.698774+0000","last_clean":"2024-03-12T03:57:18.698774+0000","last_became_active":"2024-03-12T03:55:33.607056+0000","last_became_peered":"2024-03-12T03:55:33.607056+0000","last_unstale":"2024-03-12T03:57:18.698774+0000","last_undegraded":"2024-03-12T03:57:18.698774+0000","last_fullsized":"2024-03-12T03:57:18.698774+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:57:11.234757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:11.234757+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:33.709557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"480'30","reported_seq":992,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899211+0000","last_change":"2024-03-12T03:56:17.121977+0000","last_active":"2024-03-12T03:57:28.899211+0000","last_peered":"2024-03-12T03:57:28.899211+0000","last_clean":"2024-03-12T03:57:28.899211+0000","last_became_active":"2024-03-12T03:46:25.283193+0000","last_became_peered":"2024-03-12T03:46:25.283193+0000","last_unstale":"2024-03-12T03:57:28.899211+0000","last_undegraded":"2024-03-12T03:57:28.899211+0000","last_fullsized":"2024-03-12T03:57:28.899211+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'30","last_scrub_stamp":"2024-03-12T03:56:17.121881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:17.121881+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:45.220394+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1170,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698523+0000","last_change":"2024-03-12T03:56:50.485904+0000","last_active":"2024-03-12T03:57:18.698523+0000","last_peered":"2024-03-12T03:57:18.698523+0000","last_clean":"2024-03-12T03:57:18.698523+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:57:18.698523+0000","last_undegraded":"2024-03-12T03:57:18.698523+0000","last_fullsized":"2024-03-12T03:57:18.698523+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:50.485798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:50.485798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:07.963554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":913,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.697846+0000","last_change":"2024-03-12T03:56:10.594428+0000","last_active":"2024-03-12T03:57:18.697846+0000","last_peered":"2024-03-12T03:57:18.697846+0000","last_clean":"2024-03-12T03:57:18.697846+0000","last_became_active":"2024-03-12T03:56:10.593885+0000","last_became_peered":"2024-03-12T03:56:10.593885+0000","last_unstale":"2024-03-12T03:57:18.697846+0000","last_undegraded":"2024-03-12T03:57:18.697846+0000","last_fullsized":"2024-03-12T03:57:18.697846+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:05.095186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:05.095186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:30.734749+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":973,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899900+0000","last_change":"2024-03-12T03:57:19.359191+0000","last_active":"2024-03-12T03:57:28.899900+0000","last_peered":"2024-03-12T03:57:28.899900+0000","last_clean":"2024-03-12T03:57:28.899900+0000","last_became_active":"2024-03-12T03:55:33.591483+0000","last_became_peered":"2024-03-12T03:55:33.591483+0000","last_unstale":"2024-03-12T03:57:28.899900+0000","last_undegraded":"2024-03-12T03:57:28.899900+0000","last_fullsized":"2024-03-12T03:57:28.899900+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:19.359061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:19.359061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:32.773713+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"480'39","reported_seq":953,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910368+0000","last_change":"2024-03-12T03:57:06.709997+0000","last_active":"2024-03-12T03:57:26.910368+0000","last_peered":"2024-03-12T03:57:26.910368+0000","last_clean":"2024-03-12T03:57:26.910368+0000","last_became_active":"2024-03-12T03:46:25.301700+0000","last_became_peered":"2024-03-12T03:46:25.301700+0000","last_unstale":"2024-03-12T03:57:26.910368+0000","last_undegraded":"2024-03-12T03:57:26.910368+0000","last_fullsized":"2024-03-12T03:57:26.910368+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:57:06.709862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:06.709862+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:23.704696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":793,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600384+0000","last_change":"2024-03-12T03:56:40.322391+0000","last_active":"2024-03-12T03:57:28.600384+0000","last_peered":"2024-03-12T03:57:28.600384+0000","last_clean":"2024-03-12T03:57:28.600384+0000","last_became_active":"2024-03-12T03:55:21.708360+0000","last_became_peered":"2024-03-12T03:55:21.708360+0000","last_unstale":"2024-03-12T03:57:28.600384+0000","last_undegraded":"2024-03-12T03:57:28.600384+0000","last_fullsized":"2024-03-12T03:57:28.600384+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:40.322233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:40.322233+0000","last_clean_scrub_stamp":"2024-03-12T03:56:40.322233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:52.102003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1134,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910879+0000","last_change":"2024-03-12T03:56:47.281484+0000","last_active":"2024-03-12T03:57:26.910879+0000","last_peered":"2024-03-12T03:57:26.910879+0000","last_clean":"2024-03-12T03:57:26.910879+0000","last_became_active":"2024-03-12T03:55:21.693975+0000","last_became_peered":"2024-03-12T03:55:21.693975+0000","last_unstale":"2024-03-12T03:57:26.910879+0000","last_undegraded":"2024-03-12T03:57:26.910879+0000","last_fullsized":"2024-03-12T03:57:26.910879+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:47.281351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:47.281351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:00.376002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1152,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698198+0000","last_change":"2024-03-12T03:57:01.569282+0000","last_active":"2024-03-12T03:57:18.698198+0000","last_peered":"2024-03-12T03:57:18.698198+0000","last_clean":"2024-03-12T03:57:18.698198+0000","last_became_active":"2024-03-12T03:55:33.590451+0000","last_became_peered":"2024-03-12T03:55:33.590451+0000","last_unstale":"2024-03-12T03:57:18.698198+0000","last_undegraded":"2024-03-12T03:57:18.698198+0000","last_fullsized":"2024-03-12T03:57:18.698198+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:01.569167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:51.952189+0000","last_clean_scrub_stamp":"2024-03-12T03:57:01.569167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:25.165249+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"480'18","reported_seq":819,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696564+0000","last_change":"2024-03-12T03:56:13.317181+0000","last_active":"2024-03-12T03:57:18.696564+0000","last_peered":"2024-03-12T03:57:18.696564+0000","last_clean":"2024-03-12T03:57:18.696564+0000","last_became_active":"2024-03-12T03:56:10.596455+0000","last_became_peered":"2024-03-12T03:56:10.596455+0000","last_unstale":"2024-03-12T03:57:18.696564+0000","last_undegraded":"2024-03-12T03:57:18.696564+0000","last_fullsized":"2024-03-12T03:57:18.696564+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'18","last_scrub_stamp":"2024-03-12T03:56:13.317058+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:56:13.317058+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:37.426135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":143,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.732851+0000","last_change":"2024-03-12T03:56:37.428462+0000","last_active":"2024-03-12T03:57:26.732851+0000","last_peered":"2024-03-12T03:57:26.732851+0000","last_clean":"2024-03-12T03:57:26.732851+0000","last_became_active":"2024-03-12T03:55:33.589300+0000","last_became_peered":"2024-03-12T03:55:33.589300+0000","last_unstale":"2024-03-12T03:57:26.732851+0000","last_undegraded":"2024-03-12T03:57:26.732851+0000","last_fullsized":"2024-03-12T03:57:26.732851+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:37.428362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:37.428362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:50.038301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1132,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696391+0000","last_change":"2024-03-12T03:56:24.182756+0000","last_active":"2024-03-12T03:57:18.696391+0000","last_peered":"2024-03-12T03:57:18.696391+0000","last_clean":"2024-03-12T03:57:18.696391+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:57:18.696391+0000","last_undegraded":"2024-03-12T03:57:18.696391+0000","last_fullsized":"2024-03-12T03:57:18.696391+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:24.182659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:56:24.182659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:42.764946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":151,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.732263+0000","last_change":"2024-03-12T03:57:06.618834+0000","last_active":"2024-03-12T03:57:26.732263+0000","last_peered":"2024-03-12T03:57:26.732263+0000","last_clean":"2024-03-12T03:57:26.732263+0000","last_became_active":"2024-03-12T03:55:33.583091+0000","last_became_peered":"2024-03-12T03:55:33.583091+0000","last_unstale":"2024-03-12T03:57:26.732263+0000","last_undegraded":"2024-03-12T03:57:26.732263+0000","last_fullsized":"2024-03-12T03:57:26.732263+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:06.618725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:06.618725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:34.036837+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"480'39","reported_seq":855,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698413+0000","last_change":"2024-03-12T03:56:30.347547+0000","last_active":"2024-03-12T03:57:18.698413+0000","last_peered":"2024-03-12T03:57:18.698413+0000","last_clean":"2024-03-12T03:57:18.698413+0000","last_became_active":"2024-03-12T03:56:10.593751+0000","last_became_peered":"2024-03-12T03:56:10.593751+0000","last_unstale":"2024-03-12T03:57:18.698413+0000","last_undegraded":"2024-03-12T03:57:18.698413+0000","last_fullsized":"2024-03-12T03:57:18.698413+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:56:30.347454+0000","last_deep_scrub":"345'33","last_deep_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_clean_scrub_stamp":"2024-03-12T03:56:30.347454+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:36.016747+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":143,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.732908+0000","last_change":"2024-03-12T03:56:00.477835+0000","last_active":"2024-03-12T03:57:26.732908+0000","last_peered":"2024-03-12T03:57:26.732908+0000","last_clean":"2024-03-12T03:57:26.732908+0000","last_became_active":"2024-03-12T03:55:33.601416+0000","last_became_peered":"2024-03-12T03:55:33.601416+0000","last_unstale":"2024-03-12T03:57:26.732908+0000","last_undegraded":"2024-03-12T03:57:26.732908+0000","last_fullsized":"2024-03-12T03:57:26.732908+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:00.477725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:56:00.477725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":143,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.732696+0000","last_change":"2024-03-12T03:56:54.577180+0000","last_active":"2024-03-12T03:57:26.732696+0000","last_peered":"2024-03-12T03:57:26.732696+0000","last_clean":"2024-03-12T03:57:26.732696+0000","last_became_active":"2024-03-12T03:55:33.584266+0000","last_became_peered":"2024-03-12T03:55:33.584266+0000","last_unstale":"2024-03-12T03:57:26.732696+0000","last_undegraded":"2024-03-12T03:57:26.732696+0000","last_fullsized":"2024-03-12T03:57:26.732696+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:54.577057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:54.577057+0000","last_clean_scrub_stamp":"2024-03-12T03:56:54.577057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:20.726214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":807,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599991+0000","last_change":"2024-03-12T03:57:05.486512+0000","last_active":"2024-03-12T03:57:28.599991+0000","last_peered":"2024-03-12T03:57:28.599991+0000","last_clean":"2024-03-12T03:57:28.599991+0000","last_became_active":"2024-03-12T03:55:21.701401+0000","last_became_peered":"2024-03-12T03:55:21.701401+0000","last_unstale":"2024-03-12T03:57:28.599991+0000","last_undegraded":"2024-03-12T03:57:28.599991+0000","last_fullsized":"2024-03-12T03:57:28.599991+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:05.486432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_clean_scrub_stamp":"2024-03-12T03:57:05.486432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:31.934392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"480'74","reported_seq":1136,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898583+0000","last_change":"2024-03-12T03:56:46.048356+0000","last_active":"2024-03-12T03:57:28.898583+0000","last_peered":"2024-03-12T03:57:28.898583+0000","last_clean":"2024-03-12T03:57:28.898583+0000","last_became_active":"2024-03-12T03:55:21.700045+0000","last_became_peered":"2024-03-12T03:55:21.700045+0000","last_unstale":"2024-03-12T03:57:28.898583+0000","last_undegraded":"2024-03-12T03:57:28.898583+0000","last_fullsized":"2024-03-12T03:57:28.898583+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'74","last_scrub_stamp":"2024-03-12T03:56:46.048258+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:56:46.048258+0000","objects_scrubbed":11,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:46.723052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":97,"num_read_kb":84,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":978,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899647+0000","last_change":"2024-03-12T03:56:27.043313+0000","last_active":"2024-03-12T03:57:28.899647+0000","last_peered":"2024-03-12T03:57:28.899647+0000","last_clean":"2024-03-12T03:57:28.899647+0000","last_became_active":"2024-03-12T03:46:25.279131+0000","last_became_peered":"2024-03-12T03:46:25.279131+0000","last_unstale":"2024-03-12T03:57:28.899647+0000","last_undegraded":"2024-03-12T03:57:28.899647+0000","last_fullsized":"2024-03-12T03:57:28.899647+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:27.043216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:56:27.043216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:33.284291+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":891,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.695882+0000","last_change":"2024-03-12T03:56:18.310139+0000","last_active":"2024-03-12T03:57:18.695882+0000","last_peered":"2024-03-12T03:57:18.695882+0000","last_clean":"2024-03-12T03:57:18.695882+0000","last_became_active":"2024-03-12T03:56:10.596064+0000","last_became_peered":"2024-03-12T03:56:10.596064+0000","last_unstale":"2024-03-12T03:57:18.695882+0000","last_undegraded":"2024-03-12T03:57:18.695882+0000","last_fullsized":"2024-03-12T03:57:18.695882+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:18.310042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:56:18.310042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:39.876897+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":936,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899071+0000","last_change":"2024-03-12T03:57:23.356928+0000","last_active":"2024-03-12T03:57:28.899071+0000","last_peered":"2024-03-12T03:57:28.899071+0000","last_clean":"2024-03-12T03:57:28.899071+0000","last_became_active":"2024-03-12T03:46:25.281081+0000","last_became_peered":"2024-03-12T03:46:25.281081+0000","last_unstale":"2024-03-12T03:57:28.899071+0000","last_undegraded":"2024-03-12T03:57:28.899071+0000","last_fullsized":"2024-03-12T03:57:28.899071+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:23.356793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:20.224988+0000","last_clean_scrub_stamp":"2024-03-12T03:57:23.356793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:40.552641+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"480'32","reported_seq":1189,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.696230+0000","last_change":"2024-03-12T03:56:00.435054+0000","last_active":"2024-03-12T03:57:18.696230+0000","last_peered":"2024-03-12T03:57:18.696230+0000","last_clean":"2024-03-12T03:57:18.696230+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:57:18.696230+0000","last_undegraded":"2024-03-12T03:57:18.696230+0000","last_fullsized":"2024-03-12T03:57:18.696230+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'32","last_scrub_stamp":"2024-03-12T03:56:00.434912+0000","last_deep_scrub":"480'32","last_deep_scrub_stamp":"2024-03-12T03:56:00.434912+0000","last_clean_scrub_stamp":"2024-03-12T03:56:00.434912+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:27.892690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":848,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600793+0000","last_change":"2024-03-12T03:56:38.400409+0000","last_active":"2024-03-12T03:57:28.600793+0000","last_peered":"2024-03-12T03:57:28.600793+0000","last_clean":"2024-03-12T03:57:28.600793+0000","last_became_active":"2024-03-12T03:55:33.601946+0000","last_became_peered":"2024-03-12T03:55:33.601946+0000","last_unstale":"2024-03-12T03:57:28.600793+0000","last_undegraded":"2024-03-12T03:57:28.600793+0000","last_fullsized":"2024-03-12T03:57:28.600793+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:38.400328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:56:38.400328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:48.633312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1154,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909707+0000","last_change":"2024-03-12T03:55:55.621341+0000","last_active":"2024-03-12T03:57:26.909707+0000","last_peered":"2024-03-12T03:57:26.909707+0000","last_clean":"2024-03-12T03:57:26.909707+0000","last_became_active":"2024-03-12T03:55:33.596868+0000","last_became_peered":"2024-03-12T03:55:33.596868+0000","last_unstale":"2024-03-12T03:57:26.909707+0000","last_undegraded":"2024-03-12T03:57:26.909707+0000","last_fullsized":"2024-03-12T03:57:26.909707+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:55:55.621256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_clean_scrub_stamp":"2024-03-12T03:55:55.621256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":887,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-12T03:57:18.698147+0000","last_change":"2024-03-12T03:56:10.597488+0000","last_active":"2024-03-12T03:57:18.698147+0000","last_peered":"2024-03-12T03:57:18.698147+0000","last_clean":"2024-03-12T03:57:18.698147+0000","last_became_active":"2024-03-12T03:56:10.597117+0000","last_became_peered":"2024-03-12T03:56:10.597117+0000","last_unstale":"2024-03-12T03:57:18.698147+0000","last_undegraded":"2024-03-12T03:57:18.698147+0000","last_fullsized":"2024-03-12T03:57:18.698147+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:55:41.674862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_clean_scrub_stamp":"2024-03-12T03:55:41.674862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1434,"num_read_kb":1429,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2033792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1807,"num_read_kb":1532,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":4182,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},"log_size":1257,"ondisk_log_size":1257,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070759,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004772,"kb_used_data":3164,"kb_used_omap":1,"kb_used_meta":1001598,"kb_avail":92740380,"statfs":{"total":95995035648,"available":94966149120,"internally_reserved":0,"allocated":3239936,"data_stored":1945113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1025636810},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332396,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":926668,"kb_used_data":2820,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92818484,"statfs":{"total":95995035648,"available":95046127616,"internally_reserved":0,"allocated":2887680,"data_stored":1615470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":946010566},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575173,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1080852,"kb_used_data":2888,"kb_used_omap":1,"kb_used_meta":1077950,"kb_avail":92664300,"statfs":{"total":95995035648,"available":94888243200,"internally_reserved":0,"allocated":2957312,"data_stored":1624065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1103821253},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":521,"seq":2237677961244,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":764928,"kb_used_data":684788,"kb_used_omap":4,"kb_used_meta":80123,"kb_avail":92980224,"statfs":{"total":95995035648,"available":95211749376,"internally_reserved":0,"allocated":701222912,"data_stored":699915931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":82046628},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411647,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":976912,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92768240,"statfs":{"total":95995035648,"available":94994677760,"internally_reserved":0,"allocated":3153920,"data_stored":1879281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":330,"seq":1417339207815,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":539208,"kb_used_data":3452,"kb_used_omap":13,"kb_used_meta":535730,"kb_avail":93205944,"statfs":{"total":95995035648,"available":95442886656,"internally_reserved":0,"allocated":3534848,"data_stored":2213772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13485,"internal_metadata":548588371},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953783,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":805928,"kb_used_data":2848,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":92939224,"statfs":{"total":95995035648,"available":95169765376,"internally_reserved":0,"allocated":2916352,"data_stored":1616973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":822344133},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659441,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":991124,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92754028,"statfs":{"total":95995035648,"available":94980124672,"internally_reserved":0,"allocated":3485696,"data_stored":2206261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1011415493},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:57:31.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:57:31.666 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:57:31.997 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:31 smithi169 ceph-mon[29344]: pgmap v1220: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:57:31.997 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:31 smithi169 ceph-mon[29344]: 4.5 scrub starts 2024-03-12T03:57:31.997 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:31 smithi169 ceph-mon[29344]: 4.5 scrub ok 2024-03-12T03:57:31.997 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:31 smithi169 ceph-mon[29344]: from='client.16206 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:57:31.998 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:31 smithi169 ceph-mon[39949]: pgmap v1220: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:57:31.998 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:31 smithi169 ceph-mon[39949]: 4.5 scrub starts 2024-03-12T03:57:31.998 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:31 smithi169 ceph-mon[39949]: 4.5 scrub ok 2024-03-12T03:57:31.998 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:31 smithi169 ceph-mon[39949]: from='client.16206 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:57:31.998 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:31 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:57:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:57:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:31 smithi173 ceph-mon[33544]: pgmap v1220: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-12T03:57:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:31 smithi173 ceph-mon[33544]: 4.5 scrub starts 2024-03-12T03:57:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:31 smithi173 ceph-mon[33544]: 4.5 scrub ok 2024-03-12T03:57:32.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:31 smithi173 ceph-mon[33544]: from='client.16206 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:57:32.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:32 smithi169 ceph-mon[29344]: 1.6 deep-scrub starts 2024-03-12T03:57:32.923 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:32 smithi169 ceph-mon[29344]: 1.6 deep-scrub ok 2024-03-12T03:57:32.924 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:32 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:32.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:32 smithi169 ceph-mon[39949]: 1.6 deep-scrub starts 2024-03-12T03:57:32.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:32 smithi169 ceph-mon[39949]: 1.6 deep-scrub ok 2024-03-12T03:57:32.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:32 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:32 smithi173 ceph-mon[33544]: 1.6 deep-scrub starts 2024-03-12T03:57:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:32 smithi173 ceph-mon[33544]: 1.6 deep-scrub ok 2024-03-12T03:57:33.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:32 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:33.507 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-12T03:57:33.509 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-12T03:57:33.876 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:33 smithi169 ceph-mon[29344]: pgmap v1221: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:33.876 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:33 smithi169 ceph-mon[29344]: 6.1a scrub starts 2024-03-12T03:57:33.877 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:33 smithi169 ceph-mon[29344]: 6.1a scrub ok 2024-03-12T03:57:33.877 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:33 smithi169 ceph-mon[29344]: 5.17 scrub starts 2024-03-12T03:57:33.877 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:33 smithi169 ceph-mon[29344]: 5.17 scrub ok 2024-03-12T03:57:33.877 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:33 smithi169 ceph-mon[39949]: pgmap v1221: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:33.877 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:33 smithi169 ceph-mon[39949]: 6.1a scrub starts 2024-03-12T03:57:33.877 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:33 smithi169 ceph-mon[39949]: 6.1a scrub ok 2024-03-12T03:57:33.877 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:33 smithi169 ceph-mon[39949]: 5.17 scrub starts 2024-03-12T03:57:33.877 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:33 smithi169 ceph-mon[39949]: 5.17 scrub ok 2024-03-12T03:57:33.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:57:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug there is no tcmu-runner data available 2024-03-12T03:57:33.986 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:33 smithi173 ceph-mon[33544]: pgmap v1221: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:33 smithi173 ceph-mon[33544]: 6.1a scrub starts 2024-03-12T03:57:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:33 smithi173 ceph-mon[33544]: 6.1a scrub ok 2024-03-12T03:57:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:33 smithi173 ceph-mon[33544]: 5.17 scrub starts 2024-03-12T03:57:33.987 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:33 smithi173 ceph-mon[33544]: 5.17 scrub ok 2024-03-12T03:57:34.264 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":1221,"stamp":"2024-03-12T03:57:32.348735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":3263,"num_read_kb":2983,"num_write":1200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":2041295,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1772,"ondisk_log_size":1772,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":436,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7090508,"kb_used_data":706540,"kb_used_omap":26,"kb_used_meta":6383845,"kb_avail":742870708,"statfs":{"total":767960285184,"available":760699604992,"internally_reserved":0,"allocated":723496960,"data_stored":713069522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27493,"internal_metadata":6537057435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002370"},"pg_stats":[{"pgid":"1.9","version":"83'128","reported_seq":1482,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898545+0000","last_change":"2024-03-12T03:56:30.032245+0000","last_active":"2024-03-12T03:57:28.898545+0000","last_peered":"2024-03-12T03:57:28.898545+0000","last_clean":"2024-03-12T03:57:28.898545+0000","last_became_active":"2024-03-12T03:46:25.289005+0000","last_became_peered":"2024-03-12T03:46:25.289005+0000","last_unstale":"2024-03-12T03:57:28.898545+0000","last_undegraded":"2024-03-12T03:57:28.898545+0000","last_fullsized":"2024-03-12T03:57:28.898545+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:30.032166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:56:30.032166+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:36.663339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"480'35","reported_seq":852,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899275+0000","last_change":"2024-03-12T03:56:10.592384+0000","last_active":"2024-03-12T03:57:28.899275+0000","last_peered":"2024-03-12T03:57:28.899275+0000","last_clean":"2024-03-12T03:57:28.899275+0000","last_became_active":"2024-03-12T03:56:10.592008+0000","last_became_peered":"2024-03-12T03:56:10.592008+0000","last_unstale":"2024-03-12T03:57:28.899275+0000","last_undegraded":"2024-03-12T03:57:28.899275+0000","last_fullsized":"2024-03-12T03:57:28.899275+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'35","last_scrub_stamp":"2024-03-12T03:56:09.034513+0000","last_deep_scrub":"344'25","last_deep_scrub_stamp":"2024-03-12T03:51:46.582596+0000","last_clean_scrub_stamp":"2024-03-12T03:56:09.034513+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":916,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898987+0000","last_change":"2024-03-12T03:56:55.006481+0000","last_active":"2024-03-12T03:57:28.898987+0000","last_peered":"2024-03-12T03:57:28.898987+0000","last_clean":"2024-03-12T03:57:28.898987+0000","last_became_active":"2024-03-12T03:55:21.712481+0000","last_became_peered":"2024-03-12T03:55:21.712481+0000","last_unstale":"2024-03-12T03:57:28.898987+0000","last_undegraded":"2024-03-12T03:57:28.898987+0000","last_fullsized":"2024-03-12T03:57:28.898987+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:55.006399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:55.006399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:12.462881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"534'15","reported_seq":1129,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910153+0000","last_change":"2024-03-12T03:57:16.543011+0000","last_active":"2024-03-12T03:57:26.910153+0000","last_peered":"2024-03-12T03:57:26.910153+0000","last_clean":"2024-03-12T03:57:26.910153+0000","last_became_active":"2024-03-12T03:55:33.605103+0000","last_became_peered":"2024-03-12T03:55:33.605103+0000","last_unstale":"2024-03-12T03:57:26.910153+0000","last_undegraded":"2024-03-12T03:57:26.910153+0000","last_fullsized":"2024-03-12T03:57:26.910153+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'15","last_scrub_stamp":"2024-03-12T03:57:16.542923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:57:16.542923+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:34.874127+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":896,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.900339+0000","last_change":"2024-03-12T03:56:10.598958+0000","last_active":"2024-03-12T03:57:28.900339+0000","last_peered":"2024-03-12T03:57:28.900339+0000","last_clean":"2024-03-12T03:57:28.900339+0000","last_became_active":"2024-03-12T03:56:10.598335+0000","last_became_peered":"2024-03-12T03:56:10.598335+0000","last_unstale":"2024-03-12T03:57:28.900339+0000","last_undegraded":"2024-03-12T03:57:28.900339+0000","last_fullsized":"2024-03-12T03:57:28.900339+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:08.628663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:08.628663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"480'45","reported_seq":738,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.900453+0000","last_change":"2024-03-12T03:57:17.402882+0000","last_active":"2024-03-12T03:57:28.900453+0000","last_peered":"2024-03-12T03:57:28.900453+0000","last_clean":"2024-03-12T03:57:28.900453+0000","last_became_active":"2024-03-12T03:56:10.596339+0000","last_became_peered":"2024-03-12T03:56:10.596339+0000","last_unstale":"2024-03-12T03:57:28.900453+0000","last_undegraded":"2024-03-12T03:57:28.900453+0000","last_fullsized":"2024-03-12T03:57:28.900453+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'45","last_scrub_stamp":"2024-03-12T03:57:17.402760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:17.402760+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:44.777798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":621,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.601056+0000","last_change":"2024-03-12T03:57:00.461615+0000","last_active":"2024-03-12T03:57:28.601056+0000","last_peered":"2024-03-12T03:57:28.601056+0000","last_clean":"2024-03-12T03:57:28.601056+0000","last_became_active":"2024-03-12T03:56:23.593598+0000","last_became_peered":"2024-03-12T03:56:23.593598+0000","last_unstale":"2024-03-12T03:57:28.601056+0000","last_undegraded":"2024-03-12T03:57:28.601056+0000","last_fullsized":"2024-03-12T03:57:28.601056+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:00.461483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:23.646819+0000","last_clean_scrub_stamp":"2024-03-12T03:57:00.461483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:19.367641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1136,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910328+0000","last_change":"2024-03-12T03:56:45.251733+0000","last_active":"2024-03-12T03:57:26.910328+0000","last_peered":"2024-03-12T03:57:26.910328+0000","last_clean":"2024-03-12T03:57:26.910328+0000","last_became_active":"2024-03-12T03:55:21.693557+0000","last_became_peered":"2024-03-12T03:55:21.693557+0000","last_unstale":"2024-03-12T03:57:26.910328+0000","last_undegraded":"2024-03-12T03:57:26.910328+0000","last_fullsized":"2024-03-12T03:57:26.910328+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:45.251645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:45.251645+0000","last_clean_scrub_stamp":"2024-03-12T03:56:45.251645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:11.281164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":894,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910529+0000","last_change":"2024-03-12T03:56:56.850437+0000","last_active":"2024-03-12T03:57:26.910529+0000","last_peered":"2024-03-12T03:57:26.910529+0000","last_clean":"2024-03-12T03:57:26.910529+0000","last_became_active":"2024-03-12T03:46:25.293293+0000","last_became_peered":"2024-03-12T03:46:25.293293+0000","last_unstale":"2024-03-12T03:57:26.910529+0000","last_undegraded":"2024-03-12T03:57:26.910529+0000","last_fullsized":"2024-03-12T03:57:26.910529+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:56.850341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:55:49.916580+0000","last_clean_scrub_stamp":"2024-03-12T03:56:56.850341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:58.133532+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"388'129","reported_seq":1078,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599393+0000","last_change":"2024-03-12T03:56:14.471554+0000","last_active":"2024-03-12T03:57:28.599393+0000","last_peered":"2024-03-12T03:57:28.599393+0000","last_clean":"2024-03-12T03:57:28.599393+0000","last_became_active":"2024-03-12T03:55:36.583655+0000","last_became_peered":"2024-03-12T03:55:36.583655+0000","last_unstale":"2024-03-12T03:57:28.599393+0000","last_undegraded":"2024-03-12T03:57:28.599393+0000","last_fullsized":"2024-03-12T03:57:28.599393+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":4,"last_scrub":"388'129","last_scrub_stamp":"2024-03-12T03:56:14.471428+0000","last_deep_scrub":"388'129","last_deep_scrub_stamp":"2024-03-12T03:49:17.168603+0000","last_clean_scrub_stamp":"2024-03-12T03:56:14.471428+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1050752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"480'39","reported_seq":701,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.900202+0000","last_change":"2024-03-12T03:57:04.552469+0000","last_active":"2024-03-12T03:57:28.900202+0000","last_peered":"2024-03-12T03:57:28.900202+0000","last_clean":"2024-03-12T03:57:28.900202+0000","last_became_active":"2024-03-12T03:57:01.702972+0000","last_became_peered":"2024-03-12T03:57:01.702972+0000","last_unstale":"2024-03-12T03:57:28.900202+0000","last_undegraded":"2024-03-12T03:57:28.900202+0000","last_fullsized":"2024-03-12T03:57:28.900202+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:57:04.552347+0000","last_deep_scrub":"95'23","last_deep_scrub_stamp":"2024-03-12T03:35:06.028013+0000","last_clean_scrub_stamp":"2024-03-12T03:57:04.552347+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:24.574091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1168,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898702+0000","last_change":"2024-03-12T03:56:54.585455+0000","last_active":"2024-03-12T03:57:28.898702+0000","last_peered":"2024-03-12T03:57:28.898702+0000","last_clean":"2024-03-12T03:57:28.898702+0000","last_became_active":"2024-03-12T03:39:33.264661+0000","last_became_peered":"2024-03-12T03:39:33.264661+0000","last_unstale":"2024-03-12T03:57:28.898702+0000","last_undegraded":"2024-03-12T03:57:28.898702+0000","last_fullsized":"2024-03-12T03:57:28.898702+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:54.585345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:12.928199+0000","last_clean_scrub_stamp":"2024-03-12T03:56:54.585345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:06.172179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":153,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.601271+0000","last_change":"2024-03-12T03:57:21.505521+0000","last_active":"2024-03-12T03:57:28.601271+0000","last_peered":"2024-03-12T03:57:28.601271+0000","last_clean":"2024-03-12T03:57:28.601271+0000","last_became_active":"2024-03-12T03:55:33.603297+0000","last_became_peered":"2024-03-12T03:55:33.603297+0000","last_unstale":"2024-03-12T03:57:28.601271+0000","last_undegraded":"2024-03-12T03:57:28.601271+0000","last_fullsized":"2024-03-12T03:57:28.601271+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:21.505391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:57:21.505391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:24.482071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":937,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899726+0000","last_change":"2024-03-12T03:55:59.072404+0000","last_active":"2024-03-12T03:57:28.899726+0000","last_peered":"2024-03-12T03:57:28.899726+0000","last_clean":"2024-03-12T03:57:28.899726+0000","last_became_active":"2024-03-12T03:55:33.592064+0000","last_became_peered":"2024-03-12T03:55:33.592064+0000","last_unstale":"2024-03-12T03:57:28.899726+0000","last_undegraded":"2024-03-12T03:57:28.899726+0000","last_fullsized":"2024-03-12T03:57:28.899726+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:55:59.072302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:55:59.072302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":641,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599617+0000","last_change":"2024-03-12T03:57:26.629490+0000","last_active":"2024-03-12T03:57:28.599617+0000","last_peered":"2024-03-12T03:57:28.599617+0000","last_clean":"2024-03-12T03:57:28.599617+0000","last_became_active":"2024-03-12T03:55:36.584043+0000","last_became_peered":"2024-03-12T03:55:36.584043+0000","last_unstale":"2024-03-12T03:57:28.599617+0000","last_undegraded":"2024-03-12T03:57:28.599617+0000","last_fullsized":"2024-03-12T03:57:28.599617+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:57:26.629393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:57:26.629393+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:38.494636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"480'39","reported_seq":1018,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.911191+0000","last_change":"2024-03-12T03:56:05.475015+0000","last_active":"2024-03-12T03:57:26.911191+0000","last_peered":"2024-03-12T03:57:26.911191+0000","last_clean":"2024-03-12T03:57:26.911191+0000","last_became_active":"2024-03-12T03:46:25.287373+0000","last_became_peered":"2024-03-12T03:46:25.287373+0000","last_unstale":"2024-03-12T03:57:26.911191+0000","last_undegraded":"2024-03-12T03:57:26.911191+0000","last_fullsized":"2024-03-12T03:57:26.911191+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:56:05.474875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:05.474875+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:30.806764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":88,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898556+0000","last_change":"2024-03-12T03:56:39.478674+0000","last_active":"2024-03-12T03:57:28.898556+0000","last_peered":"2024-03-12T03:57:28.898556+0000","last_clean":"2024-03-12T03:57:28.898556+0000","last_became_active":"2024-03-12T03:56:10.594485+0000","last_became_peered":"2024-03-12T03:56:10.594485+0000","last_unstale":"2024-03-12T03:57:28.898556+0000","last_undegraded":"2024-03-12T03:57:28.898556+0000","last_fullsized":"2024-03-12T03:57:28.898556+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:39.478583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:39.478583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:54.448383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"561'16","reported_seq":1136,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.900283+0000","last_change":"2024-03-12T03:56:28.303821+0000","last_active":"2024-03-12T03:57:28.900283+0000","last_peered":"2024-03-12T03:57:28.900283+0000","last_clean":"2024-03-12T03:57:28.900283+0000","last_became_active":"2024-03-12T03:56:10.598971+0000","last_became_peered":"2024-03-12T03:56:10.598971+0000","last_unstale":"2024-03-12T03:57:28.900283+0000","last_undegraded":"2024-03-12T03:57:28.900283+0000","last_fullsized":"2024-03-12T03:57:28.900283+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"561'16","last_scrub_stamp":"2024-03-12T03:56:28.303716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:28.303716+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:35.829965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":145,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600205+0000","last_change":"2024-03-12T03:56:55.718648+0000","last_active":"2024-03-12T03:57:28.600205+0000","last_peered":"2024-03-12T03:57:28.600205+0000","last_clean":"2024-03-12T03:57:28.600205+0000","last_became_active":"2024-03-12T03:55:33.598673+0000","last_became_peered":"2024-03-12T03:55:33.598673+0000","last_unstale":"2024-03-12T03:57:28.600205+0000","last_undegraded":"2024-03-12T03:57:28.600205+0000","last_fullsized":"2024-03-12T03:57:28.600205+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:55.718508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:55.718508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:17.770072+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1150,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899633+0000","last_change":"2024-03-12T03:56:08.360269+0000","last_active":"2024-03-12T03:57:28.899633+0000","last_peered":"2024-03-12T03:57:28.899633+0000","last_clean":"2024-03-12T03:57:28.899633+0000","last_became_active":"2024-03-12T03:46:25.286985+0000","last_became_peered":"2024-03-12T03:46:25.286985+0000","last_unstale":"2024-03-12T03:57:28.899633+0000","last_undegraded":"2024-03-12T03:57:28.899633+0000","last_fullsized":"2024-03-12T03:57:28.899633+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:08.360125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:08.360125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"480'54","reported_seq":1038,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898629+0000","last_change":"2024-03-12T03:56:53.075930+0000","last_active":"2024-03-12T03:57:28.898629+0000","last_peered":"2024-03-12T03:57:28.898629+0000","last_clean":"2024-03-12T03:57:28.898629+0000","last_became_active":"2024-03-12T03:55:21.699982+0000","last_became_peered":"2024-03-12T03:55:21.699982+0000","last_unstale":"2024-03-12T03:57:28.898629+0000","last_undegraded":"2024-03-12T03:57:28.898629+0000","last_fullsized":"2024-03-12T03:57:28.898629+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'54","last_scrub_stamp":"2024-03-12T03:56:53.075818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:53.075818+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:03.040971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":86,"num_read_kb":72,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":921,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899480+0000","last_change":"2024-03-12T03:56:51.027613+0000","last_active":"2024-03-12T03:57:28.899480+0000","last_peered":"2024-03-12T03:57:28.899480+0000","last_clean":"2024-03-12T03:57:28.899480+0000","last_became_active":"2024-03-12T03:55:21.699521+0000","last_became_peered":"2024-03-12T03:55:21.699521+0000","last_unstale":"2024-03-12T03:57:28.899480+0000","last_undegraded":"2024-03-12T03:57:28.899480+0000","last_fullsized":"2024-03-12T03:57:28.899480+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:51.027531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:55:42.078673+0000","last_clean_scrub_stamp":"2024-03-12T03:56:51.027531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:58.955126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":914,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899082+0000","last_change":"2024-03-12T03:56:12.307612+0000","last_active":"2024-03-12T03:57:28.899082+0000","last_peered":"2024-03-12T03:57:28.899082+0000","last_clean":"2024-03-12T03:57:28.899082+0000","last_became_active":"2024-03-12T03:56:10.591396+0000","last_became_peered":"2024-03-12T03:56:10.591396+0000","last_unstale":"2024-03-12T03:57:28.899082+0000","last_undegraded":"2024-03-12T03:57:28.899082+0000","last_fullsized":"2024-03-12T03:57:28.899082+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:12.307536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:46:56.092011+0000","last_clean_scrub_stamp":"2024-03-12T03:56:12.307536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":1332,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899879+0000","last_change":"2024-03-12T03:56:41.481708+0000","last_active":"2024-03-12T03:57:28.899879+0000","last_peered":"2024-03-12T03:57:28.899879+0000","last_clean":"2024-03-12T03:57:28.899879+0000","last_became_active":"2024-03-12T03:55:57.919888+0000","last_became_peered":"2024-03-12T03:55:57.919888+0000","last_unstale":"2024-03-12T03:57:28.899879+0000","last_undegraded":"2024-03-12T03:57:28.899879+0000","last_fullsized":"2024-03-12T03:57:28.899879+0000","mapping_epoch":200,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":555,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:41.481584+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:48:15.895638+0000","last_clean_scrub_stamp":"2024-03-12T03:56:41.481584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:49.341253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"501'13","reported_seq":1211,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898578+0000","last_change":"2024-03-12T03:56:25.043446+0000","last_active":"2024-03-12T03:57:28.898578+0000","last_peered":"2024-03-12T03:57:28.898578+0000","last_clean":"2024-03-12T03:57:28.898578+0000","last_became_active":"2024-03-12T03:46:25.288674+0000","last_became_peered":"2024-03-12T03:46:25.288674+0000","last_unstale":"2024-03-12T03:57:28.898578+0000","last_undegraded":"2024-03-12T03:57:28.898578+0000","last_fullsized":"2024-03-12T03:57:28.898578+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'13","last_scrub_stamp":"2024-03-12T03:56:25.043305+0000","last_deep_scrub":"501'13","last_deep_scrub_stamp":"2024-03-12T03:56:25.043305+0000","last_clean_scrub_stamp":"2024-03-12T03:56:25.043305+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:42.133137+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"480'56","reported_seq":736,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898939+0000","last_change":"2024-03-12T03:56:10.593554+0000","last_active":"2024-03-12T03:57:28.898939+0000","last_peered":"2024-03-12T03:57:28.898939+0000","last_clean":"2024-03-12T03:57:28.898939+0000","last_became_active":"2024-03-12T03:56:10.593076+0000","last_became_peered":"2024-03-12T03:56:10.593076+0000","last_unstale":"2024-03-12T03:57:28.898939+0000","last_undegraded":"2024-03-12T03:57:28.898939+0000","last_fullsized":"2024-03-12T03:57:28.898939+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'56","last_scrub_stamp":"2024-03-12T03:55:53.492164+0000","last_deep_scrub":"200'28","last_deep_scrub_stamp":"2024-03-12T03:39:36.078165+0000","last_clean_scrub_stamp":"2024-03-12T03:55:53.492164+0000","objects_scrubbed":8,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":62,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"320'2","reported_seq":823,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600490+0000","last_change":"2024-03-12T03:57:08.510133+0000","last_active":"2024-03-12T03:57:28.600490+0000","last_peered":"2024-03-12T03:57:28.600490+0000","last_clean":"2024-03-12T03:57:28.600490+0000","last_became_active":"2024-03-12T03:55:21.706162+0000","last_became_peered":"2024-03-12T03:55:21.706162+0000","last_unstale":"2024-03-12T03:57:28.600490+0000","last_undegraded":"2024-03-12T03:57:28.600490+0000","last_fullsized":"2024-03-12T03:57:28.600490+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'2","last_scrub_stamp":"2024-03-12T03:57:08.510029+0000","last_deep_scrub":"320'2","last_deep_scrub_stamp":"2024-03-12T03:57:08.510029+0000","last_clean_scrub_stamp":"2024-03-12T03:57:08.510029+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:27.698894+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1154,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899904+0000","last_change":"2024-03-12T03:57:10.342740+0000","last_active":"2024-03-12T03:57:28.899904+0000","last_peered":"2024-03-12T03:57:28.899904+0000","last_clean":"2024-03-12T03:57:28.899904+0000","last_became_active":"2024-03-12T03:39:33.264760+0000","last_became_peered":"2024-03-12T03:39:33.264760+0000","last_unstale":"2024-03-12T03:57:28.899904+0000","last_undegraded":"2024-03-12T03:57:28.899904+0000","last_fullsized":"2024-03-12T03:57:28.899904+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:10.342625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:49:54.471078+0000","last_clean_scrub_stamp":"2024-03-12T03:57:10.342625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:34.149055+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":1174,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899331+0000","last_change":"2024-03-12T03:56:44.141366+0000","last_active":"2024-03-12T03:57:28.899331+0000","last_peered":"2024-03-12T03:57:28.899331+0000","last_clean":"2024-03-12T03:57:28.899331+0000","last_became_active":"2024-03-12T03:56:44.141085+0000","last_became_peered":"2024-03-12T03:56:44.141085+0000","last_unstale":"2024-03-12T03:57:28.899331+0000","last_undegraded":"2024-03-12T03:57:28.899331+0000","last_fullsized":"2024-03-12T03:57:28.899331+0000","mapping_epoch":574,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:37.088818+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:47:11.313179+0000","last_clean_scrub_stamp":"2024-03-12T03:56:37.088818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:53.781815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"534'14","reported_seq":1056,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600225+0000","last_change":"2024-03-12T03:56:50.346734+0000","last_active":"2024-03-12T03:57:28.600225+0000","last_peered":"2024-03-12T03:57:28.600225+0000","last_clean":"2024-03-12T03:57:28.600225+0000","last_became_active":"2024-03-12T03:55:33.590974+0000","last_became_peered":"2024-03-12T03:55:33.590974+0000","last_unstale":"2024-03-12T03:57:28.600225+0000","last_undegraded":"2024-03-12T03:57:28.600225+0000","last_fullsized":"2024-03-12T03:57:28.600225+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'14","last_scrub_stamp":"2024-03-12T03:56:50.346128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:50.346128+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:05.737400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"480'36","reported_seq":1249,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910789+0000","last_change":"2024-03-12T03:56:09.526675+0000","last_active":"2024-03-12T03:57:26.910789+0000","last_peered":"2024-03-12T03:57:26.910789+0000","last_clean":"2024-03-12T03:57:26.910789+0000","last_became_active":"2024-03-12T03:55:33.603004+0000","last_became_peered":"2024-03-12T03:55:33.603004+0000","last_unstale":"2024-03-12T03:57:26.910789+0000","last_undegraded":"2024-03-12T03:57:26.910789+0000","last_fullsized":"2024-03-12T03:57:26.910789+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'36","last_scrub_stamp":"2024-03-12T03:56:09.526562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:09.526562+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:32.351498+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":145,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600368+0000","last_change":"2024-03-12T03:56:15.488701+0000","last_active":"2024-03-12T03:57:28.600368+0000","last_peered":"2024-03-12T03:57:28.600368+0000","last_clean":"2024-03-12T03:57:28.600368+0000","last_became_active":"2024-03-12T03:55:33.599234+0000","last_became_peered":"2024-03-12T03:55:33.599234+0000","last_unstale":"2024-03-12T03:57:28.600368+0000","last_undegraded":"2024-03-12T03:57:28.600368+0000","last_fullsized":"2024-03-12T03:57:28.600368+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:15.488604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:54:40.122207+0000","last_clean_scrub_stamp":"2024-03-12T03:56:15.488604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":809,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600001+0000","last_change":"2024-03-12T03:57:10.590959+0000","last_active":"2024-03-12T03:57:28.600001+0000","last_peered":"2024-03-12T03:57:28.600001+0000","last_clean":"2024-03-12T03:57:28.600001+0000","last_became_active":"2024-03-12T03:55:21.708570+0000","last_became_peered":"2024-03-12T03:55:21.708570+0000","last_unstale":"2024-03-12T03:57:28.600001+0000","last_undegraded":"2024-03-12T03:57:28.600001+0000","last_fullsized":"2024-03-12T03:57:28.600001+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:10.590876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:10.590876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:13.222979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":756,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.601442+0000","last_change":"2024-03-12T03:57:26.305933+0000","last_active":"2024-03-12T03:57:28.601442+0000","last_peered":"2024-03-12T03:57:28.601442+0000","last_clean":"2024-03-12T03:57:28.601442+0000","last_became_active":"2024-03-12T03:56:23.595099+0000","last_became_peered":"2024-03-12T03:56:23.595099+0000","last_unstale":"2024-03-12T03:57:28.601442+0000","last_undegraded":"2024-03-12T03:57:28.601442+0000","last_fullsized":"2024-03-12T03:57:28.601442+0000","mapping_epoch":568,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:57:26.305830+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:56:25.421410+0000","last_clean_scrub_stamp":"2024-03-12T03:57:26.305830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:28.545922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":621,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.601267+0000","last_change":"2024-03-12T03:56:39.402275+0000","last_active":"2024-03-12T03:57:28.601267+0000","last_peered":"2024-03-12T03:57:28.601267+0000","last_clean":"2024-03-12T03:57:28.601267+0000","last_became_active":"2024-03-12T03:56:23.588475+0000","last_became_peered":"2024-03-12T03:56:23.588475+0000","last_unstale":"2024-03-12T03:57:28.601267+0000","last_undegraded":"2024-03-12T03:57:28.601267+0000","last_fullsized":"2024-03-12T03:57:28.601267+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:39.402174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.893865+0000","last_clean_scrub_stamp":"2024-03-12T03:56:39.402174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:50.792838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"480'45","reported_seq":909,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600380+0000","last_change":"2024-03-12T03:56:33.463204+0000","last_active":"2024-03-12T03:57:28.600380+0000","last_peered":"2024-03-12T03:57:28.600380+0000","last_clean":"2024-03-12T03:57:28.600380+0000","last_became_active":"2024-03-12T03:55:33.594248+0000","last_became_peered":"2024-03-12T03:55:33.594248+0000","last_unstale":"2024-03-12T03:57:28.600380+0000","last_undegraded":"2024-03-12T03:57:28.600380+0000","last_fullsized":"2024-03-12T03:57:28.600380+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'45","last_scrub_stamp":"2024-03-12T03:56:33.463059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:33.463059+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:50.125553+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":61,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1188,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910243+0000","last_change":"2024-03-12T03:57:04.227889+0000","last_active":"2024-03-12T03:57:26.910243+0000","last_peered":"2024-03-12T03:57:26.910243+0000","last_clean":"2024-03-12T03:57:26.910243+0000","last_became_active":"2024-03-12T03:55:33.599159+0000","last_became_peered":"2024-03-12T03:55:33.599159+0000","last_unstale":"2024-03-12T03:57:26.910243+0000","last_undegraded":"2024-03-12T03:57:26.910243+0000","last_fullsized":"2024-03-12T03:57:26.910243+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:04.227792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:40:32.631336+0000","last_clean_scrub_stamp":"2024-03-12T03:57:04.227792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:20.574992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1153,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909819+0000","last_change":"2024-03-12T03:56:54.348787+0000","last_active":"2024-03-12T03:57:26.909819+0000","last_peered":"2024-03-12T03:57:26.909819+0000","last_clean":"2024-03-12T03:57:26.909819+0000","last_became_active":"2024-03-12T03:39:33.290809+0000","last_became_peered":"2024-03-12T03:39:33.290809+0000","last_unstale":"2024-03-12T03:57:26.909819+0000","last_undegraded":"2024-03-12T03:57:26.909819+0000","last_fullsized":"2024-03-12T03:57:26.909819+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:54.348690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:55:35.758421+0000","last_clean_scrub_stamp":"2024-03-12T03:56:54.348690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:08.111339+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":756,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:31.402603+0000","last_change":"2024-03-12T03:57:31.402633+0000","last_active":"2024-03-12T03:57:31.402603+0000","last_peered":"2024-03-12T03:57:31.402603+0000","last_clean":"2024-03-12T03:57:31.402603+0000","last_became_active":"2024-03-12T03:56:23.588993+0000","last_became_peered":"2024-03-12T03:56:23.588993+0000","last_unstale":"2024-03-12T03:57:31.402603+0000","last_undegraded":"2024-03-12T03:57:31.402603+0000","last_fullsized":"2024-03-12T03:57:31.402603+0000","mapping_epoch":568,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:57:31.402524+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:57:31.402524+0000","last_clean_scrub_stamp":"2024-03-12T03:57:31.402524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:38.667139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":145,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.601376+0000","last_change":"2024-03-12T03:56:30.427738+0000","last_active":"2024-03-12T03:57:28.601376+0000","last_peered":"2024-03-12T03:57:28.601376+0000","last_clean":"2024-03-12T03:57:28.601376+0000","last_became_active":"2024-03-12T03:55:33.594941+0000","last_became_peered":"2024-03-12T03:55:33.594941+0000","last_unstale":"2024-03-12T03:57:28.601376+0000","last_undegraded":"2024-03-12T03:57:28.601376+0000","last_fullsized":"2024-03-12T03:57:28.601376+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:30.427669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:01.257725+0000","last_clean_scrub_stamp":"2024-03-12T03:56:30.427669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:38.116057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"480'36","reported_seq":928,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600636+0000","last_change":"2024-03-12T03:56:15.547771+0000","last_active":"2024-03-12T03:57:28.600636+0000","last_peered":"2024-03-12T03:57:28.600636+0000","last_clean":"2024-03-12T03:57:28.600636+0000","last_became_active":"2024-03-12T03:55:21.704595+0000","last_became_peered":"2024-03-12T03:55:21.704595+0000","last_unstale":"2024-03-12T03:57:28.600636+0000","last_undegraded":"2024-03-12T03:57:28.600636+0000","last_fullsized":"2024-03-12T03:57:28.600636+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'36","last_scrub_stamp":"2024-03-12T03:56:15.547648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:15.547648+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:41.592339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"288'6","reported_seq":2602,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:29.901528+0000","last_change":"2024-03-12T03:56:44.159681+0000","last_active":"2024-03-12T03:57:29.901528+0000","last_peered":"2024-03-12T03:57:29.901528+0000","last_clean":"2024-03-12T03:57:29.901528+0000","last_became_active":"2024-03-12T03:55:21.696299+0000","last_became_peered":"2024-03-12T03:55:21.696299+0000","last_unstale":"2024-03-12T03:57:29.901528+0000","last_undegraded":"2024-03-12T03:57:29.901528+0000","last_fullsized":"2024-03-12T03:57:29.901528+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6","last_scrub_stamp":"2024-03-12T03:56:44.159579+0000","last_deep_scrub":"288'6","last_deep_scrub_stamp":"2024-03-12T03:44:11.061362+0000","last_clean_scrub_stamp":"2024-03-12T03:56:44.159579+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:03.675553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027317629999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1439,"num_read_kb":1434,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":842,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599619+0000","last_change":"2024-03-12T03:57:15.593193+0000","last_active":"2024-03-12T03:57:28.599619+0000","last_peered":"2024-03-12T03:57:28.599619+0000","last_clean":"2024-03-12T03:57:28.599619+0000","last_became_active":"2024-03-12T03:55:21.707906+0000","last_became_peered":"2024-03-12T03:55:21.707906+0000","last_unstale":"2024-03-12T03:57:28.599619+0000","last_undegraded":"2024-03-12T03:57:28.599619+0000","last_fullsized":"2024-03-12T03:57:28.599619+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:15.593111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:42:21.445443+0000","last_clean_scrub_stamp":"2024-03-12T03:57:15.593111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:43.830117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1170,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910049+0000","last_change":"2024-03-12T03:57:23.999876+0000","last_active":"2024-03-12T03:57:26.910049+0000","last_peered":"2024-03-12T03:57:26.910049+0000","last_clean":"2024-03-12T03:57:26.910049+0000","last_became_active":"2024-03-12T03:55:33.597111+0000","last_became_peered":"2024-03-12T03:55:33.597111+0000","last_unstale":"2024-03-12T03:57:26.910049+0000","last_undegraded":"2024-03-12T03:57:26.910049+0000","last_fullsized":"2024-03-12T03:57:26.910049+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:23.999783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:57:23.999783+0000","last_clean_scrub_stamp":"2024-03-12T03:57:23.999783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:43.215614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1313,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909935+0000","last_change":"2024-03-12T03:56:48.152070+0000","last_active":"2024-03-12T03:57:26.909935+0000","last_peered":"2024-03-12T03:57:26.909935+0000","last_clean":"2024-03-12T03:57:26.909935+0000","last_became_active":"2024-03-12T03:56:48.151784+0000","last_became_peered":"2024-03-12T03:56:48.151784+0000","last_unstale":"2024-03-12T03:57:26.909935+0000","last_undegraded":"2024-03-12T03:57:26.909935+0000","last_fullsized":"2024-03-12T03:57:26.909935+0000","mapping_epoch":154,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":584,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:06.515586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:56:06.515586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1166,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899226+0000","last_change":"2024-03-12T03:56:34.028265+0000","last_active":"2024-03-12T03:57:28.899226+0000","last_peered":"2024-03-12T03:57:28.899226+0000","last_clean":"2024-03-12T03:57:28.899226+0000","last_became_active":"2024-03-12T03:55:33.606416+0000","last_became_peered":"2024-03-12T03:55:33.606416+0000","last_unstale":"2024-03-12T03:57:28.899226+0000","last_undegraded":"2024-03-12T03:57:28.899226+0000","last_fullsized":"2024-03-12T03:57:28.899226+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:34.028165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:54:38.655865+0000","last_clean_scrub_stamp":"2024-03-12T03:56:34.028165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:36.072198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"569'42","reported_seq":1191,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:30.377162+0000","last_change":"2024-03-12T03:57:30.377206+0000","last_active":"2024-03-12T03:57:30.377162+0000","last_peered":"2024-03-12T03:57:30.377162+0000","last_clean":"2024-03-12T03:57:30.377162+0000","last_became_active":"2024-03-12T03:56:23.594618+0000","last_became_peered":"2024-03-12T03:56:23.594618+0000","last_unstale":"2024-03-12T03:57:30.377162+0000","last_undegraded":"2024-03-12T03:57:30.377162+0000","last_fullsized":"2024-03-12T03:57:30.377162+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'42","last_scrub_stamp":"2024-03-12T03:57:30.377083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:30.377083+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:32.019154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":931,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898722+0000","last_change":"2024-03-12T03:56:28.079262+0000","last_active":"2024-03-12T03:57:28.898722+0000","last_peered":"2024-03-12T03:57:28.898722+0000","last_clean":"2024-03-12T03:57:28.898722+0000","last_became_active":"2024-03-12T03:55:33.583727+0000","last_became_peered":"2024-03-12T03:55:33.583727+0000","last_unstale":"2024-03-12T03:57:28.898722+0000","last_undegraded":"2024-03-12T03:57:28.898722+0000","last_fullsized":"2024-03-12T03:57:28.898722+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:28.079184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:28.079184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:35.089911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1456,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898582+0000","last_change":"2024-03-12T03:56:54.242374+0000","last_active":"2024-03-12T03:57:28.898582+0000","last_peered":"2024-03-12T03:57:28.898582+0000","last_clean":"2024-03-12T03:57:28.898582+0000","last_became_active":"2024-03-12T03:55:45.582168+0000","last_became_peered":"2024-03-12T03:55:45.582168+0000","last_unstale":"2024-03-12T03:57:28.898582+0000","last_undegraded":"2024-03-12T03:57:28.898582+0000","last_fullsized":"2024-03-12T03:57:28.898582+0000","mapping_epoch":521,"log_start":"83'128","ondisk_log_start":"83'128","created":20,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:54.242270+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-12T03:55:27.740894+0000","last_clean_scrub_stamp":"2024-03-12T03:56:54.242270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:20.112391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":153,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.601466+0000","last_change":"2024-03-12T03:57:07.689399+0000","last_active":"2024-03-12T03:57:28.601466+0000","last_peered":"2024-03-12T03:57:28.601466+0000","last_clean":"2024-03-12T03:57:28.601466+0000","last_became_active":"2024-03-12T03:55:33.594123+0000","last_became_peered":"2024-03-12T03:55:33.594123+0000","last_unstale":"2024-03-12T03:57:28.601466+0000","last_undegraded":"2024-03-12T03:57:28.601466+0000","last_fullsized":"2024-03-12T03:57:28.601466+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:07.689305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:07.689305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:33.080629+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":897,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910170+0000","last_change":"2024-03-12T03:57:01.702738+0000","last_active":"2024-03-12T03:57:26.910170+0000","last_peered":"2024-03-12T03:57:26.910170+0000","last_clean":"2024-03-12T03:57:26.910170+0000","last_became_active":"2024-03-12T03:55:33.605033+0000","last_became_peered":"2024-03-12T03:55:33.605033+0000","last_unstale":"2024-03-12T03:57:26.910170+0000","last_undegraded":"2024-03-12T03:57:26.910170+0000","last_fullsized":"2024-03-12T03:57:26.910170+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:01.702666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:03.517277+0000","last_clean_scrub_stamp":"2024-03-12T03:57:01.702666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:10.606822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"480'27","reported_seq":1237,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.911120+0000","last_change":"2024-03-12T03:57:01.294235+0000","last_active":"2024-03-12T03:57:26.911120+0000","last_peered":"2024-03-12T03:57:26.911120+0000","last_clean":"2024-03-12T03:57:26.911120+0000","last_became_active":"2024-03-12T03:55:33.597995+0000","last_became_peered":"2024-03-12T03:55:33.597995+0000","last_unstale":"2024-03-12T03:57:26.911120+0000","last_undegraded":"2024-03-12T03:57:26.911120+0000","last_fullsized":"2024-03-12T03:57:26.911120+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'27","last_scrub_stamp":"2024-03-12T03:57:01.294138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:01.294138+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:25.799015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":872,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910663+0000","last_change":"2024-03-12T03:56:46.662889+0000","last_active":"2024-03-12T03:57:26.910663+0000","last_peered":"2024-03-12T03:57:26.910663+0000","last_clean":"2024-03-12T03:57:26.910663+0000","last_became_active":"2024-03-12T03:55:21.695439+0000","last_became_peered":"2024-03-12T03:55:21.695439+0000","last_unstale":"2024-03-12T03:57:26.910663+0000","last_undegraded":"2024-03-12T03:57:26.910663+0000","last_fullsized":"2024-03-12T03:57:26.910663+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:46.662767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:19.077015+0000","last_clean_scrub_stamp":"2024-03-12T03:56:46.662767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:57.506544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1173,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910064+0000","last_change":"2024-03-12T03:57:11.174578+0000","last_active":"2024-03-12T03:57:26.910064+0000","last_peered":"2024-03-12T03:57:26.910064+0000","last_clean":"2024-03-12T03:57:26.910064+0000","last_became_active":"2024-03-12T03:39:33.291327+0000","last_became_peered":"2024-03-12T03:39:33.291327+0000","last_unstale":"2024-03-12T03:57:26.910064+0000","last_undegraded":"2024-03-12T03:57:26.910064+0000","last_fullsized":"2024-03-12T03:57:26.910064+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:11.174451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:31.765257+0000","last_clean_scrub_stamp":"2024-03-12T03:57:11.174451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:36.638785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"90'2","reported_seq":1154,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899297+0000","last_change":"2024-03-12T03:56:09.083821+0000","last_active":"2024-03-12T03:57:28.899297+0000","last_peered":"2024-03-12T03:57:28.899297+0000","last_clean":"2024-03-12T03:57:28.899297+0000","last_became_active":"2024-03-12T03:46:25.284168+0000","last_became_peered":"2024-03-12T03:46:25.284168+0000","last_unstale":"2024-03-12T03:57:28.899297+0000","last_undegraded":"2024-03-12T03:57:28.899297+0000","last_fullsized":"2024-03-12T03:57:28.899297+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-12T03:56:09.083707+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-12T03:49:18.522103+0000","last_clean_scrub_stamp":"2024-03-12T03:56:09.083707+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:38.582457+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1471,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899489+0000","last_change":"2024-03-12T03:56:40.160673+0000","last_active":"2024-03-12T03:57:28.899489+0000","last_peered":"2024-03-12T03:57:28.899489+0000","last_clean":"2024-03-12T03:57:28.899489+0000","last_became_active":"2024-03-12T03:55:31.751073+0000","last_became_peered":"2024-03-12T03:55:31.751073+0000","last_unstale":"2024-03-12T03:57:28.899489+0000","last_undegraded":"2024-03-12T03:57:28.899489+0000","last_fullsized":"2024-03-12T03:57:28.899489+0000","mapping_epoch":521,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:40.160527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:56:40.160527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:56.297858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":789,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600432+0000","last_change":"2024-03-12T03:56:53.379669+0000","last_active":"2024-03-12T03:57:28.600432+0000","last_peered":"2024-03-12T03:57:28.600432+0000","last_clean":"2024-03-12T03:57:28.600432+0000","last_became_active":"2024-03-12T03:55:21.699805+0000","last_became_peered":"2024-03-12T03:55:21.699805+0000","last_unstale":"2024-03-12T03:57:28.600432+0000","last_undegraded":"2024-03-12T03:57:28.600432+0000","last_fullsized":"2024-03-12T03:57:28.600432+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:53.379580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:53.379580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:01.283519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"480'66","reported_seq":617,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909988+0000","last_change":"2024-03-12T03:57:13.124386+0000","last_active":"2024-03-12T03:57:26.909988+0000","last_peered":"2024-03-12T03:57:26.909988+0000","last_clean":"2024-03-12T03:57:26.909988+0000","last_became_active":"2024-03-12T03:48:48.462111+0000","last_became_peered":"2024-03-12T03:48:48.462111+0000","last_unstale":"2024-03-12T03:57:26.909988+0000","last_undegraded":"2024-03-12T03:57:26.909988+0000","last_fullsized":"2024-03-12T03:57:26.909988+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'66","last_scrub_stamp":"2024-03-12T03:57:13.124253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:13.124253+0000","objects_scrubbed":11,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:18.729923+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":92,"num_read_kb":76,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1063,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899667+0000","last_change":"2024-03-12T03:57:06.566628+0000","last_active":"2024-03-12T03:57:28.899667+0000","last_peered":"2024-03-12T03:57:28.899667+0000","last_clean":"2024-03-12T03:57:28.899667+0000","last_became_active":"2024-03-12T03:56:10.597221+0000","last_became_peered":"2024-03-12T03:56:10.597221+0000","last_unstale":"2024-03-12T03:57:28.899667+0000","last_undegraded":"2024-03-12T03:57:28.899667+0000","last_fullsized":"2024-03-12T03:57:28.899667+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:06.566531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:57:06.566531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:08.850445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"591'3","reported_seq":158,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599517+0000","last_change":"2024-03-12T03:57:12.507635+0000","last_active":"2024-03-12T03:57:28.599517+0000","last_peered":"2024-03-12T03:57:28.599517+0000","last_clean":"2024-03-12T03:57:28.599517+0000","last_became_active":"2024-03-12T03:55:33.600104+0000","last_became_peered":"2024-03-12T03:55:33.600104+0000","last_unstale":"2024-03-12T03:57:28.599517+0000","last_undegraded":"2024-03-12T03:57:28.599517+0000","last_fullsized":"2024-03-12T03:57:28.599517+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"591'3","last_scrub_stamp":"2024-03-12T03:57:12.507496+0000","last_deep_scrub":"591'3","last_deep_scrub_stamp":"2024-03-12T03:57:12.507496+0000","last_clean_scrub_stamp":"2024-03-12T03:57:12.507496+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:24.442235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00053615100000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":959,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899448+0000","last_change":"2024-03-12T03:56:31.149845+0000","last_active":"2024-03-12T03:57:28.899448+0000","last_peered":"2024-03-12T03:57:28.899448+0000","last_clean":"2024-03-12T03:57:28.899448+0000","last_became_active":"2024-03-12T03:46:25.284416+0000","last_became_peered":"2024-03-12T03:46:25.284416+0000","last_unstale":"2024-03-12T03:57:28.899448+0000","last_undegraded":"2024-03-12T03:57:28.899448+0000","last_fullsized":"2024-03-12T03:57:28.899448+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:31.149736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:31.149736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:38.499883+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":498,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599442+0000","last_change":"2024-03-12T03:56:34.407728+0000","last_active":"2024-03-12T03:57:28.599442+0000","last_peered":"2024-03-12T03:57:28.599442+0000","last_clean":"2024-03-12T03:57:28.599442+0000","last_became_active":"2024-03-12T03:55:41.798428+0000","last_became_peered":"2024-03-12T03:55:41.798428+0000","last_unstale":"2024-03-12T03:57:28.599442+0000","last_undegraded":"2024-03-12T03:57:28.599442+0000","last_fullsized":"2024-03-12T03:57:28.599442+0000","mapping_epoch":533,"log_start":"83'128","ondisk_log_start":"83'128","created":149,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-12T03:56:34.407625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:31:05.352360+0000","last_clean_scrub_stamp":"2024-03-12T03:56:34.407625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:01.957883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"480'32","reported_seq":850,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600686+0000","last_change":"2024-03-12T03:56:45.357569+0000","last_active":"2024-03-12T03:57:28.600686+0000","last_peered":"2024-03-12T03:57:28.600686+0000","last_clean":"2024-03-12T03:57:28.600686+0000","last_became_active":"2024-03-12T03:55:21.711814+0000","last_became_peered":"2024-03-12T03:55:21.711814+0000","last_unstale":"2024-03-12T03:57:28.600686+0000","last_undegraded":"2024-03-12T03:57:28.600686+0000","last_fullsized":"2024-03-12T03:57:28.600686+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'32","last_scrub_stamp":"2024-03-12T03:56:45.357462+0000","last_deep_scrub":"111'20","last_deep_scrub_stamp":"2024-03-12T03:36:29.192310+0000","last_clean_scrub_stamp":"2024-03-12T03:56:45.357462+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:06.317851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":145,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600489+0000","last_change":"2024-03-12T03:56:16.473068+0000","last_active":"2024-03-12T03:57:28.600489+0000","last_peered":"2024-03-12T03:57:28.600489+0000","last_clean":"2024-03-12T03:57:28.600489+0000","last_became_active":"2024-03-12T03:55:33.600251+0000","last_became_peered":"2024-03-12T03:55:33.600251+0000","last_unstale":"2024-03-12T03:57:28.600489+0000","last_undegraded":"2024-03-12T03:57:28.600489+0000","last_fullsized":"2024-03-12T03:57:28.600489+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:16.472956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:48.160371+0000","last_clean_scrub_stamp":"2024-03-12T03:56:16.472956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":887,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910773+0000","last_change":"2024-03-12T03:56:42.622502+0000","last_active":"2024-03-12T03:57:26.910773+0000","last_peered":"2024-03-12T03:57:26.910773+0000","last_clean":"2024-03-12T03:57:26.910773+0000","last_became_active":"2024-03-12T03:46:25.284556+0000","last_became_peered":"2024-03-12T03:46:25.284556+0000","last_unstale":"2024-03-12T03:57:26.910773+0000","last_undegraded":"2024-03-12T03:57:26.910773+0000","last_fullsized":"2024-03-12T03:57:26.910773+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:42.622408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:55:40.638645+0000","last_clean_scrub_stamp":"2024-03-12T03:56:42.622408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:52.326675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":819,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599960+0000","last_change":"2024-03-12T03:56:46.353344+0000","last_active":"2024-03-12T03:57:28.599960+0000","last_peered":"2024-03-12T03:57:28.599960+0000","last_clean":"2024-03-12T03:57:28.599960+0000","last_became_active":"2024-03-12T03:55:21.704614+0000","last_became_peered":"2024-03-12T03:55:21.704614+0000","last_unstale":"2024-03-12T03:57:28.599960+0000","last_undegraded":"2024-03-12T03:57:28.599960+0000","last_fullsized":"2024-03-12T03:57:28.599960+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:46.353210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:46.353210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:05.838936+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"480'67","reported_seq":984,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.601157+0000","last_change":"2024-03-12T03:57:21.328416+0000","last_active":"2024-03-12T03:57:28.601157+0000","last_peered":"2024-03-12T03:57:28.601157+0000","last_clean":"2024-03-12T03:57:28.601157+0000","last_became_active":"2024-03-12T03:56:23.593912+0000","last_became_peered":"2024-03-12T03:56:23.593912+0000","last_unstale":"2024-03-12T03:57:28.601157+0000","last_undegraded":"2024-03-12T03:57:28.601157+0000","last_fullsized":"2024-03-12T03:57:28.601157+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'67","last_scrub_stamp":"2024-03-12T03:57:21.328261+0000","last_deep_scrub":"480'67","last_deep_scrub_stamp":"2024-03-12T03:57:21.328261+0000","last_clean_scrub_stamp":"2024-03-12T03:57:21.328261+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:28.167134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":75,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1165,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899122+0000","last_change":"2024-03-12T03:56:50.267570+0000","last_active":"2024-03-12T03:57:28.899122+0000","last_peered":"2024-03-12T03:57:28.899122+0000","last_clean":"2024-03-12T03:57:28.899122+0000","last_became_active":"2024-03-12T03:55:33.608753+0000","last_became_peered":"2024-03-12T03:55:33.608753+0000","last_unstale":"2024-03-12T03:57:28.899122+0000","last_undegraded":"2024-03-12T03:57:28.899122+0000","last_fullsized":"2024-03-12T03:57:28.899122+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:50.267429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:50.267429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:18.288518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":871,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909924+0000","last_change":"2024-03-12T03:56:30.638765+0000","last_active":"2024-03-12T03:57:26.909924+0000","last_peered":"2024-03-12T03:57:26.909924+0000","last_clean":"2024-03-12T03:57:26.909924+0000","last_became_active":"2024-03-12T03:55:33.605330+0000","last_became_peered":"2024-03-12T03:55:33.605330+0000","last_unstale":"2024-03-12T03:57:26.909924+0000","last_undegraded":"2024-03-12T03:57:26.909924+0000","last_fullsized":"2024-03-12T03:57:26.909924+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:30.638685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:30.638685+0000","last_clean_scrub_stamp":"2024-03-12T03:56:30.638685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:59.877289+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1164,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898457+0000","last_change":"2024-03-12T03:56:29.070314+0000","last_active":"2024-03-12T03:57:28.898457+0000","last_peered":"2024-03-12T03:57:28.898457+0000","last_clean":"2024-03-12T03:57:28.898457+0000","last_became_active":"2024-03-12T03:46:25.292122+0000","last_became_peered":"2024-03-12T03:46:25.292122+0000","last_unstale":"2024-03-12T03:57:28.898457+0000","last_undegraded":"2024-03-12T03:57:28.898457+0000","last_fullsized":"2024-03-12T03:57:28.898457+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:29.070214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:29.070214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:52.987816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":878,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909795+0000","last_change":"2024-03-12T03:56:41.632325+0000","last_active":"2024-03-12T03:57:26.909795+0000","last_peered":"2024-03-12T03:57:26.909795+0000","last_clean":"2024-03-12T03:57:26.909795+0000","last_became_active":"2024-03-12T03:55:21.696267+0000","last_became_peered":"2024-03-12T03:55:21.696267+0000","last_unstale":"2024-03-12T03:57:26.909795+0000","last_undegraded":"2024-03-12T03:57:26.909795+0000","last_fullsized":"2024-03-12T03:57:26.909795+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:41.632248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:41.632248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:02.586420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"480'36","reported_seq":697,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899363+0000","last_change":"2024-03-12T03:57:21.405755+0000","last_active":"2024-03-12T03:57:28.899363+0000","last_peered":"2024-03-12T03:57:28.899363+0000","last_clean":"2024-03-12T03:57:28.899363+0000","last_became_active":"2024-03-12T03:56:10.596698+0000","last_became_peered":"2024-03-12T03:56:10.596698+0000","last_unstale":"2024-03-12T03:57:28.899363+0000","last_undegraded":"2024-03-12T03:57:28.899363+0000","last_fullsized":"2024-03-12T03:57:28.899363+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'36","last_scrub_stamp":"2024-03-12T03:57:21.405624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:21.405624+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:23.762568+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":48,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1153,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899351+0000","last_change":"2024-03-12T03:56:07.020102+0000","last_active":"2024-03-12T03:57:28.899351+0000","last_peered":"2024-03-12T03:57:28.899351+0000","last_clean":"2024-03-12T03:57:28.899351+0000","last_became_active":"2024-03-12T03:33:21.105589+0000","last_became_peered":"2024-03-12T03:33:21.105589+0000","last_unstale":"2024-03-12T03:57:28.899351+0000","last_undegraded":"2024-03-12T03:57:28.899351+0000","last_fullsized":"2024-03-12T03:57:28.899351+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:07.020001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:07.020001+0000","last_clean_scrub_stamp":"2024-03-12T03:56:07.020001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:32.228800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":873,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899158+0000","last_change":"2024-03-12T03:56:12.268938+0000","last_active":"2024-03-12T03:57:28.899158+0000","last_peered":"2024-03-12T03:57:28.899158+0000","last_clean":"2024-03-12T03:57:28.899158+0000","last_became_active":"2024-03-12T03:46:25.280784+0000","last_became_peered":"2024-03-12T03:46:25.280784+0000","last_unstale":"2024-03-12T03:57:28.899158+0000","last_undegraded":"2024-03-12T03:57:28.899158+0000","last_fullsized":"2024-03-12T03:57:28.899158+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:12.268821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:12.268821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:30.062989+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"268'9","reported_seq":1445,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899912+0000","last_change":"2024-03-12T03:57:18.230392+0000","last_active":"2024-03-12T03:57:28.899912+0000","last_peered":"2024-03-12T03:57:28.899912+0000","last_clean":"2024-03-12T03:57:28.899912+0000","last_became_active":"2024-03-12T03:33:25.178250+0000","last_became_peered":"2024-03-12T03:33:25.178250+0000","last_unstale":"2024-03-12T03:57:28.899912+0000","last_undegraded":"2024-03-12T03:57:28.899912+0000","last_fullsized":"2024-03-12T03:57:28.899912+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9","last_scrub_stamp":"2024-03-12T03:57:18.230267+0000","last_deep_scrub":"268'9","last_deep_scrub_stamp":"2024-03-12T03:43:23.985362+0000","last_clean_scrub_stamp":"2024-03-12T03:57:18.230267+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:42.715131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"480'60","reported_seq":173,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599306+0000","last_change":"2024-03-12T03:57:16.473157+0000","last_active":"2024-03-12T03:57:28.599306+0000","last_peered":"2024-03-12T03:57:28.599306+0000","last_clean":"2024-03-12T03:57:28.599306+0000","last_became_active":"2024-03-12T03:55:33.602816+0000","last_became_peered":"2024-03-12T03:55:33.602816+0000","last_unstale":"2024-03-12T03:57:28.599306+0000","last_undegraded":"2024-03-12T03:57:28.599306+0000","last_fullsized":"2024-03-12T03:57:28.599306+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'60","last_scrub_stamp":"2024-03-12T03:57:16.473039+0000","last_deep_scrub":"90'36","last_deep_scrub_stamp":"2024-03-12T03:34:39.183984+0000","last_clean_scrub_stamp":"2024-03-12T03:57:16.473039+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:21.899462+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1150,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899597+0000","last_change":"2024-03-12T03:56:38.190361+0000","last_active":"2024-03-12T03:57:28.899597+0000","last_peered":"2024-03-12T03:57:28.899597+0000","last_clean":"2024-03-12T03:57:28.899597+0000","last_became_active":"2024-03-12T03:55:21.696530+0000","last_became_peered":"2024-03-12T03:55:21.696530+0000","last_unstale":"2024-03-12T03:57:28.899597+0000","last_undegraded":"2024-03-12T03:57:28.899597+0000","last_fullsized":"2024-03-12T03:57:28.899597+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-12T03:56:38.190211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:38.190211+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:47.569585+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":145,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599729+0000","last_change":"2024-03-12T03:56:36.388606+0000","last_active":"2024-03-12T03:57:28.599729+0000","last_peered":"2024-03-12T03:57:28.599729+0000","last_clean":"2024-03-12T03:57:28.599729+0000","last_became_active":"2024-03-12T03:55:33.603264+0000","last_became_peered":"2024-03-12T03:55:33.603264+0000","last_unstale":"2024-03-12T03:57:28.599729+0000","last_undegraded":"2024-03-12T03:57:28.599729+0000","last_fullsized":"2024-03-12T03:57:28.599729+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:36.388511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:36:20.147488+0000","last_clean_scrub_stamp":"2024-03-12T03:56:36.388511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:47.725284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"480'24","reported_seq":953,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899564+0000","last_change":"2024-03-12T03:56:19.085452+0000","last_active":"2024-03-12T03:57:28.899564+0000","last_peered":"2024-03-12T03:57:28.899564+0000","last_clean":"2024-03-12T03:57:28.899564+0000","last_became_active":"2024-03-12T03:46:25.288573+0000","last_became_peered":"2024-03-12T03:46:25.288573+0000","last_unstale":"2024-03-12T03:57:28.899564+0000","last_undegraded":"2024-03-12T03:57:28.899564+0000","last_fullsized":"2024-03-12T03:57:28.899564+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'24","last_scrub_stamp":"2024-03-12T03:56:19.085344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:19.085344+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:32.626679+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":629,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600238+0000","last_change":"2024-03-12T03:56:30.402057+0000","last_active":"2024-03-12T03:57:28.600238+0000","last_peered":"2024-03-12T03:57:28.600238+0000","last_clean":"2024-03-12T03:57:28.600238+0000","last_became_active":"2024-03-12T03:56:23.588721+0000","last_became_peered":"2024-03-12T03:56:23.588721+0000","last_unstale":"2024-03-12T03:57:28.600238+0000","last_undegraded":"2024-03-12T03:57:28.600238+0000","last_fullsized":"2024-03-12T03:57:28.600238+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:30.401941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:30.401941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:42.819705+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":145,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600750+0000","last_change":"2024-03-12T03:55:56.532593+0000","last_active":"2024-03-12T03:57:28.600750+0000","last_peered":"2024-03-12T03:57:28.600750+0000","last_clean":"2024-03-12T03:57:28.600750+0000","last_became_active":"2024-03-12T03:55:33.592131+0000","last_became_peered":"2024-03-12T03:55:33.592131+0000","last_unstale":"2024-03-12T03:57:28.600750+0000","last_undegraded":"2024-03-12T03:57:28.600750+0000","last_fullsized":"2024-03-12T03:57:28.600750+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:55:56.532493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:55:56.532493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1140,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898913+0000","last_change":"2024-03-12T03:56:58.369068+0000","last_active":"2024-03-12T03:57:28.898913+0000","last_peered":"2024-03-12T03:57:28.898913+0000","last_clean":"2024-03-12T03:57:28.898913+0000","last_became_active":"2024-03-12T03:33:27.383229+0000","last_became_peered":"2024-03-12T03:33:27.383229+0000","last_unstale":"2024-03-12T03:57:28.898913+0000","last_undegraded":"2024-03-12T03:57:28.898913+0000","last_fullsized":"2024-03-12T03:57:28.898913+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:58.368911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:51:38.361430+0000","last_clean_scrub_stamp":"2024-03-12T03:56:58.368911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:01.957783+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"480'33","reported_seq":907,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600598+0000","last_change":"2024-03-12T03:57:12.618050+0000","last_active":"2024-03-12T03:57:28.600598+0000","last_peered":"2024-03-12T03:57:28.600598+0000","last_clean":"2024-03-12T03:57:28.600598+0000","last_became_active":"2024-03-12T03:55:21.711074+0000","last_became_peered":"2024-03-12T03:55:21.711074+0000","last_unstale":"2024-03-12T03:57:28.600598+0000","last_undegraded":"2024-03-12T03:57:28.600598+0000","last_fullsized":"2024-03-12T03:57:28.600598+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'33","last_scrub_stamp":"2024-03-12T03:57:12.617956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:12.617956+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:30.621978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1145,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899724+0000","last_change":"2024-03-12T03:56:11.088835+0000","last_active":"2024-03-12T03:57:28.899724+0000","last_peered":"2024-03-12T03:57:28.899724+0000","last_clean":"2024-03-12T03:57:28.899724+0000","last_became_active":"2024-03-12T03:33:21.105808+0000","last_became_peered":"2024-03-12T03:33:21.105808+0000","last_unstale":"2024-03-12T03:57:28.899724+0000","last_undegraded":"2024-03-12T03:57:28.899724+0000","last_fullsized":"2024-03-12T03:57:28.899724+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:11.088730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:11.088730+0000","last_clean_scrub_stamp":"2024-03-12T03:56:11.088730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:38.835221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":907,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899702+0000","last_change":"2024-03-12T03:56:15.154769+0000","last_active":"2024-03-12T03:57:28.899702+0000","last_peered":"2024-03-12T03:57:28.899702+0000","last_clean":"2024-03-12T03:57:28.899702+0000","last_became_active":"2024-03-12T03:46:25.286866+0000","last_became_peered":"2024-03-12T03:46:25.286866+0000","last_unstale":"2024-03-12T03:57:28.899702+0000","last_undegraded":"2024-03-12T03:57:28.899702+0000","last_fullsized":"2024-03-12T03:57:28.899702+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:15.154653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:35:04.007252+0000","last_clean_scrub_stamp":"2024-03-12T03:56:15.154653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":876,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910682+0000","last_change":"2024-03-12T03:56:28.668411+0000","last_active":"2024-03-12T03:57:26.910682+0000","last_peered":"2024-03-12T03:57:26.910682+0000","last_clean":"2024-03-12T03:57:26.910682+0000","last_became_active":"2024-03-12T03:39:33.252274+0000","last_became_peered":"2024-03-12T03:39:33.252274+0000","last_unstale":"2024-03-12T03:57:26.910682+0000","last_undegraded":"2024-03-12T03:57:26.910682+0000","last_fullsized":"2024-03-12T03:57:26.910682+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:28.668329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:28.668329+0000","last_clean_scrub_stamp":"2024-03-12T03:56:28.668329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:35.848173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"480'27","reported_seq":157,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600332+0000","last_change":"2024-03-12T03:56:08.461559+0000","last_active":"2024-03-12T03:57:28.600332+0000","last_peered":"2024-03-12T03:57:28.600332+0000","last_clean":"2024-03-12T03:57:28.600332+0000","last_became_active":"2024-03-12T03:55:33.604743+0000","last_became_peered":"2024-03-12T03:55:33.604743+0000","last_unstale":"2024-03-12T03:57:28.600332+0000","last_undegraded":"2024-03-12T03:57:28.600332+0000","last_fullsized":"2024-03-12T03:57:28.600332+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'27","last_scrub_stamp":"2024-03-12T03:56:08.461437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:08.461437+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":807,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600075+0000","last_change":"2024-03-12T03:57:26.545702+0000","last_active":"2024-03-12T03:57:28.600075+0000","last_peered":"2024-03-12T03:57:28.600075+0000","last_clean":"2024-03-12T03:57:28.600075+0000","last_became_active":"2024-03-12T03:55:21.705922+0000","last_became_peered":"2024-03-12T03:55:21.705922+0000","last_unstale":"2024-03-12T03:57:28.600075+0000","last_undegraded":"2024-03-12T03:57:28.600075+0000","last_fullsized":"2024-03-12T03:57:28.600075+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:26.545600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:57:26.545600+0000","last_clean_scrub_stamp":"2024-03-12T03:57:26.545600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:43.240467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"332'14","reported_seq":1227,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899942+0000","last_change":"2024-03-12T03:57:22.208353+0000","last_active":"2024-03-12T03:57:28.899942+0000","last_peered":"2024-03-12T03:57:28.899942+0000","last_clean":"2024-03-12T03:57:28.899942+0000","last_became_active":"2024-03-12T03:46:25.300463+0000","last_became_peered":"2024-03-12T03:46:25.300463+0000","last_unstale":"2024-03-12T03:57:28.899942+0000","last_undegraded":"2024-03-12T03:57:28.899942+0000","last_fullsized":"2024-03-12T03:57:28.899942+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'14","last_scrub_stamp":"2024-03-12T03:57:22.208259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:57:22.208259+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:49.861049+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":904,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.900135+0000","last_change":"2024-03-12T03:57:16.632153+0000","last_active":"2024-03-12T03:57:28.900135+0000","last_peered":"2024-03-12T03:57:28.900135+0000","last_clean":"2024-03-12T03:57:28.900135+0000","last_became_active":"2024-03-12T03:56:10.594927+0000","last_became_peered":"2024-03-12T03:56:10.594927+0000","last_unstale":"2024-03-12T03:57:28.900135+0000","last_undegraded":"2024-03-12T03:57:28.900135+0000","last_fullsized":"2024-03-12T03:57:28.900135+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:16.632055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:16.632055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:35.118892+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"480'49","reported_seq":1021,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899629+0000","last_change":"2024-03-12T03:56:07.038711+0000","last_active":"2024-03-12T03:57:28.899629+0000","last_peered":"2024-03-12T03:57:28.899629+0000","last_clean":"2024-03-12T03:57:28.899629+0000","last_became_active":"2024-03-12T03:46:25.299798+0000","last_became_peered":"2024-03-12T03:46:25.299798+0000","last_unstale":"2024-03-12T03:57:28.899629+0000","last_undegraded":"2024-03-12T03:57:28.899629+0000","last_fullsized":"2024-03-12T03:57:28.899629+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'49","last_scrub_stamp":"2024-03-12T03:56:07.038607+0000","last_deep_scrub":"403'39","last_deep_scrub_stamp":"2024-03-12T03:50:06.535772+0000","last_clean_scrub_stamp":"2024-03-12T03:56:07.038607+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"288'2","reported_seq":1180,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898944+0000","last_change":"2024-03-12T03:56:48.221182+0000","last_active":"2024-03-12T03:57:28.898944+0000","last_peered":"2024-03-12T03:57:28.898944+0000","last_clean":"2024-03-12T03:57:28.898944+0000","last_became_active":"2024-03-12T03:39:33.264609+0000","last_became_peered":"2024-03-12T03:39:33.264609+0000","last_unstale":"2024-03-12T03:57:28.898944+0000","last_undegraded":"2024-03-12T03:57:28.898944+0000","last_fullsized":"2024-03-12T03:57:28.898944+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-12T03:56:48.221074+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-12T03:44:13.060719+0000","last_clean_scrub_stamp":"2024-03-12T03:56:48.221074+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:52.324561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"575'15","reported_seq":173,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600950+0000","last_change":"2024-03-12T03:56:39.471736+0000","last_active":"2024-03-12T03:57:28.600950+0000","last_peered":"2024-03-12T03:57:28.600950+0000","last_clean":"2024-03-12T03:57:28.600950+0000","last_became_active":"2024-03-12T03:55:33.595131+0000","last_became_peered":"2024-03-12T03:55:33.595131+0000","last_unstale":"2024-03-12T03:57:28.600950+0000","last_undegraded":"2024-03-12T03:57:28.600950+0000","last_fullsized":"2024-03-12T03:57:28.600950+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'15","last_scrub_stamp":"2024-03-12T03:56:39.471637+0000","last_deep_scrub":"575'15","last_deep_scrub_stamp":"2024-03-12T03:56:39.471637+0000","last_clean_scrub_stamp":"2024-03-12T03:56:39.471637+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:45.962312+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":823,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600779+0000","last_change":"2024-03-12T03:57:16.609050+0000","last_active":"2024-03-12T03:57:28.600779+0000","last_peered":"2024-03-12T03:57:28.600779+0000","last_clean":"2024-03-12T03:57:28.600779+0000","last_became_active":"2024-03-12T03:55:21.711184+0000","last_became_peered":"2024-03-12T03:55:21.711184+0000","last_unstale":"2024-03-12T03:57:28.600779+0000","last_undegraded":"2024-03-12T03:57:28.600779+0000","last_fullsized":"2024-03-12T03:57:28.600779+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:16.608920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:16.608920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:38.533448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"587'32","reported_seq":1225,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899846+0000","last_change":"2024-03-12T03:57:01.710768+0000","last_active":"2024-03-12T03:57:28.899846+0000","last_peered":"2024-03-12T03:57:28.899846+0000","last_clean":"2024-03-12T03:57:28.899846+0000","last_became_active":"2024-03-12T03:57:01.704851+0000","last_became_peered":"2024-03-12T03:57:01.704851+0000","last_unstale":"2024-03-12T03:57:28.899846+0000","last_undegraded":"2024-03-12T03:57:28.899846+0000","last_fullsized":"2024-03-12T03:57:28.899846+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"587'32","last_scrub_stamp":"2024-03-12T03:56:59.573734+0000","last_deep_scrub":"587'32","last_deep_scrub_stamp":"2024-03-12T03:56:59.573734+0000","last_clean_scrub_stamp":"2024-03-12T03:56:59.573734+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:16.440341+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":210,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899245+0000","last_change":"2024-03-12T03:56:14.120675+0000","last_active":"2024-03-12T03:57:28.899245+0000","last_peered":"2024-03-12T03:57:28.899245+0000","last_clean":"2024-03-12T03:57:28.899245+0000","last_became_active":"2024-03-12T03:54:29.584018+0000","last_became_peered":"2024-03-12T03:54:29.584018+0000","last_unstale":"2024-03-12T03:57:28.899245+0000","last_undegraded":"2024-03-12T03:57:28.899245+0000","last_fullsized":"2024-03-12T03:57:28.899245+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:14.120574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:47:07.914160+0000","last_clean_scrub_stamp":"2024-03-12T03:56:14.120574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:37.900989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1165,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910702+0000","last_change":"2024-03-12T03:57:10.202708+0000","last_active":"2024-03-12T03:57:26.910702+0000","last_peered":"2024-03-12T03:57:26.910702+0000","last_clean":"2024-03-12T03:57:26.910702+0000","last_became_active":"2024-03-12T03:55:33.599723+0000","last_became_peered":"2024-03-12T03:55:33.599723+0000","last_unstale":"2024-03-12T03:57:26.910702+0000","last_undegraded":"2024-03-12T03:57:26.910702+0000","last_fullsized":"2024-03-12T03:57:26.910702+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:10.202628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:57:10.202628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:29.203028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":934,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898930+0000","last_change":"2024-03-12T03:56:38.047368+0000","last_active":"2024-03-12T03:57:28.898930+0000","last_peered":"2024-03-12T03:57:28.898930+0000","last_clean":"2024-03-12T03:57:28.898930+0000","last_became_active":"2024-03-12T03:55:33.580129+0000","last_became_peered":"2024-03-12T03:55:33.580129+0000","last_unstale":"2024-03-12T03:57:28.898930+0000","last_undegraded":"2024-03-12T03:57:28.898930+0000","last_fullsized":"2024-03-12T03:57:28.898930+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:38.047286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:38.047286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:57.054659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"480'21","reported_seq":214,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910282+0000","last_change":"2024-03-12T03:56:52.356434+0000","last_active":"2024-03-12T03:57:26.910282+0000","last_peered":"2024-03-12T03:57:26.910282+0000","last_clean":"2024-03-12T03:57:26.910282+0000","last_became_active":"2024-03-12T03:54:47.094495+0000","last_became_peered":"2024-03-12T03:54:47.094495+0000","last_unstale":"2024-03-12T03:57:26.910282+0000","last_undegraded":"2024-03-12T03:57:26.910282+0000","last_fullsized":"2024-03-12T03:57:26.910282+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'21","last_scrub_stamp":"2024-03-12T03:56:52.356337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:52.356337+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:16.038565+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1171,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899059+0000","last_change":"2024-03-12T03:57:12.257796+0000","last_active":"2024-03-12T03:57:28.899059+0000","last_peered":"2024-03-12T03:57:28.899059+0000","last_clean":"2024-03-12T03:57:28.899059+0000","last_became_active":"2024-03-12T03:46:25.283602+0000","last_became_peered":"2024-03-12T03:46:25.283602+0000","last_unstale":"2024-03-12T03:57:28.899059+0000","last_undegraded":"2024-03-12T03:57:28.899059+0000","last_fullsized":"2024-03-12T03:57:28.899059+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:12.257657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:57:12.257657+0000","last_clean_scrub_stamp":"2024-03-12T03:57:12.257657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:25.214738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":936,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898889+0000","last_change":"2024-03-12T03:56:41.034611+0000","last_active":"2024-03-12T03:57:28.898889+0000","last_peered":"2024-03-12T03:57:28.898889+0000","last_clean":"2024-03-12T03:57:28.898889+0000","last_became_active":"2024-03-12T03:55:21.712301+0000","last_became_peered":"2024-03-12T03:55:21.712301+0000","last_unstale":"2024-03-12T03:57:28.898889+0000","last_undegraded":"2024-03-12T03:57:28.898889+0000","last_fullsized":"2024-03-12T03:57:28.898889+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:41.034530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:55:38.964306+0000","last_clean_scrub_stamp":"2024-03-12T03:56:41.034530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:06.694594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":838,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599718+0000","last_change":"2024-03-12T03:57:27.505214+0000","last_active":"2024-03-12T03:57:28.599718+0000","last_peered":"2024-03-12T03:57:28.599718+0000","last_clean":"2024-03-12T03:57:28.599718+0000","last_became_active":"2024-03-12T03:55:33.590692+0000","last_became_peered":"2024-03-12T03:55:33.590692+0000","last_unstale":"2024-03-12T03:57:28.599718+0000","last_undegraded":"2024-03-12T03:57:28.599718+0000","last_fullsized":"2024-03-12T03:57:28.599718+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:27.505113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:39:38.537304+0000","last_clean_scrub_stamp":"2024-03-12T03:57:27.505113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:43.444121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"480'34","reported_seq":966,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899150+0000","last_change":"2024-03-12T03:57:09.158380+0000","last_active":"2024-03-12T03:57:28.899150+0000","last_peered":"2024-03-12T03:57:28.899150+0000","last_clean":"2024-03-12T03:57:28.899150+0000","last_became_active":"2024-03-12T03:46:25.287095+0000","last_became_peered":"2024-03-12T03:46:25.287095+0000","last_unstale":"2024-03-12T03:57:28.899150+0000","last_undegraded":"2024-03-12T03:57:28.899150+0000","last_fullsized":"2024-03-12T03:57:28.899150+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'34","last_scrub_stamp":"2024-03-12T03:57:09.158292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:09.158292+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:24.056512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":153,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600847+0000","last_change":"2024-03-12T03:56:52.549765+0000","last_active":"2024-03-12T03:57:28.600847+0000","last_peered":"2024-03-12T03:57:28.600847+0000","last_clean":"2024-03-12T03:57:28.600847+0000","last_became_active":"2024-03-12T03:55:33.588975+0000","last_became_peered":"2024-03-12T03:55:33.588975+0000","last_unstale":"2024-03-12T03:57:28.600847+0000","last_undegraded":"2024-03-12T03:57:28.600847+0000","last_fullsized":"2024-03-12T03:57:28.600847+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:52.549647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:52.549647+0000","last_clean_scrub_stamp":"2024-03-12T03:56:52.549647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:55.472330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1145,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898741+0000","last_change":"2024-03-12T03:56:56.241810+0000","last_active":"2024-03-12T03:57:28.898741+0000","last_peered":"2024-03-12T03:57:28.898741+0000","last_clean":"2024-03-12T03:57:28.898741+0000","last_became_active":"2024-03-12T03:33:25.178681+0000","last_became_peered":"2024-03-12T03:33:25.178681+0000","last_unstale":"2024-03-12T03:57:28.898741+0000","last_undegraded":"2024-03-12T03:57:28.898741+0000","last_fullsized":"2024-03-12T03:57:28.898741+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:56.241710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:55:35.857943+0000","last_clean_scrub_stamp":"2024-03-12T03:56:56.241710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:24.155864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":951,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899268+0000","last_change":"2024-03-12T03:56:47.140349+0000","last_active":"2024-03-12T03:57:28.899268+0000","last_peered":"2024-03-12T03:57:28.899268+0000","last_clean":"2024-03-12T03:57:28.899268+0000","last_became_active":"2024-03-12T03:55:21.711506+0000","last_became_peered":"2024-03-12T03:55:21.711506+0000","last_unstale":"2024-03-12T03:57:28.899268+0000","last_undegraded":"2024-03-12T03:57:28.899268+0000","last_fullsized":"2024-03-12T03:57:28.899268+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:47.140244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:40.191538+0000","last_clean_scrub_stamp":"2024-03-12T03:56:47.140244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:04.904871+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"480'14","reported_seq":931,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898349+0000","last_change":"2024-03-12T03:56:32.140402+0000","last_active":"2024-03-12T03:57:28.898349+0000","last_peered":"2024-03-12T03:57:28.898349+0000","last_clean":"2024-03-12T03:57:28.898349+0000","last_became_active":"2024-03-12T03:46:25.286315+0000","last_became_peered":"2024-03-12T03:46:25.286315+0000","last_unstale":"2024-03-12T03:57:28.898349+0000","last_undegraded":"2024-03-12T03:57:28.898349+0000","last_fullsized":"2024-03-12T03:57:28.898349+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'14","last_scrub_stamp":"2024-03-12T03:56:32.140315+0000","last_deep_scrub":"90'6","last_deep_scrub_stamp":"2024-03-12T03:34:34.274353+0000","last_clean_scrub_stamp":"2024-03-12T03:56:32.140315+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:43.658047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":802,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600957+0000","last_change":"2024-03-12T03:56:36.490463+0000","last_active":"2024-03-12T03:57:28.600957+0000","last_peered":"2024-03-12T03:57:28.600957+0000","last_clean":"2024-03-12T03:57:28.600957+0000","last_became_active":"2024-03-12T03:55:33.590905+0000","last_became_peered":"2024-03-12T03:55:33.590905+0000","last_unstale":"2024-03-12T03:57:28.600957+0000","last_undegraded":"2024-03-12T03:57:28.600957+0000","last_fullsized":"2024-03-12T03:57:28.600957+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:36.490334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:36.490334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:51.862253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":624,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600995+0000","last_change":"2024-03-12T03:56:26.377525+0000","last_active":"2024-03-12T03:57:28.600995+0000","last_peered":"2024-03-12T03:57:28.600995+0000","last_clean":"2024-03-12T03:57:28.600995+0000","last_became_active":"2024-03-12T03:56:23.594174+0000","last_became_peered":"2024-03-12T03:56:23.594174+0000","last_unstale":"2024-03-12T03:57:28.600995+0000","last_undegraded":"2024-03-12T03:57:28.600995+0000","last_fullsized":"2024-03-12T03:57:28.600995+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:26.377419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:26.377419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:54.712974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1153,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899549+0000","last_change":"2024-03-12T03:57:21.282705+0000","last_active":"2024-03-12T03:57:28.899549+0000","last_peered":"2024-03-12T03:57:28.899549+0000","last_clean":"2024-03-12T03:57:28.899549+0000","last_became_active":"2024-03-12T03:39:33.253418+0000","last_became_peered":"2024-03-12T03:39:33.253418+0000","last_unstale":"2024-03-12T03:57:28.899549+0000","last_undegraded":"2024-03-12T03:57:28.899549+0000","last_fullsized":"2024-03-12T03:57:28.899549+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:21.282582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:21.282582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:23.718535+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"480'39","reported_seq":952,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.601586+0000","last_change":"2024-03-12T03:57:17.350440+0000","last_active":"2024-03-12T03:57:28.601586+0000","last_peered":"2024-03-12T03:57:28.601586+0000","last_clean":"2024-03-12T03:57:28.601586+0000","last_became_active":"2024-03-12T03:56:23.595378+0000","last_became_peered":"2024-03-12T03:56:23.595378+0000","last_unstale":"2024-03-12T03:57:28.601586+0000","last_undegraded":"2024-03-12T03:57:28.601586+0000","last_fullsized":"2024-03-12T03:57:28.601586+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:57:17.350340+0000","last_deep_scrub":"94'23","last_deep_scrub_stamp":"2024-03-12T03:41:13.133173+0000","last_clean_scrub_stamp":"2024-03-12T03:57:17.350340+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:17.597381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'1","reported_seq":952,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898520+0000","last_change":"2024-03-12T03:56:56.024267+0000","last_active":"2024-03-12T03:57:28.898520+0000","last_peered":"2024-03-12T03:57:28.898520+0000","last_clean":"2024-03-12T03:57:28.898520+0000","last_became_active":"2024-03-12T03:55:21.699340+0000","last_became_peered":"2024-03-12T03:55:21.699340+0000","last_unstale":"2024-03-12T03:57:28.898520+0000","last_undegraded":"2024-03-12T03:57:28.898520+0000","last_fullsized":"2024-03-12T03:57:28.898520+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-12T03:56:56.024179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:56.024179+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:12.112374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":843,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600698+0000","last_change":"2024-03-12T03:57:24.572147+0000","last_active":"2024-03-12T03:57:28.600698+0000","last_peered":"2024-03-12T03:57:28.600698+0000","last_clean":"2024-03-12T03:57:28.600698+0000","last_became_active":"2024-03-12T03:55:21.707893+0000","last_became_peered":"2024-03-12T03:55:21.707893+0000","last_unstale":"2024-03-12T03:57:28.600698+0000","last_undegraded":"2024-03-12T03:57:28.600698+0000","last_fullsized":"2024-03-12T03:57:28.600698+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:24.572062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:57:24.572062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:26.719140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1150,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.911006+0000","last_change":"2024-03-12T03:56:49.483218+0000","last_active":"2024-03-12T03:57:26.911006+0000","last_peered":"2024-03-12T03:57:26.911006+0000","last_clean":"2024-03-12T03:57:26.911006+0000","last_became_active":"2024-03-12T03:55:33.604326+0000","last_became_peered":"2024-03-12T03:55:33.604326+0000","last_unstale":"2024-03-12T03:57:26.911006+0000","last_undegraded":"2024-03-12T03:57:26.911006+0000","last_fullsized":"2024-03-12T03:57:26.911006+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:49.483108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:56:49.483108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:13.402853+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"480'37","reported_seq":873,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600910+0000","last_change":"2024-03-12T03:57:25.584319+0000","last_active":"2024-03-12T03:57:28.600910+0000","last_peered":"2024-03-12T03:57:28.600910+0000","last_clean":"2024-03-12T03:57:28.600910+0000","last_became_active":"2024-03-12T03:55:21.711857+0000","last_became_peered":"2024-03-12T03:55:21.711857+0000","last_unstale":"2024-03-12T03:57:28.600910+0000","last_undegraded":"2024-03-12T03:57:28.600910+0000","last_fullsized":"2024-03-12T03:57:28.600910+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'37","last_scrub_stamp":"2024-03-12T03:57:25.584207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:25.584207+0000","objects_scrubbed":6,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:29.836546+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"351'2","reported_seq":1162,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909678+0000","last_change":"2024-03-12T03:57:25.002147+0000","last_active":"2024-03-12T03:57:26.909678+0000","last_peered":"2024-03-12T03:57:26.909678+0000","last_clean":"2024-03-12T03:57:26.909678+0000","last_became_active":"2024-03-12T03:46:25.283344+0000","last_became_peered":"2024-03-12T03:46:25.283344+0000","last_unstale":"2024-03-12T03:57:26.909678+0000","last_undegraded":"2024-03-12T03:57:26.909678+0000","last_fullsized":"2024-03-12T03:57:26.909678+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'2","last_scrub_stamp":"2024-03-12T03:57:25.002070+0000","last_deep_scrub":"351'2","last_deep_scrub_stamp":"2024-03-12T03:47:15.329905+0000","last_clean_scrub_stamp":"2024-03-12T03:57:25.002070+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:40.109854+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"569'13","reported_seq":419,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.601587+0000","last_change":"2024-03-12T03:57:18.355377+0000","last_active":"2024-03-12T03:57:28.601587+0000","last_peered":"2024-03-12T03:57:28.601587+0000","last_clean":"2024-03-12T03:57:28.601587+0000","last_became_active":"2024-03-12T03:56:23.594849+0000","last_became_peered":"2024-03-12T03:56:23.594849+0000","last_unstale":"2024-03-12T03:57:28.601587+0000","last_undegraded":"2024-03-12T03:57:28.601587+0000","last_fullsized":"2024-03-12T03:57:28.601587+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'13","last_scrub_stamp":"2024-03-12T03:57:18.355246+0000","last_deep_scrub":"557'11","last_deep_scrub_stamp":"2024-03-12T03:56:06.046550+0000","last_clean_scrub_stamp":"2024-03-12T03:57:18.355246+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:31.020050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":1181,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899102+0000","last_change":"2024-03-12T03:57:11.234883+0000","last_active":"2024-03-12T03:57:28.899102+0000","last_peered":"2024-03-12T03:57:28.899102+0000","last_clean":"2024-03-12T03:57:28.899102+0000","last_became_active":"2024-03-12T03:55:33.607056+0000","last_became_peered":"2024-03-12T03:55:33.607056+0000","last_unstale":"2024-03-12T03:57:28.899102+0000","last_undegraded":"2024-03-12T03:57:28.899102+0000","last_fullsized":"2024-03-12T03:57:28.899102+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-12T03:57:11.234757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:11.234757+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:33.709557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"480'30","reported_seq":992,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899211+0000","last_change":"2024-03-12T03:56:17.121977+0000","last_active":"2024-03-12T03:57:28.899211+0000","last_peered":"2024-03-12T03:57:28.899211+0000","last_clean":"2024-03-12T03:57:28.899211+0000","last_became_active":"2024-03-12T03:46:25.283193+0000","last_became_peered":"2024-03-12T03:46:25.283193+0000","last_unstale":"2024-03-12T03:57:28.899211+0000","last_undegraded":"2024-03-12T03:57:28.899211+0000","last_fullsized":"2024-03-12T03:57:28.899211+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'30","last_scrub_stamp":"2024-03-12T03:56:17.121881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:56:17.121881+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:45.220394+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1174,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899394+0000","last_change":"2024-03-12T03:56:50.485904+0000","last_active":"2024-03-12T03:57:28.899394+0000","last_peered":"2024-03-12T03:57:28.899394+0000","last_clean":"2024-03-12T03:57:28.899394+0000","last_became_active":"2024-03-12T03:39:33.265613+0000","last_became_peered":"2024-03-12T03:39:33.265613+0000","last_unstale":"2024-03-12T03:57:28.899394+0000","last_undegraded":"2024-03-12T03:57:28.899394+0000","last_fullsized":"2024-03-12T03:57:28.899394+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:50.485798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:50.485798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:07.963554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":917,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899389+0000","last_change":"2024-03-12T03:56:10.594428+0000","last_active":"2024-03-12T03:57:28.899389+0000","last_peered":"2024-03-12T03:57:28.899389+0000","last_clean":"2024-03-12T03:57:28.899389+0000","last_became_active":"2024-03-12T03:56:10.593885+0000","last_became_peered":"2024-03-12T03:56:10.593885+0000","last_unstale":"2024-03-12T03:57:28.899389+0000","last_undegraded":"2024-03-12T03:57:28.899389+0000","last_fullsized":"2024-03-12T03:57:28.899389+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:05.095186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:05.095186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:30.734749+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":973,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899900+0000","last_change":"2024-03-12T03:57:19.359191+0000","last_active":"2024-03-12T03:57:28.899900+0000","last_peered":"2024-03-12T03:57:28.899900+0000","last_clean":"2024-03-12T03:57:28.899900+0000","last_became_active":"2024-03-12T03:55:33.591483+0000","last_became_peered":"2024-03-12T03:55:33.591483+0000","last_unstale":"2024-03-12T03:57:28.899900+0000","last_undegraded":"2024-03-12T03:57:28.899900+0000","last_fullsized":"2024-03-12T03:57:28.899900+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:19.359061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:19.359061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:32.773713+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"480'39","reported_seq":953,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910368+0000","last_change":"2024-03-12T03:57:06.709997+0000","last_active":"2024-03-12T03:57:26.910368+0000","last_peered":"2024-03-12T03:57:26.910368+0000","last_clean":"2024-03-12T03:57:26.910368+0000","last_became_active":"2024-03-12T03:46:25.301700+0000","last_became_peered":"2024-03-12T03:46:25.301700+0000","last_unstale":"2024-03-12T03:57:26.910368+0000","last_undegraded":"2024-03-12T03:57:26.910368+0000","last_fullsized":"2024-03-12T03:57:26.910368+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:57:06.709862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:22.141170+0000","last_clean_scrub_stamp":"2024-03-12T03:57:06.709862+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:23.704696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":793,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600384+0000","last_change":"2024-03-12T03:56:40.322391+0000","last_active":"2024-03-12T03:57:28.600384+0000","last_peered":"2024-03-12T03:57:28.600384+0000","last_clean":"2024-03-12T03:57:28.600384+0000","last_became_active":"2024-03-12T03:55:21.708360+0000","last_became_peered":"2024-03-12T03:55:21.708360+0000","last_unstale":"2024-03-12T03:57:28.600384+0000","last_undegraded":"2024-03-12T03:57:28.600384+0000","last_fullsized":"2024-03-12T03:57:28.600384+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:40.322233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:40.322233+0000","last_clean_scrub_stamp":"2024-03-12T03:56:40.322233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:52.102003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1134,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.910879+0000","last_change":"2024-03-12T03:56:47.281484+0000","last_active":"2024-03-12T03:57:26.910879+0000","last_peered":"2024-03-12T03:57:26.910879+0000","last_clean":"2024-03-12T03:57:26.910879+0000","last_became_active":"2024-03-12T03:55:21.693975+0000","last_became_peered":"2024-03-12T03:55:21.693975+0000","last_unstale":"2024-03-12T03:57:26.910879+0000","last_undegraded":"2024-03-12T03:57:26.910879+0000","last_fullsized":"2024-03-12T03:57:26.910879+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:47.281351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:24.155889+0000","last_clean_scrub_stamp":"2024-03-12T03:56:47.281351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:00.376002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1157,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.900140+0000","last_change":"2024-03-12T03:57:01.569282+0000","last_active":"2024-03-12T03:57:28.900140+0000","last_peered":"2024-03-12T03:57:28.900140+0000","last_clean":"2024-03-12T03:57:28.900140+0000","last_became_active":"2024-03-12T03:55:33.590451+0000","last_became_peered":"2024-03-12T03:55:33.590451+0000","last_unstale":"2024-03-12T03:57:28.900140+0000","last_undegraded":"2024-03-12T03:57:28.900140+0000","last_fullsized":"2024-03-12T03:57:28.900140+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:01.569167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:51.952189+0000","last_clean_scrub_stamp":"2024-03-12T03:57:01.569167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:25.165249+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"480'18","reported_seq":823,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899770+0000","last_change":"2024-03-12T03:56:13.317181+0000","last_active":"2024-03-12T03:57:28.899770+0000","last_peered":"2024-03-12T03:57:28.899770+0000","last_clean":"2024-03-12T03:57:28.899770+0000","last_became_active":"2024-03-12T03:56:10.596455+0000","last_became_peered":"2024-03-12T03:56:10.596455+0000","last_unstale":"2024-03-12T03:57:28.899770+0000","last_undegraded":"2024-03-12T03:57:28.899770+0000","last_fullsized":"2024-03-12T03:57:28.899770+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'18","last_scrub_stamp":"2024-03-12T03:56:13.317058+0000","last_deep_scrub":"86'14","last_deep_scrub_stamp":"2024-03-12T03:34:23.795040+0000","last_clean_scrub_stamp":"2024-03-12T03:56:13.317058+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:37.426135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":145,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600384+0000","last_change":"2024-03-12T03:56:37.428462+0000","last_active":"2024-03-12T03:57:28.600384+0000","last_peered":"2024-03-12T03:57:28.600384+0000","last_clean":"2024-03-12T03:57:28.600384+0000","last_became_active":"2024-03-12T03:55:33.589300+0000","last_became_peered":"2024-03-12T03:55:33.589300+0000","last_unstale":"2024-03-12T03:57:28.600384+0000","last_undegraded":"2024-03-12T03:57:28.600384+0000","last_fullsized":"2024-03-12T03:57:28.600384+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:37.428362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:20.084273+0000","last_clean_scrub_stamp":"2024-03-12T03:56:37.428362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:50.038301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1136,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899408+0000","last_change":"2024-03-12T03:56:24.182756+0000","last_active":"2024-03-12T03:57:28.899408+0000","last_peered":"2024-03-12T03:57:28.899408+0000","last_clean":"2024-03-12T03:57:28.899408+0000","last_became_active":"2024-03-12T03:33:25.182170+0000","last_became_peered":"2024-03-12T03:33:25.182170+0000","last_unstale":"2024-03-12T03:57:28.899408+0000","last_undegraded":"2024-03-12T03:57:28.899408+0000","last_fullsized":"2024-03-12T03:57:28.899408+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:24.182659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:43:22.975800+0000","last_clean_scrub_stamp":"2024-03-12T03:56:24.182659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:42.764946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":153,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599442+0000","last_change":"2024-03-12T03:57:06.618834+0000","last_active":"2024-03-12T03:57:28.599442+0000","last_peered":"2024-03-12T03:57:28.599442+0000","last_clean":"2024-03-12T03:57:28.599442+0000","last_became_active":"2024-03-12T03:55:33.583091+0000","last_became_peered":"2024-03-12T03:55:33.583091+0000","last_unstale":"2024-03-12T03:57:28.599442+0000","last_undegraded":"2024-03-12T03:57:28.599442+0000","last_fullsized":"2024-03-12T03:57:28.599442+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:06.618725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:33:26.164901+0000","last_clean_scrub_stamp":"2024-03-12T03:57:06.618725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:34.036837+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"480'39","reported_seq":859,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899293+0000","last_change":"2024-03-12T03:56:30.347547+0000","last_active":"2024-03-12T03:57:28.899293+0000","last_peered":"2024-03-12T03:57:28.899293+0000","last_clean":"2024-03-12T03:57:28.899293+0000","last_became_active":"2024-03-12T03:56:10.593751+0000","last_became_peered":"2024-03-12T03:56:10.593751+0000","last_unstale":"2024-03-12T03:57:28.899293+0000","last_undegraded":"2024-03-12T03:57:28.899293+0000","last_fullsized":"2024-03-12T03:57:28.899293+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-12T03:56:30.347454+0000","last_deep_scrub":"345'33","last_deep_scrub_stamp":"2024-03-12T03:47:02.110517+0000","last_clean_scrub_stamp":"2024-03-12T03:56:30.347454+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:36.016747+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":145,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600253+0000","last_change":"2024-03-12T03:56:00.477835+0000","last_active":"2024-03-12T03:57:28.600253+0000","last_peered":"2024-03-12T03:57:28.600253+0000","last_clean":"2024-03-12T03:57:28.600253+0000","last_became_active":"2024-03-12T03:55:33.601416+0000","last_became_peered":"2024-03-12T03:55:33.601416+0000","last_unstale":"2024-03-12T03:57:28.600253+0000","last_undegraded":"2024-03-12T03:57:28.600253+0000","last_fullsized":"2024-03-12T03:57:28.600253+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:00.477725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:51.187976+0000","last_clean_scrub_stamp":"2024-03-12T03:56:00.477725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":145,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599924+0000","last_change":"2024-03-12T03:56:54.577180+0000","last_active":"2024-03-12T03:57:28.599924+0000","last_peered":"2024-03-12T03:57:28.599924+0000","last_clean":"2024-03-12T03:57:28.599924+0000","last_became_active":"2024-03-12T03:55:33.584266+0000","last_became_peered":"2024-03-12T03:55:33.584266+0000","last_unstale":"2024-03-12T03:57:28.599924+0000","last_undegraded":"2024-03-12T03:57:28.599924+0000","last_fullsized":"2024-03-12T03:57:28.599924+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:54.577057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:54.577057+0000","last_clean_scrub_stamp":"2024-03-12T03:56:54.577057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:20.726214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":807,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.599991+0000","last_change":"2024-03-12T03:57:05.486512+0000","last_active":"2024-03-12T03:57:28.599991+0000","last_peered":"2024-03-12T03:57:28.599991+0000","last_clean":"2024-03-12T03:57:28.599991+0000","last_became_active":"2024-03-12T03:55:21.701401+0000","last_became_peered":"2024-03-12T03:55:21.701401+0000","last_unstale":"2024-03-12T03:57:28.599991+0000","last_undegraded":"2024-03-12T03:57:28.599991+0000","last_fullsized":"2024-03-12T03:57:28.599991+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:05.486432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:58.095196+0000","last_clean_scrub_stamp":"2024-03-12T03:57:05.486432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:31.934392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"480'74","reported_seq":1136,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898583+0000","last_change":"2024-03-12T03:56:46.048356+0000","last_active":"2024-03-12T03:57:28.898583+0000","last_peered":"2024-03-12T03:57:28.898583+0000","last_clean":"2024-03-12T03:57:28.898583+0000","last_became_active":"2024-03-12T03:55:21.700045+0000","last_became_peered":"2024-03-12T03:55:21.700045+0000","last_unstale":"2024-03-12T03:57:28.898583+0000","last_undegraded":"2024-03-12T03:57:28.898583+0000","last_fullsized":"2024-03-12T03:57:28.898583+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'74","last_scrub_stamp":"2024-03-12T03:56:46.048258+0000","last_deep_scrub":"94'50","last_deep_scrub_stamp":"2024-03-12T03:35:05.042030+0000","last_clean_scrub_stamp":"2024-03-12T03:56:46.048258+0000","objects_scrubbed":11,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:46.723052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":97,"num_read_kb":84,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":978,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899647+0000","last_change":"2024-03-12T03:56:27.043313+0000","last_active":"2024-03-12T03:57:28.899647+0000","last_peered":"2024-03-12T03:57:28.899647+0000","last_clean":"2024-03-12T03:57:28.899647+0000","last_became_active":"2024-03-12T03:46:25.279131+0000","last_became_peered":"2024-03-12T03:46:25.279131+0000","last_unstale":"2024-03-12T03:57:28.899647+0000","last_undegraded":"2024-03-12T03:57:28.899647+0000","last_fullsized":"2024-03-12T03:57:28.899647+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:27.043216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:34:41.142994+0000","last_clean_scrub_stamp":"2024-03-12T03:56:27.043216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:33.284291+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":895,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898427+0000","last_change":"2024-03-12T03:56:18.310139+0000","last_active":"2024-03-12T03:57:28.898427+0000","last_peered":"2024-03-12T03:57:28.898427+0000","last_clean":"2024-03-12T03:57:28.898427+0000","last_became_active":"2024-03-12T03:56:10.596064+0000","last_became_peered":"2024-03-12T03:56:10.596064+0000","last_unstale":"2024-03-12T03:57:28.898427+0000","last_undegraded":"2024-03-12T03:57:28.898427+0000","last_fullsized":"2024-03-12T03:57:28.898427+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:18.310042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:37:18.123378+0000","last_clean_scrub_stamp":"2024-03-12T03:56:18.310042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:39.876897+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":936,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.899071+0000","last_change":"2024-03-12T03:57:23.356928+0000","last_active":"2024-03-12T03:57:28.899071+0000","last_peered":"2024-03-12T03:57:28.899071+0000","last_clean":"2024-03-12T03:57:28.899071+0000","last_became_active":"2024-03-12T03:46:25.281081+0000","last_became_peered":"2024-03-12T03:46:25.281081+0000","last_unstale":"2024-03-12T03:57:28.899071+0000","last_undegraded":"2024-03-12T03:57:28.899071+0000","last_fullsized":"2024-03-12T03:57:28.899071+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:57:23.356793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:56:20.224988+0000","last_clean_scrub_stamp":"2024-03-12T03:57:23.356793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:58:40.552641+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"480'32","reported_seq":1193,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898585+0000","last_change":"2024-03-12T03:56:00.435054+0000","last_active":"2024-03-12T03:57:28.898585+0000","last_peered":"2024-03-12T03:57:28.898585+0000","last_clean":"2024-03-12T03:57:28.898585+0000","last_became_active":"2024-03-12T03:33:23.163976+0000","last_became_peered":"2024-03-12T03:33:23.163976+0000","last_unstale":"2024-03-12T03:57:28.898585+0000","last_undegraded":"2024-03-12T03:57:28.898585+0000","last_fullsized":"2024-03-12T03:57:28.898585+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'32","last_scrub_stamp":"2024-03-12T03:56:00.434912+0000","last_deep_scrub":"480'32","last_deep_scrub_stamp":"2024-03-12T03:56:00.434912+0000","last_clean_scrub_stamp":"2024-03-12T03:56:00.434912+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":848,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.600793+0000","last_change":"2024-03-12T03:56:38.400409+0000","last_active":"2024-03-12T03:57:28.600793+0000","last_peered":"2024-03-12T03:57:28.600793+0000","last_clean":"2024-03-12T03:57:28.600793+0000","last_became_active":"2024-03-12T03:55:33.601946+0000","last_became_peered":"2024-03-12T03:55:33.601946+0000","last_unstale":"2024-03-12T03:57:28.600793+0000","last_undegraded":"2024-03-12T03:57:28.600793+0000","last_fullsized":"2024-03-12T03:57:28.600793+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:56:38.400328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:44:14.712916+0000","last_clean_scrub_stamp":"2024-03-12T03:56:38.400328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T03:57:48.633312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1154,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-12T03:57:26.909707+0000","last_change":"2024-03-12T03:55:55.621341+0000","last_active":"2024-03-12T03:57:26.909707+0000","last_peered":"2024-03-12T03:57:26.909707+0000","last_clean":"2024-03-12T03:57:26.909707+0000","last_became_active":"2024-03-12T03:55:33.596868+0000","last_became_peered":"2024-03-12T03:55:33.596868+0000","last_unstale":"2024-03-12T03:57:26.909707+0000","last_undegraded":"2024-03-12T03:57:26.909707+0000","last_fullsized":"2024-03-12T03:57:26.909707+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:55:55.621256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:50:04.534937+0000","last_clean_scrub_stamp":"2024-03-12T03:55:55.621256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":891,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-12T03:57:28.898939+0000","last_change":"2024-03-12T03:56:10.597488+0000","last_active":"2024-03-12T03:57:28.898939+0000","last_peered":"2024-03-12T03:57:28.898939+0000","last_clean":"2024-03-12T03:57:28.898939+0000","last_became_active":"2024-03-12T03:56:10.597117+0000","last_became_peered":"2024-03-12T03:56:10.597117+0000","last_unstale":"2024-03-12T03:57:28.898939+0000","last_undegraded":"2024-03-12T03:57:28.898939+0000","last_fullsized":"2024-03-12T03:57:28.898939+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:55:41.674862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:45:14.952550+0000","last_clean_scrub_stamp":"2024-03-12T03:55:41.674862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1439,"num_read_kb":1434,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2033792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1807,"num_read_kb":1532,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":4182,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},"log_size":1257,"ondisk_log_size":1257,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070760,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004800,"kb_used_data":3188,"kb_used_omap":1,"kb_used_meta":1001598,"kb_avail":92740352,"statfs":{"total":95995035648,"available":94966120448,"internally_reserved":0,"allocated":3264512,"data_stored":1958277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1025636810},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332397,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":926692,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92818460,"statfs":{"total":95995035648,"available":95046103040,"internally_reserved":0,"allocated":2912256,"data_stored":1628634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":946010566},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575173,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1080852,"kb_used_data":2888,"kb_used_omap":1,"kb_used_meta":1077950,"kb_avail":92664300,"statfs":{"total":95995035648,"available":94888243200,"internally_reserved":0,"allocated":2957312,"data_stored":1624065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1103821253},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":521,"seq":2237677961244,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":764928,"kb_used_data":684788,"kb_used_omap":4,"kb_used_meta":80123,"kb_avail":92980224,"statfs":{"total":95995035648,"available":95211749376,"internally_reserved":0,"allocated":701222912,"data_stored":699915931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":82046628},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411647,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":976912,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92768240,"statfs":{"total":95995035648,"available":94994677760,"internally_reserved":0,"allocated":3153920,"data_stored":1879281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":330,"seq":1417339207815,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":539208,"kb_used_data":3452,"kb_used_omap":13,"kb_used_meta":535730,"kb_avail":93205944,"statfs":{"total":95995035648,"available":95442886656,"internally_reserved":0,"allocated":3534848,"data_stored":2213772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13485,"internal_metadata":548588371},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953784,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":805960,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":92939192,"statfs":{"total":95995035648,"available":95169732608,"internally_reserved":0,"allocated":2940928,"data_stored":1630137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":822344133},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659442,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":991156,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92753996,"statfs":{"total":95995035648,"available":94980091904,"internally_reserved":0,"allocated":3510272,"data_stored":2219425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1011415493},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7379,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:57:34.270 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-12T03:57:34.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 677d624a-e020-11ee-95c8-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-12T03:57:34.516 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:57:34.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:34 smithi169 ceph-mon[39949]: from='client.26102 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:57:34.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:34 smithi169 ceph-mon[39949]: 5.1 scrub starts 2024-03-12T03:57:34.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:34 smithi169 ceph-mon[39949]: 5.1 scrub ok 2024-03-12T03:57:34.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:34 smithi169 ceph-mon[39949]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:34.915 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:34 smithi169 ceph-mon[29344]: from='client.26102 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:57:34.915 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:34 smithi169 ceph-mon[29344]: 5.1 scrub starts 2024-03-12T03:57:34.915 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:34 smithi169 ceph-mon[29344]: 5.1 scrub ok 2024-03-12T03:57:34.915 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:34 smithi169 ceph-mon[29344]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:35.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:34 smithi173 ceph-mon[33544]: from='client.26102 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:57:35.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:34 smithi173 ceph-mon[33544]: 5.1 scrub starts 2024-03-12T03:57:35.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:34 smithi173 ceph-mon[33544]: 5.1 scrub ok 2024-03-12T03:57:35.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:34 smithi173 ceph-mon[33544]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:36.026 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:35 smithi169 ceph-mon[29344]: pgmap v1222: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:36.026 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:35 smithi169 ceph-mon[39949]: pgmap v1222: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:36.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:35 smithi173 ceph-mon[33544]: pgmap v1222: 142 pgs: 142 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:36.767 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:57:36] ENGINE Bus STOPPING 2024-03-12T03:57:36.767 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:57:36] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:57:36.768 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:57:36] ENGINE Bus STOPPED 2024-03-12T03:57:36.768 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:57:36] ENGINE Bus STARTING 2024-03-12T03:57:36.768 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:57:36] ENGINE Serving on http://:::9283 2024-03-12T03:57:36.768 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:57:36] ENGINE Bus STARTED 2024-03-12T03:57:37.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[29344]: 3.e deep-scrub starts 2024-03-12T03:57:37.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[29344]: 3.e deep-scrub ok 2024-03-12T03:57:37.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[29344]: 4.14 deep-scrub starts 2024-03-12T03:57:37.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[29344]: 4.14 deep-scrub ok 2024-03-12T03:57:37.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1629274364' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-12T03:57:37.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[29344]: from='client.? 172.21.15.169:0/1629274364' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-12T03:57:37.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:37.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:37.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[29344]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:37.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[29344]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:57:37.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[39949]: 3.e deep-scrub starts 2024-03-12T03:57:37.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[39949]: 3.e deep-scrub ok 2024-03-12T03:57:37.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[39949]: 4.14 deep-scrub starts 2024-03-12T03:57:37.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[39949]: 4.14 deep-scrub ok 2024-03-12T03:57:37.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1629274364' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-12T03:57:37.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[39949]: from='client.? 172.21.15.169:0/1629274364' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-12T03:57:37.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:37.163 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:37.163 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[39949]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:37.163 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:36 smithi169 ceph-mon[39949]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:57:37.188 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-12T03:57:37.199 INFO:tasks.cephadm:Teardown begin 2024-03-12T03:57:37.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:57:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:36 smithi173 ceph-mon[33544]: 3.e deep-scrub starts 2024-03-12T03:57:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:36 smithi173 ceph-mon[33544]: 3.e deep-scrub ok 2024-03-12T03:57:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:36 smithi173 ceph-mon[33544]: 4.14 deep-scrub starts 2024-03-12T03:57:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:36 smithi173 ceph-mon[33544]: 4.14 deep-scrub ok 2024-03-12T03:57:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:36 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1629274364' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-12T03:57:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:36 smithi173 ceph-mon[33544]: from='client.? 172.21.15.169:0/1629274364' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-12T03:57:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:36 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:37.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:36 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:36 smithi173 ceph-mon[33544]: from='mgr.24458 172.21.15.169:0/1996127927' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:37.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:36 smithi173 ceph-mon[33544]: from='mgr.24458 ' entity='mgr.y' 2024-03-12T03:57:37.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:57:37.313 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-12T03:57:37.314 DEBUG:teuthology.orchestra.run.smithi169:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T03:57:37.347 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T03:57:37.377 INFO:tasks.cephadm:Stopping all daemons... 2024-03-12T03:57:37.377 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-12T03:57:37.378 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.a 2024-03-12T03:57:37.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:37 smithi169 systemd[1]: Stopping Ceph mon.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:37.980 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:37 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a[29340]: 2024-03-12T03:57:37.713+0000 7f70564b4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:57:37.981 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:37 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a[29340]: 2024-03-12T03:57:37.713+0000 7f70564b4700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-12T03:57:38.275 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:37 smithi169 podman[209971]: 2024-03-12 03:57:37.979022946 +0000 UTC m=+0.399966237 container died 9b290f8cbe46fb22732f566e597407cf2b564a7696de432e93df77444f7d35e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-12T03:57:38.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:38 smithi169 podman[209971]: 2024-03-12 03:57:38.372507095 +0000 UTC m=+0.793450396 container cleanup 9b290f8cbe46fb22732f566e597407cf2b564a7696de432e93df77444f7d35e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-12T03:57:38.650 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 12 03:57:38 smithi169 bash[209971]: ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-a 2024-03-12T03:57:38.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.a.service' 2024-03-12T03:57:38.836 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:57:38.836 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-12T03:57:38.836 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-12T03:57:38.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.c 2024-03-12T03:57:39.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:38 smithi169 systemd[1]: Stopping Ceph mon.c for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:39.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:39 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-c[39944]: 2024-03-12T03:57:39.159+0000 7f8da4072700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:57:39.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:39 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-c[39944]: 2024-03-12T03:57:39.159+0000 7f8da4072700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-12T03:57:39.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:39 smithi169 podman[210092]: 2024-03-12 03:57:39.364158932 +0000 UTC m=+0.308176307 container died ec8786add2919ed9d632de50254f54dba8fee6125f5dc2cd2f1fad362fb6ade3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-c, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-12T03:57:39.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:39 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:57:39] ENGINE Bus STOPPING 2024-03-12T03:57:39.423 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:39 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:57:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:57:39.424 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:39 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:57:39] ENGINE Bus STOPPED 2024-03-12T03:57:39.424 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:39 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:57:39] ENGINE Bus STARTING 2024-03-12T03:57:39.424 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:39 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:57:39] ENGINE Serving on http://:::9283 2024-03-12T03:57:39.424 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:39 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: [12/Mar/2024:03:57:39] ENGINE Bus STARTED 2024-03-12T03:57:40.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:39 smithi169 podman[210092]: 2024-03-12 03:57:39.777563622 +0000 UTC m=+0.721581068 container cleanup ec8786add2919ed9d632de50254f54dba8fee6125f5dc2cd2f1fad362fb6ade3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:57:40.097 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 12 03:57:39 smithi169 bash[210092]: ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-c 2024-03-12T03:57:40.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.c.service' 2024-03-12T03:57:40.280 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:57:40.280 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-12T03:57:40.281 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-12T03:57:40.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.b 2024-03-12T03:57:40.620 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:40 smithi173 systemd[1]: Stopping Ceph mon.b for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:40.954 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:40 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-b[33538]: 2024-03-12T03:57:40.617+0000 7feba4cee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:57:40.954 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:40 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-b[33538]: 2024-03-12T03:57:40.617+0000 7feba4cee700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-12T03:57:40.954 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:40 smithi173 podman[162035]: 2024-03-12 03:57:40.690271878 +0000 UTC m=+0.175998477 container died a135eea20e2c3620654e3c4b06984f73374a5c8d0367fb3355d1c094ddd89a10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-b, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e) 2024-03-12T03:57:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:41 smithi173 podman[162035]: 2024-03-12 03:57:41.118735413 +0000 UTC m=+0.604462011 container cleanup a135eea20e2c3620654e3c4b06984f73374a5c8d0367fb3355d1c094ddd89a10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-b, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-12T03:57:41.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 12 03:57:41 smithi173 bash[162035]: ceph-677d624a-e020-11ee-95c8-87774f69a715-mon-b 2024-03-12T03:57:41.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@mon.b.service' 2024-03-12T03:57:41.605 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:57:41.605 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-12T03:57:41.605 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-12T03:57:41.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@mgr.y 2024-03-12T03:57:41.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:41 smithi169 systemd[1]: Stopping Ceph mgr.y for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:41.923 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: ::ffff:172.21.15.173 - - [12/Mar/2024:03:57:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:57:42.244 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:57:41.923+0000 7fb39d34a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:57:42.244 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:41 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y[29569]: 2024-03-12T03:57:41.923+0000 7fb39d34a700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-12T03:57:42.244 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:41 smithi169 podman[210226]: 2024-03-12 03:57:41.955109767 +0000 UTC m=+0.154576110 container died eadae3e4956938f1e170b658b706e3429746291351b73442fd10e772f2323e26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-12T03:57:42.628 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:42 smithi169 podman[210226]: 2024-03-12 03:57:42.332876829 +0000 UTC m=+0.532343171 container cleanup eadae3e4956938f1e170b658b706e3429746291351b73442fd10e772f2323e26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:57:42.628 INFO:journalctl@ceph.mgr.y.smithi169.stdout:Mar 12 03:57:42 smithi169 bash[210226]: ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-y 2024-03-12T03:57:42.752 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-12T03:57:42.788 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:57:42.789 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-12T03:57:42.789 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-12T03:57:42.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@mgr.x 2024-03-12T03:57:43.140 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:57:42 smithi173 systemd[1]: Stopping Ceph mgr.x for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:43.410 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:57:43 smithi173 podman[162159]: 2024-03-12 03:57:43.138274376 +0000 UTC m=+0.161520346 container died d08b03e30b6b02f2089ebe167b5ad5236f6d70907a866a41c457e5d26643b0f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:57:43.736 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:57:43 smithi173 podman[162159]: 2024-03-12 03:57:43.558073015 +0000 UTC m=+0.581318985 container cleanup d08b03e30b6b02f2089ebe167b5ad5236f6d70907a866a41c457e5d26643b0f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-12T03:57:43.736 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 12 03:57:43 smithi173 bash[162159]: ceph-677d624a-e020-11ee-95c8-87774f69a715-mgr-x 2024-03-12T03:57:43.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-12T03:57:44.010 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:57:44.010 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-12T03:57:44.011 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-12T03:57:44.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.0 2024-03-12T03:57:44.308 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:44 smithi169 systemd[1]: Stopping Ceph osd.0 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:44.673 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:44 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0[201513]: 2024-03-12T03:57:44.311+0000 7fa78286d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:57:44.673 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:44 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0[201513]: 2024-03-12T03:57:44.311+0000 7fa78286d700 -1 osd.0 595 *** Got signal Terminated *** 2024-03-12T03:57:44.673 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:44 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0[201513]: 2024-03-12T03:57:44.311+0000 7fa78286d700 -1 osd.0 595 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:57:49.990 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:49 smithi169 podman[210350]: 2024-03-12 03:57:49.72050227 +0000 UTC m=+5.522400583 container died 7ff85389739434cf5ab42c0ec739c1b88b3d6aed59698a42c8d4c43876b0ee52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-12T03:57:50.423 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:50 smithi169 podman[210350]: 2024-03-12 03:57:50.171172445 +0000 UTC m=+5.973070758 container cleanup 7ff85389739434cf5ab42c0ec739c1b88b3d6aed59698a42c8d4c43876b0ee52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-12T03:57:50.423 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:50 smithi169 bash[210350]: ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0 2024-03-12T03:57:50.785 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:50 smithi169 podman[210367]: 2024-03-12 03:57:50.455923011 +0000 UTC m=+0.734225324 container remove 7ff85389739434cf5ab42c0ec739c1b88b3d6aed59698a42c8d4c43876b0ee52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-12T03:57:51.110 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:50 smithi169 podman[210457]: 2024-03-12 03:57:50.686545509 +0000 UTC m=+0.016900304 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:57:51.110 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:50 smithi169 podman[210457]: 2024-03-12 03:57:50.790030429 +0000 UTC m=+0.120385224 container create 270b264ffc8ad547091f3b5fd3a8aad346002072ed779381660be06128b2b332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-12T03:57:51.110 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:51 smithi169 podman[210457]: 2024-03-12 03:57:51.02589497 +0000 UTC m=+0.356249780 container init 270b264ffc8ad547091f3b5fd3a8aad346002072ed779381660be06128b2b332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, ceph=True) 2024-03-12T03:57:51.110 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:51 smithi169 podman[210457]: 2024-03-12 03:57:51.029343547 +0000 UTC m=+0.359698345 container start 270b264ffc8ad547091f3b5fd3a8aad346002072ed779381660be06128b2b332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-deactivate, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-12T03:57:51.111 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:51 smithi169 podman[210457]: 2024-03-12 03:57:51.109038327 +0000 UTC m=+0.439393138 container attach 270b264ffc8ad547091f3b5fd3a8aad346002072ed779381660be06128b2b332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-12T03:57:51.672 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 12 03:57:51 smithi169 podman[210457]: 2024-03-12 03:57:51.379094002 +0000 UTC m=+0.709448810 container died 270b264ffc8ad547091f3b5fd3a8aad346002072ed779381660be06128b2b332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-12T03:57:52.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.0.service' 2024-03-12T03:57:52.054 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:57:52.054 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-12T03:57:52.054 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-12T03:57:52.054 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.1 2024-03-12T03:57:52.371 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:52 smithi169 systemd[1]: Stopping Ceph osd.1 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:52.673 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:52 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1[49981]: 2024-03-12T03:57:52.365+0000 7fb2a83b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:57:52.673 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:52 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1[49981]: 2024-03-12T03:57:52.365+0000 7fb2a83b1700 -1 osd.1 595 *** Got signal Terminated *** 2024-03-12T03:57:52.673 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:52 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1[49981]: 2024-03-12T03:57:52.365+0000 7fb2a83b1700 -1 osd.1 595 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:57:55.237 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:57:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:57:54.916Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.169:8765: connect: connection refused" 2024-03-12T03:57:55.237 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:57:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:57:54.916Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.169:8765: connect: connection refused" 2024-03-12T03:57:55.237 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:57:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:57:54.919Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.169:8765: connect: connection refused" 2024-03-12T03:57:55.237 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:57:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:57:54.920Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.169:8765: connect: connection refused" 2024-03-12T03:57:55.237 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:57:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:57:54.922Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.169:8765: connect: connection refused" 2024-03-12T03:57:58.127 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:57 smithi169 podman[210564]: 2024-03-12 03:57:57.830668833 +0000 UTC m=+5.588137633 container died 8c3999f888dedbc3b4ff2f98731d13f0ce622a4dd516b0437aba16bda06bc898 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-12T03:57:58.423 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:58 smithi169 podman[210564]: 2024-03-12 03:57:58.282827468 +0000 UTC m=+6.040296264 container cleanup 8c3999f888dedbc3b4ff2f98731d13f0ce622a4dd516b0437aba16bda06bc898 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-12T03:57:58.423 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:58 smithi169 bash[210564]: ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1 2024-03-12T03:57:58.875 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:58 smithi169 podman[210581]: 2024-03-12 03:57:58.57609673 +0000 UTC m=+0.745215692 container remove 8c3999f888dedbc3b4ff2f98731d13f0ce622a4dd516b0437aba16bda06bc898 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-12T03:57:59.173 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:58 smithi169 podman[210669]: 2024-03-12 03:57:58.776723282 +0000 UTC m=+0.018130530 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:57:59.173 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:58 smithi169 podman[210669]: 2024-03-12 03:57:58.876983002 +0000 UTC m=+0.118390287 container create a3f53dabd577604eaf301a116fd8631e9e5b578e7dde1586ec6bae6d3b20a6da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-12T03:57:59.173 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:59 smithi169 podman[210669]: 2024-03-12 03:57:59.105007134 +0000 UTC m=+0.346414381 container init a3f53dabd577604eaf301a116fd8631e9e5b578e7dde1586ec6bae6d3b20a6da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:57:59.173 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:59 smithi169 podman[210669]: 2024-03-12 03:57:59.108244362 +0000 UTC m=+0.349651604 container start a3f53dabd577604eaf301a116fd8631e9e5b578e7dde1586ec6bae6d3b20a6da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1-deactivate, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-12T03:57:59.673 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:59 smithi169 podman[210669]: 2024-03-12 03:57:59.204719328 +0000 UTC m=+0.446126576 container attach a3f53dabd577604eaf301a116fd8631e9e5b578e7dde1586ec6bae6d3b20a6da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-12T03:57:59.673 INFO:journalctl@ceph.osd.1.smithi169.stdout:Mar 12 03:57:59 smithi169 podman[210669]: 2024-03-12 03:57:59.396539671 +0000 UTC m=+0.637946901 container died a3f53dabd577604eaf301a116fd8631e9e5b578e7dde1586ec6bae6d3b20a6da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5) 2024-03-12T03:58:00.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.1.service' 2024-03-12T03:58:00.219 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:58:00.220 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-12T03:58:00.220 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-12T03:58:00.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.2 2024-03-12T03:58:00.423 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:00 smithi169 systemd[1]: Stopping Ceph osd.2 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:58:00.923 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:00 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2[56055]: 2024-03-12T03:58:00.536+0000 7ff1a2074700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:58:00.923 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:00 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2[56055]: 2024-03-12T03:58:00.536+0000 7ff1a2074700 -1 osd.2 595 *** Got signal Terminated *** 2024-03-12T03:58:00.923 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:00 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2[56055]: 2024-03-12T03:58:00.536+0000 7ff1a2074700 -1 osd.2 595 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:58:06.521 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:06 smithi169 podman[210775]: 2024-03-12 03:58:06.20682824 +0000 UTC m=+5.767855761 container died 2b1d3d1032e2ce163ee80537555945f0ecb1270156b13ee51cc30a7e21aaa754 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, io.buildah.version=1.33.5) 2024-03-12T03:58:06.905 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:06 smithi169 podman[210775]: 2024-03-12 03:58:06.628110626 +0000 UTC m=+6.189138155 container cleanup 2b1d3d1032e2ce163ee80537555945f0ecb1270156b13ee51cc30a7e21aaa754 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-12T03:58:06.906 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:06 smithi169 bash[210775]: ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2 2024-03-12T03:58:07.173 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:06 smithi169 podman[210792]: 2024-03-12 03:58:06.904495324 +0000 UTC m=+0.698676308 container remove 2b1d3d1032e2ce163ee80537555945f0ecb1270156b13ee51cc30a7e21aaa754 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-12T03:58:07.692 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:07 smithi169 podman[210892]: 2024-03-12 03:58:07.336387688 +0000 UTC m=+0.018185540 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:58:07.693 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:07 smithi169 podman[210892]: 2024-03-12 03:58:07.455385749 +0000 UTC m=+0.137183601 container create 228cbe05d5b39a7a91dc586e803dbb205fe928bb39389d2a0b7262ac2281a073 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:58:07.693 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:07 smithi169 podman[210892]: 2024-03-12 03:58:07.69122063 +0000 UTC m=+0.373018482 container init 228cbe05d5b39a7a91dc586e803dbb205fe928bb39389d2a0b7262ac2281a073 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-12T03:58:07.992 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:07 smithi169 podman[210892]: 2024-03-12 03:58:07.694781669 +0000 UTC m=+0.376579522 container start 228cbe05d5b39a7a91dc586e803dbb205fe928bb39389d2a0b7262ac2281a073 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-12T03:58:07.992 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:07 smithi169 podman[210892]: 2024-03-12 03:58:07.808211891 +0000 UTC m=+0.490009772 container attach 228cbe05d5b39a7a91dc586e803dbb205fe928bb39389d2a0b7262ac2281a073 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-12T03:58:07.992 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 12 03:58:07 smithi169 podman[210892]: 2024-03-12 03:58:07.990876857 +0000 UTC m=+0.672674718 container died 228cbe05d5b39a7a91dc586e803dbb205fe928bb39389d2a0b7262ac2281a073 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:58:08.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.2.service' 2024-03-12T03:58:08.597 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:58:08.597 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-12T03:58:08.598 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-12T03:58:08.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3 2024-03-12T03:58:08.923 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:08 smithi169 systemd[1]: Stopping Ceph osd.3 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:58:09.423 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:08 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[149269]: 2024-03-12T03:58:08.922+0000 7efe88385700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:58:09.423 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:08 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[149269]: 2024-03-12T03:58:08.922+0000 7efe88385700 -1 osd.3 595 *** Got signal Terminated *** 2024-03-12T03:58:09.423 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:08 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[149269]: 2024-03-12T03:58:08.922+0000 7efe88385700 -1 osd.3 595 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:58:12.986 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:12.507+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:12.987 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:12 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:12.746+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.274607+0000 front 2024-03-12T03:57:47.274517+0000 (oldest deadline 2024-03-12T03:58:12.574647+0000) 2024-03-12T03:58:13.673 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:13 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3[149269]: 2024-03-12T03:58:13.200+0000 7efe84979700 -1 osd.3 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.208917+0000 front 2024-03-12T03:57:48.209072+0000 (oldest deadline 2024-03-12T03:58:12.909030+0000) 2024-03-12T03:58:13.733 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:13.496+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:13.733 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:13.336+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:13.986 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:13 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:13.730+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.274607+0000 front 2024-03-12T03:57:47.274517+0000 (oldest deadline 2024-03-12T03:58:12.574647+0000) 2024-03-12T03:58:14.376 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:14 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:14.027+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:14.718 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:14 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:14.374+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:14.718 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:14 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:14.533+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:14.841 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:14 smithi169 podman[211000]: 2024-03-12 03:58:14.563591992 +0000 UTC m=+5.734939443 container died aa2a33b518fa524484a9721065801e882b8211ec8605dad6e715ef09ebbd9cbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, ceph=True, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-12T03:58:14.986 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:14 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:14.715+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.274607+0000 front 2024-03-12T03:57:47.274517+0000 (oldest deadline 2024-03-12T03:58:12.574647+0000) 2024-03-12T03:58:15.173 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:14 smithi169 podman[211000]: 2024-03-12 03:58:14.997884447 +0000 UTC m=+6.169231894 container cleanup aa2a33b518fa524484a9721065801e882b8211ec8605dad6e715ef09ebbd9cbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:58:15.173 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:14 smithi169 bash[211000]: ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3 2024-03-12T03:58:15.340 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:15 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:15.053+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:15.568 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:15 smithi169 podman[211017]: 2024-03-12 03:58:15.174328049 +0000 UTC m=+0.609880027 container remove aa2a33b518fa524484a9721065801e882b8211ec8605dad6e715ef09ebbd9cbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-12T03:58:15.726 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:15 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:15.337+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:15.726 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:15 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:15.557+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:15.854 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:15 smithi169 podman[211106]: 2024-03-12 03:58:15.566847245 +0000 UTC m=+0.111053887 container create be60c3fac05bfb547f54a95ca0cbb766057bfb6a7b4a8df6a4f1feb7bdc03d09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-12T03:58:15.854 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:15 smithi169 podman[211106]: 2024-03-12 03:58:15.473892509 +0000 UTC m=+0.018099168 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:58:15.855 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:15 smithi169 podman[211106]: 2024-03-12 03:58:15.777604159 +0000 UTC m=+0.321810801 container init be60c3fac05bfb547f54a95ca0cbb766057bfb6a7b4a8df6a4f1feb7bdc03d09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-03-12T03:58:15.855 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:15 smithi169 podman[211106]: 2024-03-12 03:58:15.78071379 +0000 UTC m=+0.324920431 container start be60c3fac05bfb547f54a95ca0cbb766057bfb6a7b4a8df6a4f1feb7bdc03d09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:58:15.855 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:15 smithi169 podman[211106]: 2024-03-12 03:58:15.852596798 +0000 UTC m=+0.396803449 container attach be60c3fac05bfb547f54a95ca0cbb766057bfb6a7b4a8df6a4f1feb7bdc03d09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-12T03:58:15.986 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:15 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:15.723+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.274607+0000 front 2024-03-12T03:57:47.274517+0000 (oldest deadline 2024-03-12T03:58:12.574647+0000) 2024-03-12T03:58:16.173 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:16 smithi169 podman[211106]: 2024-03-12 03:58:16.06744746 +0000 UTC m=+0.611654098 container died be60c3fac05bfb547f54a95ca0cbb766057bfb6a7b4a8df6a4f1feb7bdc03d09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-12T03:58:16.375 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:16 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:16.065+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:16.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3.service' 2024-03-12T03:58:16.625 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:16 smithi169 podman[211132]: 2024-03-12 03:58:16.580974685 +0000 UTC m=+0.508070889 container remove be60c3fac05bfb547f54a95ca0cbb766057bfb6a7b4a8df6a4f1feb7bdc03d09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-03-12T03:58:16.625 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:16 smithi169 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-12T03:58:16.625 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:16 smithi169 systemd[1]: Stopped Ceph osd.3 for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:58:16.625 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 12 03:58:16 smithi169 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.3.service: Consumed 11min 36.398s CPU time. 2024-03-12T03:58:16.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:58:16.640 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-12T03:58:16.640 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-12T03:58:16.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.4 2024-03-12T03:58:16.671 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:16 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:16.579+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:16.671 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:16 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:16.373+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:16.971 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:16 smithi173 systemd[1]: Stopping Ceph osd.4 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:58:16.971 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:16 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:16.742+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.274607+0000 front 2024-03-12T03:57:47.274517+0000 (oldest deadline 2024-03-12T03:58:12.574647+0000) 2024-03-12T03:58:16.972 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:16 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:16.968+0000 7fe0d9179700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:58:16.972 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:16 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:16.968+0000 7fe0d9179700 -1 osd.4 595 *** Got signal Terminated *** 2024-03-12T03:58:16.972 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:16 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:16.968+0000 7fe0d9179700 -1 osd.4 595 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:58:17.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:17 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:17.053+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:17.736 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:17 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:17.553+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:17.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:17 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:17.325+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:18.093 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:17 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:17.755+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.274607+0000 front 2024-03-12T03:57:47.274517+0000 (oldest deadline 2024-03-12T03:58:12.574647+0000) 2024-03-12T03:58:18.347 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:18 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:18.091+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:18.736 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:18 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:18.564+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:18.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:18 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:18.345+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:19.128 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:18 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:18.738+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.274607+0000 front 2024-03-12T03:57:47.274517+0000 (oldest deadline 2024-03-12T03:58:12.574647+0000) 2024-03-12T03:58:19.128 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:18 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:18.738+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:56.675719+0000 front 2024-03-12T03:57:56.675679+0000 (oldest deadline 2024-03-12T03:58:18.375536+0000) 2024-03-12T03:58:19.486 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:19 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:19.126+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:19.486 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:19 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:19.126+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:19.486 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:19 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:19.307+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:19.486 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:19 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:19.307+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:19.986 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:19 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:19.555+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:19.986 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:19 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:19.555+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:52.285018+0000 front 2024-03-12T03:57:52.284996+0000 (oldest deadline 2024-03-12T03:58:18.685304+0000) 2024-03-12T03:58:19.987 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:19 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:19.689+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.274607+0000 front 2024-03-12T03:57:47.274517+0000 (oldest deadline 2024-03-12T03:58:12.574647+0000) 2024-03-12T03:58:19.987 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:19 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:19.689+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:56.675719+0000 front 2024-03-12T03:57:56.675679+0000 (oldest deadline 2024-03-12T03:58:18.375536+0000) 2024-03-12T03:58:20.486 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:20 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:20.093+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:20.486 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:20 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:20.093+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:20.487 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:20 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:20.281+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:20.487 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:20 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:20.281+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:20.986 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:20 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:20.677+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.274607+0000 front 2024-03-12T03:57:47.274517+0000 (oldest deadline 2024-03-12T03:58:12.574647+0000) 2024-03-12T03:58:20.986 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:20 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:20.677+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:56.675719+0000 front 2024-03-12T03:57:56.675679+0000 (oldest deadline 2024-03-12T03:58:18.375536+0000) 2024-03-12T03:58:20.986 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:20 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:20.540+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:20.986 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:20 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:20.540+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:52.285018+0000 front 2024-03-12T03:57:52.284996+0000 (oldest deadline 2024-03-12T03:58:18.685304+0000) 2024-03-12T03:58:21.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:21 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:21.047+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:21.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:21 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:21.047+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:21.719 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:21 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:21.493+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:21.719 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:21 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:21.493+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:52.285018+0000 front 2024-03-12T03:57:52.284996+0000 (oldest deadline 2024-03-12T03:58:18.685304+0000) 2024-03-12T03:58:21.719 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:21 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:21.312+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:21.720 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:21 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:21.312+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:21.986 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:21 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:21.717+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.274607+0000 front 2024-03-12T03:57:47.274517+0000 (oldest deadline 2024-03-12T03:58:12.574647+0000) 2024-03-12T03:58:21.986 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:21 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4[38468]: 2024-03-12T03:58:21.717+0000 7fe0d576d700 -1 osd.4 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:56.675719+0000 front 2024-03-12T03:57:56.675679+0000 (oldest deadline 2024-03-12T03:58:18.375536+0000) 2024-03-12T03:58:22.357 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:22.077+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:22.357 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:22.077+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:22.736 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:22 smithi173 podman[162294]: 2024-03-12 03:58:22.573779356 +0000 UTC m=+5.726297534 container died b58884cda3be37184e4c9880eeed0013526e89fe3eb3a13e094546ca98dc0553 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-12T03:58:22.737 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:22.354+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:22.737 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:22.354+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:22.737 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:22.517+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:22.737 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:22 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:22.517+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:52.285018+0000 front 2024-03-12T03:57:52.284996+0000 (oldest deadline 2024-03-12T03:58:18.685304+0000) 2024-03-12T03:58:23.214 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:22 smithi173 podman[162294]: 2024-03-12 03:58:22.969326602 +0000 UTC m=+6.121844780 container cleanup b58884cda3be37184e4c9880eeed0013526e89fe3eb3a13e094546ca98dc0553 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True) 2024-03-12T03:58:23.214 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:22 smithi173 bash[162294]: ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4 2024-03-12T03:58:23.214 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:23.077+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:23.214 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:23.077+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:23.483 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:23 smithi173 podman[162311]: 2024-03-12 03:58:23.212440826 +0000 UTC m=+0.632011928 container remove b58884cda3be37184e4c9880eeed0013526e89fe3eb3a13e094546ca98dc0553 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-12T03:58:23.483 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:23.355+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:23.483 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:23.355+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:23.736 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:23 smithi173 podman[162400]: 2024-03-12 03:58:23.441388846 +0000 UTC m=+0.018062407 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:58:23.737 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:23 smithi173 podman[162400]: 2024-03-12 03:58:23.563875915 +0000 UTC m=+0.140549464 container create 28d09572281f19df78d7d226b3a94a449cbd4c87c3e58130eb1bb5e2a7a094ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-2b0101e, io.buildah.version=1.33.5) 2024-03-12T03:58:23.737 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:23.480+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:23.737 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:23 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:23.480+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:52.285018+0000 front 2024-03-12T03:57:52.284996+0000 (oldest deadline 2024-03-12T03:58:18.685304+0000) 2024-03-12T03:58:24.054 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:23 smithi173 podman[162400]: 2024-03-12 03:58:23.825266183 +0000 UTC m=+0.401939735 container init 28d09572281f19df78d7d226b3a94a449cbd4c87c3e58130eb1bb5e2a7a094ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True) 2024-03-12T03:58:24.054 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:23 smithi173 podman[162400]: 2024-03-12 03:58:23.828734973 +0000 UTC m=+0.405408522 container start 28d09572281f19df78d7d226b3a94a449cbd4c87c3e58130eb1bb5e2a7a094ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4-deactivate, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-12T03:58:24.054 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:23 smithi173 podman[162400]: 2024-03-12 03:58:23.930911135 +0000 UTC m=+0.507584782 container attach 28d09572281f19df78d7d226b3a94a449cbd4c87c3e58130eb1bb5e2a7a094ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-12T03:58:24.383 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:24 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:24.051+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:24.383 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:24 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:24.051+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:24.706 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 12 03:58:24 smithi173 podman[162400]: 2024-03-12 03:58:24.419302531 +0000 UTC m=+0.995976088 container died 28d09572281f19df78d7d226b3a94a449cbd4c87c3e58130eb1bb5e2a7a094ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-12T03:58:24.706 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:24 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:24.380+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:24.707 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:24 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:24.380+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:24.707 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:24 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:24.512+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:24.707 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:24 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:24.512+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:52.285018+0000 front 2024-03-12T03:57:52.284996+0000 (oldest deadline 2024-03-12T03:58:18.685304+0000) 2024-03-12T03:58:25.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.4.service' 2024-03-12T03:58:25.104 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:58:25.104 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-12T03:58:25.104 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-12T03:58:25.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.5 2024-03-12T03:58:25.412 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:25 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:25.003+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:25.412 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:25 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:25.003+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:25.412 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:25 smithi173 systemd[1]: Stopping Ceph osd.5 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:58:25.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:25 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:25.408+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:25.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:25 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:25.408+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:25.737 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:25 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:25.473+0000 7fe262694700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:58:25.737 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:25 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:25.473+0000 7fe262694700 -1 osd.5 595 *** Got signal Terminated *** 2024-03-12T03:58:25.737 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:25 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:25.473+0000 7fe262694700 -1 osd.5 595 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:58:25.737 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:25 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:25.533+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:25.737 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:25 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:25.533+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:52.285018+0000 front 2024-03-12T03:57:52.284996+0000 (oldest deadline 2024-03-12T03:58:18.685304+0000) 2024-03-12T03:58:26.440 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:26 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:26.035+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:26.440 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:26 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:26.035+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:26.736 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:26 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:26.562+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:26.736 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:26 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:26.562+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:52.285018+0000 front 2024-03-12T03:57:52.284996+0000 (oldest deadline 2024-03-12T03:58:18.685304+0000) 2024-03-12T03:58:26.736 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:26 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:26.562+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:01.586217+0000 front 2024-03-12T03:58:01.586146+0000 (oldest deadline 2024-03-12T03:58:26.286061+0000) 2024-03-12T03:58:26.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:26 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:26.437+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:26.737 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:26 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:26.437+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:27.411 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:27 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:27.014+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:27.412 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:27 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:27.014+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:27.412 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:27 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:27.014+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:27.737 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:27 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:27.612+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:27.737 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:27 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:27.612+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:52.285018+0000 front 2024-03-12T03:57:52.284996+0000 (oldest deadline 2024-03-12T03:58:18.685304+0000) 2024-03-12T03:58:27.737 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:27 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:27.612+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:01.586217+0000 front 2024-03-12T03:58:01.586146+0000 (oldest deadline 2024-03-12T03:58:26.286061+0000) 2024-03-12T03:58:27.737 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:27 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:27.409+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:27.737 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:27 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:27.409+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:28.399 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:28 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:28.011+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:28.400 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:28 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:28.011+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:28.400 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:28 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:28.011+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:28.736 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:28 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:28.640+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:28.736 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:28 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:28.640+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:52.285018+0000 front 2024-03-12T03:57:52.284996+0000 (oldest deadline 2024-03-12T03:58:18.685304+0000) 2024-03-12T03:58:28.736 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:28 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:28.640+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:01.586217+0000 front 2024-03-12T03:58:01.586146+0000 (oldest deadline 2024-03-12T03:58:26.286061+0000) 2024-03-12T03:58:28.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:28 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:28.397+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:28.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:28 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:28.397+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:29.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:28 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:28.971+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:29.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:28 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:28.971+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:29.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:28 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:28.972+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:29.736 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:29 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:29.616+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:48.784663+0000 front 2024-03-12T03:57:48.784578+0000 (oldest deadline 2024-03-12T03:58:12.284556+0000) 2024-03-12T03:58:29.736 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:29 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:29.616+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:52.285018+0000 front 2024-03-12T03:57:52.284996+0000 (oldest deadline 2024-03-12T03:58:18.685304+0000) 2024-03-12T03:58:29.736 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:29 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5[44069]: 2024-03-12T03:58:29.616+0000 7fe25ec88700 -1 osd.5 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:01.586217+0000 front 2024-03-12T03:58:01.586146+0000 (oldest deadline 2024-03-12T03:58:26.286061+0000) 2024-03-12T03:58:29.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:29 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:29.426+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:29.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:29 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:29.426+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:29.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:29 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:29.426+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:03.114454+0000 front 2024-03-12T03:58:03.114413+0000 (oldest deadline 2024-03-12T03:58:29.014354+0000) 2024-03-12T03:58:30.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:29 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:29.937+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:30.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:29 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:29.937+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:30.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:29 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:29.937+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:30.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:30 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:30.415+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:30.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:30 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:30.415+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:30.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:30 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:30.415+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:03.114454+0000 front 2024-03-12T03:58:03.114413+0000 (oldest deadline 2024-03-12T03:58:29.014354+0000) 2024-03-12T03:58:31.236 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:31 smithi173 podman[162516]: 2024-03-12 03:58:31.108075255 +0000 UTC m=+5.770702156 container died 8291b2243b78fa7be075ffe115909fee8778b47f6afc1836a5964f2c81cda099 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, io.buildah.version=1.33.5) 2024-03-12T03:58:31.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:30 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:30.967+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:31.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:30 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:30.967+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:31.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:30 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:30.967+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:31.694 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:31 smithi173 podman[162516]: 2024-03-12 03:58:31.483060785 +0000 UTC m=+6.145687683 container cleanup 8291b2243b78fa7be075ffe115909fee8778b47f6afc1836a5964f2c81cda099 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-12T03:58:31.694 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:31 smithi173 bash[162516]: ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5 2024-03-12T03:58:31.695 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:31 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:31.388+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:31.695 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:31 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:31.388+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:31.695 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:31 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:31.388+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:03.114454+0000 front 2024-03-12T03:58:03.114413+0000 (oldest deadline 2024-03-12T03:58:29.014354+0000) 2024-03-12T03:58:31.975 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:31 smithi173 podman[162535]: 2024-03-12 03:58:31.693002395 +0000 UTC m=+0.580240564 container remove 8291b2243b78fa7be075ffe115909fee8778b47f6afc1836a5964f2c81cda099 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-12T03:58:32.236 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:32 smithi173 podman[162624]: 2024-03-12 03:58:31.914742874 +0000 UTC m=+0.017865684 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:58:32.236 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:32 smithi173 podman[162624]: 2024-03-12 03:58:32.035642485 +0000 UTC m=+0.138765282 container create b9804206818d05acc73ba950a27969b8d66a3c1e7bfb88385ebd9591fb7bf9f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-12T03:58:32.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:31 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:31.972+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:32.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:31 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:31.972+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:32.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:31 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:31.972+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:32.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:31 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:31.972+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:32.569 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:32.408+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:32.569 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:32.408+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:32.570 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:32.408+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:03.114454+0000 front 2024-03-12T03:58:03.114413+0000 (oldest deadline 2024-03-12T03:58:29.014354+0000) 2024-03-12T03:58:32.570 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:32 smithi173 podman[162624]: 2024-03-12 03:58:32.280401733 +0000 UTC m=+0.383524546 container init b9804206818d05acc73ba950a27969b8d66a3c1e7bfb88385ebd9591fb7bf9f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-12T03:58:32.570 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:32 smithi173 podman[162624]: 2024-03-12 03:58:32.28382912 +0000 UTC m=+0.386951918 container start b9804206818d05acc73ba950a27969b8d66a3c1e7bfb88385ebd9591fb7bf9f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-12T03:58:32.570 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:32 smithi173 podman[162624]: 2024-03-12 03:58:32.371667494 +0000 UTC m=+0.474790293 container attach b9804206818d05acc73ba950a27969b8d66a3c1e7bfb88385ebd9591fb7bf9f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-12T03:58:32.919 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 12 03:58:32 smithi173 podman[162624]: 2024-03-12 03:58:32.568509281 +0000 UTC m=+0.671632084 container died b9804206818d05acc73ba950a27969b8d66a3c1e7bfb88385ebd9591fb7bf9f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-12T03:58:33.185 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:32.934+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:33.186 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:32.934+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:33.186 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:32.934+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:33.186 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:32 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:32.934+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:33.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.5.service' 2024-03-12T03:58:33.246 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:58:33.247 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-12T03:58:33.247 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-12T03:58:33.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.6 2024-03-12T03:58:33.486 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:33 smithi173 systemd[1]: Stopping Ceph osd.6 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:58:33.486 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:33.418+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:33.486 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:33.418+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:33.486 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:33.418+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:03.114454+0000 front 2024-03-12T03:58:03.114413+0000 (oldest deadline 2024-03-12T03:58:29.014354+0000) 2024-03-12T03:58:33.892 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:33.595+0000 7fa43297f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:58:33.892 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:33.595+0000 7fa43297f700 -1 osd.6 595 *** Got signal Terminated *** 2024-03-12T03:58:33.892 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:33.595+0000 7fa43297f700 -1 osd.6 595 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:58:34.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:33.889+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:34.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:33.889+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:34.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:33.889+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:34.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:33 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:33.889+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:34.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:34 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:34.427+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:34.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:34 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:34.427+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:34.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:34 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:34.427+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:03.114454+0000 front 2024-03-12T03:58:03.114413+0000 (oldest deadline 2024-03-12T03:58:29.014354+0000) 2024-03-12T03:58:35.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:34 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:34.861+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:35.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:34 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:34.861+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:35.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:34 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:34.861+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:35.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:34 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:34.861+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:35.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:35 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:35.402+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:35.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:35 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:35.402+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:35.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:35 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:35.402+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:03.114454+0000 front 2024-03-12T03:58:03.114413+0000 (oldest deadline 2024-03-12T03:58:29.014354+0000) 2024-03-12T03:58:35.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:35 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:35.402+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:11.215745+0000 front 2024-03-12T03:58:11.215765+0000 (oldest deadline 2024-03-12T03:58:34.715616+0000) 2024-03-12T03:58:36.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:35 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:35.812+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:36.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:35 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:35.812+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:36.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:35 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:35.812+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:36.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:35 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:35.812+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:36.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:36 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:36.375+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:36.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:36 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:36.375+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:36.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:36 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:36.375+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:03.114454+0000 front 2024-03-12T03:58:03.114413+0000 (oldest deadline 2024-03-12T03:58:29.014354+0000) 2024-03-12T03:58:36.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:36 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:36.375+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:11.215745+0000 front 2024-03-12T03:58:11.215765+0000 (oldest deadline 2024-03-12T03:58:34.715616+0000) 2024-03-12T03:58:37.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:36 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:36.862+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:37.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:36 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:36.862+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:37.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:36 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:36.862+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:37.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:36 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:36.862+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:12.380141+0000 front 2024-03-12T03:58:12.380128+0000 (oldest deadline 2024-03-12T03:58:36.480060+0000) 2024-03-12T03:58:37.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:36 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:36.862+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:37.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:37 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:37.384+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:37.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:37 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:37.384+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:37.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:37 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:37.384+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:03.114454+0000 front 2024-03-12T03:58:03.114413+0000 (oldest deadline 2024-03-12T03:58:29.014354+0000) 2024-03-12T03:58:37.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:37 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:37.384+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:11.215745+0000 front 2024-03-12T03:58:11.215765+0000 (oldest deadline 2024-03-12T03:58:34.715616+0000) 2024-03-12T03:58:38.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:37 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:37.862+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:38.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:37 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:37.862+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:38.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:37 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:37.862+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:38.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:37 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:37.862+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:12.380141+0000 front 2024-03-12T03:58:12.380128+0000 (oldest deadline 2024-03-12T03:58:36.480060+0000) 2024-03-12T03:58:38.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:37 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:37.862+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:38.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:38 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:38.375+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:47.912383+0000 front 2024-03-12T03:57:47.912260+0000 (oldest deadline 2024-03-12T03:58:13.212253+0000) 2024-03-12T03:58:38.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:38 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:38.375+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.212688+0000 front 2024-03-12T03:57:53.212798+0000 (oldest deadline 2024-03-12T03:58:19.112685+0000) 2024-03-12T03:58:38.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:38 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:38.375+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:03.114454+0000 front 2024-03-12T03:58:03.114413+0000 (oldest deadline 2024-03-12T03:58:29.014354+0000) 2024-03-12T03:58:38.736 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:38 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6[49611]: 2024-03-12T03:58:38.375+0000 7fa42ef73700 -1 osd.6 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:11.215745+0000 front 2024-03-12T03:58:11.215765+0000 (oldest deadline 2024-03-12T03:58:34.715616+0000) 2024-03-12T03:58:39.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:38 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:38.885+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:39.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:38 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:38.885+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:39.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:38 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:38.885+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:39.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:38 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:38.885+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:12.380141+0000 front 2024-03-12T03:58:12.380128+0000 (oldest deadline 2024-03-12T03:58:36.480060+0000) 2024-03-12T03:58:39.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:38 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:38.886+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:39.548 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:39 smithi173 podman[162730]: 2024-03-12 03:58:39.279278332 +0000 UTC m=+5.816713768 container died 3ec79f244b1f842f99f54a433c34e01a5f63cc0162613d25254590595c741575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-12T03:58:39.925 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:39 smithi173 podman[162730]: 2024-03-12 03:58:39.63775886 +0000 UTC m=+6.175194309 container cleanup 3ec79f244b1f842f99f54a433c34e01a5f63cc0162613d25254590595c741575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-12T03:58:39.925 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:39 smithi173 bash[162730]: ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6 2024-03-12T03:58:40.236 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:39 smithi173 podman[162747]: 2024-03-12 03:58:39.922769422 +0000 UTC m=+0.644532863 container remove 3ec79f244b1f842f99f54a433c34e01a5f63cc0162613d25254590595c741575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-12T03:58:40.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:39 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:39.933+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:40.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:39 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:39.933+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:40.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:39 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:39.933+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:40.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:39 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:39.933+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:12.380141+0000 front 2024-03-12T03:58:12.380128+0000 (oldest deadline 2024-03-12T03:58:36.480060+0000) 2024-03-12T03:58:40.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:39 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:39.934+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:40.519 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:40 smithi173 podman[162836]: 2024-03-12 03:58:40.163012013 +0000 UTC m=+0.019291053 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:58:40.519 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:40 smithi173 podman[162836]: 2024-03-12 03:58:40.314288664 +0000 UTC m=+0.170567736 container create aea8af6d80c352fca82cf85f0346aad2999b7f8ccc617408164f9cc48553f00a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6-deactivate, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-12T03:58:40.519 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:40 smithi173 podman[162836]: 2024-03-12 03:58:40.51804041 +0000 UTC m=+0.374319436 container init aea8af6d80c352fca82cf85f0346aad2999b7f8ccc617408164f9cc48553f00a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-12T03:58:40.819 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:40 smithi173 podman[162836]: 2024-03-12 03:58:40.52103594 +0000 UTC m=+0.377314977 container start aea8af6d80c352fca82cf85f0346aad2999b7f8ccc617408164f9cc48553f00a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-12T03:58:40.819 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:40 smithi173 podman[162836]: 2024-03-12 03:58:40.601445209 +0000 UTC m=+0.457724270 container attach aea8af6d80c352fca82cf85f0346aad2999b7f8ccc617408164f9cc48553f00a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-2b0101e, GIT_BRANCH=HEAD) 2024-03-12T03:58:41.106 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 12 03:58:40 smithi173 podman[162836]: 2024-03-12 03:58:40.818147584 +0000 UTC m=+0.674426625 container died aea8af6d80c352fca82cf85f0346aad2999b7f8ccc617408164f9cc48553f00a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-03-12T03:58:41.107 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:40 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:40.955+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:41.107 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:40 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:40.955+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:41.107 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:40 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:40.955+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:41.107 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:40 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:40.955+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:12.380141+0000 front 2024-03-12T03:58:12.380128+0000 (oldest deadline 2024-03-12T03:58:36.480060+0000) 2024-03-12T03:58:41.107 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:40 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:40.956+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:41.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.6.service' 2024-03-12T03:58:41.423 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:58:41.423 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-12T03:58:41.423 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-12T03:58:41.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.7 2024-03-12T03:58:41.736 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:41 smithi173 systemd[1]: Stopping Ceph osd.7 for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:58:42.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:41 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:41.741+0000 7f09b7458700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:58:42.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:41 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:41.741+0000 7f09b7458700 -1 osd.7 595 *** Got signal Terminated *** 2024-03-12T03:58:42.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:41 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:41.741+0000 7f09b7458700 -1 osd.7 595 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:58:42.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:41 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:41.921+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:42.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:41 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:41.921+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:42.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:41 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:41.921+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:42.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:41 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:41.921+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:12.380141+0000 front 2024-03-12T03:58:12.380128+0000 (oldest deadline 2024-03-12T03:58:36.480060+0000) 2024-03-12T03:58:42.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:41 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:41.921+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:43.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:42.956+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:43.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:42.956+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:43.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:42.956+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:43.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:42.956+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:12.380141+0000 front 2024-03-12T03:58:12.380128+0000 (oldest deadline 2024-03-12T03:58:36.480060+0000) 2024-03-12T03:58:43.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:42 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:42.957+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:44.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:43.923+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:44.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:43.923+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:44.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:43.923+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:44.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:43.923+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:12.380141+0000 front 2024-03-12T03:58:12.380128+0000 (oldest deadline 2024-03-12T03:58:36.480060+0000) 2024-03-12T03:58:44.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:43 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:43.923+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:45.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:44 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:44.921+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:45.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:44 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:44.921+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:45.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:44 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:44.921+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:45.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:44 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:44.921+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:12.380141+0000 front 2024-03-12T03:58:12.380128+0000 (oldest deadline 2024-03-12T03:58:36.480060+0000) 2024-03-12T03:58:45.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:44 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:44.921+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.173:6802 osd.4 since back 2024-03-12T03:58:19.380647+0000 front 2024-03-12T03:58:19.380722+0000 (oldest deadline 2024-03-12T03:58:44.680623+0000) 2024-03-12T03:58:45.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:44 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:44.921+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:46.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:45 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:45.884+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6803 osd.0 since back 2024-03-12T03:57:49.076742+0000 front 2024-03-12T03:57:49.076895+0000 (oldest deadline 2024-03-12T03:58:13.176749+0000) 2024-03-12T03:58:46.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:45 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:45.884+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6807 osd.1 since back 2024-03-12T03:57:53.177445+0000 front 2024-03-12T03:57:53.177461+0000 (oldest deadline 2024-03-12T03:58:19.077342+0000) 2024-03-12T03:58:46.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:45 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:45.884+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6811 osd.2 since back 2024-03-12T03:58:05.478680+0000 front 2024-03-12T03:58:05.478623+0000 (oldest deadline 2024-03-12T03:58:26.578493+0000) 2024-03-12T03:58:46.236 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:45 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:45.884+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.169:6815 osd.3 since back 2024-03-12T03:58:12.380141+0000 front 2024-03-12T03:58:12.380128+0000 (oldest deadline 2024-03-12T03:58:36.480060+0000) 2024-03-12T03:58:46.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:45 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:45.884+0000 7f09b3a4c700 -1 osd.7 595 heartbeat_check: no reply from 172.21.15.173:6802 osd.4 since back 2024-03-12T03:58:19.380647+0000 front 2024-03-12T03:58:19.380722+0000 (oldest deadline 2024-03-12T03:58:44.680623+0000) 2024-03-12T03:58:46.237 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:45 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7[55399]: 2024-03-12T03:58:45.884+0000 7f09b3a4c700 -1 osd.7 595 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14529.0:1466 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e595) 2024-03-12T03:58:47.618 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:47 smithi173 podman[162941]: 2024-03-12 03:58:47.297820129 +0000 UTC m=+5.661627626 container died be9cbf241cd60fb05de56b63d7280f80e40a572dde892389209932c8cb68bac0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-12T03:58:47.618 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:47 smithi173 podman[162941]: 2024-03-12 03:58:47.616657682 +0000 UTC m=+5.980465224 container cleanup be9cbf241cd60fb05de56b63d7280f80e40a572dde892389209932c8cb68bac0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:58:47.986 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:47 smithi173 bash[162941]: ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7 2024-03-12T03:58:47.986 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:47 smithi173 podman[162974]: 2024-03-12 03:58:47.801159334 +0000 UTC m=+0.504669125 container remove be9cbf241cd60fb05de56b63d7280f80e40a572dde892389209932c8cb68bac0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-12T03:58:48.486 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:48 smithi173 podman[163068]: 2024-03-12 03:58:48.112484998 +0000 UTC m=+0.018279154 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:58:48.486 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:48 smithi173 podman[163068]: 2024-03-12 03:58:48.227007065 +0000 UTC m=+0.132801213 container create 91336b2a943c5b49b70fe4512dac658afdbc50e1031a6516cff53f363bcecd5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-03-12T03:58:48.486 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:48 smithi173 podman[163068]: 2024-03-12 03:58:48.454471839 +0000 UTC m=+0.360265987 container init 91336b2a943c5b49b70fe4512dac658afdbc50e1031a6516cff53f363bcecd5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:58:48.486 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:48 smithi173 podman[163068]: 2024-03-12 03:58:48.45787454 +0000 UTC m=+0.363668688 container start 91336b2a943c5b49b70fe4512dac658afdbc50e1031a6516cff53f363bcecd5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS) 2024-03-12T03:58:48.986 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:48 smithi173 podman[163068]: 2024-03-12 03:58:48.580105281 +0000 UTC m=+0.485899443 container attach 91336b2a943c5b49b70fe4512dac658afdbc50e1031a6516cff53f363bcecd5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-12T03:58:48.986 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 12 03:58:48 smithi173 podman[163068]: 2024-03-12 03:58:48.737238701 +0000 UTC m=+0.643032842 container died 91336b2a943c5b49b70fe4512dac658afdbc50e1031a6516cff53f363bcecd5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-12T03:58:49.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@osd.7.service' 2024-03-12T03:58:49.337 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:58:49.337 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-12T03:58:49.337 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-12T03:58:49.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-12T03:58:49.673 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:58:49 smithi169 systemd[1]: Stopping Ceph rgw.foo.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:58:49.994 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:58:49 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-rgw-foo-a[66679]: 2024-03-12T03:58:49.679+0000 7ff2786a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:58:49.994 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:58:49 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-rgw-foo-a[66679]: 2024-03-12T03:58:49.679+0000 7ff286f1bac0 -1 shutting down 2024-03-12T03:58:55.236 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:58:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:58:54.916Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.169:8765: connect: connection refused" 2024-03-12T03:58:55.236 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:58:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:58:54.917Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.169:8765: connect: connection refused" 2024-03-12T03:58:55.237 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:58:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:58:54.919Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.169:8765: connect: connection refused" 2024-03-12T03:58:55.237 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:58:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:58:54.920Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.169:8765: connect: connection refused" 2024-03-12T03:58:55.237 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:58:54 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:58:54.922Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.169:8765: connect: connection refused" 2024-03-12T03:59:00.021 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:58:59 smithi169 bash[211213]: time="2024-03-12T03:58:59Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-677d624a-e020-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-12T03:59:00.021 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:58:59 smithi169 podman[211213]: 2024-03-12 03:58:59.705035492 +0000 UTC m=+10.175321651 container died 0025bc06a0703e548bcf9283256b08a984ce108f4eeaec372e30cba18cf5951d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-12T03:59:00.274 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:59:00 smithi169 podman[211213]: 2024-03-12 03:59:00.25415266 +0000 UTC m=+10.724438829 container cleanup 0025bc06a0703e548bcf9283256b08a984ce108f4eeaec372e30cba18cf5951d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-12T03:59:00.274 INFO:journalctl@ceph.rgw.foo.a.smithi169.stdout:Mar 12 03:59:00 smithi169 bash[211213]: ceph-677d624a-e020-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-12T03:59:00.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-12T03:59:00.712 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:59:00.712 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-12T03:59:00.712 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-12T03:59:00.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-677d624a-e020-11ee-95c8-87774f69a715@prometheus.a 2024-03-12T03:59:00.995 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:00 smithi173 systemd[1]: Stopping Ceph prometheus.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:59:01.280 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:00 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:59:00.994Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:59:01.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:00 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:59:00.994Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-12T03:59:01.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:00 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:59:00.994Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-12T03:59:01.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:00 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:59:00.994Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T03:59:01.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:00 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:59:00.994Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-12T03:59:01.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:00 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:59:00.994Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-12T03:59:01.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:00 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:59:00.994Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T03:59:01.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:01 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:59:00.994Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-12T03:59:01.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:01 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:59:01.001Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-12T03:59:01.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:01 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:59:01.001Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T03:59:01.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:01 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:59:01.001Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-12T03:59:01.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:01 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a[68067]: ts=2024-03-12T03:59:01.001Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-12T03:59:01.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:01 smithi173 podman[163174]: 2024-03-12 03:59:01.00441754 +0000 UTC m=+0.104211061 container died 48619c1583bf34b4bda7d98ccfdbed6f47506c1bc2a5f7826009531171ecbbfc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:59:01.588 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:01 smithi173 podman[163174]: 2024-03-12 03:59:01.37747162 +0000 UTC m=+0.477265148 container cleanup 48619c1583bf34b4bda7d98ccfdbed6f47506c1bc2a5f7826009531171ecbbfc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:59:01.588 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 12 03:59:01 smithi173 bash[163174]: ceph-677d624a-e020-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T03:59:01.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-677d624a-e020-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-12T03:59:01.829 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:59:01.829 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-12T03:59:01.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 677d624a-e020-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T03:59:02.020 INFO:teuthology.orchestra.run.smithi169.stdout:Deleting cluster with fsid: 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:59:04.423 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:59:04 smithi169 systemd[1]: Stopping Ceph alertmanager.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:59:04.827 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:59:04 smithi169 ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a[81118]: ts=2024-03-12T03:59:04.525Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:59:04.828 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:59:04 smithi169 podman[211720]: 2024-03-12 03:59:04.566872853 +0000 UTC m=+0.139835542 container died 7561c849829672b755d52fb85640fb77a5c1387d0d1187b8776661531e87bf78 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:59:05.173 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:59:04 smithi169 podman[211720]: 2024-03-12 03:59:04.992885545 +0000 UTC m=+0.565848284 container cleanup 7561c849829672b755d52fb85640fb77a5c1387d0d1187b8776661531e87bf78 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:59:05.173 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:59:04 smithi169 bash[211720]: ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a 2024-03-12T03:59:05.673 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:59:05 smithi169 podman[211734]: 2024-03-12 03:59:05.419862326 +0000 UTC m=+0.848037749 container remove 7561c849829672b755d52fb85640fb77a5c1387d0d1187b8776661531e87bf78 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:59:05.673 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:59:05 smithi169 podman[211734]: 2024-03-12 03:59:05.49558857 +0000 UTC m=+0.923763985 volume remove 45aa4dd8f43e7447f7fb95b648003a1bac6448b7acc22a0ff0707591098db799 2024-03-12T03:59:06.173 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:59:05 smithi169 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-12T03:59:06.173 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:59:05 smithi169 systemd[1]: Stopped Ceph alertmanager.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:59:06.174 INFO:journalctl@ceph.alertmanager.a.smithi169.stdout:Mar 12 03:59:05 smithi169 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.710s CPU time. 2024-03-12T03:59:06.174 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:59:05 smithi169 systemd[1]: Stopping Ceph node-exporter.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:59:06.466 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:59:06 smithi169 podman[211850]: 2024-03-12 03:59:06.200398903 +0000 UTC m=+0.114741687 container died 07069691a7352c6e3d11e30cea279d9aacd1fbe482a708d2f151cf62c251fe8c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:59:06.818 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:59:06 smithi169 podman[211850]: 2024-03-12 03:59:06.640399921 +0000 UTC m=+0.554742715 container cleanup 07069691a7352c6e3d11e30cea279d9aacd1fbe482a708d2f151cf62c251fe8c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:59:06.818 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:59:06 smithi169 bash[211850]: ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a 2024-03-12T03:59:07.173 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:59:06 smithi169 podman[211864]: 2024-03-12 03:59:06.816653056 +0000 UTC m=+0.617724608 container remove 07069691a7352c6e3d11e30cea279d9aacd1fbe482a708d2f151cf62c251fe8c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:59:07.173 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:59:06 smithi169 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:59:07.173 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:59:07 smithi169 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-12T03:59:07.173 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:59:07 smithi169 systemd[1]: Stopped Ceph node-exporter.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:59:07.173 INFO:journalctl@ceph.node-exporter.a.smithi169.stdout:Mar 12 03:59:07 smithi169 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 4.747s CPU time. 2024-03-12T03:59:07.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 677d624a-e020-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T03:59:07.918 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T03:59:09.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:59:09 smithi173 systemd[1]: Stopping Ceph iscsi.iscsi.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:59:10.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:59:10 smithi173 ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a[60489]: debug Shutdown received 2024-03-12T03:59:20.397 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:59:20 smithi173 bash[163585]: time="2024-03-12T03:59:20Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-12T03:59:20.397 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:59:20 smithi173 podman[163585]: 2024-03-12 03:59:20.109838944 +0000 UTC m=+10.145724987 container died ac42ed100c151baa7964ed750753707225444a9d6585c443187cc6fab399a024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:59:20.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:59:20 smithi173 podman[163585]: 2024-03-12 03:59:20.497764304 +0000 UTC m=+10.533650353 container cleanup ac42ed100c151baa7964ed750753707225444a9d6585c443187cc6fab399a024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-12T03:59:20.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:59:20 smithi173 bash[163585]: ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a 2024-03-12T03:59:21.121 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:59:20 smithi173 podman[163600]: 2024-03-12 03:59:20.797496053 +0000 UTC m=+0.688257518 container remove ac42ed100c151baa7964ed750753707225444a9d6585c443187cc6fab399a024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-iscsi-iscsi-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-12T03:59:21.121 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:59:20 smithi173 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-12T03:59:21.121 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:59:20 smithi173 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-12T03:59:21.121 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:59:20 smithi173 systemd[1]: Stopped Ceph iscsi.iscsi.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:59:21.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 12 03:59:20 smithi173 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Consumed 3.536s CPU time. 2024-03-12T03:59:22.736 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:59:22 smithi173 systemd[1]: Stopping Ceph grafana.a for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:59:23.042 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:59:22 smithi173 podman[163869]: 2024-03-12 03:59:22.784070213 +0000 UTC m=+0.123830478 container died 93c2f834c6423dd67068acdfbce512d1be78c1bd64dab0139fdebf7e3f4bd6aa (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, version=8.5, description=Ceph Grafana Container, io.openshift.expose-services=, maintainer=Paul Cuzner ) 2024-03-12T03:59:23.334 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:59:23 smithi173 podman[163869]: 2024-03-12 03:59:23.123272177 +0000 UTC m=+0.463032446 container cleanup 93c2f834c6423dd67068acdfbce512d1be78c1bd64dab0139fdebf7e3f4bd6aa (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, description=Ceph Grafana Container, version=8.5, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., architecture=x86_64, io.buildah.version=1.32.0, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-12T03:59:23.334 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:59:23 smithi173 bash[163869]: ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a 2024-03-12T03:59:23.736 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:59:23 smithi173 podman[163883]: 2024-03-12 03:59:23.332806342 +0000 UTC m=+0.545710461 container remove 93c2f834c6423dd67068acdfbce512d1be78c1bd64dab0139fdebf7e3f4bd6aa (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-grafana-a, name=ubi8, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, description=Ceph Grafana Container, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, release=239.1651231664, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, version=8.5) 2024-03-12T03:59:23.736 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:59:23 smithi173 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:59:23.736 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:59:23 smithi173 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-12T03:59:23.736 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:59:23 smithi173 systemd[1]: Stopped Ceph grafana.a for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:59:23.736 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 12 03:59:23 smithi173 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@grafana.a.service: Consumed 13.175s CPU time. 2024-03-12T03:59:24.039 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:59:23 smithi173 systemd[1]: Stopping Ceph node-exporter.b for 677d624a-e020-11ee-95c8-87774f69a715... 2024-03-12T03:59:24.343 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:59:24 smithi173 podman[164000]: 2024-03-12 03:59:24.038194474 +0000 UTC m=+0.113806024 container died e391b52dfd8cddcbd4382e3ecb2927d0a041a3f2853c962962b9eb7a68618870 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:59:24.655 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:59:24 smithi173 podman[164000]: 2024-03-12 03:59:24.444575716 +0000 UTC m=+0.520187265 container cleanup e391b52dfd8cddcbd4382e3ecb2927d0a041a3f2853c962962b9eb7a68618870 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:59:24.655 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:59:24 smithi173 bash[164000]: ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b 2024-03-12T03:59:24.986 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:59:24 smithi173 podman[164012]: 2024-03-12 03:59:24.654140798 +0000 UTC m=+0.615865893 container remove e391b52dfd8cddcbd4382e3ecb2927d0a041a3f2853c962962b9eb7a68618870 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-677d624a-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:59:24.986 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:59:24 smithi173 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:59:24.986 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:59:24 smithi173 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-12T03:59:24.986 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:59:24 smithi173 systemd[1]: Stopped Ceph node-exporter.b for 677d624a-e020-11ee-95c8-87774f69a715. 2024-03-12T03:59:24.986 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 12 03:59:24 smithi173 systemd[1]: ceph-677d624a-e020-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 4.497s CPU time. 2024-03-12T03:59:25.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:59:25.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:59:25.697 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-12T03:59:25.699 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869/remote/smithi169/crash 2024-03-12T03:59:25.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/crash -- . 2024-03-12T03:59:25.755 INFO:teuthology.orchestra.run.smithi169.stderr:tar: /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T03:59:25.755 INFO:teuthology.orchestra.run.smithi169.stderr:tar: Error is not recoverable: exiting now 2024-03-12T03:59:25.757 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869/remote/smithi173/crash 2024-03-12T03:59:25.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/crash -- . 2024-03-12T03:59:25.809 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/677d624a-e020-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T03:59:25.809 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-03-12T03:59:25.810 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-12T03:59:25.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-12T03:59:25.932 INFO:teuthology.orchestra.run.smithi169.stdout:2024-03-12T03:40:00.000170+0000 mon.a (mon.0) 1089 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-12T03:59:25.932 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-12T03:59:25.932 DEBUG:teuthology.orchestra.run.smithi169:> sudo egrep '\[SEC\]' /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-12T03:59:25.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo egrep '\[ERR\]' /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-12T03:59:26.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo egrep '\[WRN\]' /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-12T03:59:26.107 INFO:teuthology.orchestra.run.smithi169.stdout:2024-03-12T03:40:00.000170+0000 mon.a (mon.0) 1089 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-12T03:59:26.107 INFO:tasks.cephadm:Compressing logs... 2024-03-12T03:59:26.107 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T03:59:26.149 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T03:59:26.198 INFO:teuthology.orchestra.run.smithi169.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-12T03:59:26.198 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T03:59:26.199 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-12T03:59:26.199 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T03:59:26.206 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-12T03:59:26.206 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T03:59:26.207 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-12T03:59:26.207 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.6.log: /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-12T03:59:26.207 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-12T03:59:26.207 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-12T03:59:26.207 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/tcmu-runner.log 2024-03-12T03:59:26.207 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T03:59:26.207 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.log 2024-03-12T03:59:26.207 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T03:59:26.207 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-12T03:59:26.208 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-12T03:59:26.209 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T03:59:26.209 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-12T03:59:26.210 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-12T03:59:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-12T03:59:26.212 INFO:teuthology.orchestra.run.smithi169.stderr: 57.4% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-12T03:59:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-12T03:59:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-12T03:59:26.213 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T03:59:26.213 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.log 2024-03-12T03:59:26.213 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-12T03:59:26.213 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T03:59:26.213 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/objectstore_tool.2.log 2024-03-12T03:59:26.213 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-12T03:59:26.222 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/objectstore_tool.2.log: /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mon.c.log: 91.9% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T03:59:26.266 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mon.b.log: 81.6% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T03:59:26.276 INFO:teuthology.orchestra.run.smithi173.stderr: 67.8% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/tcmu-runner.log.gz 2024-03-12T03:59:26.372 INFO:teuthology.orchestra.run.smithi173.stderr: 85.6% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T03:59:26.380 INFO:teuthology.orchestra.run.smithi173.stderr: 90.0% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T03:59:26.395 INFO:teuthology.orchestra.run.smithi173.stderr: 89.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T03:59:26.403 INFO:teuthology.orchestra.run.smithi169.stderr: 95.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T03:59:26.432 INFO:teuthology.orchestra.run.smithi169.stderr: 89.0% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T03:59:26.446 INFO:teuthology.orchestra.run.smithi173.stderr: 91.2% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T03:59:26.458 INFO:teuthology.orchestra.run.smithi169.stderr: 91.3% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T03:59:26.459 INFO:teuthology.orchestra.run.smithi169.stderr: 93.9% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T03:59:26.526 INFO:teuthology.orchestra.run.smithi173.stderr: 92.3% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-12T03:59:26.845 INFO:teuthology.orchestra.run.smithi169.stderr: 94.3% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/objectstore_tool.2.log.gz 2024-03-12T03:59:27.606 INFO:teuthology.orchestra.run.smithi169.stderr: 89.7% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-12T03:59:28.900 INFO:teuthology.orchestra.run.smithi173.stderr: 91.1% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-12T03:59:33.367 INFO:teuthology.orchestra.run.smithi169.stderr: 91.0% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-12T03:59:35.046 INFO:teuthology.orchestra.run.smithi169.stderr: 90.4% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-12T04:01:05.872 INFO:teuthology.orchestra.run.smithi173.stderr: 93.9% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-12T04:02:22.638 INFO:teuthology.orchestra.run.smithi173.stderr: 93.7% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-12T04:03:01.487 INFO:teuthology.orchestra.run.smithi173.stderr: 93.5% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-12T04:03:33.206 INFO:teuthology.orchestra.run.smithi173.stderr: 93.2% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-12T04:03:33.208 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-12T04:03:33.208 INFO:teuthology.orchestra.run.smithi173.stderr:real 4m7.044s 2024-03-12T04:03:33.208 INFO:teuthology.orchestra.run.smithi173.stderr:user 5m2.374s 2024-03-12T04:03:33.208 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m13.121s 2024-03-12T04:12:20.624 INFO:teuthology.orchestra.run.smithi169.stderr: 93.2% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-12T04:14:41.909 INFO:teuthology.orchestra.run.smithi169.stderr: 92.6% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-12T04:16:34.117 INFO:teuthology.orchestra.run.smithi169.stderr: 93.0% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-12T04:17:15.192 INFO:teuthology.orchestra.run.smithi169.stderr: 92.7% -- replaced with /var/log/ceph/677d624a-e020-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-12T04:17:15.194 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-12T04:17:15.194 INFO:teuthology.orchestra.run.smithi169.stderr:real 17m49.033s 2024-03-12T04:17:15.194 INFO:teuthology.orchestra.run.smithi169.stderr:user 7m7.714s 2024-03-12T04:17:15.195 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m21.212s 2024-03-12T04:17:15.195 INFO:tasks.cephadm:Archiving logs... 2024-03-12T04:17:15.195 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869/remote/smithi169/log 2024-03-12T04:17:15.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T04:18:23.535 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869/remote/smithi173/log 2024-03-12T04:18:23.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T04:19:10.443 INFO:tasks.cephadm:Removing cluster... 2024-03-12T04:19:10.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 677d624a-e020-11ee-95c8-87774f69a715 --force 2024-03-12T04:19:10.713 INFO:teuthology.orchestra.run.smithi169.stdout:Deleting cluster with fsid: 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T04:19:11.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 677d624a-e020-11ee-95c8-87774f69a715 --force 2024-03-12T04:19:11.650 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 677d624a-e020-11ee-95c8-87774f69a715 2024-03-12T04:19:12.359 INFO:tasks.cephadm:Removing cephadm ... 2024-03-12T04:19:12.359 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T04:19:12.404 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T04:19:12.419 INFO:tasks.cephadm:Teardown complete 2024-03-12T04:19:12.420 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-12T04:19:12.430 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-12T04:19:12.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-12T04:19:12.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-12T04:19:12.544 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-12T04:19:12.552 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-12T04:19:12.553 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T04:19:12.556 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T04:19:12.569 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-03-12T04:19:12.570 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-12T04:19:12.591 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T04:19:12.591 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-12T04:19:12.591 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 5 -208us[ -208us] +/- 102ms 2024-03-12T04:19:12.591 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 -1638us[-1675us] +/- 60ms 2024-03-12T04:19:12.591 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +2200us[+2200us] +/- 74ms 2024-03-12T04:19:12.591 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:19:12.612 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T04:19:12.612 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-12T04:19:12.612 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 -253us[ -286us] +/- 102ms 2024-03-12T04:19:12.612 INFO:teuthology.orchestra.run.smithi169.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -1665us[-1699us] +/- 60ms 2024-03-12T04:19:12.612 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +2147us[+2147us] +/- 74ms 2024-03-12T04:19:12.612 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:19:12.613 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-12T04:19:12.623 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-12T04:19:12.623 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-12T04:19:12.657 DEBUG:teuthology.orchestra.run.smithi169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T04:19:12.744 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T04:19:12.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T04:19:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:19:12.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T04:19:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:19:12.876 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-12T04:19:12.886 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-12T04:19:12.915 INFO:teuthology.task.internal:Duration was 4075.339949 seconds 2024-03-12T04:19:12.915 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-12T04:19:12.967 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-12T04:19:12.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T04:19:12.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T04:19:13.041 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T04:19:13.069 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T04:19:13.489 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-12T04:19:13.490 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-03-12T04:19:13.490 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T04:19:13.511 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-03-12T04:19:13.511 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T04:19:13.542 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-12T04:19:13.542 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T04:19:13.553 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T04:19:13.611 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-12T04:19:13.611 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T04:19:13.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T04:19:15.015 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-12T04:19:15.037 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-12T04:19:15.037 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T04:19:15.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T04:19:15.098 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-12T04:19:15.160 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T04:19:15.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T04:19:15.195 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-03-12T04:19:15.205 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-03-12T04:19:15.252 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T04:19:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:19:15.310 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T04:19:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:19:15.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-12T04:19:15.553 INFO:teuthology.task.internal:Transferring archived files... 2024-03-12T04:19:15.554 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869/remote/smithi169 2024-03-12T04:19:15.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T04:19:15.623 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869/remote/smithi173 2024-03-12T04:19:15.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T04:19:15.665 INFO:teuthology.task.internal:Removing archive directory... 2024-03-12T04:19:15.666 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T04:19:15.668 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T04:19:15.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-12T04:19:15.725 INFO:teuthology.task.internal:Not uploading archives. 2024-03-12T04:19:15.725 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-12T04:19:15.768 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-12T04:19:15.768 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T04:19:15.770 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T04:19:15.782 INFO:teuthology.orchestra.run.smithi169.stdout: 265312 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 04:19 /home/ubuntu/cephtest 2024-03-12T04:19:15.786 INFO:teuthology.orchestra.run.smithi173.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 04:19 /home/ubuntu/cephtest 2024-03-12T04:19:15.787 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-12T04:19:15.808 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-12T04:19:15.876 INFO:teuthology.nuke:Checking targets against current locks 2024-03-12T04:19:15.902 DEBUG:teuthology.nuke:shortname: smithi169 2024-03-12T04:19:15.903 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T04:19:15.925 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 03:01:30.638213', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGVehr8HZ5N4vY73bOvh5Z9eSmyze1Jejhc1NhvLYo7mQCQkXzcb9WKKxhZunYOw4V4+ECTo1qMqMx0iO7gVMk='} 2024-03-12T04:19:15.929 DEBUG:teuthology.nuke:shortname: smithi173 2024-03-12T04:19:15.929 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T04:19:15.948 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 03:01:30.636107', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLRR6gqvY4I1TKPA4iluTrw1r/Hv7coz9fW43/ByTE1ShCkAXmybEMEt9U8s5Em4LNPbsTG3jJTQsC+l2x0SpLc='} 2024-03-12T04:19:15.981 INFO:teuthology.orchestra.console.smithi169:Power off 2024-03-12T04:19:15.981 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T04:19:16.009 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-03-12T04:19:16.009 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:19:16.024 INFO:teuthology.orchestra.console.smithi173:Power off 2024-03-12T04:19:16.024 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T04:19:16.033 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-12T04:19:16.049 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-03-12T04:19:16.049 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:19:16.071 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-12T04:19:20.034 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:19:20.072 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:19:20.144 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-12T04:19:20.182 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-12T04:19:24.149 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:19:24.183 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:19:24.259 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-12T04:19:24.292 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-12T04:19:28.265 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:19:28.292 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:19:28.375 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-03-12T04:19:28.400 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-03-12T04:19:28.477 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-03-12T04:19:28.501 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-03-12T04:19:28.605 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} duration: 4075.339949131012 failure_reason: '"2024-03-12T03:40:00.000170+0000 mon.a (mon.0) 1089 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-12T04:19:28.606 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T04:19:28.695 INFO:teuthology.run:FAIL